I am looking for tasks that
are unsuitable for GPUs
gain significant speedup as more CPU nodes are added
don't require large data transfer or inter-thread communication between nodes.
Do any problems of this type still exist?
Parallelizing Monte Carlo simulations for non-trivial simulation dynamics should fit the bill. For example, simulating an aircraft flight using systems of ODEs and representing parameters like wind speed, wind direction, mass and inertia of the aircraft, etc using probability distributions so we can see how uncertainty of the parameters impacts performance. This problem is then tackled by Monte Carlo simulation which is embarrassingly parallel but also not something you would generally stick on a GPU due to timestep synchronization and since the dynamics are non-trivial.
@FranckDernoncourt "highly parallelizable" means a different thing. Embarrassingly-parallel conveys much more information about the parallelization pattern. And I would elevate this one from "expression" to "term", at least from my understanding of the field.
@AntonMenshov the OP seems to have defined embarrassingly parallel as "gain significant speedup as more CPU nodes are added" so what's the difference with highly parallelizable? Sorry I don't mean to argue, genuine question. I understand that in some cases, not all, embarrassingly parallel can mean linear scale up (e.g. 10 times more CPU will yield a x10 speed-up).
@FranckDernoncourt Usually, "embarassingly parallel" implies a problem that can be parallelized with very few manipulations (thus, very few - if any - dependencies are present in the problem). Thus, the parallelized tasks pretty much do not communicate with each other.
"higly parallelizable" is a problem that can be very efficiently (close to linear) be parallelized. However, the dependencies are inherently present within the tasks and they have to be broken by a programmer.
In this question, I gave an example of an embarrassingly parallel problem: parallelizing across frequencies withing a frequency sweep simulation.
An opposite case, would be, say, and FDTD simulation of a certain problem, where no embarrassingly parallel stuff happens, but parallelization is done across the mesh. There is a small communication burden on the border shared by separate tasks. However, it is minimal and can be tackled very efficiently.
This one, would be a highly-parallelizable problem.
Thus, there is information within the term "embarassingly parallel".
As per this page, there is also a term pleasingly parallel. However, I do not see why would I use it. I am perfectly fine to be pleased by embarrassingly parallel codes.
Regarding the OPs question: they might have a confusion about the term OR might not. While I don't see how that would influence the essence of the question, you are welcome to ask the clarification regarding that from the OP.
@AntonMenshov Thanks very much for the explanation and the Wikipedia link. I wasn't aware that the term seems rather well defined in that context (unlike in my field, AI, where people typically use it as a clickbait e.g. https://arxiv.org/abs/1912.08140).
From your description, it seems that "embarrassingly parallel" = "highly and easily parallelizable". Is that an accurate summary? (If so, I'd still tend to use "highly and easily parallelizable" but I guess that's just personal preference).
@NumLock Discussing about terminology is nothing to be embarrassed about. Ironically, you're the only one in this chat room hiding behind a nickname, so maybe you are the one who is embarrassed?
@FranckDernoncourt I would say that "embarassingly parallel" in scientific computing implies little to know dependency in the partitioned tasks. However, that actually would imply "highly and easily parallelizable" :)
My pleasure. The terminology and its usage certainly varies by field.
Discussion on question by Andrew: Are…
Discussion on question by Andrew: Are…