[erlang-questions] Re: Classification of concurrency patterns

[Jay Nelson]

> These are really about (1) discovering the core topology of the
> machine the runtime is executing on taking into account latencies, (2)
> affinitizing schedulers for

An alternative is to watch the execution, reducing the data set size,  
until a speed up is seen.  In this way you don't optimize based on  
how it _should_ work, but on how the system is actually performing.   
Once the optimal data set size is determined, the data can be  
automatically chopped up accordingly.

Better to detect cache (or who knows what future technology) speed up  
so that the code is automatically adaptive on alternative hardware.   
Likewise, you could build a large queue of tasks, then spawn new  
workers periodically measuring number of tasks / unit of time.  At  
some threshold the overhead will show up and slow down progress, so  
throttle back the number of open processes in the worker pool,  
dynamically deciding based performance rather than number of cores,  
or schedulers or what have you.

jay