[erlang-questions] Re: Classification of concurrency patterns

[Jay Nelson]

On May 19, 2010, at 1:17 PM, Sean Hernandez wrote:

> But in order to do adaptive execution a model of your execution
> environment needs to be built (e.g. processor, core, memory (this
> includes cache sizes too btw), execution latencies, queue lengths,
> similarity of instructions (instruction caches misses, etc) up so that
> adaptation algorithms can figure out what to tweak.

If you are just copying data from one array to another, you can
tell when a cache boundary is tripped by the slowdown in the
performance.  If you write progressively more data, measuring
the time lapse, you can find the array size that first causes the
slowdown.  You don't need to do any complicated modeling.
Then you can make your divide and conquer algorithm choose
the maximum size for each of the divide elements to get maximal
performance.

As I said earlier, the same thing can be done with a task queue
and a worker pool.  If there are 1M tasks over an hour, it is easy
to adjust the number of workers every 10 seconds for the first 10
minutes to get a feel for performance.  Then choose the optimal
number of workers for the rest of the time.

You just need to have elements of your algorithm that can be
tuned (e.g., number of processes, data sequence size in divide
and conquer, message batch size, etc).

jay