[erlang-questions] A more parallel widefinder
Tue Oct 30 10:25:41 CET 2007
Good work. It's very nice idea to find last new line in chunk and than
not concatenate binary with big prefix. But I still argue, that this
implementation is not so much like universal tool, but especial one
purpose tuned solution. I'm so busy to try rewrite it to more modular
implementation. But I will try it some time.
--Hynek (Pichi) Vychodil
On 10/29/07, Anders Nygren <> wrote:
> One more try.
> This one tries to do more in the workers and as little as possible in
> the central controller.
> The controller only
> - starts workers, and maintains a specific number of workers.
> - prints the results
> Each worker
> - opens the file and reads a chunk
> - finds the first and last newlines in the chunk
> - sends the partial line at the beginning of the chunk
> to the worker processing the previous chunk
> - processes its chunk, excluding the initial and final
> partial lines
> - receives a partial line from from the worker that handles
> the next chunk
> - concatenates and process the last line
> The sequential control part now takes ~180 ms
> A worker processing a 200kbytes chunk takes ~1.7 ms
> So it will scale better to more cores than the previous versions.
> I estimate that on Steve's 8 core it will run at ~ 0.3 s.
> It is actually slightly faster than my previous wfinder1_1 even
> on a dual core.
> erlang-questions mailing list
More information about the erlang-questions