[erlang-questions] A more parallel widefinder

[Hynek Vychodil]

Good work. It's very nice idea to find last new line in chunk and than
not concatenate binary with big prefix. But I still argue, that this
implementation is not so much like universal tool, but especial one
purpose tuned solution. I'm so busy to try rewrite it to more modular
implementation. But I will try it some time.

Congratulation anyway.

--Hynek (Pichi) Vychodil

On 10/29/07, Anders Nygren <anders.nygren@REDACTED> wrote:
> One more try.
> This one tries to do more in the workers and as little as possible in
> the central controller.
>
> The controller only
> - starts workers, and maintains a specific number of workers.
> - prints the results
>
> Each worker
> - opens the file and reads a chunk
> - finds the first and last newlines in the chunk
> - sends the partial line at the beginning of the chunk
> to the worker processing the previous chunk
> - processes its chunk, excluding the initial and final
> partial lines
> - receives a partial line from from the worker that handles
> the next chunk
> - concatenates and process the last line
>
> The sequential control part now takes ~180 ms
> A worker processing a 200kbytes chunk takes ~1.7 ms
> So it will scale better to more cores than the previous versions.
>
> I estimate that on Steve's 8 core it will run at ~ 0.3 s.
>
> It is actually slightly faster than my previous wfinder1_1 even
> on a dual core.
>
> /Anders
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://www.erlang.org/mailman/listinfo/erlang-questions
>
>