[erlang-questions] widefinder update

Thomas Lindgren <>
Sun Oct 28 17:51:15 CET 2007

--- Hynek Vychodil <> wrote:

> On 10/28/07, Thomas Lindgren
> <> wrote:
> >
> > --- Hynek Vychodil <>
> wrote:
> >
> > > Hello,
> > > These results are interesting, but I demur to
> kind
> > > of solution. Your
> > > and Steve's approach have some caveats.
> > >
> > > 1/ File is read all in memory.
> >
> > Hynek,
> >
> > This is true for some versions, but not all. The
> > 'block read' version reads the file in chunks.
> What version do you mean? tbray_blockread.erl from
> reads in chunks, but when workers are slow you run
> out of memory. Look
> at scan_file/9 cycle. There isn't limit of blocks in
> memory.

Correct, the code doesn't support it at the moment,
but neither is there an inherent problem with adding
such a limit. If you are processing terabytes of data,
this is obviously a more pressing issue than at the
moment, but you are not _required_ to read all of the
logfile at once, and you are not _unable_ to control
how many data chunks are resident at any time. All the
building blocks for a more robust file reader are
available if you need them. 

So I would say that while this code might not be
something you deploy in production, it still
definitely works as a proof of concept on how to write
efficient code that exploits multicore, etc.


Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 

More information about the erlang-questions mailing list