[erlang-questions] widefinder update

[Thomas Lindgren]

--- Hynek Vychodil <vychodil.hynek@REDACTED> wrote:

> On 10/28/07, Thomas Lindgren
> <thomasl_erlang@REDACTED> wrote:
> >
> > --- Hynek Vychodil <vychodil.hynek@REDACTED>
> wrote:
> >
> > > Hello,
> > > These results are interesting, but I demur to
> kind
> > > of solution. Your
> > > and Steve's approach have some caveats.
> > >
> > > 1/ File is read all in memory.
> >
> > Hynek,
> >
> > This is true for some versions, but not all. The
> > 'block read' version reads the file in chunks.
> 
> What version do you mean? tbray_blockread.erl from
>
http://www.erlang.org/pipermail/erlang-questions/2007-October/030118.html
> reads in chunks, but when workers are slow you run
> out of memory. Look
> at scan_file/9 cycle. There isn't limit of blocks in
> memory.

Correct, the code doesn't support it at the moment,
but neither is there an inherent problem with adding
such a limit. If you are processing terabytes of data,
this is obviously a more pressing issue than at the
moment, but you are not _required_ to read all of the
logfile at once, and you are not _unable_ to control
how many data chunks are resident at any time. All the
building blocks for a more robust file reader are
available if you need them. 

So I would say that while this code might not be
something you deploy in production, it still
definitely works as a proof of concept on how to write
efficient code that exploits multicore, etc.

Best,
Thomas



__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com