Parsing big files

[Thomas Arts]

James Hague wrote:

> > So far the builtin scanning modules cannot handle binaries.
>
> I assumed the poster was writing his own scanner.  I wrote my own scanner a
> while back for a compiler (by hand; I didn't use the unofficial Erlang
> tool), and I suspect it would have been fairly easy to make it operate on
> binary input, rather than a list.  That would be an interesting experiment.

no, I used erl_scan basically, since the text scanned was very Erlang-like.

The files I try to deal with can be as large as several 100 Mbytes. I don't think
it is worth trying to optimise the scanner for that, since in the most optimal case
I still need to store two times 100 MBytes in memory in the naive way of reading
the file once, modifying it and storing it again.

/Thomas