Parsing big files
Wed Dec 6 17:19:52 CET 2000
James Hague wrote:
> > So far the builtin scanning modules cannot handle binaries.
> I assumed the poster was writing his own scanner. I wrote my own scanner a
> while back for a compiler (by hand; I didn't use the unofficial Erlang
> tool), and I suspect it would have been fairly easy to make it operate on
> binary input, rather than a list. That would be an interesting experiment.
no, I used erl_scan basically, since the text scanned was very Erlang-like.
The files I try to deal with can be as large as several 100 Mbytes. I don't think
it is worth trying to optimise the scanner for that, since in the most optimal case
I still need to store two times 100 MBytes in memory in the naive way of reading
the file once, modifying it and storing it again.
More information about the erlang-questions