[erlang-questions] Speed of CSV parsing: how to read 1M of lines in 1 second
Sun Mar 25 20:53:58 CEST 2012
I think you should be able to parse something like this at approaching
the speed of memcpy, but you have to try hard. Have you tried writing
something that runs outside erlang and just tries to parse the file into
numbers in memory, say using re2c or ragel?
You might need to try using mmap and a thread that just reads every 16th
byte and forces the file to read in. Or double buffer with readahead.
Is the data in your operating system's VM cache already, or on disk?
More information about the erlang-questions