<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><br></div><div>open_port({spawn, "/bin/cat " ++ File}, [{line, MaxLen}, binary]) </div><div><br></div><div>will pour the file, one line at a time, into your message queue. :)</div><div><br></div><div>Wicked fast, but no flow control.</div><div><br></div><div>Fredrik Svahn made a flow-control hack for stdin back in 2008, but I don't know (a) if that's applicable to you or (b) if it ever made it into the OTP source somehow.</div><div><br></div><div><a href="http://erlang.org/pipermail/erlang-bugs/2008-December/001136.html">http://erlang.org/pipermail/erlang-bugs/2008-December/001136.html</a></div><div><br></div><div>Otherwise, opening the file in [raw, binary] mode and using re:split() ought to work reasonably well.</div><div><br></div><div>file:read_line() is indeed dreadfully slow, but is OTOH extremely nice in distributed embedded systems, as it supports IO redirection across nodes.</div><div><br></div><div>BR,</div><div>Ulf W</div><br><div><div>On 23 Mar 2012, at 11:30, Max Lapshin wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>I need to load large CSV file into memory very fast.<br><br>I've tried to use erlang parser, but my results were very bad (in fact<br> file:read_line is very slow), so I've tried to make a NIF for it.<br>Target speed is 1 microsecond per line.<br><br><br>My CSV has very strict format: only numbers, no quoting, \n in the<br>end. Also I moved parsing of integers and date into NIF.<br><br>My results are here: <a href="http://github.com/maxlapshin/csv_reader">http://github.com/maxlapshin/csv_reader</a> and I get<br>only 15 microseconds: 4,5 seconds for 300K lines CSV.<br><br>Currently I use fgets to read line by line from file. Maybe it is a<br>bad idea and I should use mmap or implement 1MB buffer for read?<br>_______________________________________________<br>erlang-questions mailing list<br><a href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a><br>http://erlang.org/mailman/listinfo/erlang-questions<br></div></blockquote></div><br></body></html>