<br><br><div class="gmail_quote">On Thu, Nov 20, 2008 at 6:12 PM, Kevin Scaldeferri <span dir="ltr"><<a href="mailto:kevin@scaldeferri.com">kevin@scaldeferri.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div style="">Pretty nice. I'm curious, how did you decide to parallelize the IO?<div></div></div></blockquote><div><br>I didn't parallelize the I/O. I loaded the whole file into a binary and then did the parallel ops by passing a sub-binary to each process based on offset and length. To be fair, I was trying to see how well the character classification routine would fare, so I deliberately left out the I/O. IMHO parallelizing I/O is a totally different kettle of fish that I'm not ready to take on right now :)<br>
<br>Ed<br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div style=""><div><br></div><div>-kevin</div><div><br></div><div><br><div><div>
<div></div><div class="Wj3C7c"><div>On Nov 20, 2008, at 2:47 PM, Edwin Fine wrote:</div><br></div></div><blockquote type="cite"><div><div></div><div class="Wj3C7c">I've previously expressed some disappointment in Erlang's worst-case scenario SMP performance (lots of messages, very little done with each message, communicating parallel SMP processes).<br>
<br>Now, to balance that out, I want to praise Erlang (and HiPE) for a virtually linear SMP speedup in a character classification test I ran.<br> <br>Using sequential and parallelized code (rpc:pmap) and a HiPE-compiled module on an Intel Q6600/Ubuntu/8GB/R12B-4, which classified every byte of a 40MB file into character classes (e.g. punct, blank), the following results were achieved:<br>
<br>Sequential: 18817812 bytes/second<br>Parallelized: 74937454 bytes/second<br>Speedup: 3.98 (on a 4-core system).<br><br>That's extremely close to linear and is pretty impressive. I found HiPE gave about a 10x speedup over BEAM. The results are with HiPE and exclude the time taken to load the file into a binary.<br>
<br>Kudos to Erlang and HiPE.<br><br>I had to do a bit of ugly code to get this level of performance, though (using dicts and arrays to keep the counts was just too slow - I fell to using a separate parameter for each of the 12 counts).<br>
<br>The outputs follow (ascsp is something I added - counts ASCII SP (32) chars, and the purpose of '8bit' is self-evident). The other character classes are defined as per <a href="http://en.wikipedia.org/wiki/Regular_expression" target="_blank">http://en.wikipedia.org/wiki/Regular_expression</a> .<br>
<br>Regards,<br>Ed<br><br>PS In the unlikely event that someone wants the code, I will gladly post it if asked.<br>------------<br><br>31> c(charclass,[native]).<br>{ok,charclass}<br>32> charclass:bm("/home/efine/erlang/otp_src_R12B-3.tar.gz")<br>
32> .<br>*** Completed run using classify_binary ***<br>File "/home/efine/erlang/otp_src_R12B-3.tar.gz" size is 42195557 bytes<br>Classified 42195557 characters<br>Breakdown:<br>[{'8bit',21296511},<br>
{alnum,10079032},<br> {alpha,8454737},<br> {ascsp,158524},<br> {blank,318577},<br> {cntrl,5379355},<br> {digit,1624295},<br> {lower,4269350},<br> {print,15519691},<br> {punct,5282135},<br> {space,963777},<br> {upper,4185387}]<br>
Speed = 18817812 bytes/sec (0.05314113995461655 us/byte)<br>ok<br>33> charclass:bm_par("/home/efine/erlang/otp_src_R12B-3.tar.gz").<br>*** Completed run using par_classify_binary ***<br>File "/home/efine/erlang/otp_src_R12B-3.tar.gz" size is 42195557 bytes<br>
Classified 42195557 characters<br>Breakdown:<br>[{'8bit',21296511},<br> {alnum,10079032},<br> {alpha,8454737},<br> {ascsp,158524},<br> {blank,318577},<br> {cntrl,5379355},<br> {digit,1624295},<br> {lower,4269350},<br>
{print,15519691},<br> {punct,5282135},<br> {space,963777},<br> {upper,4185387}]<br>Speed = 74937454 bytes/sec (0.013344461835164304 us/byte)<br><br></div></div> _______________________________________________<br>erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br><a href="http://www.erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://www.erlang.org/mailman/listinfo/erlang-questions</a></blockquote>
</div><br></div></div></blockquote></div><br>