<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hello,<div class=""><br class=""></div><div class="">The problem is ‘global’ here. It turns out that “processing” of large binary chunk at once is not very efficient using binary:replace or binary:split. The “custom” algorithm works better. Like you’ve shown this to us.</div><div class=""><br class=""></div><div class="">However, If you are pipelining the binary operation with something else (e.g. parsing, counting, whats-not) then binary bif’s wins. I am not able to speak about binary:replace, I’ve never used that at my code but binary:split without global rocks. It was capable to out-perform a custom accumulator functions. </div><div class=""><br class=""></div><div class="">Best Regards, </div><div class="">Dmitry</div><div class=""><br class=""></div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On 07 Nov 2014, at 19:50, Stu Bailey <<a href="mailto:stu.bailey@gmail.com" class="">stu.bailey@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Thank you for the feedback. That's very helpful.<div class=""><br class=""></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Fri, Nov 7, 2014 at 9:44 AM, Loïc Hoguin <span dir="ltr" class=""><<a href="mailto:essen@ninenines.eu" target="_blank" class="">essen@ninenines.eu</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Based on the code at<br class="">
<br class="">
<a href="https://github.com/erlang/otp/blob/maint/lib/stdlib/src/binary.erl#L268" target="_blank" class="">https://github.com/erlang/otp/<u class=""></u>blob/maint/lib/stdlib/src/<u class=""></u>binary.erl#L268</a><br class="">
<br class="">
It does a lot of splitting, and then a lot more splitting, and then call iolist_to_binary. It looks very inefficient.<br class="">
<br class="">
Your solution is the fastest way to do it. You also benefit from match context optimization and so your code is very fast. The only thing that could make it faster is if memory was allocated only once for the resulting binary (instead of realloc a few times)... but maybe there's already an optimization like this?<span class=""><br class="">
<br class="">
On 11/07/2014 07:33 PM, Stu Bailey wrote:<br class="">
</span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
FYI, if you want to try to replicate it, I was processing ~80 chunks of<br class="">
binary where each chunk was about ~250,000,000 bytes. I think you'll<br class="">
see the difference on just one chunk. I happen to running on a 8-core<br class="">
MacBook Pro with 16GB Ram and therefore spawned a process per chunk to<br class="">
grab all the resources on all the cores. With the hand written<br class="">
function, it worked like a charm...yay Erlang! :-) I love seeing a few<br class="">
lines of code effectively use all processing power available. Heats the<br class="">
machine up quite a bit too. :-)<br class="">
<br class="">
On Fri, Nov 7, 2014 at 9:22 AM, Stu Bailey <<a href="mailto:stu.bailey@gmail.com" target="_blank" class="">stu.bailey@gmail.com</a><br class=""></span><span class="">
<mailto:<a href="mailto:stu.bailey@gmail.com" target="_blank" class="">stu.bailey@gmail.com</a>>> wrote:<br class="">
<br class="">
I'm not planning to spend a lot of time on this right now, but the<br class="">
binary:replace(...) was chewing a tremendous amount of system time<br class="">
CPU load (and actually never finished before I got frustrated and<br class="">
killed it) and my function was reporting the CPU load as 99% user<br class="">
time (not system time) and finished in a reasonable time. I assume<br class="">
the high system time usage for binary:replace(..) is because<br class="">
binary:replace(...) is doing something manic with system calls for<br class="">
memory management or something?<br class="">
<br class="">
<br class="">
On Fri, Nov 7, 2014 at 1:44 AM, Loïc Hoguin <<a href="mailto:essen@ninenines.eu" target="_blank" class="">essen@ninenines.eu</a><br class=""></span><span class="">
<mailto:<a href="mailto:essen@ninenines.eu" target="_blank" class="">essen@ninenines.eu</a>>> wrote:<br class="">
<br class="">
binary:split and binary:replace, unlike other functions of the<br class="">
binary module, are normal Erlang functions. They also process a<br class="">
list of options before doing the actual work, so there's an<br class="">
obvious overhead compared to not doing that. In addition as has<br class="">
been pointed out, your code is more specialized so that helps too.<br class="">
<br class="">
On 11/07/2014 03:33 AM, Stu Bailey wrote:<br class="">
<br class="">
I found<br class="">
<br class=""></span>
binary:replace(BinChunk,<<"\n"<u class=""></u>__>>,<<>>,[global]).<br class="">
<br class="">
/significantly /slower than<br class="">
<br class="">
remove_pattern(BinChunk,<<>>,<<u class=""></u>__<"\n">>).<br class="">
<br class="">
with<br class="">
<br class="">
remove_pattern(<<>>,Acc,___<u class=""></u>BinPat) -><br class="">
Acc;<br class="">
remove_pattern(Bin,Acc,BinPat)<u class=""></u>__-><span class=""><br class="">
<<Byte:1/binary,Rest/binary>> = Bin,<br class="">
case Byte == BinPat of<br class=""></span>
true -> remove_pattern(Rest,Acc,__<u class=""></u>BinPat);<br class="">
false -><br class="">
remove_pattern(Rest,<<Acc/__<u class=""></u>binary,Byte/binary>>,BinPat)<span class=""><br class="">
end.<br class="">
<br class="">
That was surprising to me. The built-in binary:replace()<br class="">
was much much<br class="">
slower for larger BinChunk with lots of <<"\n">> sprinkled<br class="">
through.<br class="">
<br class="">
Thoughts?<br class="">
<br class="">
<br class=""></span>
______________________________<u class=""></u>___________________<br class="">
erlang-questions mailing list<br class="">
<a href="mailto:erlang-questions@erlang.org" target="_blank" class="">erlang-questions@erlang.org</a> <mailto:<a href="mailto:erlang-questions@erlang.org" target="_blank" class="">erlang-questions@<u class=""></u>erlang.org</a>><br class="">
<a href="http://erlang.org/mailman/__listinfo/erlang-questions" target="_blank" class="">http://erlang.org/mailman/__<u class=""></u>listinfo/erlang-questions</a><span class=""><br class="">
<<a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank" class="">http://erlang.org/mailman/<u class=""></u>listinfo/erlang-questions</a>><br class="">
<br class="">
<br class="">
--<br class="">
Loïc Hoguin<br class="">
<a href="http://ninenines.eu/" target="_blank" class="">http://ninenines.eu</a><br class="">
<br class="">
<br class="">
<br class="">
</span></blockquote><div class="HOEnZb"><div class="h5">
<br class="">
-- <br class="">
Loïc Hoguin<br class="">
<a href="http://ninenines.eu/" target="_blank" class="">http://ninenines.eu</a><br class="">
</div></div></blockquote></div><br class=""></div></div>
_______________________________________________<br class="">erlang-questions mailing list<br class=""><a href="mailto:erlang-questions@erlang.org" class="">erlang-questions@erlang.org</a><br class="">http://erlang.org/mailman/listinfo/erlang-questions<br class=""></div></blockquote></div><br class=""></div></body></html>