<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jan 22, 2015 at 6:38 PM, Roberto Ostinelli <span dir="ltr"><<a href="mailto:roberto@widetag.com" target="_blank">roberto@widetag.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Here's something I've tried which is successful in avoiding the memory increase for binaries.<div><br></div><div>Inside a loop, I used to have:</div><div><br></div><div><font face="monospace, monospace"> [...]</font></div><div><font face="monospace, monospace"> <<Body:Len/binary, "\r\n", Rest/binary>> = Data,<br></font></div><div><div><font face="monospace, monospace"> loop(Body, Rest);</font><br></div></div><div><br></div><div>Now I force a binary copy to ensure that the reference to the original full binary is easily removed:</div><div><br><font face="monospace, monospace"> [...]<br></font></div><div><div><font face="monospace, monospace"> <<Body0:Len/binary, "\r\n", Rest0/binary>> = Data,</font></div><div><font face="monospace, monospace"> Body = binary:copy(Body0),</font></div><div><font face="monospace, monospace"> Rest = binary:copy(Rest0),</font></div><div><font face="monospace, monospace"> loop(Body, Rest);</font><br></div></div><div><br></div><div><div>This seems to have stabilized the memory usage reported by <font face="monospace, monospace">erlang:memory/0</font>.<br></div><div><br></div><div>However:</div><div><ul><li>I believe this can only work if the copied binary are <i>heap</i> and not <i>ref-c</i>, is this correct?</li></ul></div></div></div></blockquote><div>Binary copy makes a new copy of the binary regardless of size, but it is only useful on ref-c binaries,</div><div>and should be used to avoid keeping references to large binaries, when you keep and use a small part </div><div>of the original binary. But in this case you make a copy of Data (-2 bytes) and thus will double up the memory until Data is gc'ed and refc reaches zero. So I do not see the point of the above nor of making it a list which will</div><div>only explode your memory even more.</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><ul><li>Unfortunately, the BEAM process reported RES memory sill keeps growing.</li></ul></div><div>Any other ideas?</div><span class="HOEnZb"><font color="#888888"><div>r.</div><div><br></div></font></span></div><div><br></div><div><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jan 22, 2015 at 6:11 PM, Roberto Ostinelli <span dir="ltr"><<a href="mailto:roberto@widetag.com" target="_blank">roberto@widetag.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thank you Robert.<div>I'm going to try a selective fullsweep_after.</div><div><br></div><div>Could this also justify the process memory increase (which is more significant)?<br></div><span><div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jan 22, 2015 at 6:00 PM, Robert Virding <span dir="ltr"><<a href="mailto:rvirding@gmail.com" target="_blank">rvirding@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>One thing you can see is that the size of the binary data is growing. This space contains the large binaries (> 64 bytes) which are sent in messages between processes. While this means that the messages become (much) smaller and faster to send it takes a much longer time to detect that they are no longer alive and can be reclaimed. Basically it takes until all the processes they have passed through does a full garbage collection. Setting fullsweep_after to 0 and doing explicit garbage collects speeds up reclaiming the binaries.<br><br></div><div>You could be much more selective in which processes you set fullsweep_after to 0 and which ones you explicitly garbage collect.<br><br></div>I don't know if the is *the* problem but it is *a* problem you have.<br><br></div>Robert</div>
</blockquote></div><br></div></div></span></div>
</blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
<br></blockquote></div><br></div></div>