<div dir="ltr">By doing forced garbage collection, I managed to reduce the overall system memory consumption from 1 GB to 190 MB in normal operation. That's over 420% memory saving. A single use session was taking 12 MB of memory at peak and holding for some time before are now getting released fast.<div><br></div><div>I didn't notice much change in CPU usage.<div><br></div><div>I found temporary variables, eg. binary_to_list of a XML data say 100 KB in size (Xmerl needs string), won't get freed for a long period of time without force garbage collection. Therefore when there are about 500 user sessions, each process consuming large memory blocks, makes the system memory usage extremely high. We plan to support a large number of user sessions, say 10000s and this memory consumption is a show stopper for us at the moment.</div><div><br></div><div>Using erlang:process_info/2, I can get the total memory usage of a process. But, is there a method to get full breakdown of the memory allocation in a process in waiting state? Eg. if it can show the memory usage of each variable in the current scope of the process, pending for garbage collection, etc., wold be ideal.</div><div><br></div><div>Are there any fast XML parsers that work with binary, instead of string?</div><div><br></div><div>- Eranga<br></div><div><br></div><div><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Sep 28, 2014 at 9:16 AM, Robert Virding <span dir="ltr"><<a href="mailto:rvirding@gmail.com" target="_blank">rvirding@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">The obvious question is whether you are sure you actually need to optimise to save memory? Premature optimisation and all that. (Actually sensible advice) Maybe reviewing algorithms and datastructures will do the trick for you.</p>
<p dir="ltr">Robert</p>
<p dir="ltr">From my Nexus<br>
</p><div class="HOEnZb"><div class="h5">
<div class="gmail_quote">On Sep 24, 2014 7:37 PM, "Eranga Udesh" <<a href="mailto:eranga.erl@gmail.com" target="_blank">eranga.erl@gmail.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks all for your advice. Let me see how I can apply them to my program. It looks like my code/logic is going to get ugly and induce a performance penalty, in order to save memory.<br><div><br></div><div>Cheers,</div><div>- Eranga</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Sep 24, 2014 at 7:58 PM, Robert Raschke <span dir="ltr"><<a href="mailto:rtrlists@googlemail.com" target="_blank">rtrlists@googlemail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>I always thought that is one of the reasons to have processes.<br>If you've got something big you want to throw away quickly, make a process for it.<br><br>$ erl<br>Erlang R16B03-1 (erts-5.10.4) [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false]<br><br>Eshell V5.10.4 (abort with ^G)<br>1> erlang:memory().<br>[{total,14148416},<br> {processes,4091608},<br> {processes_used,4091488},<br> {system,10056808},<br> {atom,194289},<br> {atom_used,172614},<br> {binary,1514552},<br> {code,4026600},<br> {ets,262688}]<br>2> Pid = spawn(<br>2> fun () -><br>2> X = binary:copy(<<1,2,3,4,5,6,7,8>>, 1024),<br>2> Y = binary:copy(X, 1024),<br>2> receive stop -> ok end<br>2> end<br>2> ).<br><0.35.0><br>3> erlang:memory().<br>[{total,22643832},<br> {processes,4203448},<br> {processes_used,4203448},<br> {system,18440384},<br> {atom,194289},<br> {atom_used,175110},<br> {binary,9685320},<br> {code,4221791},<br> {ets,267056}]<br>4> Pid ! stop.<br>stop<br>5> erlang:memory().<br>[{total,13587776},<br> {processes,4084496},<br> {processes_used,4084384},<br> {system,9503280},<br> {atom,194289},<br> {atom_used,175110},<br> {binary,748144},<br> {code,4221791},<br> {ets,267056}]<br>6> <br><br><br></div>Regards,<br>Robby<br><br></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div>On 24 September 2014 02:13, Eranga Udesh <span dir="ltr"><<a href="mailto:eranga.erl@gmail.com" target="_blank">eranga.erl@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr">Well, yes I may deliberately lie.<div><br></div><div>However my suggestion is to, instead of doing a full sweep by the garbage collector (GC) to identify data going out of scope and reclaim, can the program (or rather I) deliberately say I (the calling process) is finished using the said data, so the GC may free that part.</div><div><br></div><div>Then the GC may carry out it's own logic, which it currently does to verify if the same data is referenced by any other processes, etc., and decide if to GC or not.</div><span><font color="#888888"><div><br></div><div>= Eranga</div></font></span></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Sep 23, 2014 at 12:42 PM, Richard A. O'Keefe <span dir="ltr"><<a href="mailto:ok@cs.otago.ac.nz" target="_blank">ok@cs.otago.ac.nz</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span><br>
On 23/09/2014, at 2:24 PM, Eranga Udesh wrote:<br>
<br>
> Thanks for the information received so far.<br>
><br>
> Wouldn't it be good for Erlang to have a single object garbage collection function/bif?<br>
<br>
</span>NO. In C it's called free() and it's a major cause of errors.<br>
<span><br>
> For example, when I no longer require a large object, I force to garbage collect it, without making a full sweep?<br>
<br>
</span>Why should the garbage collector *believe* you that the "object"<br>
is free? You could be deliberately lying. You could (and<br>
probably are) be mistaken. How is it to know which bits you<br>
want to keep without doing its usual thing? In a shared heap<br>
implementation (which Erlang has had and may have again) the<br>
fact that *you've* finished with the object doesn't mean<br>
everyone *else* has. A meaningful operation should not depend<br>
on implementation details like that.<br>
<span>><br>
> As mentioned in the document, a full sweep may degrade the performance.<br>
<br>
</span>Not half as much as freeing too much would!<br>
<br>
This is micro-optimisation. Avoid passing around large<br>
objects in the first place.<br>
<br>
</blockquote></div><br></div>
</div></div><br></div></div><span>_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
<br></span></blockquote></div><br></div>
<br>_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
<br></blockquote></div><br></div>
<br>_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
<br></blockquote></div>
</div></div></blockquote></div><br></div>