[erlang-questions] Erlang Memory Question

Sun Oct 5 09:27:41 CEST 2014

By doing forced garbage collection, I managed to reduce the overall system
memory consumption from 1 GB to 190 MB in normal operation. That's over
420% memory saving. A single use session was taking 12 MB of memory at peak
and holding for some time before are now getting released fast.

I didn't notice much change in CPU usage.

I found temporary variables, eg. binary_to_list of a XML data say 100 KB in
size (Xmerl needs string), won't get freed for a long period of time
without force garbage collection. Therefore when there are about 500 user
sessions, each process consuming large memory blocks, makes the system
memory usage extremely high. We plan to support a large number of user
sessions, say 10000s and this memory consumption is a show stopper for us
at the moment.

Using erlang:process_info/2, I can get the total memory usage of a process.
But, is there a method to get full breakdown of the memory allocation in a
process in waiting state? Eg. if it can show the memory usage of each
variable in the current scope of the process, pending for garbage
collection, etc., wold be ideal.

Are there any fast XML parsers that work with binary, instead of string?

- Eranga

On Sun, Sep 28, 2014 at 9:16 AM, Robert Virding <rvirding@REDACTED> wrote:

> The obvious question is whether you are sure you actually need to optimise
> to save memory? Premature optimisation and all that. (Actually sensible
> advice) Maybe reviewing algorithms and datastructures will do the trick for
> you.
>
> Robert
>
> From my Nexus
>  On Sep 24, 2014 7:37 PM, "Eranga Udesh" <eranga.erl@REDACTED> wrote:
>
>> Thanks all for your advice. Let me see how I can apply them to my
>> program. It looks like my code/logic is going to get ugly and induce a
>> performance penalty, in order to save memory.
>>
>> Cheers,
>> - Eranga
>>
>> On Wed, Sep 24, 2014 at 7:58 PM, Robert Raschke <rtrlists@REDACTED>
>> wrote:
>>
>>> I always thought that is one of the reasons to have processes.
>>> If you've got something big you want to throw away quickly, make a
>>> process for it.
>>>
>>> $ erl
>>> Erlang R16B03-1 (erts-5.10.4) [source] [64-bit] [smp:8:8]
>>> [async-threads:10] [hipe] [kernel-poll:false]
>>>
>>> Eshell V5.10.4  (abort with ^G)
>>> 1> erlang:memory().
>>> [{total,14148416},
>>>  {processes,4091608},
>>>  {processes_used,4091488},
>>>  {system,10056808},
>>>  {atom,194289},
>>>  {atom_used,172614},
>>>  {binary,1514552},
>>>  {code,4026600},
>>>  {ets,262688}]
>>> 2> Pid = spawn(
>>> 2>   fun () ->
>>> 2>     X = binary:copy(<<1,2,3,4,5,6,7,8>>, 1024),
>>> 2>     Y = binary:copy(X, 1024),
>>> 2>     receive stop -> ok end
>>> 2>   end
>>> 2> ).
>>> <0.35.0>
>>> 3> erlang:memory().
>>> [{total,22643832},
>>>  {processes,4203448},
>>>  {processes_used,4203448},
>>>  {system,18440384},
>>>  {atom,194289},
>>>  {atom_used,175110},
>>>  {binary,9685320},
>>>  {code,4221791},
>>>  {ets,267056}]
>>> 4> Pid ! stop.
>>> stop
>>> 5> erlang:memory().
>>> [{total,13587776},
>>>  {processes,4084496},
>>>  {processes_used,4084384},
>>>  {system,9503280},
>>>  {atom,194289},
>>>  {atom_used,175110},
>>>  {binary,748144},
>>>  {code,4221791},
>>>  {ets,267056}]
>>> 6>
>>>
>>>
>>> Regards,
>>> Robby
>>>
>>>
>>> On 24 September 2014 02:13, Eranga Udesh <eranga.erl@REDACTED> wrote:
>>>
>>>> Well, yes I may deliberately lie.
>>>>
>>>> However my suggestion is to, instead of doing a full sweep by the
>>>> garbage collector (GC) to identify data going out of scope and reclaim, can
>>>> the program (or rather I) deliberately say I (the calling process) is
>>>> finished using the said data, so the GC may free that part.
>>>>
>>>> Then the GC may carry out it's own logic, which it currently does to
>>>> verify if the same data is referenced by any other processes, etc., and
>>>> decide if to GC or not.
>>>>
>>>> = Eranga
>>>>
>>>> On Tue, Sep 23, 2014 at 12:42 PM, Richard A. O'Keefe <ok@REDACTED
>>>> > wrote:
>>>>
>>>>>
>>>>> On 23/09/2014, at 2:24 PM, Eranga Udesh wrote:
>>>>>
>>>>> > Thanks for the information received so far.
>>>>> >
>>>>> > Wouldn't it be good for Erlang to have a single object garbage
>>>>> collection function/bif?
>>>>>
>>>>> NO.  In C it's called free() and it's a major cause of errors.
>>>>>
>>>>> > For example, when I no longer require a large object, I force to
>>>>> garbage collect it, without making a full sweep?
>>>>>
>>>>> Why should the garbage collector *believe* you that the "object"
>>>>> is free?  You could be deliberately lying.  You could (and
>>>>> probably are) be mistaken.  How is it to know which bits you
>>>>> want to keep without doing its usual thing?  In a shared heap
>>>>> implementation (which Erlang has had and may have again) the
>>>>> fact that *you've* finished with the object doesn't mean
>>>>> everyone *else* has.  A meaningful operation should not depend
>>>>> on implementation details like that.
>>>>> >
>>>>> > As mentioned in the document, a full sweep may degrade the
>>>>> performance.
>>>>>
>>>>> Not half as much as freeing too much would!
>>>>>
>>>>> This is micro-optimisation.  Avoid passing around large
>>>>> objects in the first place.
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> erlang-questions mailing list
>>>> erlang-questions@REDACTED
>>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>>
>>>>
>>>
>>> _______________________________________________
>>> erlang-questions mailing list
>>> erlang-questions@REDACTED
>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>
>>>
>>
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20141005/4f01f42c/attachment.htm>