[erlang-questions] process and closure memory usage

[Ladislav Lenart]

Hi,

here are my thought vut I am by no means an Erlang expert...

1. I'd say that using gen_server has no visible memory or performance
overhead compared to raw Erlang processes. It is more a question of
whether gen_server way of handling incoming messages in order as they
come suits your needs. Also you can make a raw Erlang process into an
OTP-aware process using proc_lib.

2. I think that process_info reports all memory allocated by the
process. From my experience a process roughly uses twice as much
memory as it needs. This can be quite a lot for a large number of
processes. If only a handful of processes are active in any given
time, you could use erlang:hibernate/3 for a raw Erlang process
or specify hibernate instead of timeout in gen_server return tuple.

3. This is the most interesting part. Just to clarify, by closure I
mean fun (...) -> ... end. If you are sending funs between processes
then things can get interesting because beside the code pointer also
the lexical scope of the function is sent over the wire. And in order
to be sent it has to be serialized first. During this serialization,
the "data identity" is not preserved. So if you use complex structure
that contains multiple identical parts the scope will be larger after
the serialization. This is true for any Erlang term. Example: [A, A,
A, A] where A is bound to some tuple. If you send this as a message,
the receiver will get [A1, A2, A3, A4, A5] where all received As are
equal but no longer point to the same memory location.

4. None that I know of.


HTH,

Ladislav Lenart


Matt Handler wrote:
> I'm trying to cut down on the process memory usage for a non-relational 
> database i'm working on.  essentially there are 10,000 processes (setup 
> as gen_servers) that send information to each other in various ways.  i 
> have several questions:
> 
>    1. how much of an overhead does using the gen_server model assert? 
>       if i'm not as worried about dynamic code loading, supervising,
>       etc... am i wasting memory and processing using the gen_server?
>    2. how/where can i find out more information about the memory usage
>       of a process?  i've used process_info/2 heavily to gather some of
>       this information, but my problem is that if i look at the rough
>       size of the state of a process (using erlang:iolist_size/1), it's
>       significantly less than the size of the process as reported by
>       process_info/2.  i want to know where the rest of the memory is
>       being eaten up (initially the processes are only 1k), on average,
>       the process size is about 4x the state size. 
>    3. i am passing around many closures between the nodes of the
>       database, and they are getting to be quite large (start out as
>       100bytes, end up as 40kbytes).  these are what is taking up most
>       of the memory of the state, and so i'm wondering if there is some
>       inner workings of the erlang system that make this a bad way of
>       programming in terms of memory usage.  it seems like there are
>       certain situations that make closures take up a lot of memory, but
>       i'm having trouble pinpointing when. 
>    4. is there a good tool out there for profiling the memory usage
>       rather than executiontime usage?
> 
> thanks for the help,
> -matt handler
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://www.erlang.org/mailman/listinfo/erlang-questions