[erlang-questions] process and closure memory usage

[Hynek Vychodil]

On Thu, Mar 19, 2009 at 10:42 AM, Ladislav Lenart <lenartlad@REDACTED>wrote:

> Hi,
>
> here are my thought vut I am by no means an Erlang expert...
>
> 1. I'd say that using gen_server has no visible memory or performance
> overhead compared to raw Erlang processes. It is more a question of
> whether gen_server way of handling incoming messages in order as they
> come suits your needs. Also you can make a raw Erlang process into an
> OTP-aware process using proc_lib.
>
> 2. I think that process_info reports all memory allocated by the
> process. From my experience a process roughly uses twice as much
> memory as it needs.


Doubling memory footprint depends in way how Erlang GC works. It is
trade-off between scalability and soft real-time characteristics and memory
footprint.

This can be quite a lot for a large number of
> processes. If only a handful of processes are active in any given
> time, you could use erlang:hibernate/3 for a raw Erlang process
> or specify hibernate instead of timeout in gen_server return tuple.
>
> 3. This is the most interesting part. Just to clarify, by closure I
> mean fun (...) -> ... end. If you are sending funs between processes
> then things can get interesting because beside the code pointer also
> the lexical scope of the function is sent over the wire. And in order
> to be sent it has to be serialized first. During this serialization,
> the "data identity" is not preserved. So if you use complex structure
> that contains multiple identical parts the scope will be larger after
> the serialization. This is true for any Erlang term. Example: [A, A,
> A, A] where A is bound to some tuple. If you send this as a message,
> the receiver will get [A1, A2, A3, A4, A5] where all received As are
> equal but no longer point to the same memory location.
>
> 4. None that I know of.
>
>
> HTH,
>
> Ladislav Lenart
>
>
> Matt Handler wrote:
> > I'm trying to cut down on the process memory usage for a non-relational
> > database i'm working on.  essentially there are 10,000 processes (setup
> > as gen_servers) that send information to each other in various ways.  i
> > have several questions:
> >
> >    1. how much of an overhead does using the gen_server model assert?
> >       if i'm not as worried about dynamic code loading, supervising,
> >       etc... am i wasting memory and processing using the gen_server?
> >    2. how/where can i find out more information about the memory usage
> >       of a process?  i've used process_info/2 heavily to gather some of
> >       this information, but my problem is that if i look at the rough
> >       size of the state of a process (using erlang:iolist_size/1), it's
> >       significantly less than the size of the process as reported by
> >       process_info/2.  i want to know where the rest of the memory is
> >       being eaten up (initially the processes are only 1k), on average,
> >       the process size is about 4x the state size.
> >    3. i am passing around many closures between the nodes of the
> >       database, and they are getting to be quite large (start out as
> >       100bytes, end up as 40kbytes).  these are what is taking up most
> >       of the memory of the state, and so i'm wondering if there is some
> >       inner workings of the erlang system that make this a bad way of
> >       programming in terms of memory usage.  it seems like there are
> >       certain situations that make closures take up a lot of memory, but
> >       i'm having trouble pinpointing when.
> >    4. is there a good tool out there for profiling the memory usage
> >       rather than executiontime usage?
> >
> > thanks for the help,
> > -matt handler
> >
> >
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > erlang-questions mailing list
> > erlang-questions@REDACTED
> > http://www.erlang.org/mailman/listinfo/erlang-questions
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://www.erlang.org/mailman/listinfo/erlang-questions
>



-- 
--Hynek (Pichi) Vychodil

Analyze your data in minutes. Share your insights instantly. Thrill your
boss.  Be a data hero!
Try Good Data now for free: www.gooddata.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20090319/69da62cf/attachment.htm>