[erlang-questions] [eeps] New EEP: setrlimit(2) analogue for Erlang

[Kostis Sagonas]

On 02/08/2013 02:29 PM, Björn-Egil Dahlberg wrote:
>
>>
>> The erts_debug:flat_size/1 function is fast and implemented in C, but
>> doesn't give you the true size since it ignores sharing. If Kostis
>> Sagonas' work on sharing-preserving copying operations was included in
>> OTP, we could have an efficient version of erts_debug:size/1.
>
> I haven't seen the final implementation yet. I wonder if it's complete?
> Preserved sharing in copy would be really nice as long as it is efficient.

First of all, I would like to correct Richard a bit. Although I am very 
heavily involved in the implementation of an Erlang VM that preserves 
term sharing, most of the work so far has been performed by my very good 
colleague Nikos Papaspyrou.  A paper we wrote about this work has been 
published in the 2012 ACM Erlang Workshop and can be accessed at:

   http://dx.doi.org/10.1145/2364489.2364493

or more conveniently at:

   http://user.it.uu.se/~kostis/Papers/erlang12_sharing.pdf

Although its contents are quite technical, I strongly recommend reading 
at least its introduction and Section 2 which explains all you wanted to 
know about term sharing in Erlang/OTP.


Now, regarding the actual implementation it is also publicly available 
and can be found at:

   https://github.com/nickie/otp

It's based on vanilla R16A but adds a --enable-sharing-preserving 
configure flag that can be optionally enabled to give a VM that 
preserves term sharing in intra-node process spawns and in message passing.

As far as we know it works at well as the default system and its 
overhead is extremely small in all things we have tried. (Of course, it 
also gives quite big/unbounded? speedups whenever a lot of shared terms 
are copied between processes.)

We have discussed with the OTP team the possibility of the system being 
included in R16B, so that the community has a chance to try and evaluate 
it, but no decision has been reached so far.

The only remaining issue we are aware of has to do with code unloading: 
since terms in the constant pool of a module are not copied, as the 
vanilla system does, the code loader has to either preserve these terms 
or copy them on demand when a module is purged/reloaded. The OTP team 
has promised to look into this at some point but this work has not been 
prioritized yet. But the whole system should be usable without problems 
for all applications where unloading of modules with constants does not 
take place.

Note that the implementation is restricted to intra-node term sharing: 
i.e. it does yet not support term sharing when terms are sent to 
different nodes. This is something we are currently working on and will 
most likely require the design of a new extern term format. But this is 
an orthogonal issue.

As a relatively minor point we have not tested this on Windows; though 
we suspect that there are only minor issues to possibly address there. 
Somebody with a Windows development environment should do that (hint, 
hint...).

Kostis