[eeps] New EEP: setrlimit(2) analogue for Erlang

[Michael Truog]

On 02/07/2013 07:04 PM, ok@REDACTED wrote:
>> An idea which seems like it should be included in this EEP, is simply a
>> function to determine how much real memory is consumed by an erlang
>> variable.
> Given a language with immutable data structures and free sharing,
> I am having a hard time trying to think what this might mean.
> Consider as one of the *easy* cases
>
>     L0 = [],
>     L1 = [L0|L0],
>     ...
>     L99 = [L98|L98]
>
> The amount of memory it really uses is just 99 cons cells,
> presumably 198 words.  The amount that will be calculated
> by a recursive sizer is about 2^99 (approximately).
>
> There are at least the following notions:
>  - size in external representation
>  - amount copied when sent to a local PID
>  - amount found by a recursive walker
>  - amount used taking internal cycles into account
>  - fair share considering all references in the current PID
>  - fair share including references to binaries from other PIDs.
>
> It seems to me that there is one thing that _could_ be done
> but would be a fair bit of work, and that's allocation
> profiling.  Arrange to generate different BEAM code which
> records each (amount of memory, source location where
> allocation happens) so that you can run a test case and see
> which expressions are responsible for allocating how much
> memory.
The erts_debug:size/1 that Björn-Egil mentioned is really what I want, but we would need it to be documented to have this low-level view of memory consumption be official (it would also be cool if it could be mentioned in the memory portion of the efficiency guide).

I know the view of the allocators with profiling can be done with instrument and erts_alloc_config, which provides a high-level view of memory usage by processes and allocators.  So, that part seems fine, though it is pretty complex.  Having a function like erts_debug:size/1 can help the developer avoid all the allocation complexity, and just focus on potential memory problems when errors occur, which seems like the next step after this EEP would be implemented.