[erlang-questions] How to tune erlang memory allocator to use 25 GB of ram and stop munmaping it?

[Peer Stritzinger]

On 2013-08-20 15:04:41 +0000, Lukas Larsson said:

> Hello Peer,
> 
> On Tue, Aug 20, 2013 at 12:43 PM, Peer Stritzinger <peerst@REDACTED> wrote:
> My favorite behaviour would be for Erlang to either grab all memory 
> available or at least a fixed amount on startup and hold on to it 
> forever.  Hopefully without fragmenting it too much (I know in case of 
> binary heaps that this isn't possible which is a bit disturbing to me).
> 
> Can this be achieved by the current ERTS with some parameter settings?
> 
> You could request that the main mbc (+M<S>mmbcs) should be preallocated 
> to a certain size, and this would kind of get the effect you are 
> looking for. However this is controlled on a per allocator level, so 
> you have to figure out which type of memory it is that you are using. 
> It also will only work for any mbcs allocated, all sbcs will still be 
> allocated using mmap/malloc. If you want to force only mbcs you could 
> up the sbct to very big, just remember to up lmbcs as well so that can 
> fit all of your sbcs into the mbc.
> 
> I do not recommend doing this, but it is possible.

I see, probably time is better to just optimize the system malloc for 
erlangs use cases.

> Which direction should I look if I want to add behaviour like this?
> 
> BTW:  next best thing is to malloc/free allways same size tiles this 
> would avoid outer fragmentation and could be sped up relatively easily.
> 
> The emulator already tries to do this to the best of its abilities. 
> mbcs mmap requests grow depending on how you setup lmbcs, smbcs and 
> mbcgs. sys_alloc (i.e. malloc) calls are always multiples of +Muycs.

Having allocs multiples of a fixed number helps against small scale 
fragmentation, on the larger scale if there is no memory mapping there 
is still possibility of fragmentation with this scheme.

> You could set smbcs to the same as lmbcs if you want to, this makes it 
> so that all mbcs have that same size.

I'l try this out, any downsides of doing this?

This would remove all fragmentation issues and malloc/free could be 
sped up by just having a special free list for these fixed size tiles.

Cheers
-- Peer