[erlang-questions] How to tune erlang memory allocator to use 25 GB of ram and stop munmaping it?
Mon Aug 26 13:49:51 CEST 2013
On 2013-08-20 15:04:41 +0000, Lukas Larsson said:
> Hello Peer,
> On Tue, Aug 20, 2013 at 12:43 PM, Peer Stritzinger <peerst@REDACTED> wrote:
> My favorite behaviour would be for Erlang to either grab all memory
> available or at least a fixed amount on startup and hold on to it
> forever. Hopefully without fragmenting it too much (I know in case of
> binary heaps that this isn't possible which is a bit disturbing to me).
> Can this be achieved by the current ERTS with some parameter settings?
> You could request that the main mbc (+M<S>mmbcs) should be preallocated
> to a certain size, and this would kind of get the effect you are
> looking for. However this is controlled on a per allocator level, so
> you have to figure out which type of memory it is that you are using.
> It also will only work for any mbcs allocated, all sbcs will still be
> allocated using mmap/malloc. If you want to force only mbcs you could
> up the sbct to very big, just remember to up lmbcs as well so that can
> fit all of your sbcs into the mbc.
> I do not recommend doing this, but it is possible.
I see, probably time is better to just optimize the system malloc for
erlangs use cases.
> Which direction should I look if I want to add behaviour like this?
> BTW: next best thing is to malloc/free allways same size tiles this
> would avoid outer fragmentation and could be sped up relatively easily.
> The emulator already tries to do this to the best of its abilities.
> mbcs mmap requests grow depending on how you setup lmbcs, smbcs and
> mbcgs. sys_alloc (i.e. malloc) calls are always multiples of +Muycs.
Having allocs multiples of a fixed number helps against small scale
fragmentation, on the larger scale if there is no memory mapping there
is still possibility of fragmentation with this scheme.
> You could set smbcs to the same as lmbcs if you want to, this makes it
> so that all mbcs have that same size.
I'l try this out, any downsides of doing this?
This would remove all fragmentation issues and malloc/free could be
sped up by just having a special free list for these fixed size tiles.
More information about the erlang-questions