[erlang-patches] win32 memory allocation fix / mmap emulation

[Blaine Whittle]

-----Original Message-----
From: Björn Gustavsson [mailto:bgustavsson@REDACTED] 
Sent: Thursday, December 10, 2009 10:36 PM
To: Blaine Whittle
Cc: erlang-patches@REDACTED
Subject: Re: [erlang-patches] win32 memory allocation fix / mmap emulation

>> This patch attempts to address the win32 memory fragmentation / OMM problem.

>Thanks! I have included the current version in 'pu' so that it can be
tested and reviewed.

Awesome, thank you.

>For the next version of the patch, I would want most of the
information in your email
in the commit message itself.

>>  Additionally the link step now uses LARGEADDRESSAWARE which flags the OS to allocate 3 gigs of VM to the process instead of the standard 2 (XP requires a small config change to use it, but newer OS's should be fine.  Google "largeaddressaware 3g" for more info).

> Will an emulator built in that way work on Windows XP without the config change?

Yes, It's just a flag, either the OS can ignore the 3G flag or use it.  It's also my understanding that the flag only matters on the host process, so linking the DLL's with LARGEADDRESSAWARE should have no effect.  The flag can also be set / unset in a post build step via the visual studio command line tool editbin.

This change could have been broken out into its own commit, however without the MMAP change Erlang would still die around 1.1 or 1.3 GB on Windows.  

>If the answer is no, this change should be made optional and possible
to enable using a configure option.
If this change is totally separate from the other changes, it would be
best to split it out into a separate
commit (on the same branch).


>> I left this part of the build manual as I'm unsure exactly where the defines should be placed (i.e. either standard windows defines in erl_win_sys.h, a change to the compiler params in the win32 makefile , or via a change to the configure script).

> I cannot give you a definite answer about this yet. I'll have to
consult to consult
with the experts on Windows and the memory manager.

> My gut feeling, though, is that lying about Windows having MMAP might
not be the best way to
got, as MMAP could possibly be used in other places in a future
version of OTP. So it may be
better to change the memory management code to test some other define
that covers both the
mmap() case and the virtual allocation functions on Windows. (For ease
of reviewing, it would
be a good idea to put such re-factoring into a separate commit.)

Ya, I was unsure the best way to handle the defines for that very reason.  The only files that involved the MMAP change are erl_mseg.h and erl_mseg.c.  However any win32 specific should go in win32\erl_win_sys.h and win32\sys.c

 
> Assuming that this patch works well, will there be any reason for
someone wanting to turn it off?
If the answer is yes, there should be a configure option to do that.

There are no reasons to turn it off that I'm aware of.   We switched our win32 erlang VM's to a build with the MMAP stuff enabled and so far our Erlang nodes appears to be much more stable  as we were having a large number of OOM crashes before the change. 

Blaine