[erlang-patches] Win64 memory corruption fix

[Blaine Whittle]

git fetch git://github.com/bwhittle/otp.git win-64-pointer-fix

https://github.com/bwhittle/otp/commit/1d21ce8d0287a0a50b2e42631d361f43ce14e23e

This patch should fix a number of memory corruption issues and / or crashes on Win64 that can potential occur when the Erlang VM exceeds 4 GB of ram.    The problem stems from casting pointers to unsigned long and assuming long is type that is always large enough to hold a pointer.  This assumption holds up for all platforms except windows.

Nix 32 (unsigned long) -> 32 bit                  (pointer size = unsigned long)
Nix 64 (unsigned long) -> 64 bit                  (pointer size = unsigned long)
Win 32 (unsigned long) -> 32 bit                (pointer size = unsigned long)
Win 64 (unsigned long) -> 32 bit                (pointer size != unsigned long)

To compound the problem these casts can appear to be fine on Win64 as only those pointers that reference memory above the 32 bit address space will lead to issues.    Which means you need Erlang to allocate ~ 4 GB of memory before you even have a chance or running into problems.     The issue is if you have a pointer that reference memory above the 32 bit address space on Win64 and then type cast it to a long (i.e. 32 bits) and then turn around and use that type cast value as a pointer then you'll be referencing a different memory location.   Most of the time the incorrect pointer will still reference a valid location as memory is allocated bottom up which can lead to memory corruption.

This patch has been tested heavily and has been used on production systems.  I made the changes a year ago when the Win64 Erlang VM was released (just didn't mean to wait so long to submit it.)

The patch submission page recommends that I create new test cases which I have not done.    However I have a small registry change that should be applied on any systems that execute Erlang Win64 smoke and / or unit tests.

The registry change instructs Windows to allocate memory from top down, meaning that any valid memory pointers will require a 64 bit value and any attempt to cast them to a 32 bit value followed by a dereference will produce an access violation.

http://msdn.microsoft.com/en-us/library/windows/desktop/bb613473(v=vs.85).aspx

To apply the registry setting just copy and paste the following section and place it into a <some name>.reg file and import it on each test machines followed by a reboot.
=============================================================
Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management]
"AllocationPreference"=dword:00100000.
=============================================================

With this registry change Erlang's existing unit tests should be able to catch any incorrect pointer casts by causing the VM to crash.   Every pointer will reference memory above the 32 value so type casting it to a 32 bit value and then dereferencing causes an access violation.

One potential issue with using this registry setting is that if your test machines rely on 3rd party Win64 apps it's possible they may crash on startup (that is if they contain similar type casting bugs.)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-patches/attachments/20130206/a3c92d5b/attachment.htm>