[erlang-questions] Maximising memory compactness

[Szoboszlay Dániel]

Hi,

I'd be in favor of the half-word mode:
Having multiple 32-bit VMs would multiply the beam code stored in memory  
as well. This could mean a lot of memory wasted.
Using 32-bit VMs would not solve the 4GB limit on stack/heap size. On the  
contrary, the 4GB limit is for the total memory consumption of the 32-bit  
node. In the half-word mode the limit is per process; and you could even  
move data to private ETS tables from the heap for extremely memory hungry  
processes if nothing else helps.
If parts of your code needs HiPE, consider splitting out that part into a  
separate 32-bit VM running on the same node. So you will have two nodes  
per host: one half-word node for processes handling lot of data, and one  
32-bit node with HiPE for processes doing number crunching.
Regards,
Daniel

On Fri, 04 Apr 2014 10:33:30 +0200, Olivier BOUDEVILLE  
<olivier.boudeville@REDACTED> wrote:

> Hello all,
> Let's suppose one would like to run a large, distributed (Erlang)  
> application, with many processes, and that ends up, in some cases, being  
> RAM-bound.
> In order to squeeze more processes into each node, apart from using the  
> best data-structures to reduce memory footprint and from having the  
> least active processes hibernate, I >imagine that a good option could be  
> to rely on "smaller pointers" (as they must be used internally a lot by  
> the VM, for most terms), i.e. either using the 64-bit half-word emulator  
> or >instanciating as many 32-bit VMs as needed to "pave" the actual RAM  
> of each host, and have them communicate. Apparently large gains in terms  
> of memory could be expected.
> The half-word emulator would seem the simplest/most elegant solution to  
> do so (ex: I suppose that CPU extended registers, instruction sets, etc.  
> can then be used - which wouldn't >be the case for VMs compiled for  
> 32-bit addressing; moreover 32-bit VMs would probably have to  
> communicate through the loopback or alike, involving I suppose much  
> >marshalling/demarshalling) but apparently:       - we currently cannot  
> have both the half-word emulator *and* the HiPe-based native  
> compilation, so we have to choose between RAM and CPU (don't know if  
> targeting Erllvm >could help)       - more importantly, with the  
> half-word emulator, the total RAM for stacks + heaps for processes would  
> still have to be under 4GB (as stated apparently by  
> http://www.erlang->factory.com/upload/presentations/569/Halfword_Erlang_Factory_SF_2012.pdf)
> So, my question: would relying on a set of 32-bit VMs per host, instead  
> of a single 64-bit VM, look like the best option here?
> Thanks in advance for any hint!
> Best regards,
>> Olivier.
> ---------------------------
> Olivier Boudeville
>
> EDF R&D : 1, avenue du Général de Gaulle, 92140 Clamart, France
> Département SINETICS, groupe ASICS (I2A), bureau B-226
> Office : +33 1 47 65 59 58 / Mobile : +33 6 16 83 37 22 / Fax : +33 1 47  
> 65 27 13
>
> Ce message et toutes les pièces jointes (ci-après le 'Message') sont  
> établis à l'intention exclusive des destinataires et les informations  
> qui y figurent sont strictement confidentielles. Toute >utilisation de  
> ce Message non conforme à sa destination, toute diffusion ou toute  
> publication totale ou partielle, est interdite sauf autorisation  
> expresse.
>
> Si vous n'êtes pas le destinataire de ce Message, il vous est interdit  
> de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout  
> ou partie. Si vous avez reçu ce Message par erreur, merci >de le  
> supprimer de votre système, ainsi que toutes ses copies, et de n'en  
> garder aucune trace sur quelque support que ce soit. Nous vous  
> remercions également d'en avertir immédiatement >l'expéditeur par retour  
> du message.
>
> Il est impossible de garantir que les communications par messagerie  
> électronique arrivent en temps utile, sont sécurisées ou dénuées de  
> toute erreur ou virus.
> ____________________________________________________
>
> This message and any attachments (the 'Message') are intended solely for  
> the addressees. The information contained in this Message is  
> confidential. Any use of information contained in this >Message not in  
> accord with its purpose, any dissemination or disclosure, either whole  
> or partial, is prohibited except formal approval.
>
> If you are not the addressee, you may not copy, forward, disclose or use  
> any part of it. If you have received this message in error, please  
> delete it and all copies from your system and notify the >sender  
> immediately by return message.
>
> E-mail communication cannot be guaranteed to be timely secure, error or  
> virus-free.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20140404/284c50b0/attachment.htm>