[erlang-questions] Sending a large Erlang content to a set of remote nodes

[Olivier BOUDEVILLE]

Hi,

I would have a potentially large number of Erlang terms to send to a set 
of remote VMs. Currently it is a matter of a few megabytes, but it could 
be up to a few gigabytes in the future; and the exact same content may 
have to be sent to typically a dozen of other nodes.

I was searching for a solution that would be reliable/simple/efficient to 
do so (preferably in that order), knowing that these terms could be either 
be kept in the RAM of the sender or, maybe preferably (the size of the 
data being probably roughly on par with the local RAM), as a compressed 
file on disk. 

Currently I send a binary, compressed archive thanks to a basic Erlang 
message, but I think it is not a good practice (ex: maybe the kernel ticks 
are not sent "out of band" and their delaying by larger archives could 
trigger spurious time-outs). I imagine sendfile with enough async threads 
could be a good candidate, however I am unsure that the same content 
(either as a whole or by chunks) could be read once, yet be sent to 
multiple recipients. 

Any idea? 

Thanks in advance for any hint!

Best regards,

Olivier.
---------------------------
Olivier Boudeville

EDF R&D : 1, avenue du Général de Gaulle, 92140 Clamart, France
Département SINETICS, groupe ASICS (I2A), bureau B-226
Office : +33 1 47 65 59 58 / Mobile : +33 6 16 83 37 22 / Fax : +33 1 47 
65 27 13



Ce message et toutes les pièces jointes (ci-après le 'Message') sont établis à l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme à sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse.

Si vous n'êtes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez reçu ce Message par erreur, merci de le supprimer de votre système, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions également d'en avertir immédiatement l'expéditeur par retour du message.

Il est impossible de garantir que les communications par messagerie électronique arrivent en temps utile, sont sécurisées ou dénuées de toute erreur ou virus.
____________________________________________________

This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval.

If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message.

E-mail communication cannot be guaranteed to be timely secure, error or virus-free.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20130327/6cba9f00/attachment.htm>