[erlang-questions] Sending a large Erlang content to a set of remote nodes

Thu Mar 28 18:09:42 CET 2013

Hi,

I tried the two most obvious solutions (respectively: file:read_file/1 
followed by gen_tcp:send/2, and file:sendfile/2), of course both work but 
the second is not only faster but able to cope with larger files (whereas 
the first approach would have needed a more complex sending by chunks, 
lest it fails with enomem - i.e. not enough memory).

So I will stick with as many file:sendfile/2 concurrent calls as needed on 
the same file, this should be sufficient for my use case. 

Cheers,

Olivier.
---------------------------
Olivier Boudeville

EDF R&D : 1, avenue du Général de Gaulle, 92140 Clamart, France
Département SINETICS, groupe ASICS (I2A), bureau B-226
Office : +33 1 47 65 59 58 / Mobile : +33 6 16 83 37 22 / Fax : +33 1 47 
65 27 13

vances@REDACTED 
27/03/2013 17:54

A
olivier.boudeville@REDACTED
cc
erlang-questions@REDACTED
Objet
Re: [erlang-questions] Sending a large Erlang content to a set of remote 
nodes

I have a similar use case for which I am considering using IP 
multicasting  and implementing a reliable layer on top. Basically 
broadcasting sends but buffering data until there's been time for negative 
acknowledgements.
On Mar 27, 2013 10:18 PM, "Olivier BOUDEVILLE" <olivier.boudeville@REDACTED> 
wrote:

Hi, 

I would have a potentially large number of Erlang terms to send to a set 
of remote VMs. Currently it is a matter of a few megabytes, but it could 
be up to a few gigabytes in the future; and the exact same content may 
have to be sent to typically a dozen of other nodes. 

I was searching for a solution that would be reliable/simple/efficient to 
do so (preferably in that order), knowing that these terms could be either 
be kept in the RAM of the sender or, maybe preferably (the size of the 
data being probably roughly on par with the local RAM), as a compressed 
file on disk. 

Currently I send a binary, compressed archive thanks to a basic Erlang 
message, but I think it is not a good practice (ex: maybe the kernel ticks 
are not sent "out of band" and their delaying by larger archives could 
trigger spurious time-outs). I imagine sendfile with enough async threads 
could be a good candidate, however I am unsure that the same content 
(either as a whole or by chunks) could be read once, yet be sent to 
multiple recipients. 

Any idea? 

Thanks in advance for any hint! 

Best regards, 

Olivier.
---------------------------
Olivier Boudeville

EDF R&D : 1, avenue du Général de Gaulle, 92140 Clamart, France
Département SINETICS, groupe ASICS (I2A), bureau B-226
Office : +33 1 47 65 59 58 / Mobile : +33 6 16 83 37 22 / Fax : +33 1 47 
65 27 13

Ce message et toutes les pièces jointes (ci-après le 'Message') sont 
établis à l'intention exclusive des destinataires et les informations qui 
y figurent sont strictement confidentielles. Toute utilisation de ce 
Message non conforme à sa destination, toute diffusion ou toute 
publication totale ou partielle, est interdite sauf autorisation expresse.
Si vous n'êtes pas le destinataire de ce Message, il vous est interdit de 
le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou 
partie. Si vous avez reçu ce Message par erreur, merci de le supprimer de 
votre système, ainsi que toutes ses copies, et de n'en garder aucune trace 
sur quelque support que ce soit. Nous vous remercions également d'en 
avertir immédiatement l'expéditeur par retour du message.
Il est impossible de garantir que les communications par messagerie 
électronique arrivent en temps utile, sont sécurisées ou dénuées de toute 
erreur ou virus.
____________________________________________________
This message and any attachments (the 'Message') are intended solely for 
the addressees. The information contained in this Message is confidential. 
Any use of information contained in this Message not in accord with its 
purpose, any dissemination or disclosure, either whole or partial, is 
prohibited except formal approval.
If you are not the addressee, you may not copy, forward, disclose or use 
any part of it. If you have received this message in error, please delete 
it and all copies from your system and notify the sender immediately by 
return message.
E-mail communication cannot be guaranteed to be timely secure, error or 
virus-free.

_______________________________________________
erlang-questions mailing list
erlang-questions@REDACTED
http://erlang.org/mailman/listinfo/erlang-questions

Ce message et toutes les pièces jointes (ci-après le 'Message') sont établis à l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme à sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse.

Si vous n'êtes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez reçu ce Message par erreur, merci de le supprimer de votre système, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions également d'en avertir immédiatement l'expéditeur par retour du message.

Il est impossible de garantir que les communications par messagerie électronique arrivent en temps utile, sont sécurisées ou dénuées de toute erreur ou virus.
____________________________________________________

This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval.

If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message.

E-mail communication cannot be guaranteed to be timely secure, error or virus-free.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20130328/c2dc5ded/attachment.htm>