[erlang-questions] Sending a large Erlang content to a set of remote nodes

[Joe Armstrong]

On Wed, Mar 27, 2013 at 5:48 PM, Olivier BOUDEVILLE <
olivier.boudeville@REDACTED> wrote:

>
> Hi,
>
> I would have a potentially large number of Erlang terms to send to a set
> of remote VMs. Currently it is a matter of a few megabytes, but it could be
> up to a few gigabytes in the future; and the exact same content may have to
> be sent to typically a dozen of other nodes.
>

How big is the set of remote VMs (how many dozens?) -
Is the communication bandwidth between machines symmetric.

Once two machines have got a copy they could *both* send the data to a
third.
Machine one sends the first half, machine two the second. Now three
machines have a copy

Now three machines can send a copy to a fourth, the first can send the
first third, ...
and so on.

Lookup epidemic gossip protocols.

This is a very nice exercise in parallel programming.

Cheers

/Joe



>
> I was searching for a solution that would be reliable/simple/efficient to
> do so (preferably in that order), knowing that these terms could be either
> be kept in the RAM of the sender or, maybe preferably (the size of the data
> being probably roughly on par with the local RAM), as a compressed file on
> disk.
>
> Currently I send a binary, compressed archive thanks to a basic Erlang
> message, but I think it is not a good practice (ex: maybe the kernel ticks
> are not sent "out of band" and their delaying by larger archives could
> trigger spurious time-outs). I imagine sendfile with enough async threads
> could be a good candidate, however I am unsure that the same content
> (either as a whole or by chunks) could be read once, yet be sent to
> multiple recipients.
>
> Any idea?
>
> Thanks in advance for any hint!
>
> Best regards,
>
> Olivier.
> ---------------------------
> Olivier Boudeville
>
> EDF R&D : 1, avenue du Général de Gaulle, 92140 Clamart, France
> Département SINETICS, groupe ASICS (I2A), bureau B-226
> Office : +33 1 47 65 59 58 / Mobile : +33 6 16 83 37 22 / Fax : +33 1 47
> 65 27 13
>
>
> Ce message et toutes les pièces jointes (ci-après le 'Message') sont
> établis à l'intention exclusive des destinataires et les informations qui y
> figurent sont strictement confidentielles. Toute utilisation de ce Message
> non conforme à sa destination, toute diffusion ou toute publication totale
> ou partielle, est interdite sauf autorisation expresse.
>
> Si vous n'êtes pas le destinataire de ce Message, il vous est interdit de
> le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou
> partie. Si vous avez reçu ce Message par erreur, merci de le supprimer de
> votre système, ainsi que toutes ses copies, et de n'en garder aucune trace
> sur quelque support que ce soit. Nous vous remercions également d'en
> avertir immédiatement l'expéditeur par retour du message.
>
> Il est impossible de garantir que les communications par messagerie
> électronique arrivent en temps utile, sont sécurisées ou dénuées de toute
> erreur ou virus.
> ____________________________________________________
>
> This message and any attachments (the 'Message') are intended solely for
> the addressees. The information contained in this Message is confidential.
> Any use of information contained in this Message not in accord with its
> purpose, any dissemination or disclosure, either whole or partial, is
> prohibited except formal approval.
>
> If you are not the addressee, you may not copy, forward, disclose or use
> any part of it. If you have received this message in error, please delete
> it and all copies from your system and notify the sender immediately by
> return message.
>
> E-mail communication cannot be guaranteed to be timely secure, error or
> virus-free.
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20130328/189aad5f/attachment.htm>