[erlang-questions] term_to_binary and large data structures

[Chandru]

On 27 June 2018 at 15:39, <zxq9@REDACTED> wrote:

> On 2018年6月27日水曜日 16時19分14秒 JST Aaron Seigo wrote:
> > Hello! :)
> >
> > I have a distributed (in the Erlang sense) application which often
> > produces moderately-sized maps (10k+ entries with lots of tuples in the
> > mix) which in the past have given inter-node message passing serious
> > problems: the vm would lock for a long while, use several GB of RAM, and
> > usually eventually give up. When it didn't outright crash, it would
> > produce message sizes too big to send between nodes, and/or the
> > heartbeat messages between nodes would time out resulting in breakage.
> > Running the same terms through `term_to_binary` produces similar
> > results.
>
> If you're using disterl keep in mind that inter-node heartbeats operate
> on the same channel as intern-node messages.
>
> If you send huge messages you are killing your heartbeats. That doesn't
> play out very well.
>
> The *correct* solution would be to make disterl operate over SCTP, but in
> reality SCTP isn't supported well enough cross-platform to make for a
> universal option (thanks, Microsoft).
>
> Instead, consider opening a separate TCP connection between nodes for
> tranfer of large data. That way heartbeats and small inter-node messages
> don't interfere with your huge bulk message transfers.
>
> (Yes, this is totally the kind of thing that there should be a library
> for, but it isn't something there is a lot of time for unpaid work on.
> That sucks, but there it is.)
>
>
There are a few libraries which offer this functionality. Here are two I
know of.

https://github.com/priestjim/gen_rpc

https://github.com/Bluehouse-Technology/erpc

Chandru
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20180627/7bc4e5ce/attachment.htm>