[erlang-questions] term_to_binary and large data structures

[zxq9@REDACTED]

On 2018年6月27日水曜日 16時19分14秒 JST Aaron Seigo wrote:
> Hello! :)
> 
> I have a distributed (in the Erlang sense) application which often 
> produces moderately-sized maps (10k+ entries with lots of tuples in the 
> mix) which in the past have given inter-node message passing serious 
> problems: the vm would lock for a long while, use several GB of RAM, and 
> usually eventually give up. When it didn't outright crash, it would 
> produce message sizes too big to send between nodes, and/or the 
> heartbeat messages between nodes would time out resulting in breakage. 
> Running the same terms through `term_to_binary` produces similar 
> results.

If you're using disterl keep in mind that inter-node heartbeats operate
on the same channel as intern-node messages.

If you send huge messages you are killing your heartbeats. That doesn't
play out very well.

The *correct* solution would be to make disterl operate over SCTP, but in
reality SCTP isn't supported well enough cross-platform to make for a
universal option (thanks, Microsoft).

Instead, consider opening a separate TCP connection between nodes for
tranfer of large data. That way heartbeats and small inter-node messages
don't interfere with your huge bulk message transfers.

(Yes, this is totally the kind of thing that there should be a library
for, but it isn't something there is a lot of time for unpaid work on.
That sucks, but there it is.)

-Craig