[erlang-questions] term_to_binary and large data structures

[Benoit Chesneau]

also https://gitlab.com/barrel-db/teleport



On Wednesday, June 27, 2018, Chandru <chandrashekhar.mullaparthi@REDACTED>
wrote:

> On 27 June 2018 at 15:39, <zxq9@REDACTED> wrote:
>
>> On 2018年6月27日水曜日 16時19分14秒 JST Aaron Seigo wrote:
>> > Hello! :)
>> >
>> > I have a distributed (in the Erlang sense) application which often
>> > produces moderately-sized maps (10k+ entries with lots of tuples in the
>> > mix) which in the past have given inter-node message passing serious
>> > problems: the vm would lock for a long while, use several GB of RAM,
>> and
>> > usually eventually give up. When it didn't outright crash, it would
>> > produce message sizes too big to send between nodes, and/or the
>> > heartbeat messages between nodes would time out resulting in breakage.
>> > Running the same terms through `term_to_binary` produces similar
>> > results.
>>
>> If you're using disterl keep in mind that inter-node heartbeats operate
>> on the same channel as intern-node messages.
>>
>> If you send huge messages you are killing your heartbeats. That doesn't
>> play out very well.
>>
>> The *correct* solution would be to make disterl operate over SCTP, but in
>> reality SCTP isn't supported well enough cross-platform to make for a
>> universal option (thanks, Microsoft).
>>
>> Instead, consider opening a separate TCP connection between nodes for
>> tranfer of large data. That way heartbeats and small inter-node messages
>> don't interfere with your huge bulk message transfers.
>>
>> (Yes, this is totally the kind of thing that there should be a library
>> for, but it isn't something there is a lot of time for unpaid work on.
>> That sucks, but there it is.)
>>
>>
> There are a few libraries which offer this functionality. Here are two I
> know of.
>
> https://github.com/priestjim/gen_rpc
>
> https://github.com/Bluehouse-Technology/erpc
>
> Chandru
>
>
>

-- 
Sent from my Mobile
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20180704/6c594173/attachment.htm>