[erlang-questions] term_to_binary and large data structures

[T Ty]

also Ethernet Bonding.


On Wed, Jul 4, 2018 at 11:49 AM, Benoit Chesneau <bchesneau@REDACTED>
wrote:

> also https://gitlab.com/barrel-db/teleport
>
>
>
> On Wednesday, June 27, 2018, Chandru <chandrashekhar.mullaparthi@REDACTED>
> wrote:
>
>> On 27 June 2018 at 15:39, <zxq9@REDACTED> wrote:
>>
>>> On 2018年6月27日水曜日 16時19分14秒 JST Aaron Seigo wrote:
>>> > Hello! :)
>>> >
>>> > I have a distributed (in the Erlang sense) application which often
>>> > produces moderately-sized maps (10k+ entries with lots of tuples in
>>> the
>>> > mix) which in the past have given inter-node message passing serious
>>> > problems: the vm would lock for a long while, use several GB of RAM,
>>> and
>>> > usually eventually give up. When it didn't outright crash, it would
>>> > produce message sizes too big to send between nodes, and/or the
>>> > heartbeat messages between nodes would time out resulting in breakage.
>>> > Running the same terms through `term_to_binary` produces similar
>>> > results.
>>>
>>> If you're using disterl keep in mind that inter-node heartbeats operate
>>> on the same channel as intern-node messages.
>>>
>>> If you send huge messages you are killing your heartbeats. That doesn't
>>> play out very well.
>>>
>>> The *correct* solution would be to make disterl operate over SCTP, but in
>>> reality SCTP isn't supported well enough cross-platform to make for a
>>> universal option (thanks, Microsoft).
>>>
>>> Instead, consider opening a separate TCP connection between nodes for
>>> tranfer of large data. That way heartbeats and small inter-node messages
>>> don't interfere with your huge bulk message transfers.
>>>
>>> (Yes, this is totally the kind of thing that there should be a library
>>> for, but it isn't something there is a lot of time for unpaid work on.
>>> That sucks, but there it is.)
>>>
>>>
>> There are a few libraries which offer this functionality. Here are two I
>> know of.
>>
>> https://github.com/priestjim/gen_rpc
>>
>> https://github.com/Bluehouse-Technology/erpc
>>
>> Chandru
>>
>>
>>
>
> --
> Sent from my Mobile
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20180704/70557d31/attachment.htm>