[erlang-questions] term_to_binary and large data structures

[Jesper Louis Andersen]

The map() type now has iterators, so you can gradually iterate over the map
rather than having to convert it all at once. Maybe that is what is helping
you.

However, I'd strongly recommend you start building up a scheme in which you
chunk the large messages into smaller messages with some kind of
continuation token. Large messages are bound to create trouble at some
point. You can also toy with the idea of moving the code to the data rather
than data to the code.

On Wed, Jun 27, 2018 at 4:19 PM Aaron Seigo <aseigo@REDACTED> wrote:

> Hello! :)
>
> I have a distributed (in the Erlang sense) application which often
> produces moderately-sized maps (10k+ entries with lots of tuples in the
> mix) which in the past have given inter-node message passing serious
> problems: the vm would lock for a long while, use several GB of RAM, and
> usually eventually give up. When it didn't outright crash, it would
> produce message sizes too big to send between nodes, and/or the
> heartbeat messages between nodes would time out resulting in breakage.
> Running the same terms through `term_to_binary` produces similar
> results.
>
> The good news is that in OTP 21.0 things are quite a bit better:
> serialization of the maps goes a lot quicker, memory usage is now only
> ~500MB per encoding for terms which would quickly balloon in the
> multiple GB's, ... so there is progress and that is really fantastic.
>
> Is 21.0 using something other than `term_to_binary` now for inter-node
> messages?
>
> Is it still using the "standard" external term format?
>
> Where in the OTP source tree can one find the relevant code?
>
> Cheers!
>
> --
> Aaron Seigo
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>


-- 
J.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20180627/c2dd584f/attachment.htm>