[erlang-questions] term_to_binary and large data structures
Wed Jun 27 16:19:14 CEST 2018
I have a distributed (in the Erlang sense) application which often
produces moderately-sized maps (10k+ entries with lots of tuples in the
mix) which in the past have given inter-node message passing serious
problems: the vm would lock for a long while, use several GB of RAM, and
usually eventually give up. When it didn't outright crash, it would
produce message sizes too big to send between nodes, and/or the
heartbeat messages between nodes would time out resulting in breakage.
Running the same terms through `term_to_binary` produces similar
The good news is that in OTP 21.0 things are quite a bit better:
serialization of the maps goes a lot quicker, memory usage is now only
~500MB per encoding for terms which would quickly balloon in the
multiple GB's, ... so there is progress and that is really fantastic.
Is 21.0 using something other than `term_to_binary` now for inter-node
Is it still using the "standard" external term format?
Where in the OTP source tree can one find the relevant code?
More information about the erlang-questions