[erlang-questions] term_to_binary and large data structures

[Aaron Seigo]

Hello! :)

I have a distributed (in the Erlang sense) application which often 
produces moderately-sized maps (10k+ entries with lots of tuples in the 
mix) which in the past have given inter-node message passing serious 
problems: the vm would lock for a long while, use several GB of RAM, and 
usually eventually give up. When it didn't outright crash, it would 
produce message sizes too big to send between nodes, and/or the 
heartbeat messages between nodes would time out resulting in breakage. 
Running the same terms through `term_to_binary` produces similar 
results.

The good news is that in OTP 21.0 things are quite a bit better: 
serialization of the maps goes a lot quicker, memory usage is now only 
~500MB per encoding for terms which would quickly balloon in the 
multiple GB's, ... so there is progress and that is really fantastic.

Is 21.0 using something other than `term_to_binary` now for inter-node 
messages?

Is it still using the "standard" external term format?

Where in the OTP source tree can one find the relevant code?

Cheers!

--
Aaron Seigo