[erlang-questions] term_to_binary and large data structures

[Fred Hebert]

On 06/27, Aaron Seigo wrote:
>We have maps with 10k keys that strain this system and easily saturate 
>our network. This is not "big" by any modern definition. As a 
>demonstration of this to ourselves, I wrote an Elixir library that 
>serializes terms to a more space efficient format. Where 
>`term_to_binary` creates 500MB monsters, this library conveniently 
>creates a 1.5MB binary out of the exact same data.
>

Have you tried comparing when `term_to_binary(Term, [{compressed, 9}])'?  
If you can pack 500MB of data down to 1.5 MB, chances are that 
compression could do some good things on your end.

Regards,
Fred.