[erlang-questions] term_to_binary and large data structures

[zxq9@REDACTED]

On 2018年6月27日水曜日 19時14分10秒 JST Fred Hebert wrote:
> On 06/27, Aaron Seigo wrote:
> >We have maps with 10k keys that strain this system and easily saturate 
> >our network. This is not "big" by any modern definition. As a 
> >demonstration of this to ourselves, I wrote an Elixir library that 
> >serializes terms to a more space efficient format. Where 
> >`term_to_binary` creates 500MB monsters, this library conveniently 
> >creates a 1.5MB binary out of the exact same data.
> >
> 
> Have you tried comparing when `term_to_binary(Term, [{compressed, 9}])'?  
> If you can pack 500MB of data down to 1.5 MB, chances are that 
> compression could do some good things on your end.

With ratios like that it is also highly likely that the bulk of the data
is not actually information...

-Craig