[erlang-questions] term_to_binary and large data structures

Thu Jun 28 02:32:03 CEST 2018

On 2018年6月27日水曜日 19時14分10秒 JST Fred Hebert wrote:
> On 06/27, Aaron Seigo wrote:
> >We have maps with 10k keys that strain this system and easily saturate 
> >our network. This is not "big" by any modern definition. As a 
> >demonstration of this to ourselves, I wrote an Elixir library that 
> >serializes terms to a more space efficient format. Where 
> >`term_to_binary` creates 500MB monsters, this library conveniently 
> >creates a 1.5MB binary out of the exact same data.
> >
> Have you tried comparing when `term_to_binary(Term, [{compressed, 9}])'?  
> If you can pack 500MB of data down to 1.5 MB, chances are that 
> compression could do some good things on your end.

With ratios like that it is also highly likely that the bulk of the data
is not actually information...


More information about the erlang-questions mailing list