Binary Large Erlang Dump

[Eric Newhuis]

What is the Erlang way to serialize/pickle a large stream of Erlang
terms?  We have these Erlang processes spitting out vast amounts of
Erlang terms that we would like to capture on disk for later playback.

We're talking about 20 MB per hour at least.  So we figure we'll create
one file every 20 minutes.  But we just started using Erlang and have no
idea what the best serialization format is.

Also, we need to quickly play the data back into an Erlang process.  So
we don't want to have to parse the data.  It would be nice to have some
kind of automatic stream that just regenerated messages.

We'll have a recorder that creates the files and a player that plays the
files.  I suppose, related conceptually, would be the idea of a
transaction log on disk for arbitrary Erlang messages.

Sincerely,
Eric