Parsing infinite streams style

[Jay Nelson]

Hal Snyder wrote:

 > makes me wonder about speed gains possible by
 >      a. using binaries instead of lists

Instead of ++ keep a list of binaries received so far.
You may have to split the last one if there is extra left
over.  You do not want to concatenate binaries because
you will end up copying the entire binary for each of
the elements every time you concatenate.

Binaries will only be more efficient if you can deal with
a list of them and especially if you are relaying them
out a socket.  You can just hand the list to the socket
and not have to bother with concatenating them.

The dark side of this venture is that it is much harder
to parse a handful of binaries that are broken into
arbitrary sized chunks (based on the receive semantics).

Best bet is to make it work with lists.  If the performance
is bad, retrofit a conversion to binaries -- knowing that the
debugging of the parser will be much harder, but you will
have a working version to compare to.

jay