[erlang-questions] LambdaStream's home-made memory management for high-rate binary data

[Samuel Rivas]

Joel Reymont wrote:
> For those of us unable to attend your presentation, would you kindly
> elaborate on why you needed a home-made C++ memory management solution for
> high-rate binary data?

It is more C than C++, I tend to avoid the latter as much as possible (do not
start a war with this please :).

The thing is that for low-level stream processing, I could not make bitsyntax
based code fast enough. In fact, the result was much slower than real-time. The
problem is that automatic memory handling devices (such as GC) used by
high-level languages fail for that kind of applications. At the end, we had to
choose C and handle memory by hand (that comes for free with it).

For example, a Spanish TDT transport stream usually carries 4 TV channels. The
whole stream bit-rate is 20 mbit/s. We managed to extract meta-information in
the stream that show up around each 0.1 second. That is, Erlang is able to read
the 20 mbit/s with bitsyntax, throwing away most of the bytes and parsing only
relevant data, with rate of hundreds of bits/s only. Also, that kind of data are
relatively easy to parse, because they come in fixed-size chunks of 188 bytes.

In that case bitsyntax and automatic memory handling is fast enough, and saves
us to write tons of code.

But, if you try to parse the elementary video streams, which have data fields
that are not byte-aligned, performance drops quickly.

I have experimented several times with that, but I was not able to get
acceptable results, by far. For that kind of things we had to develop a C
library, where much of the effort is not only keep a reasonable performance
(which is relatively easy), but also avoiding memory leaks, memory faults, and
other stuff that you get for free when you get closer the hardware (which is not
easy).

Of course, the fact that I did not succeed on that does not mean that it is
impossible, so I will be happy if someone says that it is possible.

Also, Per Gustafsson advanced the new bitsyntax including bitstreams for
R12. Part of the burden of parsing unaligned data is that you need extra code to
handle padding chunks. I am not sure if that may be a performance penalty too
(apart for being really boring :). I have to experiment with that to check
whether we can get code fast enough. Another alternative is waiting until a
suitable hardware evolution :)

In short. When we are not able to use Erlang to handle input data, we have to do
by hand all the cool stuff that a garbage collector and a nice virtual machine
do for you.

Regards
-- 
	Samuel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20071115/2c8f1ee9/attachment.bin>