[erlang-questions] Losing data

[Bob Cowdery]

On 02/06/2011 20:20, Jesper Louis Andersen wrote:
> On Thu, Jun 2, 2011 at 21:09, Bob Cowdery <bob@REDACTED> wrote:
>
>> The symptoms look like it just can't keep up but If a process was
>> getting behind I would expect the message q to build up. CPU is 5-7%.
>>
>> Any ideas would be very welcome.
>>
> Ladies and Gentlemen! Welcome to the Circus of the Kernel! Tonight we
> will have artist of all kinds: Jugglers! Trapezoids! Old Erlang
> programmers. A balancing act with a VAX! And above all, there will be
> UDP throwing!
>
> Your problem is that you are sending data out too fast and the kernel
> is then throwing your data away. This is especially true on Linux
> where this happens silently. On FreeBSD I have seen ENOBUFS being sent
> back when this happen, but Linux just choses to silently drop packets.
> The only way to fix this is to pace packets so they are sent at a
> slower rate - and build in a retransmit mechanism. It is the only way
> to be sure. Note that the send buffer can be ridicously small on some
> systems, so you can't rely on it. Though my code using UDP can surely
> send around 8K on Linux without too much trouble.
If the data was being thrown away after it left the application I would
expect to see the input and output sequence numbers keep in step. The
input sequence is added by the hardware device and I add the output
sequence number before transmission. I check the input sequence and
shout if there is a sequence error and the device checks the output
sequence and lights a nice little LED if it sees a sequence error. Both
show correct sequences so I'm losing blocks inside Erlang somewhere.
> Since the queue is tail-end dropping you see the problem where many
> packets are lost suddenly. The queue fill up, and them you are in for
> the surprise when the next 30 packets are just skipped and dropped.
It sounds really plausible if only it fitted the facts.
> There are a couple of fixes depending on what you want to do, but if
> possible go check out SCTP which can give you many of the things you
> want from UDP. It is kind of the hybrid between UDP and TCP and then
> not. it is also a very interesting protocol - check it out!
>
I don't have control over the hardware device so it has to be UDP. Looks
an interesting protocol though for future reference.