[erlang-questions] Losing data

[Bob Cowdery]

On 02/06/2011 22:05, Valentin Micic wrote:
> Just increase the UDP buffer size on both machines.
Why would that help? Am I missing something in my analysis? There is
only one computer, the other end is dedicated hardware. This is only
intended to work on a local network.

Bob
> BR
>
> V/
>
> On 02 Jun 2011, at 11:02 PM, Bob Cowdery wrote:
>
>> On 02/06/2011 20:20, Jesper Louis Andersen wrote:
>>> On Thu, Jun 2, 2011 at 21:09, Bob Cowdery <bob@REDACTED> wrote:
>>>
>>>> The symptoms look like it just can't keep up but If a process was
>>>> getting behind I would expect the message q to build up. CPU is 5-7%.
>>>>
>>>> Any ideas would be very welcome.
>>>>
>>> Ladies and Gentlemen! Welcome to the Circus of the Kernel! Tonight we
>>> will have artist of all kinds: Jugglers! Trapezoids! Old Erlang
>>> programmers. A balancing act with a VAX! And above all, there will be
>>> UDP throwing!
>>>
>>> Your problem is that you are sending data out too fast and the kernel
>>> is then throwing your data away. This is especially true on Linux
>>> where this happens silently. On FreeBSD I have seen ENOBUFS being sent
>>> back when this happen, but Linux just choses to silently drop packets.
>>> The only way to fix this is to pace packets so they are sent at a
>>> slower rate - and build in a retransmit mechanism. It is the only way
>>> to be sure. Note that the send buffer can be ridicously small on some
>>> systems, so you can't rely on it. Though my code using UDP can surely
>>> send around 8K on Linux without too much trouble.
>> If the data was being thrown away after it left the application I would
>> expect to see the input and output sequence numbers keep in step. The
>> input sequence is added by the hardware device and I add the output
>> sequence number before transmission. I check the input sequence and
>> shout if there is a sequence error and the device checks the output
>> sequence and lights a nice little LED if it sees a sequence error. Both
>> show correct sequences so I'm losing blocks inside Erlang somewhere.
>>> Since the queue is tail-end dropping you see the problem where many
>>> packets are lost suddenly. The queue fill up, and them you are in for
>>> the surprise when the next 30 packets are just skipped and dropped.
>> It sounds really plausible if only it fitted the facts.
>>> There are a couple of fixes depending on what you want to do, but if
>>> possible go check out SCTP which can give you many of the things you
>>> want from UDP. It is kind of the hybrid between UDP and TCP and then
>>> not. it is also a very interesting protocol - check it out!
>>>
>> I don't have control over the hardware device so it has to be UDP. Looks
>> an interesting protocol though for future reference.
>>
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions