[erlang-questions] Losing data

[Valentin Micic]

Just increase the UDP buffer size on both machines.

BR

V/

On 02 Jun 2011, at 11:02 PM, Bob Cowdery wrote:

> On 02/06/2011 20:20, Jesper Louis Andersen wrote:
>> On Thu, Jun 2, 2011 at 21:09, Bob Cowdery <bob@REDACTED> wrote:
>> 
>>> The symptoms look like it just can't keep up but If a process was
>>> getting behind I would expect the message q to build up. CPU is 5-7%.
>>> 
>>> Any ideas would be very welcome.
>>> 
>> Ladies and Gentlemen! Welcome to the Circus of the Kernel! Tonight we
>> will have artist of all kinds: Jugglers! Trapezoids! Old Erlang
>> programmers. A balancing act with a VAX! And above all, there will be
>> UDP throwing!
>> 
>> Your problem is that you are sending data out too fast and the kernel
>> is then throwing your data away. This is especially true on Linux
>> where this happens silently. On FreeBSD I have seen ENOBUFS being sent
>> back when this happen, but Linux just choses to silently drop packets.
>> The only way to fix this is to pace packets so they are sent at a
>> slower rate - and build in a retransmit mechanism. It is the only way
>> to be sure. Note that the send buffer can be ridicously small on some
>> systems, so you can't rely on it. Though my code using UDP can surely
>> send around 8K on Linux without too much trouble.
> If the data was being thrown away after it left the application I would
> expect to see the input and output sequence numbers keep in step. The
> input sequence is added by the hardware device and I add the output
> sequence number before transmission. I check the input sequence and
> shout if there is a sequence error and the device checks the output
> sequence and lights a nice little LED if it sees a sequence error. Both
> show correct sequences so I'm losing blocks inside Erlang somewhere.
>> Since the queue is tail-end dropping you see the problem where many
>> packets are lost suddenly. The queue fill up, and them you are in for
>> the surprise when the next 30 packets are just skipped and dropped.
> It sounds really plausible if only it fitted the facts.
>> There are a couple of fixes depending on what you want to do, but if
>> possible go check out SCTP which can give you many of the things you
>> want from UDP. It is kind of the hybrid between UDP and TCP and then
>> not. it is also a very interesting protocol - check it out!
>> 
> I don't have control over the hardware device so it has to be UDP. Looks
> an interesting protocol though for future reference.
> 
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions