[erlang-questions] Losing data

[Bob Cowdery]

On 03/06/2011 01:08, Edmond Begumisa wrote:
> On Fri, 03 Jun 2011 05:09:40 +1000, Bob Cowdery
> <bob@REDACTED> wrote:
>
>> Can anyone give me some hints on how to debug this problem. I know its a
>> long-shot but maybe someone has experience of a similar problem.
>>
>> I have a real-time system that reads from a device using UDP and writes
>> back to the same device using UDP. The data processing chain comprises 6
>> processes:
>>
>> Reader - Decoder - Input to linked-in-driver - output from
>> linked-in-driver - Encoder - Writer
>>
>
> Wondering: is there any particular reason why the reader and decoder
> are separate processes, and the encoder plus writer are also separate
> processes?
My initial though was the reader and writer are pretty simple processes
and if there was a throughput problem I wanted it to show up as a
message queue building and not as dropped packets. I think its structure
over efficiency at the moment.
>
> If the reader+decoder were one single process (reader module calling
> decoder module) and the encoder+writer another single process, you
> could reduce the amount of message-passing and get better performance.
> Probably won't solve your immediate problem since you're not
> witnessing message-bottlenecks (yet) but worth looking into.
If I can solve this problem then it will get pushed much harder as I
ramp up the sample rate and I might then hit a wall and I will have to
optimise.
>
> - Edmond -
>
>
>> With the parameters I'm using I should be writing the same number of
>> blocks that I am reading. There are sequence numbers on both input and
>> output blocks. For a while this is what happens, then I start to lose
>> blocks. I believe in snatches rather than a regular pattern. I can
>> sometimes hear the disruption in the output and if I stop at that point
>> I've lost blocks, as many as 40 in one case. However it can run on way
>> longer than that with no lost blocks.
>>
>> I've checked every stage of the code with debug statements but
>> everything seems to pass through correctly.  I've also checked the
>> message queue length at each stage and it never builds up on any
>> process. I have no catch-all message handlers. The messages are all
>> large binary.
>>
>> The symptoms look like it just can't keep up but If a process was
>> getting behind I would expect the message q to build up. CPU is 5-7%.
>>
>> Any ideas would be very welcome.
>>
>> Bob
>>
>>
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
>
>