[erlang-questions] Losing data

[Edmond Begumisa]

On Fri, 03 Jun 2011 05:09:40 +1000, Bob Cowdery <bob@REDACTED>  
wrote:

> Can anyone give me some hints on how to debug this problem. I know its a
> long-shot but maybe someone has experience of a similar problem.
>
> I have a real-time system that reads from a device using UDP and writes
> back to the same device using UDP. The data processing chain comprises 6
> processes:
>
> Reader - Decoder - Input to linked-in-driver - output from
> linked-in-driver - Encoder - Writer
>

Wondering: is there any particular reason why the reader and decoder are  
separate processes, and the encoder plus writer are also separate  
processes?

If the reader+decoder were one single process (reader module calling  
decoder module) and the encoder+writer another single process, you could  
reduce the amount of message-passing and get better performance. Probably  
won't solve your immediate problem since you're not witnessing  
message-bottlenecks (yet) but worth looking into.

- Edmond -


> With the parameters I'm using I should be writing the same number of
> blocks that I am reading. There are sequence numbers on both input and
> output blocks. For a while this is what happens, then I start to lose
> blocks. I believe in snatches rather than a regular pattern. I can
> sometimes hear the disruption in the output and if I stop at that point
> I've lost blocks, as many as 40 in one case. However it can run on way
> longer than that with no lost blocks.
>
> I've checked every stage of the code with debug statements but
> everything seems to pass through correctly.  I've also checked the
> message queue length at each stage and it never builds up on any
> process. I have no catch-all message handlers. The messages are all
> large binary.
>
> The symptoms look like it just can't keep up but If a process was
> getting behind I would expect the message q to build up. CPU is 5-7%.
>
> Any ideas would be very welcome.
>
> Bob
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions


-- 
Using Opera's revolutionary e-mail client: http://www.opera.com/mail/