[erlang-questions] Losing data

[Robert Virding]

My first instinct reply was the same as jlouis, check if you are losing packets in UDP. Because to cite the bible, er Wikipedia:

"UDP uses a simple transmission model without implicit handshaking dialogues for providing reliability, ordering, or data integrity. Thus, UDP provides an unreliable service and datagrams may arrive out of order, appear duplicated, or go missing without notice."

Robert

----- "Bob Cowdery" <bob@REDACTED> wrote:

> On 03/06/2011 01:08, Edmond Begumisa wrote:
> > On Fri, 03 Jun 2011 05:09:40 +1000, Bob Cowdery
> > <bob@REDACTED> wrote:
> >
> >> Can anyone give me some hints on how to debug this problem. I know
> its a
> >> long-shot but maybe someone has experience of a similar problem.
> >>
> >> I have a real-time system that reads from a device using UDP and
> writes
> >> back to the same device using UDP. The data processing chain
> comprises 6
> >> processes:
> >>
> >> Reader - Decoder - Input to linked-in-driver - output from
> >> linked-in-driver - Encoder - Writer
> >>
> >
> > Wondering: is there any particular reason why the reader and
> decoder
> > are separate processes, and the encoder plus writer are also
> separate
> > processes?
> My initial though was the reader and writer are pretty simple
> processes
> and if there was a throughput problem I wanted it to show up as a
> message queue building and not as dropped packets. I think its
> structure
> over efficiency at the moment.
> >
> > If the reader+decoder were one single process (reader module
> calling
> > decoder module) and the encoder+writer another single process, you
> > could reduce the amount of message-passing and get better
> performance.
> > Probably won't solve your immediate problem since you're not
> > witnessing message-bottlenecks (yet) but worth looking into.
> If I can solve this problem then it will get pushed much harder as I
> ramp up the sample rate and I might then hit a wall and I will have
> to
> optimise.
> >
> > - Edmond -
> >
> >
> >> With the parameters I'm using I should be writing the same number
> of
> >> blocks that I am reading. There are sequence numbers on both input
> and
> >> output blocks. For a while this is what happens, then I start to
> lose
> >> blocks. I believe in snatches rather than a regular pattern. I can
> >> sometimes hear the disruption in the output and if I stop at that
> point
> >> I've lost blocks, as many as 40 in one case. However it can run on
> way
> >> longer than that with no lost blocks.
> >>
> >> I've checked every stage of the code with debug statements but
> >> everything seems to pass through correctly.  I've also checked the
> >> message queue length at each stage and it never builds up on any
> >> process. I have no catch-all message handlers. The messages are
> all
> >> large binary.
> >>
> >> The symptoms look like it just can't keep up but If a process was
> >> getting behind I would expect the message q to build up. CPU is
> 5-7%.
> >>
> >> Any ideas would be very welcome.
> >>
> >> Bob
> >>
> >>
> >> _______________________________________________
> >> erlang-questions mailing list
> >> erlang-questions@REDACTED
> >> http://erlang.org/mailman/listinfo/erlang-questions
> >
> >
> 
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions