[erlang-questions] Losing data

[Bob Cowdery]

On 02/06/2011 23:12, Mike Oxford wrote:
> I'm guessing you've already tried increasing SASL debugging, and using
> dbg:tracer()?
> How about putting a debugger (gdb or the like) on your driver?
> Have you tried synthetic testing (eg, loop the same packet into your
> system, injected just after 
> the UDP layer and then check the output on the other side?"
> You say that you're breaking the packets up for processing, and
> putting them back together, with 
> sequencing staying the same... but you're losing packets?  That seems
> only possible if your
> hardware isn't actually sending it (or there's a bug in your
> sequencing numbers/checks.)
>
I started out with a simulator (in Erlang) and when I was happy moved
onto the hardware. I think I will move back to the simulator because I
can then see both ends, and I can vary the data rate easily and should
be able to reproduce the fault. Nothing makes sense at the moment
because I'm pretty sure the sequence numbering is sound and because of
it I can pretty much state I'm not losing packets either in reception or
transmission and don't have queues building in Erlang but blocks are
still going missing.
> Have you put Ethereal/Wireshark/tcpdump etc on the wire during the
> tests?  Have you seen a pause
> in output?
>
I've used Wireshark but the disruption in output is not seen because its
missing data but the sequence is correct. It's not a silent pause.
> Hard to tell from this far away.  =/
I know, but sometime just talking through the problem helps to clarify.
>
> On Thu, Jun 2, 2011 at 2:02 PM, Bob Cowdery <bob@REDACTED
> <mailto:bob@REDACTED>> wrote:
>
>     On 02/06/2011 20:26, Mike Oxford wrote:
>>     Can you pin down where you think you're losing the data?  You
>>     state a hard number....where in the chain are you measuring?
>>
>     I've been trying to do that for several days but it's really hard
>     to track it all the way through in one go because the buffer size
>     changes in different stages and its not even multiples, it only
>     comes together in fixed size blocks at each end. When I check each
>     stage individually I'm failing to find any problem. I see I've
>     lost blocks overall but the stage I'm looking at has no breaks in
>     the pattern. They all seem to process the data correctly - and
>     most of the time the input and output keep pace. There is no code
>     in the path that could just junk blocks.
>
>>     Usually in audio-processing you're going to run into issues in
>>     two main places; buffer starvation and codec corruption.
>>     Buffer starvation is pretty self-explanitory -- something in the
>>     chain cannot keep up, usually due to CPU load, lock contention or
>>     other data flow interruption.
>>     Codec corruption happens when the codec tries to use the new
>>     incoming packets against it's stored run-state.  If 
>>     something happens (out of sequence, null packets, etc) the codec
>>     will usually do one of two things:  do its best and give you
>>     corrupted audio or just drop the packet on the floor as if it
>>     never got it.
>>
>     This is akin to audio processing but the signal processing is
>     under my control so there are no unknowns in the chain.
>
>>     One more caveat with audio: output packet size is not usually
>>     equal to input packet size.  The most codecs do
>>     lossy-compression.  Just one more thing to check.
>>
>     I'm using 48k input and output rate for the test.
>>     -mox 
>>
>>     On Thu, Jun 2, 2011 at 12:09 PM, Bob Cowdery
>>     <bob@REDACTED <mailto:bob@REDACTED>> wrote:
>>
>>         Can anyone give me some hints on how to debug this problem. I
>>         know its a
>>         long-shot but maybe someone has experience of a similar problem.
>>
>>         I have a real-time system that reads from a device using UDP
>>         and writes
>>         back to the same device using UDP. The data processing chain
>>         comprises 6
>>         processes:
>>
>>         Reader - Decoder - Input to linked-in-driver - output from
>>         linked-in-driver - Encoder - Writer
>>
>>         With the parameters I'm using I should be writing the same
>>         number of
>>         blocks that I am reading. There are sequence numbers on both
>>         input and
>>         output blocks. For a while this is what happens, then I start
>>         to lose
>>         blocks. I believe in snatches rather than a regular pattern.
>>         I can
>>         sometimes hear the disruption in the output and if I stop at
>>         that point
>>         I've lost blocks, as many as 40 in one case. However it can
>>         run on way
>>         longer than that with no lost blocks.
>>
>>         I've checked every stage of the code with debug statements but
>>         everything seems to pass through correctly.  I've also
>>         checked the
>>         message queue length at each stage and it never builds up on any
>>         process. I have no catch-all message handlers. The messages
>>         are all
>>         large binary.
>>
>>         The symptoms look like it just can't keep up but If a process was
>>         getting behind I would expect the message q to build up. CPU
>>         is 5-7%.
>>
>>         Any ideas would be very welcome.
>>
>>         Bob
>>
>>
>>         _______________________________________________
>>         erlang-questions mailing list
>>         erlang-questions@REDACTED <mailto:erlang-questions@REDACTED>
>>         http://erlang.org/mailman/listinfo/erlang-questions
>>
>>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20110603/6f919e17/attachment.htm>