[erlang-questions] Packet size parsing

[Tony Garnock-Jones]

Hi Christian,

That certainly works: in fact, we're currently simulating this in our 
AMQP server, using prim_inet:async_recv/3 to alternate between asking 
for headers and bodies. (rabbit_reader:switch_callback/3, for the 
interested reader.)

The reason I asked about a specialised option is because having the 
packets effortlessly delivered to us is likely to be quite a bit faster 
than constantly chopping and changing, with several messages 
back-and-forth for each parsed packet, and inet_drv.c already has almost 
exactly the same (kind of) logic as we're duplicating down at the erlang 
level.

I see, though, that to handle the PostgreSQL protocol as well, it would 
have to become

   {packet, N, Offset, Correction}

e.g.,

  - for AMQP 0-8, {packet, 4, 3, +1},
  - for AMQP 0-10 (most recent draft), {packet, 2, 2, -12}, and
  - for PostgreSQL (as you described it), {packet, 4, 1, -4}.

The revised general packet reader would be:

  - read N + Offset bytes
  - let Len = the big-endian N-byte word at Offset
  - wait for Len + Correction more bytes
  - pass a tuple of {<<the N+Offset header>>, <<the body>>}
    to the application

You're right, this is starting to look more complicated. I'd still like 
to hear how much pain might be involved in *doing* it, of course :-)

Having looked at INET_LOPT_PACKET in prim_inet.erl and inet_drv.c I get 
the feeling the hardest part is going to be passing the three parameters 
over to C-land!

Regards,
   Tony



Christian S wrote:
> A slightly more generalized way to handle protocols with a header+payload is
> to be able to tell the driver to collect B bytes before it sends a message.
> 
> The postgresql protocol has a header with
> <<MessageType:8/integer, Length:32/big-endian-integer>>
> then followed by (Length-4) bytes of payload.
> 
> So it would be nice to first tell it to send me a 5 byte header, not
> more and not less,
> then tell it to send me the payload size of data, still not more and not less.
> 
> I get uneasy about so much protocol stuff being done in C code. That
> so easy to get
> wrong and get wrong so it is exploitable. Just asking C to collect a
> certain amount of
> bytes before it sends the process a message is not much code to get wrong.
> 
> On Feb 8, 2008 5:59 PM, Tony Garnock-Jones <tonyg@REDACTED> wrote:
>> Hi all,
>>
>>
>> How difficult might it be to support, in the C inet runtime, a new kind
>> of {packet, N} option for sockets,
>>
>>   {packet, N, Offset}
>>
>> where
>>   N - number of bytes of big-endian length indicator
>>   Offset - byte offset within the packet of the length indicator
>>
>