[erlang-questions] BERT vs protobuf in the erlang world

Jon Watte jwatte@REDACTED
Thu Aug 25 00:15:57 CEST 2011


You solve that in one of two ways:
1) Create a message that is the "PDU" that contains the union of everything
that could be sent,
or
2) Create a wrapper that is just "type" followed by "binary data that is a
marshaled message."

If you believe that the entirety of a protocol should be described by that
protocol, then 1) is actually a fine solution. The preserve-and-forward
semantic of unknown field ids means that forward compatibility is still
possible.

message MyProtocolPDU {
  optional SomeMessage some_message = 1;
  optional AnotherMessage other_message = 2;
  ...
}

So, encoding a MyProtocolPDU that contains a SomeMessage will end up being
different than encoding a MyProtocolPDU that contains an AnotherMessage.
At the high level, you have to know that there is only one member in each
MyProtocolPDU; I find that to be quite acceptable.

Sincerely,

jw

--
Americans might object: there is no way we would sacrifice our living
standards for the benefit of people in the rest of the world. Nevertheless,
whether we get there willingly or not, we shall soon have lower consumption
rates, because our present rates are unsustainable.



On Tue, Aug 23, 2011 at 4:22 PM, Anthony Molinaro <
anthonym@REDACTED> wrote:

> Protobuf is not what I would call self describing, if you have something
> like
>
> message foo {
>  required int32 bar = 1;
> }
>
> message foo2 {
>  optional int32 bar2 = 1;
> }
>
> and create an instance of each and then send it remotely, you can not
> distinguish between the 2 messages, unless the remote side know when
> one you mean to send.
>
> Here's an example in erlang
>
> Erlang R14B03 (erts-5.8.4) [source] [64-bit] [smp:2:2] [rq:2]
> [async-threads:0]
> [hipe] [kernel-poll:false]
>
> Eshell V5.8.4  (abort with ^G)
> 1> rr(foo_pb).
> [foo,foo2]
> 2> foo_pb:encode(#foo{bar=42}).
> <<"\b*">>
> 3> foo_pb:encode(#foo2{bar2=42}).
> <<"\b*">>
>
> To me a self-describing protocol is something which also encoded the names
> of the messages and fields, so you could send it through to the other side
> and it would be unambiguous.  An example of this is a system I wrote many
> years ago called lwes (www.lwes.org), which does not require code
> generation
> because messages are self describing.
>
> Now this does mean messages are longer, but sometimes that is okay.
>
> -Anthony
>
> On Tue, Aug 23, 2011 at 01:17:12PM -0700, Jon Watte wrote:
> > Protobuf is fully self-described, and fully forwards/backwards
> compatible,
> > in that it can carry through "unknown" members of a structure, including
> the
> > semantic of the fields. ASN1 is generally not fully self-described,
> because
> > it only describes type, not semantic, for each field.
> >
> > Sincerely,
> >
> > jw
> >
> > --
> > Americans might object: there is no way we would sacrifice our living
> > standards for the benefit of people in the rest of the world.
> Nevertheless,
> > whether we get there willingly or not, we shall soon have lower
> consumption
> > rates, because our present rates are unsustainable.
> >
> >
> >
> > On Tue, Aug 23, 2011 at 3:03 AM, Vincent de Phily <
> > vincent.dephily@REDACTED> wrote:
> >
> > > On Monday 22 August 2011 11:31:44 Mike Oxford wrote:
> > > > BERT is pretty much a "known good quantity" but protobuf is more
> > > > efficient on the wire.
> > > >
> > > > Anyone have thoughts on which to go with?
> > > >
> > > > Also, along the BERT side, does anyone have recommendations on AS3
> and
> > > > Java implementations of the marshallers?
> > > >
> > > > The downside(s) to protobuf - hard IDLs and you have to build the
> > > > project and generators.
> > > > The downside(s) to BERT - less efficient on the wire (verbosity.)
> > >
> > > Also take a look at bjson, and one of my favourite : msgpack. BERT is
> > > rather
> > > unique in supporting atoms, which *nice* and can reduce encoded size a
> lot
> > > (but that's less significant if you compress). For encoded size, I have
> yet
> > > to
> > > see anything beat UPER (asn1).
> > >
> > > A few things to consider :
> > > * Bench size and speed with a mockup of your data : different formats
> are
> > >  better for different types of data. Add compression to your tests.
> > > * Think about format extensibility. It's easy enough with
> self-described
> > >  formats like bert or msgpack, but gets more complicated with protobuf
> or
> > >  asn1.
> > > * The initial hurdle of dealing with a grammar and generators is well
> paid
> > > off
> > >  by having data validation and nicer api (mostly via #records{}) later
> on.
> > > * If you need to speak with the outside world, self-describing formats
> will
> > >  make your life easyer.
> > > --
> > > Vincent de Phily
> > > _______________________________________________
> > > erlang-questions mailing list
> > > erlang-questions@REDACTED
> > > http://erlang.org/mailman/listinfo/erlang-questions
> > >
>
> > _______________________________________________
> > erlang-questions mailing list
> > erlang-questions@REDACTED
> > http://erlang.org/mailman/listinfo/erlang-questions
>
>
> --
> ------------------------------------------------------------------------
> Anthony Molinaro                           <anthonym@REDACTED>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20110824/bce5a761/attachment.htm>


More information about the erlang-questions mailing list