[erlang-questions] Chinese whisper problem
Thu Dec 1 03:05:37 CET 2011
protobufs, thrift and other "human-readable binary protocol" is the
message packets,decoding and encoding fast speed is fast.
2011/12/1 envelopes envelopes <sunwood360@REDACTED>
> I remember there is also messagepack that supports erlang, It has good
> In addition, there is Avro.
> On Nov 30, 2011 3:07 PM, "Anthony Molinaro" <anthonym@REDACTED>
>> On Fri, Nov 25, 2011 at 04:18:09PM +0100, Joe Armstrong wrote:
>> > I guess you all know about Chinese whispers.
>> > You send a whispered message round a circle and see if you get the same
>> > message back.
>> > The computer variant is as follows:
>> > Send a complex data structure round a circle, and see if you get the
>> > message back.
>> Sorry I'm late to the party with this one, but was out of town for a bit.
>> I have used thrift, protobuffs and lwes to do binary multi-language
>> message exchange, which is not quite the whispers problem but close.
>> They all contain serializers and deserializers and each support different
>> sorts of things.
>> Thrift is mainly meant for RPC, so if all you want is a binary
>> format, it's a bit tricky (mostly in erlang, less so in java), in that you
>> have to create a memory serializer and construct the terms then do a
>> term_to_binary on them, to get a binary you could do what you want with.
>> On the other hand it supports many languages (include erlang and
>> and several protocols (sockets and http being the most common). It can
>> fairly complex structures like maps (which become dict's in erlang),
>> different sized ints, floats, etc.
>> Protobufs also off fairly complex structures (although not maps, you sort
>> have to do those yourself with repeated structures which will become lists
>> of records, so not quite as useful), as well as strings, ints, floats and
>> binary data. It does not include any RPC based mechanism, and you
>> get serializers/deserializers which get/give binaries, so you can use it
>> with whatever you want. Riak uses this as its faster interface via a
>> and I've implemented server to server communication over HTTP which is
>> LWES (http://lwes.github.com/) is a system and protocol I first developed
>> in 1998 and which was open sourced in 2008. It's mostly meant for passing
>> maps in a fire and forget fashion over UDP (or multicast UDP), it's a bit
>> bulkier on the wire than thrift and protobuffs as the serialization format
>> is self describing (key names are sent along with values). It also only
>> has support in java for floats and arrays (but those will probably make
>> their way into the other languages as time permits). It currently
>> has native bindings for C, erlang, java, and .Net, as well as SWIG based
>> bindings for perl, python and ruby. It's a great system for quickly
>> passing maps between different programming languages where you can afford
>> to loose some data here or there. Also, unlike thrift/protobuff it does
>> not require any precompilation of modules. While we don't have a
>> version yet, we may be developing one soon (although it most likely won't
>> is extremely stable and has been in production systems serving trillions
>> of events for almost 13 years (every penny ever made by
>> search advertising system was via lwes).
>> I've used all three of these systems in production web systems so can
>> that they all work and at large scale (ten's of thousands to hundred's of
>> thousands of requests per second).
>> I've not used Bert, BSON, Avro, MessagePack or Etch all of which do
>> similiar things but offer slightly different feature sets.
>> Hope that helps,
>> Anthony Molinaro <anthonym@REDACTED>
>> erlang-questions mailing list
> erlang-questions mailing list
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the erlang-questions