[erlang-questions] Chinese whisper problem

[Anthony Molinaro]

On Fri, Nov 25, 2011 at 04:18:09PM +0100, Joe Armstrong wrote:
> I guess you all know about Chinese whispers.
> 
> You send a whispered message round a circle and see if you get the same
> message back.
> 
> The computer variant is as follows:
> 
> Send a complex data structure round a circle, and see if you get the same
> message back.

Sorry I'm late to the party with this one, but was out of town for a bit.

I have used thrift, protobuffs and lwes to do binary multi-language
message exchange, which is not quite the whispers problem but close.
They all contain serializers and deserializers and each support different
sorts of things.

Thrift is mainly meant for RPC, so if all you want is a binary serialization
format, it's a bit tricky (mostly in erlang, less so in java), in that you
have to create a memory serializer and construct the terms then do a
term_to_binary on them, to get a binary you could do what you want with.
On the other hand it supports many languages (include erlang and javascript),
and several protocols (sockets and http being the most common).  It can have
fairly complex structures like maps (which become dict's in erlang), lists,
different sized ints, floats, etc.

Protobufs also off fairly complex structures (although not maps, you sort of
have to do those yourself with repeated structures which will become lists
of records, so not quite as useful), as well as strings, ints, floats and
binary data.  It does not include any RPC based mechanism, and you basically
get serializers/deserializers which get/give binaries, so you can use it
with whatever you want.  Riak uses this as its faster interface via a socket
and I've implemented server to server communication over HTTP which is
easy.

LWES (http://lwes.github.com/) is a system and protocol I first developed
in 1998 and which was open sourced in 2008.  It's mostly meant for passing
maps in a fire and forget fashion over UDP (or multicast UDP), it's a bit
bulkier on the wire than thrift and protobuffs as the serialization format
is self describing (key names are sent along with values).  It also only
has support in java for floats and arrays (but those will probably make 
their way into the other languages as time permits).  It currently
has native bindings for C, erlang, java, and .Net, as well as SWIG based
bindings for perl, python and ruby.  It's a great system for quickly
passing maps between different programming languages where you can afford
to loose some data here or there.  Also, unlike thrift/protobuff it does
not require any precompilation of modules.  While we don't have a javascript
version yet, we may be developing one soon (although it most likely won't
be able to do UDP from a javascript client which is unfortunate).  LWES
is extremely stable and has been in production systems serving trillions
of events for almost 13 years (every penny ever made by Goto/Overture/Yahoo's
search advertising system was via lwes).

I've used all three of these systems in production web systems so can attest
that they all work and at large scale (ten's of thousands to hundred's of
thousands of requests per second).

I've not used Bert, BSON, Avro, MessagePack or Etch all of which do
similiar things but offer slightly different feature sets.

Hope that helps,

-Anthony

-- 
------------------------------------------------------------------------
Anthony Molinaro                           <anthonym@REDACTED>