[erlang-questions] Erlang suitability

[Garrett Smith]

Hi Ovid,

Thanks for the very interesting use case!

On Fri, May 18, 2012 at 4:00 AM, Ovid <curtis_ovid_poe@REDACTED> wrote:
> Hi there,
>
> We've a system that run across 75 servers and needs to be highly performant,
> fault-tolerant, scalable and shares persistent data across all 75 servers.
> We're investigating Erlang/Mnesia (which we don't know) because it sounds
> tailor-made for our situation.

The previous comments about 75 being "a bit much" are warranted. In
distributed mode, Erlang establishes a "fully connected mesh" where
each node tries to maintain connections to every other node. This
N(N-1) scheme adds up quickly and caps the practical limits of these
clusters.

You can partition the clusters to limit their size, but you're moving
into a more complex topology. It's probably worth looking at though
since you get so much out of distributed Erlang for this problem.

> We are not using Erlang for our first implementation, but are instead
> hacking together a solution from known technologies including Perl, MySQL
> and Redis. We're considering Erlang for our future work.
>
> We have two primary needs: Each box can bid on an auction and potentially
> spend a tiny amount of money and each of the 75 boxes will receive
> notifications of a small amount of money spent if they win the auction (the
> auction notification will probably not be sent to the box bidding in the
> auction).
>
> Use case 1: If the *total* of all of those small amounts exceeds a daily cap
> or an all-time cap, all 75 boxes must immediately stop spending bidding in
> auctions. It seems that each box can run a separate Erlang process and write
> out "winning bid" information to an Mnesia database and all boxes can read
> the total amount spent from that to determine if it should stop bidding.

You might also look at Riak for this.

Also, something like doozerd.

Or a consensus algorithm like Paxos, if you don't mind the extra work
in building smarter clients.

> This seems trivial to set up.
>
> Use case 2: we periodically need to reauthenticate to the auction system. We
> MUST NOT have all 75 boxes trying to reauthenticate at the same time because
> we will be locked out of the system if we do this. Having a central box
> handling reauthentication is a single point of failure that we would like to
> avoid, but we don't know what design pattern Erlang would use to ensure that
> only one of the 75 Erlang instances would attempt to reauthenticate at any
> one time (all 75 boxes can share the same authentication token).

This could also be framed as a consensus problem -- you'd elect a node
to reauthenticate among a set of candidates.

> Use case 1 is clearly perfect for Erlang. Use case 2 is less clear to us.
> We're going to be spending a fair amount of time hacking together a
> non-Erlang solution, but the benefit is using known technologies that don't
> depend on the "one guy who knows the system".

This is a great approach!

One tech you might look at to glue these pieces together is 0MQ. This
would let you use whatever languages you'd want, hacking something
that just works, then replace pieces as needed without breaking your
network protocols.

You'll have a learning curve using 0MQ, but this is because you'll be
learning core messaging patterns, not so much 0MQ itself, which is
very simple. I think all of that learning will be indispensable for
your team as you move along.

> At full scale, we anticipate billions of auctions per day.
>
> Does anyone care to comment about things we should look at? Particularly, is
> use case 2 not something appropriate for Erlang? A tiny amount of sample
> code would be lovely (note: I am somewhat comforable with Prolog, so Erlang
> looks fairly straightforward for me, aside from understanding the message
> passing).

If you haven't already, take a look at this excellent document from Google:

http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/archive/chubby-osdi06.pdf

Distributed algorithms are a PITA (for me anyway) and the Chubby lock
service is a great "cheat" that simplifies a lot of the harder
problems.

Doozer is one of the few implementations of something Chubby-like that
I'm aware of.

> Cheers,
> Ovid

Best of luck!

Garrett