[erlang-questions] "actor database" - architectural strategy question

[Miles Fidelman]

Ahh... an existence proof.  VERY helpful.  Thanks!

Motiejus Jakštys wrote:
>
> On Mon, Feb 17, 2014 at 7:04 PM, Miles Fidelman 
> <mfidelman@REDACTED <mailto:mfidelman@REDACTED>> wrote:
> >
> > If I wanted to model this as a standard database, or serializing 
> state into
> > a traditional database, I wouldn't be asking the questions I asked. Can
> > anybody talk to the questions I actually asked, about:
> > - handling large numbers of actors that might persist for years, or 
> decades
> > (where actor = Erlang-style process)
> > - backup up/restoring state of long-running actors that might crash
> > - multi-cast messaging among actors
>
> Hi,
>
> Some time ago I was part of a team which created software to manage 
> phone number migration between mobile operators. Say you want to 
> change your cell phone provider (mandatory in EU and in many other 
> countries). We were the entity responsible for that process.
>
> One portability request is one process. At any time we could have had 
> up to 1M processes (practically it was much lower, but we used this 
> number when designing the system). A "portability process" is a finite 
> state machine with communication in SOAP between two or three parties 
> with many internal state changes.
>
> A single process could last from a few hours up to few months (median 
> ~3-4 days), each state up to 10-100KB of uncompressed text (mean ~15KB 
> uncompressed).
>
> Having Erlang processes allowed very nice things like straightforward 
> programming of state transitions during timeouts.
>
> Strict consistency requirements meant we had checkpoints in a 
> key-value store for every operation for every process, which was 
> managed globally. From that checkpoints it was possible to re-create 
> state replying all actions.
>
> We did not really manage to fully implement a proper addressing 
> mechanism for non-volatile message sending. We invented our own PIDs 
> which had some sort of address / node ownership information. The 
> mechanism was complex and imperfect, nothing really to learn from. 
> AMQP might be a good candidate though.
>
> Note that some of the details above are not exactly true (esp. 
> numbers), because I can't remember all the details.
>
> A few remarks:
> 1. Do *not* store full state after you change it. Implement a diff 
> mechanism on your abstract state tree (it's strictly defined, right?), 
> test it using PropEr and use that. If you require fast recovery in 
> case of crash, checkpoint is ok, but never drop the old state. You 
> might dispute the state transition after months, go fix the bug and 
> want to re-run a particular process transitions again next year... Ugh.
> 2. Long-lived processes (weeks+) are perfectly fine for Erlang VM. 
> Just make sure to hibernate them after some minutes of inactivity. You 
> can easily have hundreds of thousands, consume basically no CPU and 
> just enough memory to keep the internal state.
>
> Regards,
> Motiejus
>


-- 
In theory, there is no difference between theory and practice.
In practice, there is.   .... Yogi Berra