[erlang-questions] design pattern question for messaging

[Miles Fidelman]

Hi Rich,

Jesper's writeup is interesting.

I guess a question that now comes to mind is: How does RabbitMQ 
implement message persistence?  After all, it's written in Erlang.  Off 
to do some research (unless someone can point to a good architectural 
reference).

Thanks,

Miles

Youngkin, Rich wrote:
> Hi,
>
> An application I'm working on has a similar requirement regarding 
> message persistence.  We elected to use RabbitMQ.  One notable 
> difference between this app and the one you're describing is we're not 
> moving a message between states as in a work-flow model. But as Dmitry 
> said below, "if the message persistency is a MUST then you have to 
> look into a reliable message broker..." or some other way to manage 
> persistence.
>
> I may have missed it, but in Jesper's reply I didn't see the 
> persistence requirement addressed.  Jesper has a nice write-up "On 
> Erlang, State and Crashes that you might find useful 
> (http://jlouisramblings.blogspot.com/2010/11/on-erlang-state-and-crashes.html). 
>
>
> Cheers,
> Rich
>
> Date: Tue, 22 Jul 2014 22:58:08 +0300
> From: Dmitry Kolesnikov <dmkolesnikov@REDACTED 
> <mailto:dmkolesnikov@REDACTED>>
> To: Miles Fidelman <mfidelman@REDACTED 
> <mailto:mfidelman@REDACTED>>
> Cc: erlang-questions Questions <erlang-questions@REDACTED 
> <mailto:erlang-questions@REDACTED>>
> Subject: Re: [erlang-questions] design pattern question for messaging
>   system
> Message-ID: <B61EC9A8-DC4B-48F1-84A2-D8FCE65C3994@REDACTED 
> <mailto:B61EC9A8-DC4B-48F1-84A2-D8FCE65C3994@REDACTED>>
> Content-Type: text/plain; charset=windows-1252
>
> Hello,
>
> "Except, that model kind of falls down because Erlang message are 
> unreliable by design, and don't persist in the event of a process 
> crash (much less a node crash).?
>
> This is a trade off you have to accept. If the message persistency is 
> MUST then you have to look into reliable message broker solution then.
>
> I think you already articulate the basic pattern, which is used in 
> many applications
>
>  ?[ mq ]?[ worker ]?[ mq ]-[ worker ]?
>
> You are right if you keep messages within mailbox then crash of 
> process destroys mailbox. Thus, you need an intermediate process to 
> hold messages. The intermediate process does not do any work except 
> enqueue / dequeue operation. Therefore, the number of failures is 
> limited. The worker is pool of processes Jesper Louis gave you list 
> pool libraries, I can give one more if you need ;-)
>
> I am using a similar pattern at one of my application. The biggest 
> problem is node crash due to OOM or external factors. I am trying to 
> solve it by duplicating the processing path into N-distinct node (my 
> app uses last-write wins but you might use other conflict resolution 
> technique).
>
> You can extend the pattern by having persistent mq to check-point 
> intermediate results to survive node crash but I found it complicated.
>
> ?[ p-mq ]?[ worker ]?[ mq ]?[ worker ]?[ mq ]?[ worker ]?[ p-mq ]-
>
> Best Regards,
> Dmitry
>
> On 22 Jul 2014, at 14:53, Miles Fidelman <mfidelman@REDACTED 
> <mailto:mfidelman@REDACTED>> wrote:
>
> > Hi Folks,
> >
> > So far, I've mostly been experimenting w/ Erlang, and using 
> Erlang-based technology (notably CouchDB).  As I'm thinking about a 
> new application, I'm having trouble getting my hands around an 
> appropriate design pattern.  I wonder if anybody might be able to 
> point me in the right direction.
> >
> > The application is message handling (back to that in a minute).  I 
> realize that I have a pretty good idea how to handle some kinds of 
> applications in a highly concurrent fashion, such as:
> > - modeling/simulation (obviously, each entity - such as a vehicle - is 
> a process) - this is what led me to Erlang in the first place
> > - protocol engines as state machines - e.g., spawn a process for each 
> tcp connection
> > - transaction systems - spawn a process for each transaction
> > - transaction oriented
> >
> > But I'm looking at a work flow application that maps onto a 
> paper-forms-based model.  It's a classic queuing system - work 
> elements move from queue to queue as they're worked on.  The obvious 
> first thought is:
> > - a process for each queue
> > - a worker process for each work step
> > - a message for each piece of work-in-process -- moving from queue to 
> queue via the worker processes
> >
> > Except, that model kind of falls down because Erlang message are 
> unreliable by design, and don't persist in the event of a process 
> crash (much less a node crash).
> >
> > My first two thoughts are:
> > - spawn a process for each queue entry, pass around the PIDs
> > - use Mnesia to hold the queues
> >
> > But neither of those feels quite right.  This must be a solved 
> problem, but I'm hitting a blind spot.  So... what is the design 
> pattern for queuing systems and/or reliable message passing in Erlang?
> >
> > Any good examples to look at?  Good presentation slides or reference 
> materials to review?
> >
> > Thanks very much,
> >
> > Miles Fidelman


-- 
In theory, there is no difference between theory and practice.
In practice, there is.   .... Yogi Berra