[erlang-questions] design pattern question for messaging

[Leonard Boyce]

On Wed, Jul 23, 2014 at 10:53 AM, Jesper Louis Andersen
<jesper.louis.andersen@REDACTED> wrote:
>
> On Wed, Jul 23, 2014 at 4:26 PM, Youngkin, Rich
> <richard.youngkin@REDACTED> wrote:
>>
>> I may have missed it, but in Jesper's reply I didn't see the persistence
>> requirement addressed.
>
>
> One way to approach persistence is to have each process track its state on
> disk whenever it is sure its state is consistent. The trick is to keep the
> amount of persist-calls low so most of the way forward moves on volatile
> messages which are fast. In the event of a crash you reset to the last known
> stable point and continue on from there.
>
> For smaller systems, tracking state in mnesia is an obvious candidate. For
> larger systems, you can use something like Riak.
>
> RabbitMQ implements its own persistence layer and just makes sure that data
> are sync'ed to disk before it accepts the message from the client. In the
> event of an error, RabbitMQ can then reconstruct its queue state from the
> disk log. This is one of the reasons never-persisted messages are much
> faster in RabbitMQ than messages you persist on disk.
>

Just thought I'd throw in my 2c as we use RabbitMQ exactly for this.

We use durable queues/exchanges and delivery_mode = 2 (persisted)
messages. Each message carries its own 'State' as it moves between
nodes and queues. Messages are only ack'd after the results are sent
onto the next stage (yes, there is a possible race condition with
duplicate delivery bu that's handled elsewhere) so that crashes in
nodes or processes allow for another worker on some other node to
handle the message.

While it's not the 'fastest' method it has been incredibly stable for
almost 3 years of 24/7 operation.

Leonard

> --
> J.
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>