[erlang-questions] Considering a Generic Transaction System in Erlang

[Jörgen Brandt]

Hello,

On 20.10.2015 12:53, Jesper Louis Andersen wrote:
>
> On Fri, Oct 9, 2015 at 12:07 AM, Jörgen Brandt
> <brandjoe@REDACTED <mailto:brandjoe@REDACTED>> wrote:
>
> Consider an application (with transient software faults),
> processing user queries.
>
>
> I think this is the central point of the mail. One of the usual
> Erlang approaches is to rely on stable storage and checkpointing
> for these kinds of work units.
>
> Stable storage means that you can write to disk, sync data, and
> then you have a linearizable point in time after which data are
> safe on the disk and can be re-read.

Sounds good.

>
> Checkpointing means to track on stable storage whenever the system
> reaches some safe invariant state.

Given, (after failure) a service is restarted with constant
parameters, its state is constituted only by the requests it received.
So the reestablishment of a state and the resending of all requests
should be exchangeable (like in a redo log).

Now, if you can guarantee/assume that messages are uncorrelated
throughout the application, you only need to redo the requests which
have been unreplied so far.

I find the view that state is just the sum of all messages quite
appealing because appending a message to a message history is a local
action whereas snapshotting and storing the state of a whole
application is a global action (and, thus, harder to accomplish).

>
> In order to make sure progress happens, even under the possibility
> of transient errors, one must figure out how to checkpoint
> invariant state such that it can be recovered. In some cases, you
> don't need truly persistent storage, but can simply ship the state
> to another process, or even system.

Agreed, not for all applications one needs persistent storage.

>
> The other ingredient you need is that of idempotent operations.
> That is, you can rerun an operation/subtask and it will produce the
> same result as before, or it will return early with the answer if
> the answer is already evaluated and cached.

You mean that operations are deterministic and side-effect free.
Something I assume (but do not enforce) in my application.

>
> Given these two ingredients, you don't need to use transactions.
> You just reinject unfulfilled obligations into the system and you
> track which obligations that deadline.
>

Checkpointing is a substitute for transactions. But is it a preferable
one?

> A project like Apache Storm implements this workflow, and I'm 99%
> sure a lot of Erlang projects exist for this, though the names
> eludes me right now.
>

Perhaps I have been a bit unspecific in what I wanted to know. I am
thankful, though, for your reply.

Cheers
Jörgen

>
> -- J.