[erlang-questions] Disk-backed log

[Scott Lystig Fritchie]

> On Sat, Jun 18, 2016 at 3:54 AM, John Smith <
4crzen62cwqszy68g7al@REDACTED> wrote:

> For one of my systems in the financial area, I am in need of a
> disk-backed log that I could use as a backend for an Event
> Sourcing/CQRS store. Recently, I have read a bit about Kafka [1] and
> it seems like a good fit but, unfortunately, it is on JVM (written
> in Scala, to be exact) and depends heavily on ZooKeeper [2] for
> distribution, while I would prefer something similar for an Erlang
> ecosystem. [...]

Howdy.  I'd started working on a project quite similar to what you
described, when I was at Basho Japan.  That project has been cancelled
and the Basho Japan development office closed, but the source still
exists in a GitHub repo at https://github.com/basho/machi with an
APLv2 license.  The current state of the 'master' branch is messy,
since the cancellation came during three different refactorings that
never were fully finished & merged.  But IIRC it works and is likely
correct, most of the time.  ^_^

The master branch does have a dependency on eleveldb which depends on
the C++ code in the LevelDB fork that Basho uses for eleveldb.  If
that's a dealbreaker, just go backward in the repo's history a few
months more to find the eleveldb-free version.

The low-level API is UNIX'ish & Kafka'ish.  It does not have the
higher-level abstraction of the key-value store that you want.  But
it's easy to build that on top of a strictly ordered log.

> In other words, I would like to create something like a *Minimum
> Viable Log* (in Kafka style), only in Erlang/Elixir.

Heh, defining "MVL" is tricky.  A non-distributed version is pretty
easy.  (As Kafka devs discovered early on.)  A distributed version
that doesn't reorder log entries and doesn't lose log entries is not
easy.  (As most distributed systems devs tends to discover.  ^_^)

If the Machi code doesn't suit you, I wouldn't be surprised.  However,
the design papers in the 'doc' directory (look for the PDF files) can
give you a lot of background for why Machi was designed the way it is.
If that rationale fits your "MVL", then I'm glad the docs & code are
helpful.

-Scott
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20160618/f14cc6d4/attachment.htm>