[erlang-questions] Disk-backed log
Scott Lystig Fritchie
Sat Jun 18 21:41:42 CEST 2016
> On Sat, Jun 18, 2016 at 3:54 AM, John Smith <
> For one of my systems in the financial area, I am in need of a
> disk-backed log that I could use as a backend for an Event
> Sourcing/CQRS store. Recently, I have read a bit about Kafka  and
> it seems like a good fit but, unfortunately, it is on JVM (written
> in Scala, to be exact) and depends heavily on ZooKeeper  for
> distribution, while I would prefer something similar for an Erlang
> ecosystem. [...]
Howdy. I'd started working on a project quite similar to what you
described, when I was at Basho Japan. That project has been cancelled
and the Basho Japan development office closed, but the source still
exists in a GitHub repo at https://github.com/basho/machi with an
APLv2 license. The current state of the 'master' branch is messy,
since the cancellation came during three different refactorings that
never were fully finished & merged. But IIRC it works and is likely
correct, most of the time. ^_^
The master branch does have a dependency on eleveldb which depends on
the C++ code in the LevelDB fork that Basho uses for eleveldb. If
that's a dealbreaker, just go backward in the repo's history a few
months more to find the eleveldb-free version.
The low-level API is UNIX'ish & Kafka'ish. It does not have the
higher-level abstraction of the key-value store that you want. But
it's easy to build that on top of a strictly ordered log.
> In other words, I would like to create something like a *Minimum
> Viable Log* (in Kafka style), only in Erlang/Elixir.
Heh, defining "MVL" is tricky. A non-distributed version is pretty
easy. (As Kafka devs discovered early on.) A distributed version
that doesn't reorder log entries and doesn't lose log entries is not
easy. (As most distributed systems devs tends to discover. ^_^)
If the Machi code doesn't suit you, I wouldn't be surprised. However,
the design papers in the 'doc' directory (look for the PDF files) can
give you a lot of background for why Machi was designed the way it is.
If that rationale fits your "MVL", then I'm glad the docs & code are
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the erlang-questions