[ANN] ram, an in-memory distributed KV store

[Benoit Chesneau]

I forgot to describe strategies:

* non blocking LRU, LRU cache with eviction over ets. Eviction is done
asynchronously using garbage collection. A pool of keys among the less used
are evicted. This prevents evicting the keys too often and increases the
latency. The algorithm is similar to the one used in Redis [1]
* volatile: pure cache maintained in a gen process. Keys to be evicted are
maintained in a list
* tier file: part of the cache is maintained in memory, part is maintained
on disk for persistence in an append only manner. This allows the cache
tobe persistent across upgrades/restarts.

Note: you can pass your own cache module if you need to. You can also pass
a hook to be executed on eviction on all caches backends.

Distribution is done by maintaining a group of peers. For now it is using
PG2 and erlang distribution. This is similar to Group Cache. The
distribution backend itself is pluggable. For some customers a true P2P
distribution is used for example.

Anyway I may release the stable version if anyone sees an interest in it.

Benoît

[1] http://antirez.com/news/109


On Wed, Dec 22, 2021 at 2:23 PM Benoit Chesneau <bchesneau@REDACTED> wrote:

> while we are here let's add cached to the comparison:
> https://gitlab.com/barrel-db/lab/cached
>
> Only the experiment is public for now. It has different strategies to
> store k/v in memory and distribute it. Distribution is plugable and by
> default rely on erlang distribution.
>
> Quick sample:
> https://gitlab.com/barrel-db/lab/cached/-/blob/master/test/cached_SUITE.erl#L47
>
>
> Benoît
>
> On Wed, Dec 22, 2021 at 1:51 PM Attila Rajmund Nohl <
> attila.r.nohl@REDACTED> wrote:
>
>> Roberto Ostinelli <ostinelli@REDACTED> ezt írta (időpont: 2021. dec.
>> 21., K, 14:57):
>> >
>> > Let’s write a database! Well not really, but I think it’s a little sad
>> that there doesn’t seem to be a simple in-memory distributed KV database in
>> Erlang. Many times all I need is a consistent distributed ETS table.
>> >
>> > The two main ones I normally consider are:
>> >
>> > Riak which is great, it handles loads of data and is based on DHTs.
>> This means that when there are cluster changes there is a need for
>> redistribution of data and the process needs to be properly managed, with
>> handoffs and so on. It is really great but it’s eventually consistent and
>> on many occasions it may be overkill when all I’m looking for is a simple
>> in-memory ACI(not D) KV solution which can have 100% of its data replicated
>> on every node.
>> > mnesia which could be it, but unfortunately requires special attention
>> when initializing tables and making them distributed (which is tricky),
>> handles net splits very badly, needs hacks to resolve conflicts, and does
>> not really support dynamic clusters (additions can be kind of ok, but for
>> instance you can’t remove nodes unless you stop the app).
>> > …other solutions? In general people end up using Foundation DB or REDIS
>> (which has master-slave replication), so external from the beam. Pity, no?
>>
>> Have you seen this: https://gitlab.com/leapsight/plum_db ? It's only
>> eventually consistent, but if you want distribution and availability
>> even in case of network partitioning, you won't get consistency...
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20211222/d3309dcf/attachment.htm>