[erlang-questions] ETS’s performances and limits?

[Mikael Pettersson]

On Sat, Sep 1, 2018 at 10:53 PM, Frank Muller
<frank.muller.erl@REDACTED> wrote:
> Hi,
>
> Is ETS capable of handling 1 million key/value (or more)? My keys are 36B
> and all my values are local PIDs. My ETS is mostly read-only!!!

Yes, that's tiny.  Using integers as keys would be best.

> What’s the largest ETS one has to deal with?

You're only limited by available RAM.  We routinely handle tables with
several 100s of million records in each, and we have many such tables
in each node.

> Is there any limitations or point when the ETS’s perf degrades?

ETS tables with complex keys can degrade drastically under high
insertion rates, but are Ok when mostly read with with occasional (say
<< 1k/s) insertions.

> I’m on CentOS7, Erlang 20.x & 21.x with 256GB of RAM.

You may want to disable transparent hugepages.  There's also a default
limit to the number of individual memory mappings per Linux process
that you might
run into with larger nodes, we did and it was a nightmare because it
caused random unexpected VM deaths with no diagnostics of the real
culprit.  (Solution:
bump the kernel's max_map_count, but be advised the limit is there
because some memory map operations in the kernel don't scale well to
large address spaces).

/Mikael