[erlang-questions] Private ets table vs. Process directory

[Joe Armstrong]

In order to even think about your question I'd need certain data -
words like "huge" as in "huge amounts of copying" and "limited numbers
of processes"
etc. do not convey much meaning.

Huge means different things to different people - to some people Huge
means Gbytes
(I talked the other day to somebody who used the word Huge - and I
said "how big"
he said tens of PetaBytes)

To me huge means a data structure that is larger than the RAM on my machine
(which is 16GB) - so not only do you have to say what you meant by huge but also
how your numbers relate to your machine(s).

Also how long do you have to do what? - Handling huge amounts of data
is easy if you have a big enough disks and enough time - you also need
to say (roughly) how long you have to do what (are we talking seconds,
milliseconds,
hours, days???)

The more numbers you add to questions like this the better answers
you'll get :-)

Cheers

/Joe



On Wed, Feb 7, 2018 at 5:56 PM, Charles Hixson
<charleshixsn@REDACTED> wrote:
> When should a private ets table be preferred over the process directory?
>
> To give some context, I'm expecting to has nearly as many processes as I can
> run, and that each one will need internal mutable state.  Also, that the
> mutable state will be complex (partially because of the limited number of
> processes), so passing the state as function parameters would entail huge
> amounts of copying.  (Essentially I'd be modifying nodes deep within trees.)
>
> Mutable state would allow me to avoid the copying, and the state is not
> exported from the process.  I'm concerned that a huge number of private ets
> tables would use excessive memory, decreasing the number of processes I
> could use...but all the references keep saying not to use the process
> directory.
>
> I'm still designing things now, so this is the ideal time to decide.  An
> alternative is that I could use a single public ets table, with each process
> only accessing its own data, but I suspect that might involve a lot of
> locking overhead, even though in principle nothing should need to be locked.
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions