[erlang-questions] Handling a large number of DETS and ETS tables

[Jonathan Haddad]

Hey folks,

I'm creating an application that will handle several ets or dets tables at
once.  I won't know the names of the tables before hand, they'll be
dynamically created.

Each table will most likely correspond to a domain name that's being tracked
within the system, but there might be more depending on what limits I hit.
 I'm slightly concerned about DETS 2GB limit, and we want to be able to move
individual domains into different nodes.  (rev 2 is having the system do
that automatically).

What's a good way of dynamically managing which ones I have?  I'm
considering 2 methods so far.

1. Use list_to_atom to convert the domain name to an atom, and use that to
register a process for that table.  Each process manages a single table.

2. Have a single process which has all tables in a list, and uses the
appropriate one for each request.  This introduces a bottleneck at that
process and seems like I'd want to avoid this one.  Not to mention I'm
somewhat defeating the purpose of using a language built for concurrency.

What should I be considering as far as I/O issues are concerned?  How does
an ETS write compare to a simple disk write?   Does it hold the data in
memory for reads?

Thanks,
Jon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20081111/d11fc64b/attachment.htm>