[erlang-questions] How to speed up mnesia startup?

[Richard O'Keefe]

On 12/04/2012, at 9:26 PM, Ulf Wiger wrote:

> 
> Hmm, the first thing that stands out is the inordinate number of tables.
> 
> Out of curiosity, I tried figuring out what is considered a reasonable number of tables for other database management systems:
> 
> - Oracle: over 10,000 tables considered insane
>   (http://asktom.oracle.com/pls/asktom/f?p=100:11:::::P11_QUESTION_ID:26039880025641)

That page has no claim that *Oracle* cannot handle it.
The arguments given are
 - that it seems extremely unlikely that people could "design,
   implement, or maintain" a system with 10,000 distinct tables
   "personally".

   Of course it does not follow that a team could not design,
   implement, and maintain an ALGEBRA by means of which 10,000
   relation values could be constructed and queried.

 - the specific problem looked as if it would fit a two table
   design

   That does not mean the OP's problem would suit such a design,
   although it well might.

 - "flooding the shared pool with 100's of thousands of SQL statements
   for one, that would be the first one - doesn't scale very well at all."

   This doesn't apply to Erlang/Mnesia.

 - You'd have to use dynamic SQL rather than stored procedures in SQL.

   This doesn't apply to Erlang/Mnesia either.

> 
> - Postgres: 10,000 tables works fine; practical limit (on ext3) probably 32k
>   (http://blog.endpoint.com/2008/11/10000-databases-on-postgresql-cluster.html)

That page actually talks about 10,000 DATABASES in one cluster.
Since each database had 1-5 tables, the test tried about
30,000 TABLES.  The 32k limit applies to databases, not tables.
Somebody called "Michael" commented that they had 20,000 tables
in a single database and someone called "wstrzalka" claimed to
have a "similar installation".
> 
> - MySQL Server: no apparent problems
>   (http://stackoverflow.com/questions/610920/maximum-number-of-workable-tables-in-sql-server-and-mysql)

The page asks about (Microsoft) "SQL Server" and also about "MySQL".
"Andy Ansryan" said "I have had a database with 2 million tables"
with "no performance hit at all".  As far as I can see all the
answers were about Microsoft SQL Server, not about MySQL.

For what it's worth,
http://www.firebirdfaq.org/faq61/
says that the maximum number of tables in Firebird is 32k.

>> Here is my conditions:
>>   - DB exists local
>>   - more than 60000 tables
>>   - sum size is about 28 Giga Bytes.
>>   - most tables created options with record_name, attributes(and record_info()) and disc_only_copies.


60,000 tables is, for example, more than Firebird can handle.
The average table size is only 500 kB.
How big are the tuples?  How many tuples per table?

I'm pretty sure that there is some structure in the problem domain
that's not being exploited here.