mnesia is overloaded

[Anders Nygren]

Hi
Today something happened that I dont quite understand.
We ran a test that was supposed to send 25000 messages, where every
message leads to 5-10 mnesia writes.
When the test was finished we found that only about 4000 of the messages were
processed completely and there were no traces of the others. 
(There is of course a chance that we manage to screw something up on our own 
but tests with 1000-2000 messages have never had any problems.)

The only messages we got in the log was couple of warnings from mnesia.
Mnesia is overloaded: {dump_log,write_threshold}

Searching the archives I found that this was discussed back in 2000
when Sean Hinde said in message  
http://www.erlang.org/ml-archive/erlang-questions/200007/msg00099.html

"It also makes a big difference whether you have dump_log_load_regulation set
to true or false. If true, the dumper process runs much more slowly which
will make the error report more likely, If false the dumper runs at full
speed but the emulator itself can pause noticeably while it waits for lots of
disc intensive bifs to complete.

I tend to run with it set to true as I need to guarantee read response
times."

Which sounds like true means to run the dumps at a lower priority.

But the mnesia documentation says
"Controls if the log dumps should be performed as fast as possible or if the 
dumper should do its own load regulation."

Which is not exactly clear what true is supposed to mean.

Is there any tuning that can be done to improve performance with lots of 
writes?


The part that has me worried is that mnesia:info() showed that one 
table that is used as a persistant message queue had 25 records, but
 > mnesia:dirty_first(pq_msg).
'$end_of_table'
and adding new rows did not change it.
After 
mnesia:stop().
mnesia:start().

It found the 25 queued messages again.

Anders Nygren