job regulation (was Re: Mnesia disk performance (was RE: multi-attribute mnesia indexes?))
Wed Jan 3 10:24:43 CET 2001
On Tue, 2 Jan 2001, Shawn Pearce wrote:
>I for one would like to be able to force my transactions (write that
>is) to wait for the log writer to finish writing them if the problem
>is that mnesia will overload like this. Should I step up the
>frequency of my log writer in order to force it to write more
>frequently, and hopefully at least simulate that the writes are
>waiting for the log?
The right way to do this is probably to introduce job scheduling for
At AXD 301, we've had reason to think very hard about load regulation
-- with pretty fantastic results, I might add: the AXD 301 should be
almost impervious to denial-of-service attacks, and easily meets the
BellCore requirement of 90% throughput at 150% continuous load.
What we've found though, is that for really effective load regulation,
_all_ significant jobs in the system should be made queuable in a job
scheduler. Since we can't do this with mnesia (no hooks for load
regulation), our job scheduler samples the "background load" and takes
this into account.
I would really like to se a generic framework for plugging in a load
regulator in an OTP system. The default behaviour should be "no
regulation", and it should be possible to use different load
regulators (but with common semantics) for different products.
For mnesia, I lean towards the following three measures:
- provide for synchronous transactions:
these would work similarly to sync_dirty, i.e. they return when
the transaction is committed on all nodes.
- make it possible/configurable to force a log dump before returning
to the caller. I suggested a wrapper before (still haven't tested
it myself). Would that method work?
- make it possible to hook log dumps into a load regulator. This
would require a framework for load regulation as mentioned above.
To complete the picture, the programs starting transactions would of
course also be load regulated. As mnesia cannot know the priority and
total cost of a transaction, this would have to be done by the
Ulf Wiger tfn: +46 8 719 81 95
Senior System Architect mob: +46 70 519 81 95
Strategic Product & System Management ATM Multiservice Networks
Data Backbone & Optical Services Division Ericsson Telecom AB
More information about the erlang-questions