[erlang-questions] Mnesia sorting/limiting

[Paul Mineiro]

On Thu, 11 Jun 2009, Ulf Wiger wrote:

> Barry Mitchelson wrote:
> >
> > ...aside from this specific problem,
> > the more general question I'm trying to find an answer is how to
> > select data from an mnesia table ordered by a certain element and
> > limited without having to go through the entire table when I perform
> > the sort. I'm not sure if I'm trying to shoehorn my SQL thinking into
> > this, or if I'm not aware of the solution.
>
> The easiest way to do this would be to use an ordered set,
> using an entry like {{Priority, Timestamp}, Reference},
> where Reference is some reference to whatever it is that
> you have queued.
>
> If you want to have ascending order on priority and LIFO
> on time, you can use the following trick:
>
> timestamp() ->
>     {A,B,C} = erlang:now(),
>     {-A,-B,-C}.
>
> (FIFO is more common, of course, but sometimes LIFO makes more
> sense in practice, along the lines of "serve the ones most
> likely to succeed".)
>
> Then you can use mnesia:dirty_first(Tab) to pick the first item
> from the queue.
>
> If you want to limit the number of entries, mnesia:table_info(Tab,size)
> will give you the number of objects, and you can delete the last item
> when you push a new one, once you've hit the limit.

This will work, and if the volume is low it'll work great, and it's easy
to get going.  When we scaled this up, we discovered that we were doing
one or both of the following alot: 1) insert at the front or 2) delete
from the back.  Since the underlying implementation is trying to be a
balanced tree, this leads to alot of rebalancing, which slows things down
(doubly so since our ordered set was on disk). We switched to specialized
data structures at that point.

I feel obliged to mention that premature optimization is evil.  The
point of this post is just to give you something to measure to see if a
problem is occurring.  Please start as Ulf recommends.

-- p

>
> BR,
> Ulf W
>
>
> --
> Ulf Wiger
> CTO, Erlang Training & Consulting Ltd
> http://www.erlang-consulting.com
>
> ________________________________________________________________
> erlang-questions mailing list. See http://www.erlang.org/faq.html
> erlang-questions (at) erlang.org
>
>