[erlang-questions] dets disk i/o

[Dmitry Kolesnikov]

leveldb uses LSM-tree while dets linear hash like schema.
I do agree about leveldb usage in general. 
However, I am looking for explicit linear hashing storage.

- Dmitry
 

On Sep 8, 2013, at 11:05 PM, OvermindDL1 <overminddl1@REDACTED> wrote:

> I would recommend leveldb if you need local and fast storage, there is a great erlang library for it too.
> 
> On Sep 8, 2013 2:03 PM, "Janos Hary" <janos.n.hary@REDACTED> wrote:
> Dimitry,
> 
> Not exactly what you asked, but I measured the write performance of mnesia
> disc_only bag table, which is backed by dets. I used the following function
> to generate records, where I can set the amount of repeating keys by
> ?REPCOUNT:
> 
> fill_modulo(_, Tab, 0, _, _, AccessMod) ->
>         {ok, tab_size(Tab, AccessMod)};
> fill_modulo(run, Tab, N, Max, T, AccessMod) ->
>         T1 = progress(N, Max, T),
>         R = #trec{ dbid = N rem ?REPCOUNT, data1 = <<"Hello World!">>, data2
> = Max - N },
>         case write_transaction(Tab, R, AccessMod) of
>                 ok ->
>                         fill_modulo(run, Tab, N-1, Max, T1, AccessMod);
>                 error ->
>                         {error, Max-N}
>         end.
> 
> This is the result of inserting 4M records with ?REPCOUNT set to 10000 (each
> line shows the running time of the given 5% of the whole task):
> ......... 11.01s at 5%
> ......... 11.44s at 10%
> ......... 12.39s at 15%
> ......... 12.37s at 20%
> ......... 12.89s at 25%
> ......... 13.53s at 30%
> ......... 14.09s at 35%
> ......... 14.04s at 40%
> ......... 15.07s at 45%
> ......... 14.84s at 50%
> ......... 15.52s at 55%
> ......... 15.49s at 60%
> ......... 16.57s at 65%
> ......... 16.25s at 70%
> ......... 17.60s at 75%
> ......... 16.89s at 80%
> ......... 17.41s at 85%
> ......... 18.63s at 90%
> ......... 17.77s at 95%
> .........
> tab7 -> (ok) modulo insert into bag 4000000 of 4000000 in 306.931000s
> 
> And this is the result of the same test ran with ?REPCOUNT set to 100:
> ......... 30.34s at 5%
> ......... 70.68s at 10%
> ......... 117.20s at 15%
> ......... 163.29s at 20%
> ......... 203.08s at 25%
> ......... 246.20s at 30%
> ... and giving up waiting.
> 
> Since then I learned some other problems using mnesia/dets in this manner
> (see my post 'mnesia silent failure and table corruption' and Ahmed Omar's
> reply), and I stopped further investigation. Now I'm evaluating Berkely DB
> for the task.
> 
> Best Regards,
> Janos
> 
> -----Original Message-----
> From: erlang-questions-bounces@REDACTED
> [mailto:erlang-questions-bounces@REDACTED] On Behalf Of Dmitry Kolesnikov
> Sent: Sunday, September 08, 2013 9:08 PM
> To: erlang-questions Questions
> Subject: Re: [erlang-questions] dets disk i/o
> 
> ping!
> It looks that I should measure it by myself.
> 
> Best Regards,
> Dmitry
> 
> On Sep 4, 2013, at 12:09 AM, Dmitry Kolesnikov <dmkolesnikov@REDACTED>
> wrote:
> 
> > Hello,
> >
> > Does any one run a disk i/o analysis on dets tables?
> > I am very curious about bag case, write scenario.
> >
> > Let's assume that I have a key which is associated with multiple entries.
> > What is dets behaviour when I am writing a new entry to same key.
> > Does it make append-only to data slot? or Does it read/write whole slot?
> >
> > Best Regards,
> > Dmitry
> 
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
> 
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20130908/e537de23/attachment.htm>