[erlang-questions] What database can I use with erlang to store 100M records and select single one in 0.1s

[Kevin Scaldeferri]

I'm not sure what's "non-standard" about what the OP is trying to do.   
A select on a properly indexed key should be quite fast on any  
database system you choose.  I have to assume that he doesn't have his  
schema set up properly.


-k


On Feb 17, 2009, at 1:27 PM, lenz wrote:

> hi,
>
> searching for solutions to this type of problem is a challenge and  
> there are many ways to solve it. give this site (http://highscalability.com 
> ) a thorough read and you might find something.
> again, i am with jan, try CouchDB and it might pay off to use a unix  
> based machine for stuff like that. i think you are trying to trick  
> your OS to do things it is not meant for. not going into flame wars  
> about windows here but a basic linux setup is really done in no time  
> and gives you way more control for an application that is "non  
> standard" :-)
>
> cheers
> lenz
>
> On Wed, Feb 18, 2009 at 10:26 AM, lenz <norbu09@REDACTED> wrote:
> hi,
>
> searching for solutions to this type of problem is a challenge and  
> there are many ways to solve it. give this site (http://highscalability.com 
> ) a thorough read and you might find something.
> again, i am with jan, try CouchDB and it might pay off to use a unix  
> based machine for stuff like that. i think you are trying to trick  
> your OS to do things it is not meant for. not going into flame wars  
> about windows here but a basic linux setup is really done in no time  
> and gives you way more control for an application that is "non  
> standard" :-)
>
> cheers
> lenz
>
> On Wed, Feb 18, 2009 at 7:42 AM, Joe Armstrong <erlang@REDACTED>  
> wrote:
> What are the types of the keys and records? Is there are systematic  
> structure in
> the keys? are the records fixed or variable lengths? How about using  
> a SSD?
>
> /Joe Armstrong
>
>
> 2009/2/17 Scott Zhang <getyourcontacts@REDACTED>:
> > Hi.
> >
> > I opened a question about mnesia days ago. Now I am facing another  
> question.
> >
> > I am going to scrape large amount of data and put them into  
> database.
> > Previously, my program was written in .NET/C# and saved the  
> records into
> > SqlServer, after 10 days running, I have 63M records in database  
> which make
> > SqlServer really slow, select single record need more than 1  
> minute which is
> > unacceptable.
> >
> > And days ago, I tried to save 7M records in mnesia with table type=
> > disc_only_copies. I tested , select single one from that talbe  
> need more
> > than 20 seconds which is unacceptable too. ( I don't want to  
> conclude
> > mnesia's performance is worse than SqlServer.)
> >
> > Lenz suggested me to use couchDB, just checked, couchDB depends on  
> too much
> > things and these depends make it hard to configure on window system.
> >
> > So here, does anyone has a good suggestion for a database can fit  
> 63M(this
> > time may get 2billion) and select one node within 1 sec?
> >
> > Thanks.
> >
> > Regards.
> >
> > Scott
> >
> > _______________________________________________
> > erlang-questions mailing list
> > erlang-questions@REDACTED
> > http://www.erlang.org/mailman/listinfo/erlang-questions
> >
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://www.erlang.org/mailman/listinfo/erlang-questions
>
>
>
> -- 
> iWantMyName.com
> painless domain registration (finally)
>
>
>
> -- 
> iWantMyName.com
> painless domain registration (finally)
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://www.erlang.org/mailman/listinfo/erlang-questions

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20090217/10165b2e/attachment.htm>