[erlang-questions] What database can I use with erlang to store 100M records and select single one in 0.1s

[Chandru]

Hi,

2009/2/17 Scott Zhang <getyourcontacts@REDACTED>

> Hi.
>
> I opened a question about mnesia days ago. Now I am facing another
> question.
>
> I am going to scrape large amount of data and put them into database.
> Previously, my program was written in .NET/C# and saved the records into
> SqlServer, after 10 days running, I have 63M records in database which make
> SqlServer really slow, select single record need more than 1 minute which is
> unacceptable.
>
> And days ago, I tried to save 7M records in mnesia with table type=
> disc_only_copies. I tested , select single one from that talbe need more
> than 20 seconds which is unacceptable too. ( I don't want to conclude
> mnesia's performance is worse than SqlServer.)
>
Have you tried to fragment the table? It is not clear from your question
whether you are querying mnesia using the primary key, or matching using
select statements. We currently have an mnesia database storing 40M+ records
with lookup times in the order of a few milliseconds.

cheers
Chandru
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20090302/ecb01706/attachment.htm>