I'll try.<br><br><div><span class="gmail_quote">On 28/03/06, <b class="gmail_sendername">Sanjaya Vitharana</b> <<a href="mailto:sanjaya@wavenet.lk">sanjaya@wavenet.lk</a>> wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div style="direction: ltr;">
<div><font face="Arial" size="2">Hi ... !!!</font></div>
<div><font face="Arial" size="2"></font> </div>
<div><font face="Arial" size="2">What will be the best way to handle 3 million
records (size of the record = 1K) in mnesia with 4GB RAM. Please
help anyone with such experience.</font></div></div></blockquote><div><br>We have an mnesia database with 25 million records in 128 fragments split across 2 erlang nodes. Server has 8GB of RAM and the two nodes use about 4GB together.
<br></div><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div style="direction: ltr;"><div><font face="Arial" size="2">Currently I'm testing with HP Server with 2GB RAM
(there are plenty of harddisk space).</font></div>
<div><font face="Arial" size="2"></font> </div>
<div><font face="Arial" size="2">I'm using beow to create the table, but getting
problems when the table getting bigger (~350000 records).</font></div>
<div><font face="Arial" size="2"></font><font face="Arial" size="2"></font> </div>
<div><font face="Arial" size="2">mnesia:create_table(profile_db,[<br> {disc_copies,
NodeList},<br> {type,
ordered_set}, <br> {index,
[type, last_update_date, first_creation_date,
fax_no]},<br> {frag_properties,
[{n_fragments, 30},{n_disc_copies,
1}]},<br> {attributes,
record_info(fields,
profile_db)}<br>
]),</font></div></div></blockquote><div><br>Bear in mind that when you have an ordered set fragmented table, each fragment is an ordered set. The property does not apply across all tables.<br><br></div><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div style="direction: ltr;"><div><font face="Arial" size="2">Problems: (little bit details added to the end of
the file, but may be not sufficient, if anyone needs more details I can
send)</font></div>
<div><font face="Arial" size="2"></font> </div>
<div><font face="Arial" size="2">1.) unexpected restarts by heart. I have increase
the heart beat timeout from 30 to 60 & 90. It will bring me from (~100000
receords to ~350000 records). But again it comes again this time</font></div>
<div><font face="Arial" size="2"></font> </div>
<div><font face="Arial" size="2">2.) some unexpected errors, which was not happend
earlier (I mean upto the current size of the DB)</font></div>
<div><font face="Arial" size="2">2.1)
{aborted,{no_exists,profile_db_frag25}}</font></div></div></blockquote><div><br>This is strange. It seems to suggest that this fragment isn't available yet. Are all tables fully loaded before you start populating? Check using the mnesia:wait_for_tables function.
<br><br></div><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div style="direction: ltr;"><div><font face="Arial" size="2">2.2) ** exited:
{timeout,{gen_server,call,[vm_prof_db_svr,db_backup_once]}} **</font></div></div></blockquote><div><br>The backup is taking quite a long time. Have you tried increasing the timeout? <br></div><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div style="direction: ltr;"><div><font face="Arial" size="2">2.3) error_info:
{{failed,{error,{file_error,<br>
"/usr2/omni_vm_prof/db/vmdb/db/backup/db_back_2006-3-28_14-3-4.BUPTMP",<br>
enoent}}},<br>
[{disk_log,open,1}]}</font></div></div></blockquote><div><br>enoent - The temporary backup file does not exist. Dunno why. <br></div><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div style="direction: ltr;"><div><font face="Arial" size="2">2.4) {error,{"Cannot prepare checkpoint (replica
not available)",
[profile_db_frag10,{{1143,528317,121399},vmdb@prof}]}}</font></div></div></blockquote><div><br><br>Looks like your fragments are spread across a few nodes and one of the fragments is not available - are all nodes connected to each other.
<br></div><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div style="direction: ltr;"><div><font face="Arial" size="2">2.5) eheap_alloc: Cannot allocate 122441860 bytes
of memory (of type
"heap").<br>
Aborted</font></div></div></blockquote><div><br>Your node ran out of memory. You seem to have quite a lot of secondary indices. Bear in mind that each one will consume more memory. It is trying to allocate about 122MB of memory. Have you tried this on a machine with 4GB of RAM.
<br><br></div><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div style="direction: ltr;"><div><font face="Arial" size="2">I have idea to changing the below properties
and try, but I don't no this will be the best way or not.</font></div>
<div><font face="Arial" size="2">disc_copies -> disc_only_copies</font></div></div></blockquote><div><br>Performance will suffer.<br> <br></div><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div style="direction: ltr;"><div><font face="Arial" size="2">ordered_set
-> set (of course I could not find any direct function for this in mnesia
reference manual, are there any way ?)</font></div></div></blockquote><div><br>You will have to delete the table and recreate it.<br> <br></div><br></div>hth<br>Chandru<br>