<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=iso-8859-1">
<META content="MSHTML 6.00.2600.0" name=GENERATOR>
<STYLE></STYLE>
</HEAD>
<BODY bgColor=#ffffff>
<DIV><FONT face=Arial size=2>Hi ... !!!</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>What will be the best way to handle 3 million
records (size of the record = 1K) in mnesia with 4GB RAM. Please
help anyone with such experience.</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>Currently I'm testing with HP Server with 2GB RAM
(there are plenty of harddisk space).</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>I'm using beow to create the table, but getting
problems when the table getting bigger (~350000 records).</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial
size=2>mnesia:create_table(profile_db,[<BR> {disc_copies,
NodeList},<BR> {type,
ordered_set}, <BR> {index,
[type, last_update_date, first_creation_date,
fax_no]},<BR> {frag_properties,
[{n_fragments, 30},{n_disc_copies,
1}]},<BR> {attributes,
record_info(fields,
profile_db)}<BR>
]),</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>Problems: (little bit details added to the end of
the file, but may be not sufficient, if anyone needs more details I can
send)</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>1.) unexpected restarts by heart. I have increase
the heart beat timeout from 30 to 60 & 90. It will bring me from (~100000
receords to ~350000 records). But again it comes again this time</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>2.) some unexpected errors, which was not happend
earlier (I mean upto the current size of the DB)</FONT></DIV>
<DIV><FONT face=Arial size=2>2.1)
{aborted,{no_exists,profile_db_frag25}}</FONT></DIV>
<DIV><FONT face=Arial size=2>2.2) ** exited:
{timeout,{gen_server,call,[vm_prof_db_svr,db_backup_once]}} **</FONT></DIV>
<DIV><FONT face=Arial size=2>2.3) error_info:
{{failed,{error,{file_error,<BR>
"/usr2/omni_vm_prof/db/vmdb/db/backup/db_back_2006-3-28_14-3-4.BUPTMP",<BR>
enoent}}},<BR>
[{disk_log,open,1}]}</FONT></DIV>
<DIV><FONT face=Arial size=2>2.4) {error,{"Cannot prepare checkpoint (replica
not available)",
[profile_db_frag10,{{1143,528317,121399},vmdb@prof}]}}</FONT></DIV>
<DIV><FONT face=Arial size=2>2.5) eheap_alloc: Cannot allocate 122441860 bytes
of memory (of type
"heap").<BR>
Aborted<BR></FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>I have idea to changing the below properties
and try, but I don't no this will be the best way or not.</FONT></DIV>
<DIV><FONT face=Arial size=2>disc_copies -> disc_only_copies<BR>ordered_set
-> set (of course I could not find any direct function for this in mnesia
reference manual, are there any way ?)<BR></FONT><FONT face=Arial
size=2></FONT><FONT face=Arial size=2></FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>So Please help anyone with experience of large
Data Bases.</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>Regards,</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>Sanjaya Vitharana</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV>
<DIV><FONT face=Arial
size=2>-----------------------------------------------------------------------------------------------------------------------------</FONT></DIV>
<DIV><FONT face=Arial size=2>1.)</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>DB Server Started<BR>DB will be backup in 59536
secs<BR>heart: Tue Mar 28 10:28:47 2006: Erlang has closed.<BR>heart: Tue Mar 28
10:28:52 2006: Unable to kill old process, kill failed (tried multiple
times).<BR>heart: Tue Mar 28 10:28:52 2006: Executed
"/etc/rc.d/init.d/vm_prof.system start". Terminating.</FONT></DIV>
<DIV>
<DIV><FONT face=Arial
size=2>-----------------------------------------------------------------------------------------------------------------------------</FONT></DIV></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>2.1)
{aborted,{no_exists,profile_db_frag25}}</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>=ERROR REPORT==== 28-Mar-2006::10:28:39 ===<BR>**
Generic server vm_prof_db_svr terminating <BR>** Last message in was
{count,"main"}<BR>** When Server state ==
{state,"/usr2/omni_vm_prof/db/vmdb",1400000}<BR>** Reason for termination ==
<BR>** {aborted,{no_exists,profile_db_frag25}}<BR>DB Server
Starting</FONT></DIV>
<DIV> </DIV>
<DIV><FONT face=Arial size=2>=CRASH REPORT==== 28-Mar-2006::10:28:39
===<BR> crasher:<BR> pid:
<0.52.0><BR> registered_name:
vm_prof_db_svr<BR> error_info:
{aborted,{no_exists,profile_db_frag25}}<BR> initial_call:
{gen,init_it,<BR>
[gen_server,<BR>
<0.51.0>,<BR>
<0.51.0>,<BR>
{local,vm_prof_db_svr},<BR>
vm_prof_db_svr,<BR>
[],<BR>
[]]}<BR> ancestors:
[<0.51.0>,<0.50.0>]<BR> messages:
[]<BR> links:
[<0.51.0>,<0.63.0>]<BR> dictionary:
[]<BR> trap_exit: false<BR> status:
running<BR> heap_size: 377<BR> stack_size:
21<BR> reductions: 3667<BR> neighbours:=ERROR REPORT====
28-Mar-2006::10:28:39 ===<BR>** Generic server vm_prof_db_svr terminating <BR>**
Last message in was {count,"main"}<BR>** When Server state ==
{state,"/usr2/omni_vm_prof/db/vmdb",1400000}<BR>** Reason for termination ==
<BR>** {aborted,{no_exists,profile_db_frag25}}<BR>DB Server
Starting</FONT></DIV>
<DIV> </DIV>
<DIV><FONT face=Arial size=2>=CRASH REPORT==== 28-Mar-2006::10:28:39
===<BR> crasher:<BR> pid:
<0.52.0><BR> registered_name:
vm_prof_db_svr<BR> error_info:
{aborted,{no_exists,profile_db_frag25}}<BR> initial_call:
{gen,init_it,<BR>
[gen_server,<BR>
<0.51.0>,<BR>
<0.51.0>,<BR>
{local,vm_prof_db_svr},<BR>
vm_prof_db_svr,<BR>
[],<BR>
[]]}<BR> ancestors:
[<0.51.0>,<0.50.0>]<BR> messages:
[]<BR> links:
[<0.51.0>,<0.63.0>]<BR> dictionary:
[]<BR> trap_exit: false<BR> status:
running<BR> heap_size: 377<BR> stack_size:
21<BR> reductions: 3667<BR> neighbours:</FONT></DIV>
<DIV><FONT face=Arial size=2><FONT face=Arial
size=2>-----------------------------------------------------------------------------------------------------------------------------</FONT></FONT></DIV>
<DIV><FONT face=Arial size=2>2.2) </FONT></DIV>
<DIV><FONT face=Arial size=2>(<A href="mailto:vmdb@prof)2">vmdb@prof)2</A>>
gen_server:call(vm_prof_db_svr,db_backup_once).<BR>** exited:
{timeout,{gen_server,call,[vm_prof_db_svr,db_backup_once]}} **</FONT></DIV>
<DIV><FONT face=Arial size=2>
<DIV><FONT face=Arial size=2><FONT face=Arial
size=2>-----------------------------------------------------------------------------------------------------------------------------</FONT></FONT></DIV>
<DIV>2.3)</DIV>
<DIV> </DIV>
<DIV>gen_server:call(vm_prof_db_svr,db_backup_once).<BR>DB will backup in to:
"/usr2/omni_vm_prof/db/vmdb/db/backup/db_back_2006-3-28_14-3-4"<BR>ok<BR>(<A
href="mailto:vmdb@prof)2">vmdb@prof)2</A>> <BR>=CRASH REPORT====
28-Mar-2006::14:03:04 ===<BR> crasher:<BR> pid:
<0.266.0><BR> registered_name: []<BR>
error_info:
{{failed,{error,{file_error,<BR>
"/usr2/omni_vm_prof/db/vmdb/db/backup/db_back_2006-3-28_14-3-4.BUPTMP",<BR>
enoent}}},<BR>
[{disk_log,open,1}]}<BR> initial_call:
{disk_log,init,[<0.69.0>,<0.70.0>]}<BR> ancestors:
[disk_log_sup,kernel_safe_sup,kernel_sup,<0.9.0>]<BR>
messages: []<BR> links: [<0.69.0>]<BR>
dictionary: []<BR> trap_exit: true<BR>
status: running<BR> heap_size: 610<BR>
stack_size: 21<BR> reductions: 598<BR> neighbours:</DIV>
<DIV> </DIV>
<DIV>=SUPERVISOR REPORT==== 28-Mar-2006::14:03:04
===<BR> Supervisor:
{local,disk_log_sup}<BR> Context:
child_terminated<BR> Reason:
{{failed,{error,{file_error,<BR>
"/usr2/omni_vm_prof/db/vmdb/db/backup/db_back_2006-3-28_14-3-4.BUPTMP",<BR>
enoent}}},<BR>
[{disk_log,open,1}]}<BR> Offender:
[{pid,<0.266.0>},<BR>
{name,disk_log},<BR>
{mfa,{disk_log,istart_link,[<0.70.0>]}},<BR>
{restart_type,temporary},<BR>
{shutdown,1000},<BR>
{child_type,worker}]</DIV>
<DIV> </DIV>
<DIV>DB backup failed:
{file_error,"/usr2/omni_vm_prof/db/vmdb/db/backup/db_back_2006-3-28_14-3-4.BUPTMP",<BR>
enoent}</DIV>
<DIV> </DIV>
<DIV>=ERROR REPORT==== 28-Mar-2006::14:03:04 ===<BR>Mnesia(<A
href="mailto:vmdb@prof">vmdb@prof</A>): ** ERROR ** Failed to abort backup.
mnesia_backup:abort_write["/usr2/omni_vm_prof/db/vmdb/db/backup/db_back_2006-3-28_14-3-4"]
->
{'EXIT',<BR>
{badarg,<BR>
[{mnesia_backup,<BR>
abort_write,<BR>
1},<BR>
{mnesia_log,<BR>
abort_write,<BR>
4},<BR>
{mnesia_log,<BR>
do_backup_master,<BR>
1},<BR>
{mnesia_log,<BR>
backup_master,<BR>
2}]}}<BR>---------------------------------------------------------------------------------------------</DIV>
<DIV>2.4)</DIV>
<DIV><BR>[root@prof root]# /usr2/omni_vm_prof/system/bin/to_erl
/usr2/omni_vm_prof/pipe/<BR>Attaching to
/usr2/omni_vm_prof/pipe/erlang.pipe.2183066 (^D to exit)</DIV>
<DIV> </DIV>
<DIV>vm_prof_db:db_backup("/usr2/omni_vm_prof").<BR>DB will backup in to:
"/usr2/omni_vm_prof/db/backup/db_back_2006-3-28_12-45-17"<BR>DB backup failed:
{"Cannot prepare checkpoint (replica not
available)",<BR>
[profile_db_frag10,{{1143,528317,121399},vmdb@prof}]}<BR>{error,{"Cannot prepare
checkpoint (replica not
available)",<BR>
[profile_db_frag10,{{1143,528317,121399},vmdb@prof}]}}</DIV>
<DIV>---------------------------------------------------------------------------------------------</DIV>
<DIV>2.5)<BR>(<A href="mailto:vmdb@prof)13">vmdb@prof)13</A>>
mnesia:table_info(profile_db,size).<BR>Crash dump was written to:
erl_crash.dump<BR>eheap_alloc: Cannot allocate 122441860 bytes of memory (of
type
"heap").<BR>
Aborted<BR>You have new mail in /var/spool/mail/root<BR>[root@prof vmdb]# </DIV>
<DIV>---------------------------------------------------------------------------------------------<BR></DIV></DIV></FONT></DIV></BODY></HTML>