Large DBs, mnesia_frag ????

Sanjaya Vitharana <>
Tue Mar 28 10:32:54 CEST 2006


Hi ... !!!

What will be the best way to handle 3 million records (size of the record = 1K) in mnesia with 4GB RAM. Please help anyone with such experience.

Currently I'm testing with HP Server with 2GB RAM (there are plenty of harddisk space).

I'm using beow to create the table, but getting problems when the table getting bigger (~350000 records).

mnesia:create_table(profile_db,[
                      {disc_copies, NodeList},
                      {type, ordered_set}, 
                      {index, [type, last_update_date, first_creation_date, fax_no]},
                      {frag_properties, [{n_fragments, 30},{n_disc_copies, 1}]},
                      {attributes, record_info(fields, profile_db)}
                      ]),

Problems: (little bit details added to the end of the file, but may be not sufficient, if anyone needs more details I can send)

1.) unexpected restarts by heart. I have increase the heart beat timeout from 30 to 60 & 90. It will bring me from (~100000 receords to ~350000 records). But again it comes again this time

2.) some unexpected errors, which was not happend earlier (I mean upto the current size of the DB)
2.1) {aborted,{no_exists,profile_db_frag25}}
2.2) ** exited: {timeout,{gen_server,call,[vm_prof_db_svr,db_backup_once]}} **
2.3)     error_info: {{failed,{error,{file_error,
                                     "/usr2/omni_vm_prof/db/vmdb/db/backup/db_back_2006-3-28_14-3-4.BUPTMP",
                                     enoent}}},
                  [{disk_log,open,1}]}
2.4) {error,{"Cannot prepare checkpoint (replica not available)", [profile_db_frag10,{{1143,528317,121399}}]}}
2.5) eheap_alloc: Cannot allocate 122441860 bytes of memory (of type "heap").
                                                                        Aborted



I have idea to changing the below properties and try, but I don't no this will be the best way or not.
disc_copies -> disc_only_copies
ordered_set -> set (of course I could not find any direct function for this in mnesia reference manual, are there any way ?)


So Please help anyone with experience of large Data Bases.


Regards,

Sanjaya Vitharana


-----------------------------------------------------------------------------------------------------------------------------
1.)

DB Server Started
DB will be backup in 59536 secs
heart: Tue Mar 28 10:28:47 2006: Erlang has closed.
heart: Tue Mar 28 10:28:52 2006: Unable to kill old process, kill failed (tried multiple times).
heart: Tue Mar 28 10:28:52 2006: Executed "/etc/rc.d/init.d/vm_prof.system start". Terminating.
-----------------------------------------------------------------------------------------------------------------------------

2.1) {aborted,{no_exists,profile_db_frag25}}

=ERROR REPORT==== 28-Mar-2006::10:28:39 ===
** Generic server vm_prof_db_svr terminating 
** Last message in was {count,"main"}
** When Server state == {state,"/usr2/omni_vm_prof/db/vmdb",1400000}
** Reason for termination == 
** {aborted,{no_exists,profile_db_frag25}}
DB Server Starting

=CRASH REPORT==== 28-Mar-2006::10:28:39 ===
  crasher:
    pid: <0.52.0>
    registered_name: vm_prof_db_svr
    error_info: {aborted,{no_exists,profile_db_frag25}}
    initial_call: {gen,init_it,
                      [gen_server,
                       <0.51.0>,
                       <0.51.0>,
                       {local,vm_prof_db_svr},
                       vm_prof_db_svr,
                       [],
                       []]}
    ancestors: [<0.51.0>,<0.50.0>]
    messages: []
    links: [<0.51.0>,<0.63.0>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 377
    stack_size: 21
    reductions: 3667
  neighbours:=ERROR REPORT==== 28-Mar-2006::10:28:39 ===
** Generic server vm_prof_db_svr terminating 
** Last message in was {count,"main"}
** When Server state == {state,"/usr2/omni_vm_prof/db/vmdb",1400000}
** Reason for termination == 
** {aborted,{no_exists,profile_db_frag25}}
DB Server Starting

=CRASH REPORT==== 28-Mar-2006::10:28:39 ===
  crasher:
    pid: <0.52.0>
    registered_name: vm_prof_db_svr
    error_info: {aborted,{no_exists,profile_db_frag25}}
    initial_call: {gen,init_it,
                      [gen_server,
                       <0.51.0>,
                       <0.51.0>,
                       {local,vm_prof_db_svr},
                       vm_prof_db_svr,
                       [],
                       []]}
    ancestors: [<0.51.0>,<0.50.0>]
    messages: []
    links: [<0.51.0>,<0.63.0>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 377
    stack_size: 21
    reductions: 3667
  neighbours:
-----------------------------------------------------------------------------------------------------------------------------
2.2) 
()2> gen_server:call(vm_prof_db_svr,db_backup_once).
** exited: {timeout,{gen_server,call,[vm_prof_db_svr,db_backup_once]}} **
-----------------------------------------------------------------------------------------------------------------------------
2.3)

gen_server:call(vm_prof_db_svr,db_backup_once).
DB will backup in to: "/usr2/omni_vm_prof/db/vmdb/db/backup/db_back_2006-3-28_14-3-4"
ok
()2> 
=CRASH REPORT==== 28-Mar-2006::14:03:04 ===
  crasher:
    pid: <0.266.0>
    registered_name: []
    error_info: {{failed,{error,{file_error,
                                     "/usr2/omni_vm_prof/db/vmdb/db/backup/db_back_2006-3-28_14-3-4.BUPTMP",
                                     enoent}}},
                  [{disk_log,open,1}]}
    initial_call: {disk_log,init,[<0.69.0>,<0.70.0>]}
    ancestors: [disk_log_sup,kernel_safe_sup,kernel_sup,<0.9.0>]
    messages: []
    links: [<0.69.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 610
    stack_size: 21
    reductions: 598
  neighbours:

=SUPERVISOR REPORT==== 28-Mar-2006::14:03:04 ===
     Supervisor: {local,disk_log_sup}
     Context:    child_terminated
     Reason:     {{failed,{error,{file_error,
                                     "/usr2/omni_vm_prof/db/vmdb/db/backup/db_back_2006-3-28_14-3-4.BUPTMP",
                                     enoent}}},
                  [{disk_log,open,1}]}
     Offender:   [{pid,<0.266.0>},
                  {name,disk_log},
                  {mfa,{disk_log,istart_link,[<0.70.0>]}},
                  {restart_type,temporary},
                  {shutdown,1000},
                  {child_type,worker}]

DB backup failed: {file_error,"/usr2/omni_vm_prof/db/vmdb/db/backup/db_back_2006-3-28_14-3-4.BUPTMP",
                              enoent}

=ERROR REPORT==== 28-Mar-2006::14:03:04 ===
Mnesia(): ** ERROR ** Failed to abort backup. mnesia_backup:abort_write["/usr2/omni_vm_prof/db/vmdb/db/backup/db_back_2006-3-28_14-3-4"] -> {'EXIT',
                                                                                                                                                         {badarg,
                                                                                                                                                             [{mnesia_backup,
                                                                                                                                                                  abort_write,
                                                                                                                                                                  1},
                                                                                                                                                              {mnesia_log,
                                                                                                                                                                  abort_write,
                                                                                                                                                                  4},
                                                                                                                                                              {mnesia_log,
                                                                                                                                                                  do_backup_master,
                                                                                                                                                                  1},
                                                                                                                                                              {mnesia_log,
                                                                                                                                                                  backup_master,
                                                                                                                                                                  2}]}}
---------------------------------------------------------------------------------------------
2.4)

[ root]# /usr2/omni_vm_prof/system/bin/to_erl /usr2/omni_vm_prof/pipe/
Attaching to /usr2/omni_vm_prof/pipe/erlang.pipe.2183066 (^D to exit)

vm_prof_db:db_backup("/usr2/omni_vm_prof").
DB will backup in to: "/usr2/omni_vm_prof/db/backup/db_back_2006-3-28_12-45-17"
DB backup failed: {"Cannot prepare checkpoint (replica not available)",
                   [profile_db_frag10,{{1143,528317,121399}}]}
{error,{"Cannot prepare checkpoint (replica not available)",
        [profile_db_frag10,{{1143,528317,121399}}]}}
---------------------------------------------------------------------------------------------
2.5)
()13> mnesia:table_info(profile_db,size).
Crash dump was written to: erl_crash.dump
eheap_alloc: Cannot allocate 122441860 bytes of memory (of type "heap").
                                                                        Aborted
You have new mail in /var/spool/mail/root
[ vmdb]# 
---------------------------------------------------------------------------------------------

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20060328/281f9e45/attachment.html>


More information about the erlang-questions mailing list