[erlang-questions] Mnesia crashes and restarts the Erlang VM repeatedly

[Arun]

Hi,

     I am facing some serious issues with mnesia database. We have an 
erlang application using mnesia database as back-end. The application 
runs on the hardware and the IP address of the hardware will be changing 
based on its connectivity and so the erlang node name. Some times as 
soon as the node changes am getting an error log as given below and the 
VM restarts repeatedly. But if I delete the mnesia related files and 
restart again, application runs smoothly. For me data reliability and 
persistence are the prime concern so I can not go with the above 
mentioned method. Can somebody kindly assist me how to resolve this 
issue with out deleting mnesia schema.

Current working directory:/root/rel-3/galaxy/lib/galaxy-1.0.0/priv
  Release number as string:3
PHY: mdio@REDACTED:01 - Link is Up - 100/Full
36524 01999.282  2001391.0     71.4  532699753591315.4  11871.3         0
Exec: /usr/lib/erlang/erts-8.0/bin/erlexec -boot /root/rel-3/galaxy/releases/0.0.1/galaxy -mode embedded -boot_var ERTS_LIB_DIR /usr/lib/erlang/erts-8.0/../lib -config /root/rel-3/galaxy/releases/0.0.1/sys.config -args_file /root/rel-3/galaxy/vm.args -- console
Root: /root/rel-3/galaxy
/root/rel-3/galaxy
Erlang/OTP 19 [erts-8.0] [source] [async-threads:10] [kernel-poll:false]


=INFO REPORT==== 17-Nov-2016::12:42:59 ===
disk_log: repairing "/root/mnesia_database/DECISION_TAB.LOG" ...
Mnesia('scu_plus@REDACTED'): Data may be missing, Corrupt logfile deleted: "/root/mnesia_database/LATEST.LOG", {not_a_log_file,
                                                                                                                   "/root/mnesia_database/LATEST.LOG"}

=ERROR REPORT==== 17-Nov-2016::12:43:00 ===
Mnesia('scu_plus@REDACTED'): ** ERROR ** (core dumped to file:"/root/rel-3/galaxy/MnesiaCore.scu_plus@REDACTED")
  ** FATAL ** Failed to merge schema: Bad cookie in table definition configuration_table: 'scu_plus@REDACTED' = {cstruct,configuration_table,set,[],['scu_plus@REDACTED'],[],[],0,read_write,false,[],[],false,gen_record,[key,value,time_stamp],[],[],[],{{1479494951549325042,-134217623,1},'scu_plus@REDACTED'},{{2,0},[]}},'scu_plus@REDACTED' = {cstruct,configuration_table,set,[],['scu_plus@REDACTED'],[],[],0,read_write,false,[],[],false,gen_record,[key,value,time_stamp],[],[],[],{{1479575314787900797,-134217651,1},'scu_plus@REDACTED'},{{2,0},[]}}


=ERROR REPORT==== 17-Nov-2016::12:43:10 ===
** Generic server mnesia_monitor terminating
** Last message in was {'EXIT',<0.411.0>,killed}
** When Server state == {state,<0.411.0>,[],[],true,[],undefined,[],[]}
** Reason for termination ==
** killed

=ERROR REPORT==== 17-Nov-2016::12:43:10 ===
** Generic server mnesia_recover terminating
** Last message in was {'EXIT',<0.411.0>,killed}
** When Server state == {state,<0.411.0>,undefined,undefined,undefined,0,
                                false,true,[]}
** Reason for termination ==
** killed

=ERROR REPORT==== 17-Nov-2016::12:43:10 ===
** Generic server mnesia_snmp_sup terminating
** Last message in was {'EXIT',<0.411.0>,killed}
** When Server state == {state,
                             {local,mnesia_snmp_sup},
                             simple_one_for_one,
                             [{child,undefined,mnesia_snmp_sup,
                                  {mnesia_snmp_hook,start,[]},
                                  transient,3000,worker,
                                  [mnesia_snmp_sup,mnesia_snmp_hook,
                                   supervisor]}],
                             undefined,0,86400000,[],0,mnesia_snmp_sup,[]}
** Reason for termination ==
** killed

=ERROR REPORT==== 17-Nov-2016::12:43:10 ===
** Generic server mnesia_subscr terminating
** Last message in was {'EXIT',<0.411.0>,killed}
** When Server state == {state,<0.411.0>,12307}
** Reason for termination ==
** {{aborted,{node_not_running,'scu_plus@REDACTED'}},
     [{mnesia,abort,1,[{file,"mnesia.erl"},{line,318}]},
      {mnesia_lib,del,2,[{file,"mnesia_lib.erl"},{line,485}]},
      {mnesia_subscr,do_handle_exit,1,[{file,"mnesia_subscr.erl"},{line,508}]},
      {mnesia_subscr,handle_exit,2,[{file,"mnesia_subscr.erl"},{line,502}]},
      {mnesia_subscr,do_prepare_stop,2,[{file,"mnesia_subscr.erl"},{line,526}]},
      {mnesia_subscr,terminate,2,[{file,"mnesia_subscr.erl"},{line,304}]},
      {gen_server,try_terminate,3,[{file,"gen_server.erl"},{line,629}]},
      {gen_server,terminate,7,[{file,"gen_server.erl"},{line,795}]}]}

=ERROR REPORT==== 17-Nov-2016::12:43:10 ===
Mnesia('scu_plus@REDACTED'): ** ERROR ** mnesia_event got unexpected event: {'EXIT',
                                                                                <0.413.0>,
                                                                                {aborted,
                                                                                 {node_not_running,
                                                                                  'scu_plus@REDACTED'}}}

=INFO REPORT==== 17-Nov-2016::12:43:10 ===
     application: mnesia
     exited: {{shutdown,{failed_to_start_child,mnesia_kernel_sup,killed}},
              {mnesia_sup,start,[normal,[]]}}
     type: permanent
{"Kernel pid terminated",application_controller,"{application_start_failure,mnesia,{{shutdown,{failed_to_start_child,mnesia_kernel_sup,killed}},{mnesia_sup,start,[normal,[]]}}}"}

Crash dump is being written to: erl_crash.dump...


Thanks in advance,
Arun
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20161214/4c0380a1/attachment.htm>