[erlang-questions] Mnesia inconsistency with R11B-1

[Fredrik Thulin]

Hi

I upgraded from R10B-10 to R11B-1 about a week ago. Today I got a 
problem report that seems to be because I have inconsistencies in my 
Mnesia table 'phone'.

On node #1, a record is present that is not present on node #2 :

  (root@REDACTED)49> rpc:call('appserver@REDACTED', phone,  
     get_sipuser_locations, ["username"]).
  {ok,[{siplocationdb_e,{sipurl,"sip",
     ...
                        1159532776}]}

  (root@REDACTED)50> rpc:call('appserver@REDACTED', phone, 
     get_sipuser_locations, ["username"]).
  {ok,[]}
  (root@REDACTED)51>  

This table is written to from at least four different nodes, but always 
using transactions and never with dirty writes. What should I do to 
figure out the cause of this? I will have to restart the nodes at the 
latest before Monday morning to get everything working again, but if 
you guys are quick I can do diagnostics on the running system =).

It seems to me that there is no partitioned network (both nodes list the 
same nodes as 'running').

Another indicator of the differences, and also mnesia:info() output from 
the two nodes :

(root@REDACTED)55> length( rpc:call('appserver@REDACTED', db_util, 
tab_to_list, [phone]) ).
63
(root@REDACTED)56> length( rpc:call('appserver@REDACTED', db_util, 
tab_to_list, [phone]) ).
58
(root@REDACTED)57> rpc:call('appserver@REDACTED', mnesia, info, []).
---> Processes holding locks <---
---> Processes waiting for locks <---
---> Participant transactions <---
---> Coordinator transactions <---
---> Uncertain transactions <---
---> Active tables <---
schema         : with 10       records occupying 1569     words of mem
call           : with 0        records occupying 279      words of mem
===> System info in version "4.3.2", debug level = none <===
opt_disc. Directory "/var/yxa/mnesia/appserver" is NOT used.
use fallback at restart = false
running db nodes   = 
['root@REDACTED','eventserver@REDACTED','incomingproxy@REDACTED','pstnproxy@REDACTED','appserver@REDACTED','outgoingproxy@REDACTED','outgoingproxy@REDACTED','incomingproxy@REDACTED','eventserver@REDACTED','pstnproxy@REDACTED','appserver@REDACTED']
stopped db nodes   = ['root@REDACTED','pstnproxy@REDACTED']
master node tables = []
remote             = [cpl_script_graph,
                      eventdata,
                      forward,
                      gruu,
                      numbers,
                      phone,
                      regexproute,
                      user]
ram_copies         = [call,schema]
disc_copies        = []
disc_only_copies   = []
[{'appserver@REDACTED',ram_copies},
{'appserver@REDACTED',ram_copies}] = [call]
[{'appserver@REDACTED',ram_copies},
 {'appserver@REDACTED',ram_copies},
 {'eventserver@REDACTED',ram_copies},
 {'eventserver@REDACTED',ram_copies},
 {'incomingproxy@REDACTED',disc_copies},
 {'incomingproxy@REDACTED',disc_copies},
 {'outgoingproxy@REDACTED',ram_copies},
 {'outgoingproxy@REDACTED',ram_copies},
 {'pstnproxy@REDACTED',ram_copies},
 {'pstnproxy@REDACTED',ram_copies},
 {'root@REDACTED',ram_copies}] = [schema]
[{'eventserver@REDACTED',ram_copies},
 {'eventserver@REDACTED',ram_copies},
 {'incomingproxy@REDACTED',disc_copies},
 {'incomingproxy@REDACTED',disc_copies}] = [eventdata]
[{'incomingproxy@REDACTED',disc_copies},
 {'incomingproxy@REDACTED',disc_copies}] = [gruu,
                                              cpl_script_graph,
                                              regexproute,
                                              phone,
                                              forward,
                                              numbers,
                                              user]
93 transactions committed, 0 aborted, 9 restarted, 0 logged to disc
0 held locks, 0 in queue; 0 local transactions, 0 remote
0 transactions waits for other nodes: []
ok
(root@REDACTED)58> rpc:call('appserver@REDACTED', mnesia, info, []).
---> Processes holding locks <---
---> Processes waiting for locks <---
---> Participant transactions <---
---> Coordinator transactions <---
---> Uncertain transactions <---
---> Active tables <---
schema         : with 10       records occupying 1569     words of mem
call           : with 0        records occupying 279      words of mem
===> System info in version "4.3.2", debug level = none <===
opt_disc. Directory "/var/yxa/mnesia/appserver" is NOT used.
use fallback at restart = false
running db nodes   = 
['root@REDACTED','appserver@REDACTED','eventserver@REDACTED','outgoingproxy@REDACTED','pstnproxy@REDACTED','incomingproxy@REDACTED','eventserver@REDACTED','pstnproxy@REDACTED','incomingproxy@REDACTED','outgoingproxy@REDACTED','appserver@REDACTED']
stopped db nodes   = ['root@REDACTED','pstnproxy@REDACTED']
master node tables = []
remote             = [cpl_script_graph,
                      eventdata,
                      forward,
                      gruu,
                      numbers,
                      phone,
                      regexproute,
                      user]
ram_copies         = [call,schema]
disc_copies        = []
disc_only_copies   = []
[{'appserver@REDACTED',ram_copies},
{'appserver@REDACTED',ram_copies}] = [call]
[{'appserver@REDACTED',ram_copies},
 {'appserver@REDACTED',ram_copies},
 {'eventserver@REDACTED',ram_copies},
 {'eventserver@REDACTED',ram_copies},
 {'incomingproxy@REDACTED',disc_copies},
 {'incomingproxy@REDACTED',disc_copies},
 {'outgoingproxy@REDACTED',ram_copies},
 {'outgoingproxy@REDACTED',ram_copies},
 {'pstnproxy@REDACTED',ram_copies},
 {'pstnproxy@REDACTED',ram_copies},
 {'root@REDACTED',ram_copies}] = [schema]
[{'eventserver@REDACTED',ram_copies},
 {'eventserver@REDACTED',ram_copies},
 {'incomingproxy@REDACTED',disc_copies},
 {'incomingproxy@REDACTED',disc_copies}] = [eventdata]
[{'incomingproxy@REDACTED',disc_copies},
 {'incomingproxy@REDACTED',disc_copies}] = [gruu,
                                              cpl_script_graph,
                                              regexproute,
                                              phone,
                                              forward,
                                              numbers,
                                              user]
759 transactions committed, 0 aborted, 0 restarted, 0 logged to disc
0 held locks, 0 in queue; 0 local transactions, 0 remote
0 transactions waits for other nodes: []
ok
(root@REDACTED)59>                          

/Fredrik