[erlang-questions] Mnesia - Partitioned network problem

[Valentin Micic]

Q1: This may sound wired, but this is not necessarily a networking problem!

A couple of years ago, I've been conducting some tests using two machines connected "back-to-back" (*), thus limiting the network to two directly connected nodes. My mnesia environment consisted of two highly fragmented disk-only tables (256 and 768 dets files) hosted by two nodes, having only one interacting with a "traffic source" (let's call it ACTIVE), whilst using the second for replication only. Then, I've exposed the ACTIVE mnesia node to a continuous traffic, using a great deal of inserts, pushing the rate as high as 600 inserts per second (Before I forget -- I've used a huge data set, say, more than 300 million records)

After running smoothly for a while (that is, until the general population per dets file reached approx 200,000 records), I've noticed the ACTIVE mnesia node warned about the network partitioning, although there was no reason to do so (e.g. cable was still connected ;-)). After a considerable fact finding effort, I've concluded that it was the disk activity (that caused CPU to go into the uninterruptible sleep state waiting for disk I/O to complete) causing the erts to slow-down, hence build-up process message queue, which in turn caused mnesia to report network partitioning -- not because response hasn't been received from the peer node, but because the response has not been processed on time. Since the remote node did not have the same issue with CPU load, it was able to "see" the peer mnesia node.

Indeed, I've been able to fix the issue (at least short term), by increasing the file I/O thread pool (+A flag).

Q2: see above :-)

Kind regards

V/

(*) that is, by connecting two machines using a single cable, thus avoiding any additional points of failure.



On 05 Sep 2011, at 11:48 AM, Erik Seres wrote:

> Hello,
> 
> I am running distributed mnesia on a cluster of two servers, x@REDACTED and x@REDACTED All interaction with mnesia occurs on x@REDACTED while x@REDACTED just replicates from x@REDACTED
> 
> Due to an intermittent network failure, replication had stopped. In the logs for x@REDACTED was the following message:
> 
> =ERROR REPORT==== 23-Aug-2011::23:56:37 ===
> ** Node 'x@REDACTED' not responding **
> ** Removing (timedout) connection **
> 
> This message went unnoticed until x@REDACTED was restarted for an unrelated reason and it x@REDACTED logged the following error at startup:
> 
> =ERROR REPORT==== 24-Aug-2011::14:22:09 ===
> Mnesia('x@REDACTED'): ** ERROR ** mnesia_event got {inconsistent_database, running_partitioned_network, 'x@REDACTED'}
> 
> 
> Question 1: How is it possible that only x@REDACTED detected the disconnect from x@REDACTED and x@REDACTED did not detect a disconnect from x@REDACTED?
> Question 2: How is it possible that x@REDACTED did not log the running_paritioned_network error?
> 
> I realize that the answer to Q1 may lead to an answer to Q2, as well.
> 
> Note, that neither server was set as master.
> 
> Thank you.
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions