[erlang-questions] Asynchronous operations slow down the system

[Zhongmiao Li]

Hi all,

I am experiencing a problem that in my program, executing some background operations seems to slow down other operations running in foreground….

I am implementing a transactional datastore on top of Erlang. A transaction reads bunches of data (read phase) and then enters a commit phase that write data to the data store. To provide fault tolerance, in the commit phase, written data is synchronously replicated to a few Erlang nodes, running in different physical machines. Client is notified after the replication is finished. 

Since replication is slow, I make replication asynchronous: client is notified as soon as the data is written to its local node and the replication process just continues running.
Ideally, this should improve throughput, as now the client doesn’t wait replication to finish and can execute more operations. However, I benchmarked with 1 thread as client and I don’t see difference in the throughput between synchronous mode and asynchronous mode.

After checking more carefully of the metrics, I found that the commit phase of asynchronous mode is faster than before, but somehow the read phase becomes slower. The following are the average executing time for read and commit in microsecond of these two modes:
Synchronous:  Read 8381 Commit 4529
Asynchronous: Read  10874 Commit 1287

Machines are not overloaded yet, because each machine has 2 CPUs and the CPU utilization for datastore is only about 75%. And read and replication is totally decoupled: client only accesses a data storage fsm to read; replication is handled by individual replication fsms, so data storage fsm won’t be bothered (and possibly slowed down) during replication. I can only guess that it might be related with scheduling or asynchronous threads, but I couldn’t solve it so far….

Any suggestions and help are really appreciated! Thank you in advance!

Thank you,
Li


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20160104/42046e8b/attachment.htm>