Erlang Event Loop Question

[Evans, Matthew]

I'm hoping that this got missed because I sent it on a Friday afternoon.

If someone knows how to benchmark this it'll be great (I tried a simple echo - server type application, but that'll include network slowness too)...

________________________________
From: Evans, Matthew
Sent: Friday, March 19, 2010 10:50 AM
To: erlang-questions@REDACTED
Subject: Erlang Event Loop Question

Hi All,

I'm wondering about the inner workings of the Erlang event loop. Is the event loop single threaded?

We have a service (the VM has epoll enabled) that is doing a lot of TCP (HTTP) ingesting and parsing of large files (25MB of data or more per second). It is also doing a lot of IPC, both to other Erlang nodes, and via a proprietary transport to C++ applications. We currently support two methods to ingest these files: straight TCP (gen_tcp), and a specialized linked-in driver (which passes the epoll socket to the Erlang event loop with driver_select and passes the driver-created socket to gen_tcp; we also have a pool of threads for this driver).

My concern is that under load the event loop is getting very busy resulting in time-sensitive IPC messages getting delayed/slowed down.

I could get around this by having the linked-in driver not do driver_select and instead read from the socket and do driver_output_binary to pass the data to an Erlang process when it has accumulated a large enough buffer (say 100K). I would however like to get an understanding of the event loop before we try this, and see if there are any ways we can measure its performance to see if it is indeed a bottleneck.

Anyone have any suggestions?

Regards

Matt