Any pending improvements to async thread pool?
Scott Lystig Fritchie
Fri May 16 22:17:48 CEST 2003
Hi there. I've a question for the OTP team: are there any
improvements to the async thread pool that might be appearing in a
future Erlang release?
Last night I was bitten by a "feature" that I've known about for ages
but had forgotten about.(*) If I have a driver D that utilizes the
async thread pool, and if that driver gives an async thread a task T
that takes many seconds to finish, then it's quite possible to block
task T' by another driver (also using the async thread pool) until
task T has finished.
As an example, say I've got a driver that executes the C library's
sleep(200) call using an async thread.(**) Then, I try launching the
debugger (or "pman" or "appmon" or whatever). The application won't
run for 200 seconds...
... because the "efile_drv" driver is also using the async thread
pool. The task of operating on one of the beam files is assigned by
efile_drv to the same thread that's executing the sleep(200). The
result is a lot of wasted time.
The current scheme for mapping tasks to a particular thread is, IMHO,
broken. A short-term solution is to manage my own async worker
thread(s) rather than using the async thread pool already provided.
However, given that "efile_drv" can end up blocking *itself* by trying
to operate on slow media (CD-ROM, NFS, Flash-based storage, or even a
really busy local on-disk file system(***)), I think it ought to be
fixed. If there are other idle threads available, they should be
(*) I mentioned this in my Erlang Driver Toolkit paper: section 7.3,
(**) That's the first driver using async threads I wrote. :-)
However, any computation that takes a long time is sufficient.
(***) Back when I was running USENET News servers, I had file systems
so busy that a single open(), link, or unlink() system call sometimes
took 3/4 of a second. Each. I don't recommend workloads that high.
More information about the erlang-questions