Advantages of a large number of threads cf other approaches?
Tue Feb 17 15:40:53 CET 2004
Mickael Remond wrote:
> Thread can be created more quickly. However, I do not know to which
> extend the 2.6 thread are lighter than 2.4.
> If the memory taken for each thread is still important, this is a direct
> and quickly reached limitation to the number of thread you can manage.
AFAIK the overhead of a Unix process over a thread is stuff like the
command line, environment variables, and a table of file descriptors. I
don't know how the exact layout is, but this shouldn't amount to more
than a handful of KBytes.
That's still more than the CPU context that must be swapped to RAM when
switching threads, but not too much - modern processors typically take
several hundred bytes or more.
> You end up filling up the memory too quickly and soon start deterioating
I don't think so - if you take a server with 1 GB RAM, the process vs.
thread overhead will cut the number of processes you can serve from tens
of billions to billions (under the worst-case assumption that threads
don't use any local data on their own).
As soon as every thread allocates a KByte of memory, the memory overhead
diminishes to a factor of two, and it decreases further as each thread
uses more memory.
But even in the worst case, I suspect that the true bottleneck is the
CPU, not RAM.
The quoted IBM paper gave numbers for a concrete test run: a server with
9 GB of RAM and eight 700-MHz processors had a near-100% CPU usage, but
just 36% memory usage (when serving 623 pages per second).
More interesting is that server performance increased by a factor of
six (!). Given that Yaws performance was ahead of Apache by a factor of
roughly 2.5 (at least on the benchmarks posted by Joe), it would be very
interesting to see how much Yaws profits from the new Linux kernel.
Of course, since my server is a low-bandwidth machine, I'd be more
interested in the results of a uniprocessor benchmark :-)
Joe - would you or Ali be willing and able to publish the test
configuration&data so that others with equipment and time at hand could
repeat the tests?
Currently looking for a new job.
More information about the erlang-questions