[erlang-questions] Erlang Web Servers challenge

[Jesper Louis Andersen]

On Wed, Jul 6, 2011 at 13:42, Zabrane Mickael <zabrane3@REDACTED> wrote:
> What are you trying to show with this benchmark? That is, what is the
> goal of it?
>
> At the end of the day, I'll try to compare Erlang Web servers with others
> competitors (not written in Erlang).

That was not the answer I had in mind when phrasing the question. I'll
try to be more precise: What are you going to measure? If you are
going to count requests, then the problem is that a 100 bytes transfer
is way less than the size of the average HTTP header. In other words,
you will be measuring who cheats and does not provide full headers as
per the HTTP spec. If you are going to measure throughput, on a single
machine, then 'ab' and whatever web server you have will fight for the
resources and thus show nothing at all.

What I am after is why you think your benchmark will give you any
indication of the goal you had in mind. What is performance to you? I
am interested in your answer because it is important to me if I want
to use the results for anything. It may be flawed or not, that is not
the concern for me. I am interested in the reason for why you laid out
the rules of the game as your did.

I disagree that you can see web servers as competitors to each other.
There is always a greater purpose behind writing a web server. From
the point where it is a personal exercise (can I do it?), to providing
a web server in your favorite language for ease of integration, to
writing a server because you want to earn money. It is not as much a
competition as it is a convenience of use. Or said otherwise, many of
the competitors are not even aware they are competing :)

> I'm aware of that rule (Steve Vinoski
> post: http://steve.vinoski.net/blog/2011/05/09/erlang-web-server-benchmarking/).
> Thanks anyway.

What are you going to do about it? Those who will have multiple CPUs
in their machine and pins 'ab' to one and gives Erlang free reign over
the others will definitely be at an advantage here. In fact, those who
disable SMP from the Erlang runtime might also be faster. If you ran
many tests and then used a bootstrapping technique, then you might be
able to tell if the localhost test generator tool interferes too much
with the results for instance. That is one way to circumvent that
problem.

> Why it's so hard to just play the game guys?

We are playing the game on another level. Our game is "The scientific
method and experimental testing". To be precise, I think your test has
way to many confounding factors to underpin your hypothesis that your
experiment can be used to show anything about web servers in
comparison to each other. When you design a simple experiment, it is
mandatory in our game to point out exactly what the confounding
factors are and what trouble there will be when drawing conclusions
from the test itself. I have pointed out some factors and Joe has
pointed out some different factors. That is all.

-- 
J.