[erlang-questions] Erlang http servers

[Loïc Hoguin]

On 10/01/2012 04:30 PM, Kenneth Lundin wrote:
> I agree with Serge that it is somewhat unfortunate that there are so
> many initiatives
> where people write their own implementation of something we already
> have included in
> the Erlang/OTP distribution and doing this without even suggesting or
> asking us if we could
> incorporate some missing feature or improve the current implementation
> in various ways.

It happens a lot. People tend to review available options, and if none 
fits their needs, write something themselves without consulting any of 
the authors. Sometimes they're right to do so, because the answer 
they'll get is obvious, and sometimes not.

There's a few reasons this happens for this specific case:

* HTTP is simple, it's easier to go ahead and experiment on a new 
project than on an existing one.
* If the experiment succeeds and the implementation has to differ 
significantly from other projects, it's not worth backporting since 
you'll just run into a wall of backward compatibility requirements.
* OTP has a long release process. Many companies need their fixes to be 
included quickly to avoid maintaining huge branches for their changes. 
They generally agree in return to keep track of changes that might 
happen during the product's development.
* Inets is seriously well hidden. I only heard about it after releasing 
Cowboy. I knew about Yaws/Mochiweb/Misultin but not the inets HTTP 
server. Many people I talked to didn't seem to know it existed.
* Sometimes removing features is an improvement. Inets is big, it's only 
natural that lightweight alternatives appear. Same happened to Apache, 
and it's no surprise that Inets is quite similar to Apache from a user 
point of view.

Note that while not everyone uses inets, most servers do use 
erlang:decode_packet/3 which does most of the job. Cowboy doesn't 
anymore, because it needed backward incompatible changes to 
erlang:decode_packet/3. I didn't ask your thoughts, simply because the 
experiment I did hinted that doing the parsing in full Erlang was a 
better idea: this change results in improved HiPE optimizations and 
better reduction management which results in better latency under load.

> In Erlang/OTP we have keep strict compatibility with many of our APIs
> which is both a strength and sometimes a burden (when we are not happy
> with the old API) but there are mostly ways around that if there is
> strong case for a change.

It's a big strength for people looking for stability and a big burden 
for people looking for reactivity. Can't really blame the second group 
for finding solutions to their problems.

> We still actively develop inets with its http client and server and I
> am not sure that
> all the reasons to develop alternatives still exist.

But like you said the level of support for applications included in OTP 
vary widely. It is also not clearly indicated which parts are the focus 
of recent works (there is a small roadmap for the most important stuff 
but that's about it), nor can it be deduced from reading tickets in a 
bug tracker or similar collaboration tool.

Which brings me to a very important point: please start using a public 
bug tracker! It doesn't have to be 100% public (as I'm sure you have 
some issues that are internal only) but it would help a lot if people 
could see the existing issues instead of stumbling upon the same bugs 
again and again. It probably also would help with contributions.

> I think it is of great value to have a relative complete functionality
> in Erlang/OTP so that
> the user can implement basic functionality without having to search
> for separate implementations of, with unknown quality and completeness
> at least for a beginner.

I would prefer Erlang/OTP to focus on the core features and leave the 
satellite features to third parties. OTP has an HTTP server but doesn't 
have proper Unicode support. OTP has OpenGL but doesn't have proper 
timezones support. These 2 examples missing from OTP are some of many 
things that are too complex for most people to implement, and that would 
not benefit from having alternatives.

Let's take another example. While taking a quick glance at the modules 
included in OTP I discovered there is a GD implementation in the percept 
application. I'm pretty sure most people didn't know it was there, as 
they usually end up using a NIF. It's quite unexpected that it's in OTP.

Finding things in OTP unexpectedly is a quite common occurence, though.

If the user had a package management application (perhaps working 
similarly to rebar's deps, = per project/release, no site-wide install 
by default) along with a package index website, it could quite easily 
find what he needs. It wouldn't be of unknown quality or completeness, 
simply because adding a comment and rating system to a package index 
website is fairly simple to do.

Plus, he can make an opinion for himself, instead of just using whatever 
someone decided for him. You might include a project in OTP that fits 
your criteria for quality, but it might be crap for someone else. A more 
democratic system where people choose what they use based on other 
users' feedback and ratings is not only a lot more powerful but also 
allows to bring some light on some more obscure projects that are 
sometimes quite hard to find armed only with Google.

It makes sense to have a single choice for big corporate environments, 
not so much for the rest of the world.

-- 
Loïc Hoguin
Erlang Cowboy
Nine Nines
http://ninenines.eu