[erlang-questions] [ANN] Map/Reduce in Erlang and Python - Disco 0.1

[Bob Ippolito]

On Thu, Sep 11, 2008 at 11:00 AM, Ville H Tuulos
<ville.h.tuulos@REDACTED> wrote:
> ext Bob Ippolito wrote:
>>
>> Why the choice of SCGI? It seems like it would be a lot simpler not to
>> require an external web server. Erlang does HTTP on its own pretty
>> well, in my experience :)
>
> Thanks Bob for the comment (: I have used Mochiweb quite happily(*) in
> another Erlang project of mine. There are people already working on Disco to
> make it work with Mochiweb.
>
> The web server is mainly used to serve large files to workers; SCGI is just
> used to forward control requests to the Erlang process. Using an external
> web server for IO intensive jobs was a safe choice in the first place. I'd
> be happy to replace it with, say, Mochiweb, if it handle the load.
>
>
> Ville
>
> (*) I think I had a problem with Mochiweb: It uses the raw mode for sockets,
> and it seems that there's a 16M limit for gen_tcp:recv() in that case:
>
> http://www.erlang.org/pipermail/erlang-questions/2006-September/022907.html
>
> If I remember correctly, this caused any HTTP POST requests larger than 16M
> fail. Please correct me if I'm wrong.

Well sure, that just means that you shouldn't receive the entire POST
body in one shot from the socket directly. Take a look at the
mochiweb_multipart.erl code for example. Req:recv_body() has a size
limit of 1MB because it should only really be used as a convenience
when dealing with small bodies, otherwise you should be using some
kind of streaming API that brings the data in a chunk at a time.
Chunks obviously should not be bigger than 16MB, more like a few kb :)

-bob