[erlang-questions] inets HTTP client does not persist connections when POSTing

[Leif Hedstrom]

Edwin Fine wrote:
> I read the inets code, and if I understand it correctly, it seems that 
> when a non-idempotent operation such as POST is performed, httpc does 
> not reuse the existing connection (httpc_manager:select_session/4 
> returns no_connection, which starts a new handler). I am not sure if 
> this is the correct thing to do, because my understanding is that the 
> restriction on non-idempotent operations applies to pipelining, not 
> persistent connections. I can see no reason why one should not be able 
> to do a series of consecutive POSTs on the same socket, as long as one 
> waits for a reply before posting the next request.

I don't think this is extremely unreasonable, it's basically a defense 
mechanism to avoid issues with the non-idempotent request putting the 
server/connection in an undesirable state. The reasoning generally seems 
to be that a "fresh" connection to the server is less likely to be 
terminated prematurely (or arbitrarily). Also remember a server always 
has the right to close a KA connection (particularly after a 
non-idempotent request).

One way to improve on this could be to use the "100-continue" header 
(Expect: 100-continue) before sending a POST body on a Keep-Alive 
connection. This obviously introduces more latency, which can negate all 
benefits of Keep-Alive (or even make it worse). On high latency 
networks, with many of POSTs on a KA connection, it's probably 
beneficial even with the overhead of "100-continue" (due to 3-way 
handshake + TCP congestion control on new connections).

Now, I do agree that it seems a little draconian that the library 
doesn't let the user (developer) make these decisions. I haven't looked 
at the code at all, but assuming this is what it does,  I'm guessing it 
was done to make sure people don't fall into this pit of nastiness 
accidentally. It's a reasonable default for sure, but ought to be 
possible to override,  by someone well familiar with the dangers (and 
how to handle them).

Cheers,

-- Leif