[erlang-questions] Streaming Data using httpc

[John Duffy]

Fred

By the way... a great book! It is constantly open on my desk at the moment as I try to get to grips with Erlang.

Kind regards

John Duffy


----Original message----
>From : mononcqc@REDACTED
Date : 21/04/2015 - 16:41 (GMTDT)
To : antoine.koener@REDACTED
Cc : jb_duffy@REDACTED, erlang-questions@REDACTED
Subject : Re: [erlang-questions] Streaming Data using httpc

On 04/21, Antoine Koener wrote:
>
>The problem seems to be the server because if you use -v for curl you will observe something like this:
>
> [...]
>
>As you can see there's some strange headers:
>Connection: close
>The connection is not closed because it's a stream.

The connection:close header is used to say that once the request is 
done, and to avoid attempting to reuse it as keep-alive for a follow-up 
request. There's nothing bad about it, and it is not relevant to the 
fact you are streaming. The streaming is part of chunked encoding (which 
has no content length known ahead of time, and is self-delimiting), 
related to the body, not the connection.

>
>Transfer-Encoding: chunked
>What I see is absolutely not chunked transfer, it's a bunch of json lines...
>Chunks should be preceded by the size (hex encoded) and \r\n
>

That's because you need to use `--raw` in curl to avoid seeing the 
content body already decoded. `-v` will decode chunked content and 
display it as it comes. You can also try a tcpdump to see the raw data.

>It might be interesting to report those problems to the developers of 
>this service :)
>

Don't. Nothing you specified there is actually a problem unless they 
advertise keep-alive connections forever.