[erlang-questions] Streaming Data using httpc

[John Duffy]

Fred, Antoine

Thank you for your replies. I will investigate further...

Kind regards

John


----Original message----
>From : mononcqc@REDACTED
Date : 21/04/2015 - 16:41 (GMTDT)
To : antoine.koener@REDACTED
Cc : jb_duffy@REDACTED, erlang-questions@REDACTED
Subject : Re: [erlang-questions] Streaming Data using httpc

On 04/21, Antoine Koener wrote:
>
>The problem seems to be the server because if you use -v for curl you will observe something like this:
>
> [...]
>
>As you can see there's some strange headers:
>Connection: close
>The connection is not closed because it's a stream.

The connection:close header is used to say that once the request is 
done, and to avoid attempting to reuse it as keep-alive for a follow-up 
request. There's nothing bad about it, and it is not relevant to the 
fact you are streaming. The streaming is part of chunked encoding (which 
has no content length known ahead of time, and is self-delimiting), 
related to the body, not the connection.

>
>Transfer-Encoding: chunked
>What I see is absolutely not chunked transfer, it's a bunch of json lines...
>Chunks should be preceded by the size (hex encoded) and \r\n
>

That's because you need to use `--raw` in curl to avoid seeing the 
content body already decoded. `-v` will decode chunked content and 
display it as it comes. You can also try a tcpdump to see the raw data.

>It might be interesting to report those problems to the developers of 
>this service :)
>

Don't. Nothing you specified there is actually a problem unless they 
advertise keep-alive connections forever.