[erlang-questions] Amazon S3 / streaming HTTP upload

[Chandru]

2010/1/19 Anton Krasovsky <anton.krasovsky@REDACTED>

> I'm looking for a way to upload/download largish binary files
> (photos/videos) to Amazon S3.
>
> Thanks to streaming option in inets http client, downloading these
> files seems pretty straitforward.
> However uploading to S3 is more problematic because streaming is not
> supported for put/post requests
> and entire file has to be read into memory before I can post it.
>
> Would anyone know about http client/Amazon S3 library that would work
> without buffering of the entire file?
>
>
ibrowse supports streaming for requests by taking a fun as input for the
Body.

ibrowse:send_req(Url, Headers, Method, Body).

An example invocation would be:

-record(stream_state, {iod, filename}).

stream_file(#stream_state{iod = undefined, filename = Filename} = State) ->
    {ok, Iod} = file:open(Filename, [read, raw]),
    stream_file(State#stream_state{iod = Iod});
stream_file(#stream_state{iod = Iod}) ->
    case file:read(Iod, 8192) of
        {ok, Data} ->
            {ok, Data, State};
        eof ->
             file:close(Iod),
             eof
    end.

Body = {fun stream_file/1, #stream_state{filename = "/file/to/stream"}},
ibrowse:send_req("http://blah/blah", [], put, Body).

cheers
Chandru