[erlang-questions] zlib problems in R14B03 when processing gzip encoded HTTP responses

[Shaun Kruger]

I have an  HTTP proxy server that I have added gzip support to.  I am sending the header "Accept-Encoding: gzip", but what comes back doesn't always unzip with zlib.  I learned early on that I need to receive the whole gzip block before I can unzip using zlib:gunzip() so I always ask the server for mode data each time I can't unzip the block.  My problem at the moment is that there are some sites where I can never unzip the gzip encoded data while there are others where I can.

I am not entirely sure from the documentation which of the zlib decompress functions I should be using.  However, I have determined that when I have this problem none of them will work to decode the data.  I am experiencing this problem only on certain sites.  I have enabled logging so I am aware when gzip is working and when it is not.  There are some sites where the main page gzip works, but other pages do not.

I have called the three main unzip, uncompress, and gunzip functions in the zlib module and all of them fail the same way.

I am wondering if anyone can suggest something else for me to try as I'm running out of leads.  I have to admit that I don't know compression as well as I know other things like HTTP so I may just be missing something basic here.

Read below to see my debugging output when I call the three zlib functions.  I don't know for sure if it will help, but it should help identify if I'm making any basic mistakes.

Shaun

?ERROR_MSG("unzip: ~p~n",[catch zlib:unzip(GZ)]),
?ERROR_MSG("uncompress: ~p~n",[catch zlib:uncompress(GZ)]),
?ERROR_MSG("gunzip: ~p~n",[catch zlib:gunzip(GZ)]),

=ERROR REPORT==== 26-Oct-2011::08:25:08 ===
unzip: {'EXIT',{data_error,[{zlib,call,3},
                            {zlib,inflate,2},
                            {zlib,unzip,1},
                            {proxy_pass,handle_info,3},
                            {gen_fsm,handle_msg,8},
                            {proc_lib,init_p_do_apply,3}]}}

=ERROR REPORT==== 26-Oct-2011::08:25:08 ===
uncompress: {'EXIT',{data_error,[{zlib,call,3},
                                 {zlib,inflate,2},
                                 {zlib,uncompress,1},
                                 {proxy_pass,handle_info,3},
                                 {gen_fsm,handle_msg,8},
                                 {proc_lib,init_p_do_apply,3}]}}

=ERROR REPORT==== 26-Oct-2011::08:25:08 ===
gunzip: {'EXIT',{data_error,[{zlib,call,3},
                             {zlib,gunzip,1},
                             {proxy_pass,handle_info,3},
                             {gen_fsm,handle_msg,8},
                             {proc_lib,init_p_do_apply,3}]}}