[erlang-questions] zlib design flaw?

[Håkan Huss]

2014-09-24 10:26 GMT+02:00 Park, Sungjin <jinni.park@REDACTED>:

> Guess not.
> Personally, I the method 2 among my suggestion before is the most
> consistent and generic API design in this sense - streaming chunks in
> inflating too.
> Actually, zlib.erl is implemented in streaming chunks from a port driver
> to the calling process.  The problem is that the calling process collects
> all the chunks indefinitely and there is no way to intervene.
>
> Of course, you could feed zlib:inflate one small chunk of the input at a
time. Assuming the previously quoted bound of compression is correct,
taking say 1kB chunks would limit the inflated chunk size to ~1MB. If you
are really careful you can do one byte of input at a time.


> On Wed, Sep 24, 2014 at 5:09 PM, Richard A. O'Keefe <ok@REDACTED>
> wrote:
>
>> I'm a little bit puzzled here.
>> According to zlib.h, the underlying C code works with
>> input and output buffers.  Decompression is done in
>> chunks, and one reason for a chunk stopping is that the
>> output buffer has filled up.  I would expect a zlib
>> interface to say "give me the decompressed information
>> in chunks no bigger than this, as and when I ask for them".
>>
>> I suspect that the idea is to avoid getting "partially inflated" bytes.
Limiting the inflated size would require a more elaborate internal state in
zlib, and also more corner cases in the API.

Looking at RFC 1950 and RFC 1951, since reflecting on the fact
>> that you are supposed to be able to create zlib compressed
>> stuff by streaming blocks through the library without
>> announcing the total size in advance, it really doesn't look
>> as though there can be a way to get the size without
>> actually decompressing.
>>
>> Is there any possibility of adjusting the overall protocol to
>> transmit the size in advance?
>>
>> If the compressed data is actually zip or gzip, size information is
present in the metadata. However, a malicious user can fake this, as would
probably be the case with any solution transmitting the size. I guess you
could create a cryptographic protocol which guarantees that the transmitted
size is correct, but that may be overkill...

>
>
>
> --
> Park, Sungjin
>
> -------------------------------------------------------------------------------------------------------------------
> Peculiar travel suggestions are dancing lessons from god.
>   -- The Books of Bokonon
>
> -------------------------------------------------------------------------------------------------------------------
>
> /Håkan Huss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20140924/4727ceba/attachment.htm>