[erlang-questions] zlib design flaw?

[Tony Rogvall]

On 24 sep 2014, at 10:17, Robert Wilkinson <bob@REDACTED> wrote:

> On Wed, Sep 24, 2014 at 11:55:18AM +0900, Park, Sungjin wrote:
>> 
>> We recently had some malicious packets which were not very big in the first
>> place but inflated to really big ones - hundreds of megabytes each.  As a
>> result, the server crashed with out-of-memory by the processes calling
>> zlib:inflate/2.  Urgency forced us to make a custom NIF library with
>> inflation size limit.  We also studied erlang reference manual but couldn't
>> find anything useful.  The zlib library source code shows even
>> zlib:setBufSize/2 does not prevent producing very big binaries.
>> 
>> Not being able to know how big the data would become after inflation, it
>> should be a quite common problem.  So I'm curious if I missed something
>> very simple and nice.  Is there anything like that?
> 
> Hi Sungjin 
> 
> The articles referenced at http://en.wikipedia.org/wiki/Zip_bomb
> should give you some insight into the problem, in general.
> 

Thanks. Cool stuff :-)

The following is also a fun version ( I am not the only one to blame for api design faults :-)

> xmerl_scan:file("lol.xml").

where "lol.xml" contains:

<?xml version="1.0"?>
<!DOCTYPE lolz [
 <!ENTITY lol "lol">
 <!ELEMENT lolz (#PCDATA)>
 <!ENTITY lol1 "&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;">
 <!ENTITY lol2 "&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;">
 <!ENTITY lol3 "&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;">
 <!ENTITY lol4 "&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;">
 <!ENTITY lol5 "&lol4;&lol4;&lol4;&lol4;&lol4;&lol4;&lol4;&lol4;&lol4;&lol4;">
 <!ENTITY lol6 "&lol5;&lol5;&lol5;&lol5;&lol5;&lol5;&lol5;&lol5;&lol5;&lol5;">
 <!ENTITY lol7 "&lol6;&lol6;&lol6;&lol6;&lol6;&lol6;&lol6;&lol6;&lol6;&lol6;">
 <!ENTITY lol8 "&lol7;&lol7;&lol7;&lol7;&lol7;&lol7;&lol7;&lol7;&lol7;&lol7;">
 <!ENTITY lol9 "&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;">
]>
<lolz>&lol9;</lolz>


The following code could be used to try out various zlib chunk sizes. 


inflate(Date) ->
    inflate(Date,1024).

inflate(Data,ChunkSize) when is_binary(Data) ->
    Z = zlib:open(),
    ok = zlib:inflateInit(Z),
    inflate_(Z, Data, ChunkSize, 0, []).

inflate_(Z, Data, ChunkSize, InflatedSize, Acc) ->
    case Data of
        <<Chunk:ChunkSize/binary,Tail/binary>> ->
            List = zlib:inflate(Z, Chunk),
            Size = erlang:iolist_size(List),
            InflatedSize1 = InflatedSize + Size,
            io:format("sz = ~w, inflated size = ~w\n", [Size, InflatedSize1]),
            inflate_(Z, Tail, ChunkSize, InflatedSize1, [List | Acc]);
        <<LastChunk/binary>> ->
            List = zlib:inflate(Z, LastChunk),
            zlib:inflateEnd(Z),
            zlib:close(Z),
            Size = erlang:iolist_size(List),
            InflatedSize1 = InflatedSize + Size,
            io:format("sz = ~w, inflated size = ~w\n", [Size, InflatedSize1]),
            list_to_binary(lists:reverse([List|Acc]))
    end.

/Tony


> Bob
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions

"Installing applications can lead to corruption over time. Applications gradually write over each other's libraries, partial upgrades occur, user and system errors happen, and minute changes may be unnoticeable and difficult to fix"