[erlang-bugs] Segfault in erl_interface attempting to decode certain large binaries

[Serge Aleynikov]

You can use the following tutorial to help you started with ei:
http://www.trapexit.org/How_to_use_ei_to_marshal_binary_terms_in_port_programs

I haven't seen any newer erl_encode/erl_decode implementations.  Frankly 
I've started doing that myself in C++ and got the recursive encoding 
part, but haven't done the decoder and had to put this project on hold 
due to other priorities.

Serge

Jonathan Gray wrote:
> Serge,
> 
> Sorry I was not more clear.  This is a port program with two remote systems
> communicating via simple TCP packets (size header, buffer).
> 
> Thanks for the advice.  The documentation suggests that erl_interface is
> actually making use of the ei code.  I'm not sure if that's correct because
> I see other resources on the web that say they are different (ei is new,
> erl_interface is old).  However the error does exist in a malloc() call so
> perhaps that can be avoided doing the term decodes myself?
> 
> Unfortunately what I need (already have working but segfaulting on a large
> decode) is a generic parser that converts ErlBinary into a number of
> different formats (JSON, Python, C struct tree) and vice versa.  The ETERM
> representation gave me an easy way to make recursive converters.  Recreating
> this generic behavior using ei seems like I'll be recreating functions of
> erl_interface and losing the ETERM representation so I'll need to rewrite my
> converters in a completely different way. 
> 
> Rewriting it to go directly from binary would be a good thing in the long
> run, just not something I was planning on doing at this stage.  Has anyone
> ever written a newer generic erl_encode/erl_decode using the latest ei?
> 
> I will take a closer look at ei and report back.
> 
> Thanks for your help.
> 
> Jonathan
> 
> 
> -----Original Message-----
> From: erlang-bugs-bounces@REDACTED [mailto:erlang-bugs-bounces@REDACTED]
> On Behalf Of Serge Aleynikov
> Sent: Wednesday, July 23, 2008 3:52 AM
> To: jlist@REDACTED
> Cc: erlang-bugs@REDACTED
> Subject: Re: [erlang-bugs] Segfault in erl_interface attempting to decode
> certain large binaries
> 
> Jonathan,
> 
> It's not clear from your description if you're using erl_interface to 
> build a driver or a port program.  While I don't have an answer to your 
> direct question, perhaps if you are writing a C port program you can use 
> ei instead of erl_interface, and if you are writing a driver, you can 
> use driver_output_term() / driver_send_term() and corresponding 
> ErlDrvTermData* structures to pass data to/from the emulator (*). This 
> is the fastest way to communicate with the emulator's port owner process.
> 
> (*) See: http://www.erlang.org/doc/man/erl_driver.html
> and also the source code of inet_drv.c in the distribution for various 
> LOAD_*() macros that simplify working with ErlDrvTermData structures.
> 
> Serge
> 
> 
> 
> 
> jlist@REDACTED wrote:
>> All,
>>
>> I have a TCP interface between an Erlang system and a C system.  Both
>> send/receive marshaled binary Erlang terms and I have not had any problems
>> to date.
>>
>> Today I began doing some more serious testing with larger chunks of binary
>> to be decoded in C.
>>
>> We ran into a bug (it seems) with erl_interface 3.5.5.4 that is causing it
>> to segfault during decoding.  The backtrace looks like this:
>>
>> Program received signal SIGSEGV, Segmentation fault.
>> [Switching to Thread 46912496233216 (LWP 4091)]
>> 0x00000000004032c4 in _erl_free_term ()
>> (gdb) bt
>> #0  0x00000000004032c4 in _erl_free_term ()
>> #1  0x000000000040496b in erl_decode_it ()
>> #2  0x0000000000404937 in erl_decode_it ()
>> #3  0x0000000000404c93 in erl_decode_it ()
>> #4  0x0000000000404c93 in erl_decode_it ()
>> #5  0x0000000000405311 in erl_decode ()
>> #6  0x0000000000401b38 in main (argc=1, argv=0x7fff51ec7938) at
> badtest.c:28
>> The unfortunate part is that the way this large binary term is generated
>> cannot be done in any kind of sample code (it's being pulled off an
> external
>> database).
>>
>> Testing code:  http://jgray.la/erlang/erl_decode_segfault_test.tar.gz
>>
>> However, I have created a set of test files in C which recreate the
>> segfault.  I stored the binary in a flat file (as 'badbinary') and have a
>> testing program which reads it off disk and attempts to decode it.  To
> prove
>> the approach is sane (and that this segfault is related to something
> strange
>> about the decoding of this particular binary, not the size or general
> format
>> of the binary) there is a 'goodbinary' file and testing program for that.
>>
>> To use the test code:
>>
>> Untar/Ungzip the file.  You may need to edit the Makefile to fix the paths
>> to your erl_interface library.
>>
>> 'make' and then you can:
>>
>> ./badtest  (this reads 'badbinary' and attempts to decode, causes
> segfault)
>> ./goodtest (this reads 'goodbinary' and successfully decodes it)  [nearly
>> identical code to badtest.c but reads different file w/ different size]
>>
>> Also included is
>>
>> ./makegoodbin (a simple program that generates a large ETERM in an
> identical
>> format to the badbinary but contains duplicated binary data everywhere) 
>>
>>
>> Notes:
>>
>> * The marshaled binary erlang term being sent to C can be successfully
>> decoded/unmarshaled from within Erlang without a problem
>> * This is reproducible with many different large erlang terms generated
> from
>> our database queries.  'makegoodbin.c' creates a term identical in format
> to
>> those causing problems, however it does not have the random distribution
> of
>> binary sizes and content, and so I'm not able to reproduce the problem in
>> this way.
>> * The entire system, end-to-end including this decoding step, works
>> perfectly in most cases.  However when the data goes into the 100k+ range,
>> the segfaults start to happen.  That's why I created the 'makegoodbin'
> which
>> follows the same format.  Unfortunately that works even at sizes of >1MB
>> adding to the confusion of the problem.
>>
>>
>> Any help is appreciated.  Thanks.
>>
>> I apologize if this is a repost.  I never saw my original post hit the
> list
>> and did not receive any responses.
>>
>> Jonathan Gray
>> Streamy Inc.
>>
>> _______________________________________________
>> erlang-bugs mailing list
>> erlang-bugs@REDACTED
>> http://www.erlang.org/mailman/listinfo/erlang-bugs
>>
> 
> _______________________________________________
> erlang-bugs mailing list
> erlang-bugs@REDACTED
> http://www.erlang.org/mailman/listinfo/erlang-bugs
> 
>