[erlang-bugs] Unloading then reloading crypto.so causes erlang core dump on Solaris 11

[XinFeng Liu]

This I don't understand at all.

> reload() in crypto.c should only be called if the module is upgraded, not
> if it is unloaded and later loaded again.
>
> /Sverker, Erlang/OTP
>
> Thanks for pointing this. I'm sorry I made a wrong conclusion for the
workaround in modifying reload() function. I wronly modfied reload() in
15B02.

I have some new findings on this issue:
I find the latest 15B02 (without any modification) does not cause core dump
in running couchDB test suite. Digging this issue further, there's a subtle
difference between 15B02 and 15B01 in crypto.so.
In 15B01, crypto.so explicitly link libssl.so, while in 15B02 it does not.

And more importantly, the libssl.so built by Sun/Oracle seems built with
"-z nodelete" meaning RTLD_NODELETE. ("elfdump -d" can show that).
In 15B01, loading crypto.so causes libssl.so to be loaded, since libssl.so
depends on libcrypto.so,  libcrypto.so is somehow promoted to RTLD_NODELETE
(using solaris runtime LD debugger can show this). So, libcrypto.so is
unloadable in dlclose().
In 15B02, when running couchDB test suite, unloading crypto.so causes
libcrypto.so unloaded too, then later reloading both crypto.so and
libcrypto.so would not cause previous problem.

A new question, each time loading crypto.so will cause load() to be called,
then it means CRYPTO_set_mem_functions() should be called again, I assume
it should correctly set the callback funcs. But from instruction-level
tracing and the src, it simply returned in line 129 because
"!allow_customize" is true.

(dbx) stepi
t@REDACTED (l@REDACTED) stopped in CRYPTO_set_mem_functions at 0xfd053b0c
0xfd053b0c: CRYPTO_set_mem_functions+0x0034:    retl

    125 int CRYPTO_set_mem_functions(void *(*m)(size_t), void *(*r)(void *,
size_t),
    126         void (*f)(void *))
    127         {
    128         if (!allow_customize)
    129                 return 0;
    130         if ((m == 0) || (r == 0) || (f == 0))
    131                 return 0;
    132         malloc_func=m; malloc_ex_func=default_malloc_ex;
    133         realloc_func=r; realloc_ex_func=default_realloc_ex;
    134         free_func=f;
    135         malloc_locked_func=m;
malloc_locked_ex_func=default_malloc_locked_ex;
    136         free_locked_func=f;
    137         return 1;
    138         }
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-bugs/attachments/20120907/5d41479f/attachment.htm>