[erlang-bugs] Mnesia/R15B: TYPE ASSERTION FAILED, erl_term.c line 109 (when stopping mnesia)

[Winston Smith]

On Tue, Apr 3, 2012 at 4:25 AM, Gustav Simonsson
<gustav.simonsson@REDACTED> wrote:
> A few ideas which might be of some use:
>
> Try starting your node with the mnesia "debug" or "trace" level debugging
> (http://www.erlang.org/doc/apps/mnesia/Mnesia_chap5.html#id76624). That might show
> at which part of the shutdown the crash occurs.

I did try this earlier today, I can see mnesia startup, but as soon as
I issue mnesia:stop(), it crashes before *any* further trace output:


# erts-5.9/bin/erl -mnesia debug trace
Eshell V5.9  (abort with ^G)
1> mnesia:start().
Mnesia(nonode@REDACTED): mnesia_monitor starting: <0.39.0>
Mnesia(nonode@REDACTED): Version: "4.6"
Mnesia(nonode@REDACTED): Env access_module: mnesia
Mnesia(nonode@REDACTED): Env auto_repair: true
Mnesia(nonode@REDACTED): Env backup_module: mnesia_backup
Mnesia(nonode@REDACTED): Env debug: trace
Mnesia(nonode@REDACTED): Env dir: "/home/avr32/mynode/Mnesia.nonode@REDACTED"
Mnesia(nonode@REDACTED): Env dump_log_load_regulation: false
Mnesia(nonode@REDACTED): Env dump_log_time_threshold: 180000
Mnesia(nonode@REDACTED): Env dump_log_update_in_place: true
Mnesia(nonode@REDACTED): Env dump_log_write_threshold: 1000
Mnesia(nonode@REDACTED): Env embedded_mnemosyne: false
Mnesia(nonode@REDACTED): Env event_module: mnesia_event
Mnesia(nonode@REDACTED): Env extra_db_nodes: []
Mnesia(nonode@REDACTED): Env ignore_fallback_at_startup: false
Mnesia(nonode@REDACTED): Env fallback_error_function: {mnesia,lkill}
Mnesia(nonode@REDACTED): Env max_wait_for_decision: infinity
Mnesia(nonode@REDACTED): Env schema_location: opt_disc
Mnesia(nonode@REDACTED): Env core_dir: false
Mnesia(nonode@REDACTED): Env pid_sort_order: false
Mnesia(nonode@REDACTED): Env no_table_loaders: 2
Mnesia(nonode@REDACTED): Env dc_dump_limit: 4
Mnesia(nonode@REDACTED): Env send_compressed: 0
Mnesia(nonode@REDACTED): Mnesia debug level set to trace
Mnesia(nonode@REDACTED): mnesia_subscr starting: <0.40.0>
Mnesia(nonode@REDACTED): mnesia_locker starting: <0.41.0>
Mnesia(nonode@REDACTED): mnesia_recover starting: <0.42.0>
Mnesia(nonode@REDACTED): mnesia_tm starting: <0.43.0>
Mnesia(nonode@REDACTED): Schema initiated from: default
Mnesia(nonode@REDACTED): mnesia_controller starting: <0.47.0>
Mnesia(nonode@REDACTED): mnesia_late_loader starting: <0.48.0>
Mnesia(nonode@REDACTED): Intend to load tables: []
Mnesia(nonode@REDACTED): Mnesia started, 2 seconds
ok
2> mnesia:stop().
Segmentation fault



I also ran strace(1) against the beam process while doing the
mnesia:stop(), interestingly, I see a reference to /etc/TZ and a call
to gettimeofday() (perhaps to prepare a timestamp for a log?) just
before it faults:

...
epoll_wait(3, {{EPOLLIN, {u32=0, u64=256}}}, 256, 0) = 1
clock_gettime(CLOCK_MONOTONIC, {6650, 485055333}) = 0
read(0, "mnesia:stop().\n"..., 65536)   = 15
clock_gettime(CLOCK_MONOTONIC, {6650, 499541961}) = 0
clock_gettime(CLOCK_MONOTONIC, {6650, 502889961}) = 0
clock_gettime(CLOCK_MONOTONIC, {6650, 506291561}) = 0
time(NULL)                              = 1333462397
open("/etc/TZ", O_RDONLY)               = 6
read(6, "EST5EDT\n"..., 68)             = 8
close(6)                                = 0
time(NULL)                              = 1333462397
open("/etc/TZ", O_RDONLY)               = 6
read(6, "EST5EDT\n"..., 68)             = 8
close(6)                                = 0
epoll_wait(3, {}, 256, 0)               = 0
clock_gettime(CLOCK_MONOTONIC, {6650, 526816132}) = 0
gettimeofday({1333462397, 301868}, NULL) = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
Process 621 detached


NOTE: I'm starting erl here with +K true and +A 5, although omitting
them makes no difference.



> When you create a table, you can give the compressed option in the storage_properties
> tuple, though I don't know of any low-memory options for mnesia itself.
>
> If you want to look at what the Erlang allocators have allocated you can use
> erlang:system_info({allocator, ets_alloc}). and then look at the blocks_size tuple
> which is on the format {blocks_size, CurrentSize, MaxSizeSinceLastAllocation, MaxSizeSinceEmulatorStarted}.
> The values are in bytes.

The system I'm using only has 32MB of RAM and I had thought that this
was perhaps a problem with OTP running out of memory.  However it
seems that my Erlang node fits quite nicely into about 6MB, so I don't
actually seem to be running out of system memory (I'm monitoring via
the free(1) command).

I also added 16MB of swap just to ensure that this wasn't an
out-of-memory condition.

To summarize, I'm seeing two slightly different behaviors:
    1) When I am running my node, upon shutdown I get the TYPE_ASSERTION_FAILED
    2) When running just erl and doing an mnesia:start() immediately
followed by mnesia:stop(), I get a segfault

I'm not sure why there are differences between #1 and #2, I haven't
been able to pin that down yet.

In the case of #2, I see the following in the system log:


beam[539]: segfault at 00c0001c pc 00047aba sp 7fa36424 ecr 24


If I bring back the core file to my build box and fire up the cross
compiled gdb, the stack is trashed, but if I look at address 00047aba
(pc), I seem to be in the middle of a symbol called cmp:

0x00047ab8 <cmp+628>:	mov r5,lr
0x00047aba <cmp+630>:	ld.w r3,lr[0x0]                     <<<-------- SEGV here
0x00047abc <cmp+632>:	bfextu r10,r3,0x2,0x4

So it looks like it could be a bad pointer.  However, I can't find a
symbol called cmp anywhere in beam, or the libraries it is linked
with:

# ldd erts-5.9/bin/beam
	libutil.so.0 => /lib/libutil.so.0 (0x2aab2000)
	libdl.so.0 => /lib/libdl.so.0 (0x2aab4000)
	libm.so.0 => /lib/libm.so.0 (0x2aab7000)
	libncurses.so.5 => /usr/lib/libncurses.so.5 (0x2aac5000)
	librt.so.0 => /lib/librt.so.0 (0x2aae5000)
	libc.so.0 => /lib/libc.so.0 (0x2aae7000)
	ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0x2aaab000)


So I'm not 100% sure where gdb is getting this "cmp" symbol from.

Next step is to try to build a debug version of OTP, so hopefully I
can get some more useful information from gdb.

Thanks!


W.