[erlang-questions] no next heap size found: 18446744071896091830, offset 0

[Scott Lystig Fritchie]

Hi.  This was a fun one coming from an OpenSolaris 64-bit box running
Erlang/OTP R14B03.  Searching my archive of erlang-questions and
erlang-bugs mailing lists hasn't found a match on 'no next heap size'
string ... except for a formatting bug where the heap size is
reported as negative.

    [root@REDACTED /var/log/riak]# head -20 erl_crash.dump.slf.0
    =erl_crash_dump:0.1
    Thu Jan  5 01:03:52 2012
    Slogan: no next heap size found: 18446744071896091830, offset 0
    System version: Erlang R14B03 (erts-5.8.4) [source] [64-bit] [smp:16:16] [rq:16] [async-threads:571] [kernel-poll:true]
    Compiled: Mon Jul 25 18:05:12 2011
    Taints: eleveldb,crypto,bitcask_nifs
    Atoms: 14581
    =memory
    total: 34190115856
    processes: 33980165896
    processes_used: 33980126736
    system: 209949960
    atom: 986689
    atom_used: 982563
    binary: 154769656
    code: 9219672
    ets: 712592
    =hash_table:atom_tab
    size: 9643
    used: 7479

Riak was running at the time, and a zillion error messages were
generated & sent to the error logger by gen_gsm and handled by Andrew
Thompson's "lager" application.  The 'lager_crash_log' process ended up
with 31K messages in its Erlang mailbox, which is quite a lot ... and
the app was generating a few thousand error messages per second, which
meant that we were probably going to run out of memory anyway due to
Erlang mailbox growth.

However, having a memory allocation fail for the reason shown above
isn't good.  Has anyone else seen this error on R14B03?  Or later?

-Scott

P.S. Here's the scoop on the memory hog proc:

    =proc:<0.43.0>
    State: Garbing
    Name: lager_crash_log
    Spawned as: proc_lib:init_p/5
    Last scheduled in for: io_lib_format:iolist_to_chars/1
    Spawned by: <0.40.0>
    Started: Fri Dec 30 00:29:38 2011
    Message queue length: 31694
    Number of heap fragments: 0
    Heap fragment data: 0
    Link list: [#Port<0.33849890>, <0.40.0>]
    Reductions: 8749820516
    Stack+heap: 4148490785
    OldHeap: 0
    Heap unused: 3072286602
    OldHeap unused: 0
    Program counter: 0x000000000386f1c0 (io_lib_format:iolist_to_chars/1 + 8)
    CP: 0x0000000000000000 (invalid)