<div dir="ltr">Was thinking perhaps it was a 32-bit installation, which can run out of memory in situations like that very quickly.<div><br></div><div>Besides OS-level memory examination tools, the best (in my opinion) tools for diagnosing issues like this are eper (<a href="https://github.com/massemanet/eper">https://github.com/massemanet/eper</a>) and recon (<a href="http://ferd.github.io/recon/">http://ferd.github.io/recon/</a>). The redoubtable and inestimably worthy Mr. Hebert gives an excellent overview of the debugging process in situations just like this in his free ebook "Stuff Goes Bad: Erlang in Anger" (<a href="http://www.erlang-in-anger.com/">http://www.erlang-in-anger.com/</a>).</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Feb 11, 2015 at 3:32 PM, Gene Sher <span dir="ltr"><<a href="mailto:corticalcomputer@gmail.com" target="_blank">corticalcomputer@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Though as noted it happened on various Erlang installations for me, the one I am currently using and on which I also get the Seg Fault is:<div><br></div><div><div>Erlang/OTP 17 [erts-6.3] [source-f9282c6] [64-bit] [smp:24:24] [async-threads:10] [hipe] [kernel-poll:false]</div></div><div><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Feb 11, 2015 at 6:29 PM, Felix Gallo <span dir="ltr"><<a href="mailto:felixgallo@gmail.com" target="_blank">felixgallo@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">What does it say when you type 'erl' at the command line? Example:<div><br></div><div><div>Erlang/OTP 17 [erts-6.0] [source] [64-bit] [smp:2:2] [async-threads:10] [hipe] [kernel-poll:false]</div><div><br></div><div>Eshell V6.0 (abort with ^G)</div><div>1></div></div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div>On Wed, Feb 11, 2015 at 2:54 PM, Gene Sher <span dir="ltr"><<a href="mailto:corticalcomputer@gmail.com" target="_blank">corticalcomputer@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr">Hello List,<div><br></div><div>Hardware: E5-Xeon 2697 v2, 32GB of RAM.</div><div>OSes tried: Xubuntu 14.04.1 LTS, CentOS 7, Ubuntu 12.04 LTS</div><div>Erlang versions the code was tried on: Erlang/OTP 17, R16, & R14</div><div><br></div><div>I have an issue where every time I use processes which contain within themselves large data structures (Large deep learning single process nodes), after just a minute or so Erlang core dumps. The amount of ram used is only about 2GB, so it can't be the system running out of memory, and its only using about 10 cores, since I'm only running 10 such processes. Now the same code, the same program, the same platform, functions without a problem when I keep these processes small (substantially smaller monolithic NN-module in each process). Everything is written purely in Erlang (No NIFs were involved in this particular NN code).</div><div><br></div><div>What exactly is happening? is something running out of space? Can anyone recommend what option during the Erlang startup I should perhaps modify to alleviate the issue?</div><div><br></div><div>There are no crushdump files that I can find, but I did get a core backtrace produced during one of these crashes when I was running erts-5.10.4, here is a partial paste of it:</div><div><br></div><div><div>ccpp-2015-02-11-09\:18\:41-2273/core_backtrace:</div><div>{ "signal": 11</div><div>, "executable": "/usr/lib64/erlang/erts-5.10.4/bin/beam.smp"</div><div>, "stacktrace":</div><div> [ { "crash_thread": true</div><div> , "frames":</div><div> [ { "address": 5352736</div><div> , "build_id": "69494bd95d056f5549e80b6fe507e55af574137f"</div><div> , "build_id_offset": 1158432</div><div> , "function_name": "sweep_one_area"</div><div> , "file_name": "/usr/lib64/erlang/erts-5.10.4/bin/beam.smp"</div><div> }</div><div> , { "address": 5367589</div><div> , "build_id": "69494bd95d056f5549e80b6fe507e55af574137f"</div><div> , "build_id_offset": 1173285</div><div> , "function_name": "erts_garbage_collect"</div><div> , "file_name": "/usr/lib64/erlang/erts-5.10.4/bin/beam.smp"</div><div> }</div><div> , { "address": 5369251</div><div> , "build_id": "69494bd95d056f5549e80b6fe507e55af574137f"</div><div> , "build_id_offset": 1174947</div><div> , "function_name": "erts_gc_after_bif_call"</div><div> , "file_name": "/usr/lib64/erlang/erts-5.10.4/bin/beam.smp"</div><div> }</div><div> , { "address": 5871217</div><div> , "build_id": "69494bd95d056f5549e80b6fe507e55af574137f"</div><div> , "build_id_offset": 1676913</div><div> , "function_name": "nbif_3_gc_after_bif"</div><div> , "file_name": "/usr/lib64/erlang/erts-5.10.4/bin/beam.smp"</div><div> } ]</div><div> }</div><div> , { "frames":</div><div> [ { "address": 1101651978</div><div> , "build_id_offset": 1101651978</div><div> } ]</div><div> }</div><div> , { "frames":</div><div> [ { "address": 139994957883141</div><div> , "build_id": "18562ee0363bc9bd7101610bd86469aa426d0c44"</div><div> , "build_id_offset": 46853</div><div> , "function_name": "pthread_cond_wait@@GLIBC_2.3.2"</div><div> , "file_name": "/lib64/libpthread.so.0"</div><div> }</div><div> , { "address": 6128777</div><div> , "build_id": "69494bd95d056f5549e80b6fe507e55af574137f"</div><div> , "build_id_offset": 1934473</div><div> , "function_name": "ethr_cond_wait"</div><div> , "file_name": "/usr/lib64/erlang/erts-5.10.4/bin/beam.smp"</div><div> }</div><div> , { "address": 4665919</div><div> , "build_id": "69494bd95d056f5549e80b6fe507e55af574137f"</div><div> , "build_id_offset": 471615</div><div> , "function_name": "sys_msg_dispatcher_func"</div><div> , "file_name": "/usr/lib64/erlang/erts-5.10.4/bin/beam.smp"</div><div> }</div><div> , { "address": 6134325</div><div> , "build_id": "69494bd95d056f5549e80b6fe507e55af574137f"</div><div> , "build_id_offset": 1940021</div><div> , "function_name": "thr_wrapper"</div><div> , "file_name": "/usr/lib64/erlang/erts-5.10.4/bin/beam.smp"</div><div> }</div><div> , { "address": 139994957868531</div><div> , "build_id": "18562ee0363bc9bd7101610bd86469aa426d0c44"</div><div> , "build_id_offset": 32243</div><div> , "function_name": "start_thread"</div><div> , "file_name": "/lib64/libpthread.so.0"</div><div> }</div><div> , { "address": 139994952778157</div><div> , "build_id": "23d9f6f74c80c45a602094e5016f047bfc4d046c"</div><div> , "build_id_offset": 1008045</div><div> , "function_name": "__clone"</div><div> , "file_name": "/lib64/libc.so.6"</div><div> } ]</div><div> }</div><div> , { "frames":</div><div> [ { "address": 139994957894237</div><div> , "build_id": "18562ee0363bc9bd7101610bd86469aa426d0c44"</div><div> , "build_id_offset": 57949</div><div> , "function_name": "read"</div><div> , "file_name": "/lib64/libpthread.so.0"</div><div> }</div><div> , { "address": 5741674</div><div> , "build_id": "69494bd95d056f5549e80b6fe507e55af574137f"</div><div> , "build_id_offset": 1547370</div><div> , "function_name": "signal_dispatcher_thread_func"</div><div> , "file_name": "/usr/lib64/erlang/erts-5.10.4/bin/beam.smp"</div><div> }</div><div> , { "address": 6134325</div><div> , "build_id": "69494bd95d056f5549e80b6fe507e55af574137f"</div><div> , "build_id_offset": 1940021</div><div> , "function_name": "thr_wrapper"</div><div> , "file_name": "/usr/lib64/erlang/erts-5.10.4/bin/beam.smp"</div><div> }</div><div> , { "address": 139994957868531</div><div> , "build_id": "18562ee0363bc9bd7101610bd86469aa426d0c44"</div><div> , "build_id_offset": 32243</div><div> , "function_name": "start_thread"</div><div> , "file_name": "/lib64/libpthread.so.0"</div><div> }</div><div> , { "address": 139994952778157</div><div> , "build_id": "23d9f6f74c80c45a602094e5016f047bfc4d046c"</div><div> , "build_id_offset": 1008045</div><div> , "function_name": "__clone"</div><div> , "file_name": "/lib64/libc.so.6"</div><div> } ]</div><div> }</div></div><div>...</div><div><br></div><div>Thanks in advance for any suggestions and help,</div><div>-Gene</div></div>
<br></div></div>_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
<br></blockquote></div><br></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>