<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Adding more information,<div class="">this is related to SMP, if starting the erlang VM with -smp disable it does not crash.<br class=""><div apple-content-edited="true" class="">
<div class="">---</div><div class="">Cheers,</div><div class="">Heinz Nikolaus Gies</div><div class=""><a href="mailto:heinz@licenser.net" class="">heinz@licenser.net</a></div><div class=""><br class=""></div><br class="Apple-interchange-newline">
</div>
<br class=""><div><blockquote type="cite" class=""><div class="">On Mar 11, 2015, at 17:15, Ryan Zezeski <<a href="mailto:rzezeski@lucera.com" class="">rzezeski@lucera.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><br class="Apple-interchange-newline"><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_quote" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">On Wed, Mar 11, 2015 at 11:53 AM, Heinz Nikolaus Gies<span class="Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:heinz@licenser.net" target="_blank" class="">heinz@licenser.net</a>></span><span class="Apple-converted-space"> </span>wrote:<br class=""><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class="">Hi there is a problem with port_info it segfaults the erlang VM when two processes are executing it in parallel. We’ve been tracking this for the last two days and condensed the a crashing test down to this:</div><div class=""><br class=""></div><a href="https://gist.github.com/Licenser/41a7ffe5ca52f57a169c" target="_blank" class="">https://gist.github.com/Licenser/41a7ffe5ca52f57a169c</a><div class=""><br class=""></div><div class="">Ryan has some crash dumps:</div><div class=""><br class=""></div><div class=""><div style="border-top-width: 1px; border-top-style: solid; border-top-color: transparent; word-wrap: break-word; clear: both; padding: 0px 2rem 0.1rem 3rem; font-family: Lato, sans-serif; font-size: 15px; line-height: 22px; color: rgb(61, 60, 64); margin-top: 0.5rem;" class=""><span style="display: block; min-height: 1rem;" class=""><pre style="padding: 0.5rem; font-family: Monaco, Menlo, Consolas, 'Courier New', monospace; font-size: 0.75rem; color: rgb(51, 51, 51); border-top-left-radius: 4px; border-top-right-radius: 4px; border-bottom-right-radius: 4px; border-bottom-left-radius: 4px; margin-top: 0.5rem; margin-bottom: 0.2rem; line-height: 1.15rem; word-break: normal; word-wrap: break-word; white-space: pre-wrap; border: 1px solid rgba(0, 0, 0, 0.14902); background-color: rgb(251, 250, 248);" class="">fffffd7ff6406670 collect_heap_frags.isra.7+0x90()<br class="">fffffd7ff6406ba0 erts_garbage_collect+0x28f()<br class="">fffffd7ff6406bd0 erts_gc_after_bif_call+0xa4()<br class="">fffffd7ff6406f10 process_main+0x45da()<br class="">fffffd7ff6406f70 sched_thread_func+0xef()<br class="">fffffd7ff6406fb0 thr_wrapper+0x73()<br class="">fffffd7ff6406fe0 libc.so.1`_thrp_setup+0x8a(fffffd7ffd2a5a40)<br class="">fffffd7ff6406ff0 libc.so.1`_lwp_start()</pre></span></div><div class=""><div style="border-top-width: 1px; border-top-style: solid; border-top-color: transparent; word-wrap: break-word; clear: both; padding: 0.25rem 2rem 0.1rem 3rem; font-family: Lato, sans-serif; font-size: 15px; line-height: 22px; color: rgb(61, 60, 64);" class=""><span style="display: block; min-height: 1rem;" class=""><pre style="padding: 0.5rem; font-family: Monaco, Menlo, Consolas, 'Courier New', monospace; font-size: 0.75rem; color: rgb(51, 51, 51); border-top-left-radius: 4px; border-top-right-radius: 4px; border-bottom-right-radius: 4px; border-bottom-left-radius: 4px; margin-top: 0.5rem; margin-bottom: 0.2rem; line-height: 1.15rem; word-break: normal; word-wrap: break-word; white-space: pre-wrap; border: 1px solid rgba(0, 0, 0, 0.14902); background-color: rgb(251, 250, 248);" class="">fffffd7ff6207af0 copy_struct+0xb5()<br class="">fffffd7ff6207ba0 erts_port_info+0x292()<br class="">fffffd7ff6207bd0 erts_internal_port_info_1+0xeb()<br class="">fffffd7ff6207f10 process_main+0x45a9()<br class="">fffffd7ff6207f70 sched_thread_func+0xef()<br class="">fffffd7ff6207fb0 thr_wrapper+0x73()<br class="">fffffd7ff6207fe0 libc.so.1`_thrp_setup+0x8a(fffffd7ffd2a6240)<br class="">fffffd7ff6207ff0 libc.so.1`_lwp_start()</pre></span></div></div><div class=""><div style="border-top-width: 1px; border-top-style: solid; border-top-color: transparent; word-wrap: break-word; clear: both; padding: 0.25rem 2rem 0.1rem 3rem; font-family: Lato, sans-serif; font-size: 15px; line-height: 22px; color: rgb(61, 60, 64);" class=""><span style="display: block; min-height: 1rem;" class=""><pre style="padding: 0.5rem; font-family: Monaco, Menlo, Consolas, 'Courier New', monospace; font-size: 0.75rem; color: rgb(51, 51, 51); border-top-left-radius: 4px; border-top-right-radius: 4px; border-bottom-right-radius: 4px; border-bottom-left-radius: 4px; margin-top: 0.5rem; margin-bottom: 0.2rem; line-height: 1.15rem; word-break: normal; word-wrap: break-word; white-space: pre-wrap; border: 1px solid rgba(0, 0, 0, 0.14902); background-color: rgb(251, 250, 248);" class="">fffffd7ff73fe670 sweep_one_area+0x58()<br class="">fffffd7ff73feba0 erts_garbage_collect+0x985()<br class="">fffffd7ff73febd0 erts_gc_after_bif_call+0xa4()<br class="">fffffd7ff73fef10 process_main+0x45da()<br class="">fffffd7ff73fef70 sched_thread_func+0xef()<br class="">fffffd7ff73fefb0 thr_wrapper+0x73()<br class="">fffffd7ff73fefe0 libc.so.1`_thrp_setup+0x8a(fffffd7ffd2a1a40)<br class="">fffffd7ff73feff0 libc.so.1`_lwp_start()<br class=""><span style="display: inline-block; vertical-align: baseline; min-height: 0px; font-size: 0px; float: left; background-repeat: no-repeat no-repeat;" class="">```</span></pre></span></div><div style="border-top-width: 1px; border-top-style: solid; border-top-color: transparent; word-wrap: break-word; clear: both; padding: 0.25rem 2rem 0.1rem 3rem; font-family: Lato, sans-serif; font-size: 15px; line-height: 22px; color: rgb(61, 60, 64);" class=""><span style="font-family: arial, sans-serif; font-size: small; line-height: normal; color: rgb(34, 34, 34);" class=""> </span><span style="font-family: arial, sans-serif; font-size: small; line-height: normal; color: rgb(34, 34, 34);" class=""> </span></div></div></div></div></blockquote><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><div class=""><div style="border-top-width: 1px; border-top-style: solid; border-top-color: transparent; word-wrap: break-word; clear: both; padding: 0.25rem 2rem 0.1rem 3rem; font-family: Lato, sans-serif; font-size: 15px; line-height: 22px; color: rgb(61, 60, 64);" class=""><i style="display: inline-block; vertical-align: baseline; min-height: 0px; font-size: 0px; float: left; background-repeat: no-repeat no-repeat;" class=""><a href="https://lucera.slack.com/team/rzezeski" target="_blank" style="text-decoration: none; word-break: break-word; font-weight: 900; padding-right: 0.25rem; margin-left: 0px; font-style: normal; color: rgb(61, 60, 64) !important;" class="">zezeski</a><i style="display: inline-block; vertical-align: baseline; min-height: 0px; float: left; background-repeat: no-repeat no-repeat;" class=""></i></i><i style="display: inline-block; vertical-align: baseline; min-height: 0px; font-size: 0px; float: left; background-repeat: no-repeat no-repeat;" class="">[14:4<span style="font-family: arial, sans-serif; font-size: small; font-style: normal; line-height: normal; color: rgb(34, 34, 34);" class=""> </span><span style="font-family: arial, sans-serif; font-size: small; font-style: normal; line-height: normal; color: rgb(34, 34, 34);" class=""> </span></i></div></div></div></div></blockquote><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><div class=""><div style="border-top-width: 1px; border-top-style: solid; border-top-color: transparent; word-wrap: break-word; clear: both; padding: 0.25rem 2rem 0.1rem 3rem; font-family: Lato, sans-serif; font-size: 15px; line-height: 22px; color: rgb(61, 60, 64);" class=""><i style="display: inline-block; vertical-align: baseline; min-height: 0px; font-size: 0px; float: left; background-repeat: no-repeat no-repeat;" class=""><div class="gmail_default" style="font-size: small; font-style: normal; line-height: normal; color: rgb(34, 34, 34); font-family: monospace, monospace; display: inline;"></div></i></div></div></div></div></blockquote><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div class="gmail_default" style="font-family: monospace, monospace; display: inline;">Just to give more information: here's some info from a crash dump of the simple reproducer Heinz linked above. I simply used the chunter erl (16B02) to run this program, the chunter application is not actually running.</div></div><div class=""><div class="gmail_default" style="font-family: monospace, monospace; display: inline;"><br class=""></div></div><div class=""><div class="gmail_default" style="font-family: monospace, monospace; display: inline;">This is the situation we see most often but occasionally I've seen the copy.c `copy_struct` function try to reference an address that is _above_ the stack, i.e. not even legal.</div></div><div class=""><div class="gmail_default" style="font-family: monospace, monospace; display: inline;"><br class=""></div></div><div class=""><font face="monospace, monospace" class=""> > ::status</font></div><div class=""><font face="monospace, monospace" class=""> debugging core file of beam.smp (64-bit) from sys76</font></div><div class=""><font face="monospace, monospace" class=""> file: /opt/chunter/erts-5.10.3/bin/beam.smp</font></div><div class=""><font face="monospace, monospace" class=""> initial argv: /opt/chunter/erts-5.10.3/bin/beam.smp -- -root /opt/chunter -progname erl -- -h</font></div><div class=""><font face="monospace, monospace" class=""> threading model: native threads</font></div><div class=""><font face="monospace, monospace" class=""> status: process terminated by SIGSEGV (Segmentation Fault), addr=fffffd7ffc540000</font></div><div class=""><font face="monospace, monospace" class=""><br class=""></font></div><div class=""><font face="monospace, monospace" class=""> > $C</font></div><div class=""><font face="monospace, monospace" class=""> fffffd7ff8805670 collect_heap_frags.isra.7+0x95()</font></div><div class=""><font face="monospace, monospace" class=""> fffffd7ff8805ba0 erts_garbage_collect+0x115c()</font></div><div class=""><font face="monospace, monospace" class=""> fffffd7ff8805bd0 erts_gc_after_bif_call+0xa4()</font></div><div class=""><font face="monospace, monospace" class=""> fffffd7ff8805f10 process_main+0x45da()</font></div><div class=""><font face="monospace, monospace" class=""> fffffd7ff8805f70 sched_thread_func+0xef()</font></div><div class=""><font face="monospace, monospace" class=""> fffffd7ff8805fb0 thr_wrapper+0x73()</font></div><div class=""><font face="monospace, monospace" class=""> fffffd7ff8805fe0 libc.so.1`_thrp_setup+0x8a(fffffd7ffee6a240)</font></div><div class=""><font face="monospace, monospace" class=""> fffffd7ff8805ff0 libc.so.1`_lwp_start()</font></div><div class=""><font face="monospace, monospace" class=""><br class=""></font></div><div class=""><font face="monospace, monospace" class=""> > ::stacks</font></div><div class=""><font face="monospace, monospace" class=""> THREAD STATE SOBJ COUNT</font></div><div class=""><font face="monospace, monospace" class=""> 4 PARKED CV 10</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`cond_wait_queue+0x5b</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`__cond_wait+0xb3</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`cond_wait+0x2a</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`pthread_cond_wait+0x15</font></div><div class=""><font face="monospace, monospace" class=""> ethr_event_wait+0x63</font></div><div class=""><font face="monospace, monospace" class=""> async_main+0x275</font></div><div class=""><font face="monospace, monospace" class=""> thr_wrapper+0x73</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`_thrp_setup+0x8a</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`_lwp_start</font></div><div class=""><font face="monospace, monospace" class=""><br class=""></font></div><div class=""><font face="monospace, monospace" class=""> f PARKED CV 9</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`cond_wait_queue+0x5b</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`__cond_wait+0xb3</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`cond_wait+0x2a</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`pthread_cond_wait+0x15</font></div><div class=""><font face="monospace, monospace" class=""> ethr_event_wait+0x63</font></div><div class=""><font face="monospace, monospace" class=""> schedule+0x220d</font></div><div class=""><font face="monospace, monospace" class=""> process_main+0x143</font></div><div class=""><font face="monospace, monospace" class=""> sched_thread_func+0xef</font></div><div class=""><font face="monospace, monospace" class=""> thr_wrapper+0x73</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`_thrp_setup+0x8a</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`_lwp_start</font></div><div class=""><font face="monospace, monospace" class=""><br class=""></font></div><div class=""><font face="monospace, monospace" class=""> 16 UNPARKED <NONE> 1</font></div><div class=""><font face="monospace, monospace" class=""> erts_garbage_collect+0x115c</font></div><div class=""><font face="monospace, monospace" class=""> erts_gc_after_bif_call+0xa4</font></div><div class=""><font face="monospace, monospace" class=""> process_main+0x45da</font></div><div class=""><font face="monospace, monospace" class=""> sched_thread_func+0xef</font></div><div class=""><font face="monospace, monospace" class=""> thr_wrapper+0x73</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`_thrp_setup+0x8a</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`_lwp_start</font></div><div class=""><font face="monospace, monospace" class=""><br class=""></font></div><div class=""><font face="monospace, monospace" class=""> 17 UNPARKED <NONE> 1</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`mutex_lock_impl+0x189</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`mutex_lock+0x13</font></div><div class=""><font face="monospace, monospace" class=""> spawn_start+0x46c</font></div><div class=""><font face="monospace, monospace" class=""> erts_open_driver+0x21e</font></div><div class=""><font face="monospace, monospace" class=""> open_port+0xa52</font></div><div class=""><font face="monospace, monospace" class=""> open_port_2+0x2d</font></div><div class=""><font face="monospace, monospace" class=""> process_main+0x45a9</font></div><div class=""><font face="monospace, monospace" class=""> sched_thread_func+0xef</font></div><div class=""><font face="monospace, monospace" class=""> thr_wrapper+0x73</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`_thrp_setup+0x8a</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`_lwp_start</font></div><div class=""><font face="monospace, monospace" class=""><br class=""></font></div><div class=""><font face="monospace, monospace" class=""> 1 UNPARKED <NONE> 1</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`pselect+0x1cb</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`select+0x5a</font></div><div class=""><font face="monospace, monospace" class=""> erts_sys_main_thread+0x20</font></div><div class=""><font face="monospace, monospace" class=""> erl_start+0xcfe</font></div><div class=""><font face="monospace, monospace" class=""> main+9</font></div><div class=""><font face="monospace, monospace" class=""> _start+0x6c</font></div><div class=""><font face="monospace, monospace" class=""><br class=""></font></div><div class=""><font face="monospace, monospace" class=""> e UNPARKED <NONE> 1</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`waitpid+0x3b</font></div><div class=""><font face="monospace, monospace" class=""> child_waiter+0x30</font></div><div class=""><font face="monospace, monospace" class=""> thr_wrapper+0x73</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`_thrp_setup+0x8a</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`_lwp_start</font></div><div class=""><font face="monospace, monospace" class=""><br class=""></font></div><div class=""><font face="monospace, monospace" class=""> 2 UNPARKED <NONE> 1</font></div><div class=""><font face="monospace, monospace" class=""> signal_dispatcher_thread_func+0x29</font></div><div class=""><font face="monospace, monospace" class=""> thr_wrapper+0x73</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`_thrp_setup+0x8a</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`_lwp_start</font></div><div class=""><font face="monospace, monospace" class=""><br class=""></font></div><div class=""><font face="monospace, monospace" class=""> 15 UNPARKED <NONE> 1</font></div><div class=""><font face="monospace, monospace" class=""> spawn_start+0x60b</font></div><div class=""><font face="monospace, monospace" class=""> erts_open_driver+0x21e</font></div><div class=""><font face="monospace, monospace" class=""> open_port+0xa52</font></div><div class=""><font face="monospace, monospace" class=""> open_port_2+0x2d</font></div><div class=""><font face="monospace, monospace" class=""> process_main+0x45a9</font></div><div class=""><font face="monospace, monospace" class=""> sched_thread_func+0xef</font></div><div class=""><font face="monospace, monospace" class=""> thr_wrapper+0x73</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`_thrp_setup+0x8a</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`_lwp_start</font></div><div class=""><font face="monospace, monospace" class=""><br class=""></font></div><div class=""><font face="monospace, monospace" class=""> 3 PARKED CV 1</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`cond_wait_queue+0x5b</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`__cond_wait+0xb3</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`cond_wait+0x2a</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`pthread_cond_wait+0x15</font></div><div class=""><font face="monospace, monospace" class=""> ethr_cond_wait+9</font></div><div class=""><font face="monospace, monospace" class=""> sys_msg_dispatcher_func+0xdf</font></div><div class=""><font face="monospace, monospace" class=""> thr_wrapper+0x73</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`_thrp_setup+0x8a</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`_lwp_start</font></div><div class=""><font face="monospace, monospace" class=""><br class=""></font></div><div class=""><font face="monospace, monospace" class=""> 1b PARKED CV 1</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`cond_wait_queue+0x5b</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`__cond_wait+0xb3</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`cond_wait+0x2a</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`pthread_cond_wait+0x15</font></div><div class=""><font face="monospace, monospace" class=""> ethr_event_wait+0x63</font></div><div class=""><font face="monospace, monospace" class=""> aux_thread+0x1cd</font></div><div class=""><font face="monospace, monospace" class=""> thr_wrapper+0x73</font></div><div class=""><font face="monospace, monospace" class=""> libc.so.1`_thrp_setup+0x8a</font></div><div class=""><div class="gmail_default" style="display: inline;"><font face="monospace, monospace" class=""> libc.so.1`_lwp_start</font></div> </div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div class="gmail_default" style="font-family: monospace, monospace;">And finally here's the VA space mapping:</div><div class="gmail_default" style="font-family: monospace, monospace;"><br class=""></div><div class="gmail_default" style="font-family: monospace, monospace;"><div class="gmail_default"> # pmap /cores/core.beam.smp.92077</div><div class="gmail_default"> core '/cores/core.beam.smp.92077' of 92077: /opt/chunter/erts-5.10.3/bin/beam.smp -- -root /opt/chunter -progname</div><div class="gmail_default"> 0000000000400000 2652K r-x-- /opt/chunter/erts-5.10.3/bin/beam.smp</div><div class="gmail_default"> 00000000006A6000 296K rw--- /opt/chunter/erts-5.10.3/bin/beam.smp</div><div class="gmail_default"> 00000000006F0000 1276K rw--- [ heap ]</div><div class="gmail_default"> FFFFFD7FF6000000 2560K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FF6A00000 8192K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FF72C0000 1024K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FF7A00000 1024K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FF7E0A000 4K rw--- [ stack tid=27 ]</div><div class="gmail_default"> FFFFFD7FF8009000 4K rw--- [ stack tid=26 ]</div><div class="gmail_default"> FFFFFD7FF8208000 4K rw--- [ stack tid=25 ]</div><div class="gmail_default"> FFFFFD7FF8407000 4K rw--- [ stack tid=24 ]</div><div class="gmail_default"> FFFFFD7FF85FB000 8K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FF8604000 12K rw--- [ stack tid=23 ]</div><div class="gmail_default"> FFFFFD7FF8804000 8K rw--- [ stack tid=22 ]</div><div class="gmail_default"> FFFFFD7FF8A03000 8K rw--- [ stack tid=21 ]</div><div class="gmail_default"> FFFFFD7FF8C02000 8K rw--- [ stack tid=20 ]</div><div class="gmail_default"> FFFFFD7FF8E02000 4K rw--- [ stack tid=19 ]</div><div class="gmail_default"> FFFFFD7FF9001000 4K rw--- [ stack tid=18 ]</div><div class="gmail_default"> FFFFFD7FF9200000 4K rw--- [ stack tid=17 ]</div><div class="gmail_default"> FFFFFD7FF93FF000 4K rw--- [ stack tid=16 ]</div><div class="gmail_default"> FFFFFD7FF95FE000 4K rw--- [ stack tid=15 ]</div><div class="gmail_default"> FFFFFD7FF9600000 4096K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FF9AC0000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FF9C00000 4096K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFA1FE000 4K rw--- [ stack tid=3 ]</div><div class="gmail_default"> FFFFFD7FFA200000 2048K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFA4C0000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFA540000 1024K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFA83E000 4K rw--- [ stack tid=2 ]</div><div class="gmail_default"> FFFFFD7FFA840000 512K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFA900000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFA980000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFAA00000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFAA80000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFAB00000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFAB80000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFAC00000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFAC80000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFAD00000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFAD80000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFAE00000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFAE80000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFAF00000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFAF80000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFB000000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFB080000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFB100000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFB180000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFB200000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFB280000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFB300000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFB380000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFB400000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFB480000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFB500000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFB580000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFB600000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFB680000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFB700000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFB780000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFB800000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFB880000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFB900000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFB980000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFBA00000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFBA80000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFBB00000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFBB80000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFBC00000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFBC80000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFBD00000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFBD80000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFBE00000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFBE80000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFBF00000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFBF80000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFC000000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFC080000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFC100000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFC180000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFC200000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFC280000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFC300000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFC380000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFC400000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFC480000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFC500000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFC580000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFC600000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFC680000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFC700000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFC780000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFC800000 4096K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFCC40000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFCCC0000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFCD40000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFCDC0000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFCE40000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFCEC0000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFCF40000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFCFC0000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFD040000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFD0C0000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFD140000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFD1C0000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFD240000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFD2C0000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFD340000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFD3C0000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFD440000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFD4C0000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFD540000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFD5C0000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFD640000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFD6C0000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFD740000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFD7C0000 1024K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFD900000 1024K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFDA40000 1024K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFDB80000 1024K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFDCC0000 1024K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFDE00000 1024K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFDF40000 1024K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFE080000 1024K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFE1C0000 1024K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFE2D0000 352K r-x-- /usr/gnu/lib/amd64/libncurses.so.5.7</div><div class="gmail_default"> FFFFFD7FFE337000 20K rw--- /usr/gnu/lib/amd64/libncurses.so.5.7</div><div class="gmail_default"> FFFFFD7FFE340000 32K r-x-- /lib/amd64/librtld_db.so.1</div><div class="gmail_default"> FFFFFD7FFE358000 4K rw--- /lib/amd64/librtld_db.so.1</div><div class="gmail_default"> FFFFFD7FFE380000 580K r-x-- /usr/lib/amd64/libdtrace.so.1</div><div class="gmail_default"> FFFFFD7FFE421000 84K rw--- /usr/lib/amd64/libdtrace.so.1</div><div class="gmail_default"> FFFFFD7FFE436000 12K rw--- /usr/lib/amd64/libdtrace.so.1</div><div class="gmail_default"> FFFFFD7FFE440000 4K r-x-- /lib/amd64/libsendfile.so.1</div><div class="gmail_default"> FFFFFD7FFE451000 4K rw--- /lib/amd64/libsendfile.so.1</div><div class="gmail_default"> FFFFFD7FFE480000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFE500000 1024K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFE630000 8K r-x-- /lib/amd64/libkstat.so.1</div><div class="gmail_default"> FFFFFD7FFE642000 4K rw--- /lib/amd64/libkstat.so.1</div><div class="gmail_default"> FFFFFD7FFE680000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFE6E0000 20K r-x-- /lib/amd64/libmp.so.2</div><div class="gmail_default"> FFFFFD7FFE6F5000 4K rw--- /lib/amd64/libmp.so.2</div><div class="gmail_default"> FFFFFD7FFE700000 60K r-x-- /usr/lib/amd64/libgcc_s.so.1</div><div class="gmail_default"> FFFFFD7FFE71E000 4K rw--- /usr/lib/amd64/libgcc_s.so.1</div><div class="gmail_default"> FFFFFD7FFE71F000 4K r-x-- /lib/amd64/libdl.so.1</div><div class="gmail_default"> FFFFFD7FFE730000 4K rwx-- [ anon ]</div><div class="gmail_default"> FFFFFD7FFE740000 32K r-x-- /lib/amd64/libmd.so.1</div><div class="gmail_default"> FFFFFD7FFE758000 4K rw--- /lib/amd64/libmd.so.1</div><div class="gmail_default"> FFFFFD7FFE760000 8K r-x-- /usr/lib/amd64/libsctp.so.1</div><div class="gmail_default"> FFFFFD7FFE772000 4K rw--- /usr/lib/amd64/libsctp.so.1</div><div class="gmail_default"> FFFFFD7FFE780000 1024K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFE8BE000 4K rw--- [ stack tid=13 ]</div><div class="gmail_default"> FFFFFD7FFE8C0000 1024K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFE9EB000 4K rw--- [ stack tid=12 ]</div><div class="gmail_default"> FFFFFD7FFE9ED000 12K r-x-- /lib/amd64/libpthread.so.1</div><div class="gmail_default"> FFFFFD7FFEA1B000 4K rw--- [ stack tid=11 ]</div><div class="gmail_default"> FFFFFD7FFEA3E000 4K rw--- [ stack tid=10 ]</div><div class="gmail_default"> FFFFFD7FFEA40000 28K r-x-- /lib/amd64/libdlpi.so.1</div><div class="gmail_default"> FFFFFD7FFEA57000 4K rw--- /lib/amd64/libdlpi.so.1</div><div class="gmail_default"> FFFFFD7FFEA60000 68K r-x-- /lib/amd64/libsocket.so.1</div><div class="gmail_default"> FFFFFD7FFEA81000 4K rw--- /lib/amd64/libsocket.so.1</div><div class="gmail_default"> FFFFFD7FFEABE000 4K rw--- [ stack tid=9 ]</div><div class="gmail_default"> FFFFFD7FFEAC0000 1024K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFEBE0000 476K r-x-- /lib/amd64/libumem.so.1</div><div class="gmail_default"> FFFFFD7FFEC66000 4K rwx-- /lib/amd64/libumem.so.1</div><div class="gmail_default"> FFFFFD7FFEC77000 144K rw--- /lib/amd64/libumem.so.1</div><div class="gmail_default"> FFFFFD7FFEC9B000 44K rw--- /lib/amd64/libumem.so.1</div><div class="gmail_default"> FFFFFD7FFECC0000 4K rwx-- [ anon ]</div><div class="gmail_default"> FFFFFD7FFECEB000 4K rw--- [ stack tid=8 ]</div><div class="gmail_default"> FFFFFD7FFED0E000 4K rw--- [ stack tid=7 ]</div><div class="gmail_default"> FFFFFD7FFED10000 532K r-x-- /lib/amd64/libnsl.so.1</div><div class="gmail_default"> FFFFFD7FFEDA5000 12K rw--- /lib/amd64/libnsl.so.1</div><div class="gmail_default"> FFFFFD7FFEDA8000 32K rw--- /lib/amd64/libnsl.so.1</div><div class="gmail_default"> FFFFFD7FFEDDB000 4K rw--- [ stack tid=6 ]</div><div class="gmail_default"> FFFFFD7FFEDFE000 4K rw--- [ stack tid=5 ]</div><div class="gmail_default"> FFFFFD7FFEE00000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFEE50000 4K rwx-- [ anon ]</div><div class="gmail_default"> FFFFFD7FFEE60000 64K rwx-- [ anon ]</div><div class="gmail_default"> FFFFFD7FFEE80000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFEED5000 4K rw--- [ stack tid=14 ]</div><div class="gmail_default"> FFFFFD7FFEEE0000 396K r-x-- /lib/amd64/libm.so.2</div><div class="gmail_default"> FFFFFD7FFEF53000 8K rw--- /lib/amd64/libm.so.2</div><div class="gmail_default"> FFFFFD7FFEF7E000 4K rw--- [ stack tid=4 ]</div><div class="gmail_default"> FFFFFD7FFEF80000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFEFD0000 4K r----* [ anon ]</div><div class="gmail_default"> FFFFFD7FFEFE0000 64K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFF000000 256K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFF052000 128K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFF073000 4K rwx-- [ anon ]</div><div class="gmail_default"> FFFFFD7FFF080000 64K rwx-- [ anon ]</div><div class="gmail_default"> FFFFFD7FFF0A0000 4K rwx-- [ anon ]</div><div class="gmail_default"> FFFFFD7FFF0B0000 24K rwx-- [ anon ]</div><div class="gmail_default"> FFFFFD7FFF0C0000 4K rwx-- [ anon ]</div><div class="gmail_default"> FFFFFD7FFF0D0000 4K rwx-- [ anon ]</div><div class="gmail_default"> FFFFFD7FFF0E0000 1516K r-x-- /lib/amd64/libc.so.1</div><div class="gmail_default"> FFFFFD7FFF26B000 44K rw--- /lib/amd64/libc.so.1</div><div class="gmail_default"> FFFFFD7FFF276000 16K rw--- /lib/amd64/libc.so.1</div><div class="gmail_default"> FFFFFD7FFF280000 4K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFF290000 4K rwx-- [ anon ]</div><div class="gmail_default"> FFFFFD7FFF2A0000 4K rwx-- [ anon ]</div><div class="gmail_default"> FFFFFD7FFF2B0000 4K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFF2C0000 4K rwx-- [ anon ]</div><div class="gmail_default"> FFFFFD7FFF2D0000 4K rwx-- [ anon ]</div><div class="gmail_default"> FFFFFD7FFF2E0000 4K rwx-- [ anon ]</div><div class="gmail_default"> FFFFFD7FFF2F0000 4K rwx-- [ anon ]</div><div class="gmail_default"> FFFFFD7FFF300000 4K rwx-- [ anon ]</div><div class="gmail_default"> FFFFFD7FFF310000 4K rwx-- [ anon ]</div><div class="gmail_default"> FFFFFD7FFF320000 4K rwx-- [ anon ]</div><div class="gmail_default"> FFFFFD7FFF330000 4K rwx-- [ anon ]</div><div class="gmail_default"> FFFFFD7FFF340000 4K rwx-- [ anon ]</div><div class="gmail_default"> FFFFFD7FFF350000 4K rwx-- [ anon ]</div><div class="gmail_default"> FFFFFD7FFF360000 4K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFF370000 4K rw--- [ anon ]</div><div class="gmail_default"> FFFFFD7FFF380000 4K rwx-- [ anon ]</div><div class="gmail_default"> FFFFFD7FFF390000 4K rwx-- [ anon ]</div><div class="gmail_default"> FFFFFD7FFF398000 332K r-x-- /lib/amd64/ld.so.1</div><div class="gmail_default"> FFFFFD7FFF3FB000 8K rwx-- /lib/amd64/ld.so.1</div><div class="gmail_default"> FFFFFD7FFF3FD000 8K rwx-- /lib/amd64/ld.so.1</div><div class="gmail_default"> FFFFFD7FFFDFD000 12K rw--- [ stack ]</div><div class="gmail_default"> total 75540K</div></div><div class="gmail_default" style="font-family: monospace, monospace;"><br class=""></div><div class="gmail_default" style="font-family: monospace, monospace;"></div></div></div></div></blockquote></div><br class=""></div></body></html>