<div dir="ltr">I think I found the bug. After a fork/vfork the child process needs some initialisation. In case of vfork, this is done by executing the <font face="monospace">erl_child_setup</font> program. In case of a fork however the setup code is in the same module, <font face="monospace">sys.c</font>, where the fork happens. There are comments about how important is to keep <font face="monospace">erl_child_setup.c</font> and the relevant parts of <font face="monospace">sys.c</font> in sync.<div><br></div><div>In OTP 18 erts began to use a new signal, <font face="monospace">ERTS_SYS_SUSPEND_SIGNAL</font> internally. This signal is in fact <font face="monospace">SIGUSR2</font>. The child process has to unblock all signals used by erts as part of its initialisation. And there's an inconsistency here between the <a href="https://github.com/erlang/otp/blob/OTP-18.3.4.5/erts/emulator/sys/unix/erl_child_setup.c#L134-L136">vfork</a> and <a href="https://github.com/erlang/otp/blob/OTP-18.3.4.5/erts/emulator/sys/unix/sys.c#L1005-L1022">fork</a> cases: <font face="monospace">erl_child_setup.c</font> does not unblock <span style="font-family:monospace;font-size:10.5625px">SIGUSR2</span>.</div><div><br></div><div>And it turns out that <font face="monospace">lbzip2</font> wants to use <span style="font-family:monospace;font-size:10.5625px">SIGUSR2</span> for communication between its worker processes, but this signal is blocked when we call it from Erlang's <font face="monospace">os:cmd</font> (with vfork), so the program hangs.</div><div><br></div><div>This patch to <font face="monospace"><span style="font-size:10.5625px">erts/emulator/sys/unix/e</span></font><span style="font-family:monospace;font-size:10.5625px">rl_child_setup.c</span> solved the problem:</div><div><br></div><div><div><font face="monospace">@@ -134,6 +134,7 @@ main(int argc, char *argv[])</font></div><div><font face="monospace"> sys_sigrelease(SIGCHLD);</font></div><div><font face="monospace"> sys_sigrelease(SIGINT);</font></div><div><font face="monospace"> sys_sigrelease(SIGUSR1);</font></div><div><font face="monospace">+ sys_sigrelease(SIGUSR2);</font></div><div><font face="monospace"> </font></div><div><font face="monospace"> if (erts_spawn_executable) {</font></div><div><font face="monospace"> <span style="white-space:pre"> </span>if (argv[CS_ARGV_NO_OF_ARGS + 1] == NULL) {</font></div></div><div><br></div><div><br></div><div>Daniel</div></div><br><div class="gmail_quote"><div dir="ltr">On Thu, 27 Jul 2017 at 09:21 Dániel Szoboszlay <<a href="mailto:dszoboszlay@gmail.com">dszoboszlay@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks Dmytro, this really helped a lot!<div><br></div><div>I think the commit you pointed to is not directly related: it only changes Erlang code, and if the behaviour depends on whether you are using a release/debug build, the root cause is most probably somewhere in the C code of erts.</div><div><br></div><div>But the commit message talks about the emulator no longer using vfork, and it was a good clue: disabling vfork on 18 prevents the problem. So this one will finish:</div><div><br></div><div><font face="monospace">ERL_NO_VFORK=true erl +A0 +S 1:1 -noinput -noshell -eval 'os:cmd("tar -C /tmp/ -xf /tmp/tartest --use-compress-program=lbzip2"), init:stop().'</font><br></div><div><br></div>Thanks again for the clue, I will look into the difference between using fork/vfork in OTP 18!<div><br></div><div>Daniel</div></div><br><div class="gmail_quote"><div dir="ltr">On Thu, 27 Jul 2017 at 02:09 Dmytro Lytovchenko <<a href="mailto:dmytro.lytovchenko@gmail.com" target="_blank">dmytro.lytovchenko@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I could observe the behaviour only in R18, but not in R19 and not in R20<div>I also could not reproduce it in debug flavour of R18 emulator, but it reproduces reliably in release SMP variant.</div><div><br></div><div>The changes to os.erl between 18.3.4.5 and 19.0 include removal of os:cmd server which might somehow be related (commit <b>200247f972b012ced0c4b2c6611f091af66ebedd</b>). This commit <i>possibly</i> fixes the behavior — in R19 (build 19.0 by Kerl) the behaviour does not happen.<br></div></div><div class="gmail_extra"><br><div class="gmail_quote">2017-07-26 21:47 GMT+02:00 Dániel Szoboszlay <span dir="ltr"><<a href="mailto:dszoboszlay@gmail.com" target="_blank">dszoboszlay@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Honestly, I didn't try with other command variations. There are many commands that do not hang when run from os:cmd, regardless of the OTP version. But this particular command does hang with one OTP version, and not with the other OTP version. So the difference is in OTP, and I want to find out what has changed.<div><br></div><div>Daniel</div></div><div class="m_557495373443118240m_2842372589571276074HOEnZb"><div class="m_557495373443118240m_2842372589571276074h5"><br><div class="gmail_quote"><div dir="ltr">On Wed, 26 Jul 2017 at 21:34 Dmytro Lytovchenko <<a href="mailto:dmytro.lytovchenko@gmail.com" target="_blank">dmytro.lytovchenko@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Is it something lbzip2 related?<div>Did you try normal single-thread bzip2? (-j flag or --bzip2)</div><div>What is you use gzip? (-z or --gzip)</div></div><div class="gmail_extra"><br><div class="gmail_quote"></div></div><div class="gmail_extra"><div class="gmail_quote">2017-07-26 21:27 GMT+02:00 Dániel Szoboszlay <span dir="ltr"><<a href="mailto:dszoboszlay@gmail.com" target="_blank">dszoboszlay@gmail.com</a>></span>:<br></div></div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>I've encountered a strange problem with os:cmd when running tar and lbzip2. Steps to reproduce:</div><div><br></div><div><pre class="m_557495373443118240m_2842372589571276074m_-7968189779104302993m_-2495971627260676471m_-4595760503427139461inbox-inbox-code-java" style="margin-top:0px;margin-bottom:0px;padding:0px;max-height:30em;overflow:auto;white-space:pre-wrap;word-wrap:normal;color:rgb(51,51,51);font-size:12px;background-color:rgb(245,245,245)"># create some lbzip2 compressed data</pre><pre class="m_557495373443118240m_2842372589571276074m_-7968189779104302993m_-2495971627260676471m_-4595760503427139461inbox-inbox-code-java" style="margin-top:0px;margin-bottom:0px;padding:0px;max-height:30em;overflow:auto;white-space:pre-wrap;word-wrap:normal;color:rgb(51,51,51);font-size:12px;background-color:rgb(245,245,245)">dd <span class="m_557495373443118240m_2842372589571276074m_-7968189779104302993m_-2495971627260676471m_-4595760503427139461inbox-inbox-code-keyword" style="color:rgb(0,0,145)">if</span>=/dev/urandom of=/tmp/testfile count=10
tar -cf - -C /tmp testfile | lbzip2 -6 -n 4 | dd of=/tmp/tartest status=none</pre><pre class="m_557495373443118240m_2842372589571276074m_-7968189779104302993m_-2495971627260676471m_-4595760503427139461inbox-inbox-code-java" style="margin-top:0px;margin-bottom:0px;padding:0px;max-height:30em;overflow:auto;white-space:pre-wrap;word-wrap:normal;color:rgb(51,51,51);font-size:12px;background-color:rgb(245,245,245)"><br></pre><pre class="m_557495373443118240m_2842372589571276074m_-7968189779104302993m_-2495971627260676471m_-4595760503427139461inbox-inbox-code-java" style="margin-top:0px;margin-bottom:0px;padding:0px;max-height:30em;overflow:auto;white-space:pre-wrap;word-wrap:normal;color:rgb(51,51,51);background-color:rgb(245,245,245)"># try to extract the archive from Erlang with os:cmd</pre><pre class="m_557495373443118240m_2842372589571276074m_-7968189779104302993m_-2495971627260676471m_-4595760503427139461inbox-inbox-code-java" style="margin-top:0px;margin-bottom:0px;padding:0px;max-height:30em;overflow:auto;word-wrap:normal;background-color:rgb(245,245,245)"><font color="#333333"><span style="white-space:pre-wrap">erl -noinput -eval 'os:cmd("tar -C /tmp/ -xf /tmp/tartest --use-compress-program=lbzip2"), init:stop().'</span></font></pre><br class="m_557495373443118240m_2842372589571276074m_-7968189779104302993m_-2495971627260676471m_-4595760503427139461inbox-inbox-Apple-interchange-newline">This worked fine with OTP 17.5.6.7, but with OTP 18.3.4.5 the command hangs: lbzip2 just sits in a rt_sigsuspend syscall waiting for a USR2, PIPE or XFSZ signal. And its parent, the tar process waits in a wait4 syscall for lbzip2 to terminate.</div><div><br></div><div>I don't have at the moment any newer OTP version installed, I'm not sure how OTP 19 or 20 would behave.</div><div><br></div><div>I tried to strace the processes, but there's too much noise, I couldn't yet figure out anything interesting there.</div><div><br></div><div>I also tried to diff OTP 17 & 18, but os:cmd/1 and friends didn't change. I'm not sure about the port code, but at least the release notes didn't mention anything major. Or did I miss something? Does anyone have an idea what may have changed between these OTP versions?</div><div><br></div><div>Thanks,</div><div>Daniel</div></div>
<br></blockquote></div></div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" rel="noreferrer" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
<br></blockquote></div><br></div>
</blockquote></div>
</div></div></blockquote></div><br></div>
</blockquote></div></blockquote></div>