[erlang-questions] release handler crash during relup, sys:get_status noproc

[Richard Jones]

for the record, my initial suspicions were justified. i reproduced this by
adding a timer:sleep in release_handler_1 to make it easier to trigger.

it would only manifest if a supervisor exited (cleanly or otherwise) during
release installation. workers restarting doesn't trigger it.

if anyone else has experienced noproc crashes during release upgrades, i
recommend trying this patch:

https://github.com/erlang/otp/pull/822

On 24 August 2015 at 17:20, Richard Jones <rj@REDACTED> wrote:

> Anyone else experienced a crash like this when doing a release upgrade?
> ie, calling release_handler:install_release, with a valid relup
>
> {"init terminating in
> do_boot",{{badmatch,{error,{'EXIT',{noproc,{sys,get_status,[<6453.14610.13>]}}}}},[{erl_eval,expr,3,[]}]}}
>
> I've seen this a couple of times now (erlang 17.x) when upgrading
> production systems under load, even with a trivial relup. No idea what that
> pid was.
>
> I think it might be a race in release_handler_1 where it calls
> sys:get_status without a catch, when the process in question may have been
> a supervision tree that had legitimately shut down since the list of pids
> was fetched.
>
> ie:
>
>
> https://github.com/erlang/otp/blob/OTP-17.5.6.3/lib/sasl/src/release_handler_1.erl#L589
>
> which calls get_proc_state, which does:
>
> {status, _, {module, _}, [_, State, _, _, _]} = sys:get_status(Proc)
>
> I've not managed to make a test for this yet, planning to spam lots of
> terminate_childs to a busy supervisor while calling
> release_handler_1:get_supervised_procs to try and reproduce.
>
> If i'm right, it would only be triggered if parts of a supervision tree
> are shutting down during a release_upgrade, which perhaps isn't very
> common, depending on how dynamic the average supervision tree is in erlang
> apps.
>
> Any feedback appreciated before I spend more time studying release handler
> code :)
>
> RJ
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20150902/2db1e2e4/attachment.htm>