process waits forever spawn_opt/5

[Vyacheslav Levytskyy]

It's an old piece of code and the code is very simple: gen_server just 
spawns a function on several connected nodes. spawn_opt/5 is called by 
spawn/4, I don't call it directly.

The function doesn't interact with the gen_server that calls spawn/4, 
although I'd expect spawn/4 to run a process and return immediately 
anyway, am I wrong?

The code very literally looks like:

===

handle_cast(do_start, State) ->
     try
         my_start()
     catch
         What:Why:Stacktrace ->
             logger:warning("~p:~p~n~p", [What, Why, Stacktrace], #{mod 
=> ?MODULE, line => ?LINE})
     end,
     {noreply, State};

my_start() ->
     [spawn(Node, ?MODULE, my_action, [erlang:system_time(1)]) || Node 
<- [node() | my_cluster_nodes()]].

my_cluster_nodes() ->
     {L, _} = lists:unzip(ets:tab2list(?TBL)),
     L.

my_action(Now) ->
     % work with ETS, use an external database service

===


The code worked for a couple of years: I can't be 100% sure if this 
happened before or not, but in any case I see this situation for the 
first time. A recent change is upgrade to OTP24. The environment is 
Kubernetes, and the code runs inside docker container.

Thanks.

On 15.06.2021 22:12, Luke Bakken wrote:
> Hi Vyacheslav,
>
> It would be very helpful to share the code that spawn_opt is calling,
> or provide code to reproduce this condition.
>
> Does the function called by spawn_opt interact in _any_ way with the
> gen_server that calls spawn_opt?
>
> Thanks,
> Luke
>
> On Tue, Jun 15, 2021 at 1:09 PM Vyacheslav Levytskyy
> <v.levytskyy@REDACTED> wrote:
>> Hello,
>>
>> I'm surprised to see my gen_server process hanging forever when
>> executing spawn/4 call. Process info shows spawn_opt/5 as a current
>> function and status is waiting:
>>
>>   > process_info(P).
>> [{current_function,{erlang,spawn_opt,5}},
>>    {status,waiting},
>>    {message_queue_len,13},
>>    {trap_exit,false},
>>    {priority,normal},
>>    ...]
>>
>> Current stacktrace looks like:
>>
>>   > process_info(P, current_stacktrace).
>> {current_stacktrace,[{erlang,spawn_opt,5,[]},
>>                        {erlang,spawn,4,[]},
>>                        ...
>>                        {my_gen_server,handle_cast,2,
>>                                       [{file,"..."},
>>                                        {line,...}]},
>>                        {gen_server,try_dispatch,4,
>> [{file,"gen_server.erl"},{line,695}]},
>>                        {gen_server,handle_msg,6,
>> [{file,"gen_server.erl"},{line,771}]},
>>                        {proc_lib,init_p_do_apply,3,
>> [{file,"proc_lib.erl"},{line,226}]}]}
>>
>> I wonder what could be conditions that made spawn_opt/5 to stuck in this
>> state. Is there something I can do on my side to prevent such problems
>> in future?
>>
>> It's Erlang/OTP 24.0.2:
>>
>> Erlang/OTP 24 [erts-12.0.2] [source] [64-bit] [smp:8:8] [ds:8:8:10]
>> [async-threads:1] [jit]
>>
>> Thank you,
>> Vyacheslav
>>