Detect pid reuse

[Dinislam Salikhov]

>>> How many processes are you spawning per second?
>> ~2000 per second per node. The most of them live a few seconds or less.
> Do you mean ~2000 database connection processes crashed per second per node?

No. I mean that 2000 processes are spawned overall  A fraction of them is connection processes, indeed; but the most are processes not related to database directly (a process per client connection, for example).
________________________________________
From: Yao Bao <by@REDACTED>
Sent: Thursday, July 9, 2020 2:36 PM
To: Dinislam Salikhov
Cc: Jan Chochol; erlang-questions@REDACTED
Subject: Re: Detect pid reuse

> 在 2020年7月9日，19:14，Dinislam Salikhov <Dinislam.Salikhov@REDACTED> 写道：
>
> Hi Jan,
>
>> How many processes are you spawning per second?
>
> ~2000 per second per node. The most of them live a few seconds or less.

Do you mean ~2000 database connection processes crashed per second per node?

This means each database connection process do something and crash a few seconds later.

>
> According to the docs, the maximum is 268435456 processes. With our spawn rate it takes ~37 hours to use each pid at least once (if pids are used in round robin manner). We set the limit less than the maximum and for our settings it takes ~4 hours to use each pid at least once.
>
>> Erlang is trying quite hard to make PID unique (there is a very small probability to hit PID reuse) - isn't it possible, that there is a problem somewhere else?
>
> It took me quite a long time to find out the culprit. And it was pure luck that I could.
> We have gen_fsm processes (along with other gen_*) and they print a message they receive in handle_info if the message is unexpected. And right after the logs preceding the gen_server:call(DB, CallArgs, infinity), there was a log of a gen_fsm that {'$gen_call', CallArgs} is unexpected. Pid of gen_server (DB) and pid of gen_fsm are the same (they are logged). gen_server:call/3 hung (as gen_fsm couldn't answer it), so I had time to investigate the hanging process. So everything points to pid reuse.
>
> Dinislam Salikhov
> ________________________________________
> From: Jan Chochol <jan.chochol@REDACTED>
> Sent: Thursday, July 9, 2020 1:35 PM
> To: Dinislam Salikhov
> Cc: erlang-questions@REDACTED
> Subject: Re: Detect pid reuse
>
> Hi Dinislam,
>
> How many processes are you spawning per second?
> We (in our biggest production cluster) are spawning more than 10000
> processes per second with very variable lifetime (from milliseconds to
> hours) on the system running for more than a year and never faced a
> problem with PID reuse (our logic also depends on PID uniqueness).
> Erlang is trying quite hard to make PID unique (there is a very small
> probability to hit PID reuse) - isn't it possible, that there is a
> problem somewhere else?
>
> If you need more uniqueness you can use e.g. "erlang:make_ref/1" (same
> thing is used by "gen:call" to connect requests with responses) as
> your suggested token - I am not aware about any other workaround.
> You can also try experimenting with the size of the Erlang process
> table- it can affect the probability of PID reuse.
>
> Jan Chochol
>
>> On Thu, Jul 9, 2020 at 11:03 AM Dinislam Salikhov
>> <Dinislam.Salikhov@REDACTED> wrote:
>>
>> Unfortunately, registering a process with a name doesn't help much. It reduces a time window where the race may occur though.
>> For instance, when gen_server:call/3 is invoked, the library code calls whereis(Name) to get the pid and then sends it a message {'$gen_call,...}. So between erlang:whereis/1 and erlang:send/2, the pid may be reused (actually, it is between erlang:whereis/1 and erlang:monitor/2 followed by erlang:send/2, so we will monitor the wrong process).
>> See lib/stdlib/src/gen.erl which is used by lib/stdlib/src/gen_server.erl
>>
>>> If you have multiple connections to any given db (a pool of pools, if
>> you will), using a process group module like pg makes this easy.
>>
>> Never used it before. I'll have a look. Thanks for the reference.
>>
>> Dinislam Salikhov
>> ________________________________________
>> From: Aaron Seigo <aseigo@REDACTED>
>> Sent: Thursday, July 9, 2020 10:26 AM
>> To: Dinislam Salikhov
>> Cc: erlang-questions@REDACTED
>> Subject: Re: Detect pid reuse
>>
>>> On 2020-07-06 14:09, Dinislam Salikhov wrote:
>>> If I want to send a command to the database, I search for the pid of
>>> the corresponding connection (in supervisor's children list). And
>>
>> Perhaps register the processes with a name so that instead of searching
>> for a literal pid, which may indeed change and requires more bookkeeping
>> in your application code, you lookup the relevant connect by a name in a
>> process registry. Should the old connection go away, the new one takes
>> over the same name.
>>
>> If you have multiple connections to any given db (a pool of pools, if
>> you will), using a process group module like pg makes this easy.
>>
>> Even then, you'll obviously need to handle the failure case of the
>> process exiting between the message being sent and the response being
>> received, but at least the lookup will be consistent.
>>
>> --
>> Aaron Seigo
>