[erlang-questions] gproc scalability, shared ets, links and using terminate/2

[Ulf Wiger]

It's true that link/1 has that advantage.

The disadvantage is that if the server crashes, and is linked to, say, 100K processes, the EXIT signal will be duplicated that many times, likely forcing an OOM crash.

(Been there, done that.)

BR,
Ulf W

On 9 Nov 2011, at 11:41, Max Lapshin wrote:

> As we know, gen_server:terminate/2 function is called
> non-deterministically. You should not rely that it will be called. But
> you may hope that it will be called.
> 
> And it has very important feature: it is called in separate process.
> 
> Now a bit about gproc. While moving erlyvideo to 10 GBit limit I've
> experienced problems with scalability and using central tracking
> process.
> 
> I want to claim that monitor(process, Pid) technology has problems. It
> is very convenient in terms or reenterability and many other. But it
> has one big feature and drawback:
> it is impossible to call demonitor from other process.
> 
> Now look, what is happening. Request storm is beginning and lots of
> user sessions are created. Then it is over or something happens and
> they are going to close in a very short
> period of time.
> 
> Thousands of {'DOWN' messages are going to central tracker and it lays
> down, not possible to open new processes.
> 
> So, again: problem is in handling lots of DOWN messages in one central
> process. We should divide and conquer this situation.
> 
> I've changed this schema a bit: use good old link/1 function and call
> it in terminate/2 handler:
> 
> 
> my_session.erl:
> 
> terminate(_, #session{session_id = Id}) ->
>  gen_tracker:remove_me(flu_sessions, Id).
> 
> gen_tracker.erl:
> 
> remove_me(Zone, Id) when is_pid(Id) ->
>  unlink(whereis(Zone)),
>  delete_by_pid(Zone, Id).
> 
> 
> This approach makes all modifications in client process, which is
> still alive. gen_tracker will have to clean only those processes,
> which have died due to kill reason.
> Together with  find_or_open(Key, SpawnFun) approach it helped me a lot.
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions

Ulf Wiger, CTO, Erlang Solutions, Ltd.
http://erlang-solutions.com