[erlang-questions] gproc scalability, shared ets, links and using terminate/2

[Max Lapshin]

As we know, gen_server:terminate/2 function is called
non-deterministically. You should not rely that it will be called. But
you may hope that it will be called.

And it has very important feature: it is called in separate process.

Now a bit about gproc. While moving erlyvideo to 10 GBit limit I've
experienced problems with scalability and using central tracking
process.

I want to claim that monitor(process, Pid) technology has problems. It
is very convenient in terms or reenterability and many other. But it
has one big feature and drawback:
it is impossible to call demonitor from other process.

Now look, what is happening. Request storm is beginning and lots of
user sessions are created. Then it is over or something happens and
they are going to close in a very short
period of time.

Thousands of {'DOWN' messages are going to central tracker and it lays
down, not possible to open new processes.

So, again: problem is in handling lots of DOWN messages in one central
process. We should divide and conquer this situation.

I've changed this schema a bit: use good old link/1 function and call
it in terminate/2 handler:


my_session.erl:

terminate(_, #session{session_id = Id}) ->
  gen_tracker:remove_me(flu_sessions, Id).

gen_tracker.erl:

remove_me(Zone, Id) when is_pid(Id) ->
  unlink(whereis(Zone)),
  delete_by_pid(Zone, Id).


This approach makes all modifications in client process, which is
still alive. gen_tracker will have to clean only those processes,
which have died due to kill reason.
Together with  find_or_open(Key, SpawnFun) approach it helped me a lot.