[erlang-questions] Re: Unicast 20k messages, $500-$1000 bounty

[Paulo Sérgio Almeida]

Joel Reymont wrote:

> I traded the time taken to broadcast to a list over the time and effort 
> taken to subscribe and unsubscribe. Setup time is much longer now since 
> inserting, deleting and searching a tuple list of 20k items is garbage 
> collection galore. Traversal time of 617.352ms, 645.084ms, 871.228ms and 
> 697.16ms beats anything we have seen from ETS so far and I'm only 
> looking at a maximum of 20k subscribers per server.
> 
> I think it's better to build the list over time than to build it from 
> ets at broadcasting time.

Have you already tried using select? In my very slow very old Dell X1 
with a 1.0GHz ULV processor, extracting the list of keys from a 20k 
entry ets table takes about 5 - 8 ms.

I tried extracting using ets:foldl and was shocked. It was even much 
slower than I expected: it varies between 30 - 80 ms.

I think you don't need to compromise and build the list before hand. If 
8ms is not good enough. The solution to this case is ...

the process dictionary

(I don't have anymore pacience to hear how a sacrilege it is to use it; 
some other day I could digress about Erlang philosophy, encapsulation 
and mutable state, ...)

For now I will just say that:

- If I have 100000 processes not each of them is a gen_*. I use gen_* 
sparingly, for the main services.

- Sometimes, for processes that provide a simple service, with little 
code, the process dictionary can be the right solution to store exactly 
this kind of information that would be otherwise stored in slower global 
ets tables, or even slower immutable containers (like dict).

In this case, in my slow laptop, extracting the entries of a 20k process 
dictionary takes about 2 - 3 ms.

Regards,
Paulo