[erlang-questions] 20k messages in 4s but want to go faster!

[Joel Reymont]

On Jul 10, 2009, at 8:45 PM, Matthew Sackman wrote:

> Also, I have to say, I wouldn't use a dict. Immutable data  
> structures in
> Erlang are _slow_ when compared to, eg, ets. OTOH, if you've timed the
> fold and found it's only taking 0.2 secs, then that's clearly not the
> problem here.


I switched to ETS for my subscriber list. Two wit:

info: [{memory,14284},
        {owner,<0.305.0>},
        {heir,none},
        {name,subs},
        {size,1000},
        {node,'janus@REDACTED'},
        {named_table,false},
        {type,set},
        {keypos,1},
        {protection,protected}]
time: 26.076

info: [{memory,141821},
        {owner,<0.305.0>},
        {heir,none},
        {name,subs},
        {size,10000},
        {node,'janus@REDACTED'},
        {named_table,false},
        {type,set},
        {keypos,1},
        {protection,protected}]
time: 722.08

It's ~20ms to traverse 1000 subscribers, close to 1s for 10k and about  
2s for 20k.

It looks like around 30% of my 10k max latency is taken up by the  
traversal.

My code looks like this:

---
     Msg1 = {message, iolist_to_binary(mochijson2:encode(JSON))},
     F = fun({Pid, _}, _) -> gen_server:cast(Pid, Msg1) end,
     F1 = fun() ->
                  A = now(),
                  ets:foldr(F, ignore, State#state.subs),
                  io:format("time: ~p~n", [timer:now_diff(now(), A) /  
1000])
          end,
     spawn_link(F1),
---

I think it was faster with a dict, perhaps I should time it again.

It doesn't matter to me how long it takes to modify the data structure  
to add a new subscriber but traversal time does matter.

	Thanks, Joel

---
Mac hacker with a performance bent
http://www.linkedin.com/in/joelreymont