[erlang-questions] Re: Unicast 20k messages, $500-$1000 bounty

[Cliff Moon]

So it turns out that the optimizations which mattered most were:

compile everything with HiPE, including mochiweb
bump the process priority for publishing
have pubsub write directly to the socket during its loop

The last two send it over the top.  The reason that the wall clock time 
was so high during the ets fold was because it was getting hung up 
context switching processes.  Each message sent out ends up putting a 
process in the run queue and erlang was not allowing the fold to 
complete before context switching off of the pubsub process.  So bumping 
the priority level helps a lot with fold time, however it simply delays 
the pain of all of that context switch.  The message has to wend it's 
way through two processes before it finally gets written to a socket.  
This means that at least 3 context switches, two for erlang procs and 
one for the linked in tcp driver, must happen per publish for 20k 
messages.  So making pubsub write directly to the tcp socket is a big 
win in terms of absolute latency.  I think there are some more 
inefficiencies to tease out, but I'm unsure whether it can be gotten 
below 1s on a small instance.


Joel Reymont wrote:
>
> On Jul 16, 2009, at 6:18 PM, Cliff Moon wrote:
>
>> The problem you're seeing there is process scheduling overhead.  In 
>> the fun it sends a message and it's doing an ungodly amount of 
>> context switching.  The way to get around this is to bump the process 
>> priority level of the process which iterates the ets table.
>
>
> bumping priority to high before the broadcasting list comprehension 
> and lowering it after results in ~10ms broadcasting time and shaves 
> ~300ms off max latency. Loving it!
>
> Mac is at ~2s (one box for everything) and ec2 is ~2.7s when bots are 
> spread over 3 instances. It looks like process priority is a very 
> useful tool in one's toolbox.
>
> (debug@REDACTED)4> bot:test(flashbot, 20000, 
> 'ip-10-244-47-97', 8081).
>
> =INFO REPORT==== 16-Jul-2009::19:05:18 ===
> setup: 69312.72ms, good: 20000, bad: 0, run: 72269.01ms
>   315.0020ms | min
>   500.0000ms | 2019   -  10.10%
>  1000.0000ms | 3651   -  18.25%
>  1500.0000ms | 4433   -  22.17%
>  2000.0000ms | 5259   -  26.30%
>  2500.0000ms | 4057   -  20.29%
>  3000.0000ms | 581    -   2.90%
>  2713.3960ms | max
> ok
> (debug@REDACTED)5> bot:test(flashbot, 20000, 
> 'ip-10-244-47-97', 8081).
>
> =INFO REPORT==== 16-Jul-2009::19:08:37 ===
> setup: 65606.03ms, good: 20000, bad: 0, run: 68362.39ms
>   328.4420ms | min
>   500.0000ms | 1361   -   6.80%
>  1000.0000ms | 4231   -  21.15%
>  1500.0000ms | 4477   -  22.38%
>  2000.0000ms | 4490   -  22.45%
>  2500.0000ms | 4051   -  20.26%
>  3000.0000ms | 1390   -   6.95%
>  2716.1440ms | max
> ok
> (debug@REDACTED)6> bot:test(flashbot, 20000, 
> 'ip-10-244-47-97', 8081).
>
> =INFO REPORT==== 16-Jul-2009::19:10:45 ===
> setup: 99778.01ms, good: 19953, bad: 47, run: 104700.73ms
>   313.9300ms | min
>   500.0000ms | 1420   -   7.12%
>  1000.0000ms | 4553   -  22.82%
>  1500.0000ms | 3987   -  19.98%
>  2000.0000ms | 4777   -  23.94%
>  2500.0000ms | 4241   -  21.25%
>  3000.0000ms | 968    -   4.85%
>  3500.0000ms | 0      -   0.00%
>  4000.0000ms | 0      -   0.00%
>  4500.0000ms | 7      -   0.04%
>  4143.9490ms | max
> ok
>
> ---
> Mac hacker with a performance bent
> http://www.linkedin.com/in/joelreymont
>
>