[erlang-questions] Scaling SSL wrapped websockets - experiences

[Dmitry Kolesnikov]

Hello,

There was posts at Internet about 2M connection per node. 
http://www.phoenixframework.org/blog/the-road-to-2-million-websocket-connections

I’ve scale it up to 1M connection on AWS long time ago. This numbers is “easy” to achieve out of box for system with low traffic. I had the following traffic pattern 20 connection per second (poisson arrival rate), connections stays idle but clients shows 3 second think time (poisson rate).   

The major concern was a memory all the time. The out of box deployment on AWS large instance was capable to handle 300K connections before OOM. I’ve use hibernate feature aggressively to reduce memory demand. Finally, I’ve used 2xlarge node to run the system. 

The latency was not an issue. The message scheduling was good while CPU utilization was below 70%. However, I’ve reduce logging to error-level and made the nodes responsible for message routing only. I’ve learn that you need to keep in balance CPU/Memory to get acceptable results but this depends on your traffic.

Unfortunately, that set-up excluded SSL. I was not successful to scale SSL up to 1M due to heavier memory demand (I’ve user OTP 18 for that problem). Unfortunately, I do not recall exactly the number of concurrent SSL connection (cannot find the data sheets). It was high >>20K but <<100M for sure.  

Frankly speaking, I would design a system around ELB to terminate SSL traffic and use weighted load balancing feature to scale ELB connectivity then rush for high concurrency on single node due to operational simplicity. 

I hope, I shed some light on your problem.   

Best Regards, 
Dmitry


> On Jul 2, 2017, at 12:56 PM, Paul Peregud <paulperegud@REDACTED> wrote:
> 
> I'm looking at SSL + websocket + cowboy scaling with regards to number
> of simultaneous connections. I'm interested in relationship between
> number of connections and latency of message delivery. Is there any
> data on performance of SSL implementation in BEAM?
> 
> Best regards,
> Paul Peregud
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions