[erlang-questions] Non-blocking send in erlang: how to save memory when send one content to many people

[Zabrane Mickael]

Hi Sergej,


On Dec 15, 2011, at 8:55 AM, Rapsey wrote:

> My streaming server is in Erlang and uses this approach. Though a linked in driver is not a good idea. I started with a linked in driver but it was pretty much impossible to get it really stable. With NIFs you can use resources for sockets and stream buffers. They work beautifully.

Very interesting approcah.

> I have a NIF thread that listens on kqueue/epoll and communicates with a gen_server. Sockets are resources.

1 thread NIF per core?

> Communication from NIFs to gen_server is simple since enif_send exists, communication from gen_server to the NIF thread is done by pipes (global library read pipe and write pipe).

Could you please elaborae further on how communication is made from Erlang to NIF (the read/write pipe)?


Regards,
Zabrane


> The Erlang process that is responsible for sending data to the sockets holds a resource to the stream buffer and a list of socket resources. So the actual loop that sends the data is in Erlang and it calls a NIF with buffer and socket resource on every iteration.
> 
> 
> Sergej
> 
> On Wed, Dec 14, 2011 at 11:48 PM, Max Lapshin <max.lapshin@REDACTED> wrote:
> I'm sending live stream to 4000+ users.
> 
> Stream is saved in memory (in shared ETS table, in binaries) in chunks
> 500 KBytes each.
> 
> Client comes and asks for next chunk.
> 
> If I write server in C (or with linked-in driver), I whould just
> manage common shared circular buffer from which I will send data and
> remember
> client position in this buffer. If client is too slow to read from
> this buffer, it is disconnected and forgotten.
> 
> Such approach give ability to save memory and not to copy it in
> buffers and driver queues.
> 
> Is it possible to have something like this in erlang?
> 
> Currently, I've got limits around 3 GBit/s from one erlang node on
> loopback interface and I want to raise this limit.
> When traffic comes to this limit, memory begins very fast growing and
> when it reaches limits of RAM, massive disconnects happen.
> This is why I think, that I should save memory first without trying to
> write linked-in tcp driver =)
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
> 
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20111215/af693035/attachment.htm>