[erlang-questions] Non-blocking send in erlang: how to save memory when send one content to many people

[Rapsey]

My streaming server is in Erlang and uses this approach. Though a linked in
driver is not a good idea. I started with a linked in driver but it was
pretty much impossible to get it really stable. With NIFs you can use
resources for sockets and stream buffers. They work beautifully.

I have a NIF thread that listens on kqueue/epoll and communicates with a
gen_server. Sockets are resources. Communication from NIFs to gen_server is
simple since enif_send exists, communication from gen_server to the NIF
thread is done by pipes (global library read pipe and write pipe).

The Erlang process that is responsible for sending data to the sockets
holds a resource to the stream buffer and a list of socket resources. So
the actual loop that sends the data is in Erlang and it calls a NIF with
buffer and socket resource on every iteration.


Sergej

On Wed, Dec 14, 2011 at 11:48 PM, Max Lapshin <max.lapshin@REDACTED> wrote:

> I'm sending live stream to 4000+ users.
>
> Stream is saved in memory (in shared ETS table, in binaries) in chunks
> 500 KBytes each.
>
> Client comes and asks for next chunk.
>
> If I write server in C (or with linked-in driver), I whould just
> manage common shared circular buffer from which I will send data and
> remember
> client position in this buffer. If client is too slow to read from
> this buffer, it is disconnected and forgotten.
>
> Such approach give ability to save memory and not to copy it in
> buffers and driver queues.
>
> Is it possible to have something like this in erlang?
>
> Currently, I've got limits around 3 GBit/s from one erlang node on
> loopback interface and I want to raise this limit.
> When traffic comes to this limit, memory begins very fast growing and
> when it reaches limits of RAM, massive disconnects happen.
> This is why I think, that I should save memory first without trying to
> write linked-in tcp driver =)
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20111215/2eef976d/attachment.htm>