[erlang-questions] Terminate process if it is slow at receiving messages

Fri Sep 20 21:29:51 CEST 2013

I transcribed the entire talk content:

http://ferd.ca/rtb-where-erlang-blooms.html

The library mentioned is dispcount: https://github.com/ferd/dispcount

It allows to shed load instantly, and is meant to be used in low-latency
cases.

I've recently written PO Box: https://github.com/ferd/pobox

This one implements bounded queues and stacks for Erlang processes,
where a process middleman acts like a buffer from which you can filter
data and whatnot, and do batching at the same time. It might be somewhat
appropriate for what you need, although it won't do much to instantly
know what processes are too slow. It'll only shed load.

Regards,
Fred.

On 09/20, akonsu wrote:
> (The solution with a stack is basically what Michael Truog proposed as
> well, although he proposed a queue). Is there a reference to the Fred
> Hebert's talk at the Erlang Factory? I am new to erlang, and I do not see
> how I can in a single process receive messages, put them into another
> stack/queue and process them at the same time... spawn yet another process
> for each incoming message?
> 
> I also am not sure about sending timestamps with each message, because time
> is relative and different nodes have different clocks... besides it takes
> time for each message to get delivered, and I do not see how I can measure
> this time and disregard it.
> 
> 
> 2013/9/20 Robert Virding <robert.virding@REDACTED>
> 
> > If you are going to use timestamps the timer call you want to avoid
> > erlang:now() as its guarantees that each call generates a larger value than
> > the previous which requires synchronisation between all the schedulers in
> > Erlang VM. Somewhere I have a pathological example which goes slower the
> > more cores you use because of this.
> >
> > IIRC doing process_info(message-queue_len) also has it s problems as the
> > process does not keep a count of messages but counts them each time you
> > call. IIRC!
> >
> > An alternative solution is one presented at an Erlang Factory in San
> > Francisco by Fred Hebert used by the company he was working for. They
> > didn't want to kill the process but efficiently throw away messages that
> > had timed out. Their system did allowed them to process messages not in the
> > order they arrived and throw away messages that had timed out. Fred will
> > have to correct me here if I am wrong.
> >
> > The problem with having timestamps and receiving down the message queue
> > checking for timed out messages is that this takes time so you will lose
> > more messages because of this. Their solution was to push the messages on a
> > stack when they arrived then when you want to process a message you just
> > take the top one off the stack. If you get a timed out message then you
> > know that all those further down the stack are older so you can throw then
> > away without checking. IIRC even more they bounded the stack.
> >
> > You could augment this with a stack count so easily keep track if the
> > process is falling behind or not.
> >
> > You could use this to either kill the process or just throw away messages.
> >
> > Pretty nifty I think as it reverses how you normally expect to do this.
> >
> > Robert
> >
> >
> >
> > ----- Original Message -----
> > > From: "Michael Loftis" <mloftis@REDACTED>
> > > To: "akonsu" <akonsu@REDACTED>
> > > Cc: erlang-questions@REDACTED
> > > Sent: Friday, 20 September, 2013 8:29:09 PM
> > > Subject: Re: [erlang-questions] Terminate process if it is slow at
> > receiving  messages
> > >
> > > I never said go synchronous/use acks.  Just send a completely seperate
> > > timing message occasionally.  The mailbox queue is going to normally
> > > (well, depends a little on your code) be worked FIFO.  So throwing a
> > > timing message at the process (possibly even from a completely
> > > different process) occasionally to check it's health.
> > >
> > > On Fri, Sep 20, 2013 at 9:38 AM, akonsu <akonsu@REDACTED> wrote:
> > > > thanks. but this would generate twice as much traffic because of the
> > > > acknowledgement responses. Maybe instead of erlang messages I can use
> > > > sockets to communicate?
> > > >
> > > >
> > > > 2013/9/20 Michael Loftis <mloftis@REDACTED>
> > > >>
> > > >> Pretty simple actually...send them a message, perhaps with a timestamp
> > > >> (os:timestamp()) - the expectation is that the process responds back.
> > > >> If it does not at all within your window, kill it.  If it does, but is
> > > >> unacceptably slow, kill it.
> > > >>
> > > >> Not at all difficult.
> > > >>
> > > >> On Fri, Sep 20, 2013 at 8:08 AM, akonsu <akonsu@REDACTED> wrote:
> > > >> > Hello,
> > > >> >
> > > >> > I have posted this question to stackoverflow
> > > >> >
> > > >> > (
> > http://stackoverflow.com/questions/18895614/terminate-process-if-it-is-slow-at-receiving-messages
> > )
> > > >> > and received one helpful response, but I am still not clear about
> > the
> > > >> > right
> > > >> > way to do it.
> > > >> >
> > > >> >  I also have seen this thread:
> > > >> > http://erlang.org/pipermail/erlang-questions/2011-June/059335.html,
> > > >> > which is
> > > >> > somewhat similar but it still did not hep me.
> > > >> >
> > > >> > I would appreciate more help. Here is the question:
> > > >> >
> > > >> >
> > > >> >
> > > >> > I have a data source process that sends messages to worker
> > processes. To
> > > >> > keep memory consumption under control, I need to terminate the
> > workers
> > > >> > that
> > > >> > are slow retrieving their messages from their mailboxes.
> > > >> >
> > > >> > I am new to Erlang, I would appreciate any pointers. If this is
> > > >> > difficult to
> > > >> > accomplish with Erlang messages, maybe I can use sockets? If so, are
> > > >> > there
> > > >> > examples?
> > > >> >
> > > >> > EDIT:
> > > >> >
> > > >> > I have a registered process that reads from the web and generates a
> > lot
> > > >> > of
> > > >> > data. It sends these data to all the "subscribed" processes using
> > Erlang
> > > >> > messages. For each particular piece of data, it sends the same
> > message
> > > >> > to
> > > >> > all subscribers.
> > > >> >
> > > >> > I also have a web server that streams the data that the registered
> > > >> > process
> > > >> > reads. So, when a http client connects, the web server creates a
> > process
> > > >> > and
> > > >> > this process subscribes to the registered process and starts
> > receiving
> > > >> > its
> > > >> > messages.
> > > >> >
> > > >> > The registered process uses monitors to monitor subscribers. The
> > > >> > subscribers
> > > >> > are controlled by the web server, and when a connection is closed,
> > the
> > > >> > process that were serving this connection, dies.
> > > >> >
> > > >> > There is no acknowledgement, that is, subscribers do not respond
> > when a
> > > >> > message is sent to them. Although I can program them this way, but I
> > > >> > think
> > > >> > it is too much traffic.
> > > >> >
> > > >> > Basically I want to close the connection if a http client is too
> > slow.
> > > >> >
> > > >> > _______________________________________________
> > > >> > erlang-questions mailing list
> > > >> > erlang-questions@REDACTED
> > > >> > http://erlang.org/mailman/listinfo/erlang-questions
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >>
> > > >> "Genius might be described as a supreme capacity for getting its
> > > >> possessors
> > > >> into trouble of all kinds."
> > > >> -- Samuel Butler
> > > >
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > "Genius might be described as a supreme capacity for getting its
> > possessors
> > > into trouble of all kinds."
> > > -- Samuel Butler
> > > _______________________________________________
> > > erlang-questions mailing list
> > > erlang-questions@REDACTED
> > > http://erlang.org/mailman/listinfo/erlang-questions
> > >
> >

> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions