[erlang-questions] process per task or not ?

[Sergej Jurecko]

A process pool is completely normal.
Spawning a process per message is only a good option if processing is blocking in some way (sockets or gen_server:call).
Rate limiting is always a good idea.


Sergej

On Nov 15, 2013, at 7:34 PM, akonsu wrote:

> Hello,
> 
> I am looking for an advice on how to architect my system. I am just a beginner...
> 
> I have a process that receives messages and this process needs to parse each message, extract some information from the parsed message, and then send this information to the subscribers.
> 
> As I read somewhere, in a situation like this, people usually spawn a worker process per message, and let this worker do the work, and then send the results to the publishers.
> 
> I have an overwhelming amount of incoming messages (these are tweets that I get from a twitter stream), and I am not experienced enough to be able to tell whether this worker-per-task approach is the right way to do it. I am afraid that my application will be flooded with workers and eventually can collapse.
> 
> I was thinking that maybe I can use a worker process pool? A process pool seems unusual in an Erlang application, does it not? Although the "Learn yourself Erlang..." does have a chapter on building a pool like this.
> 
> Or maybe rate-limit the number of incoming messages?
> 
> I would really appreciate an advice or two.
> 
> Thank you
> Konstantin
> 
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions