[erlang-questions] process per task or not ?

[Michael Truog]

Spawning a short-lived process per message is done to make sure binaries get garbage collected quickly.  This can be important when your message handling does lots of binary work (e.g. with asn1).  CloudI (http://cloudi.org) does this by default with all service requests, but not info messages, though that can be changed with the service configuration options (so a request pid can be used any number of times, and an info pid can be used any number of times).

On 11/15/2013 11:00 AM, Sergej Jurecko wrote:
> A process pool is completely normal.
> Spawning a process per message is only a good option if processing is blocking in some way (sockets or gen_server:call).
> Rate limiting is always a good idea.
>
>
> Sergej
>
> On Nov 15, 2013, at 7:34 PM, akonsu wrote:
>
>> Hello,
>>
>> I am looking for an advice on how to architect my system. I am just a beginner...
>>
>> I have a process that receives messages and this process needs to parse each message, extract some information from the parsed message, and then send this information to the subscribers.
>>
>> As I read somewhere, in a situation like this, people usually spawn a worker process per message, and let this worker do the work, and then send the results to the publishers.
>>
>> I have an overwhelming amount of incoming messages (these are tweets that I get from a twitter stream), and I am not experienced enough to be able to tell whether this worker-per-task approach is the right way to do it. I am afraid that my application will be flooded with workers and eventually can collapse.
>>
>> I was thinking that maybe I can use a worker process pool? A process pool seems unusual in an Erlang application, does it not? Although the "Learn yourself Erlang..." does have a chapter on building a pool like this.
>>
>> Or maybe rate-limit the number of incoming messages?
>>
>> I would really appreciate an advice or two.
>>
>> Thank you
>> Konstantin
>>
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>