[erlang-questions] process per task or not ?

[Ludovic Demblans]

Hi,

I'm not very experienced about architecture, but i would advise the 'jobs'
erlang application[1].

You can use it in your application to handle the workers and try
differents approaches. As a first step, you could (roughly or finely i
don't know) measure how muchs messages you can handle in a given time.

I hope this helps.

Lud

[1] https://github.com/esl/jobs


Le Fri, 15 Nov 2013 19:34:37 +0100, akonsu <akonsu@REDACTED> a écrit:

> Hello,
>
> I am looking for an advice on how to architect my system. I am just a
> beginner...
>
> I have a process that receives messages and this process needs to parse
> each message, extract some information from the parsed message, and then
> send this information to the subscribers.
>
> As I read somewhere, in a situation like this, people usually spawn a
> worker process per message, and let this worker do the work, and then  
> send
> the results to the publishers.
>
> I have an overwhelming amount of incoming messages (these are tweets  
> that I
> get from a twitter stream), and I am not experienced enough to be able to
> tell whether this worker-per-task approach is the right way to do it. I  
> am
> afraid that my application will be flooded with workers and eventually  
> can
> collapse.
>
> I was thinking that maybe I can use a worker process pool? A process pool
> seems unusual in an Erlang application, does it not? Although the "Learn
> yourself Erlang..." does have a chapter on building a pool like this.
>
> Or maybe rate-limit the number of incoming messages?
>
> I would really appreciate an advice or two.
>
> Thank you
> Konstantin


-- 
Utilisant le logiciel de courrier révolutionnaire d'Opera :
http://www.opera.com/mail/