[erlang-questions] process per task or not ?

[akonsu]

thanks everyone for your help. it is clearer now.

I decided to just count my processes to make sure that I do not start more
than some maximum number.

I have one more question. Are monitors designed to be used efficiently with
a lot of very short lived processes? Are monitors inexpensive to set up? Is
this a normal erlang programming practice to use monitors for a lot of
short lived processes? Alternatively, I believe I can use spawn_linked to
start my processes, and send a message from each worker, when the worker
finishes (since otherwise I would only get a message if a linked process
crashes). How does this latter approach compare to using monitors?

thanks for help
konstantin



2013/11/15 akonsu <akonsu@REDACTED>

> Hello,
>
> I am looking for an advice on how to architect my system. I am just a
> beginner...
>
> I have a process that receives messages and this process needs to parse
> each message, extract some information from the parsed message, and then
> send this information to the subscribers.
>
> As I read somewhere, in a situation like this, people usually spawn a
> worker process per message, and let this worker do the work, and then send
> the results to the publishers.
>
> I have an overwhelming amount of incoming messages (these are tweets that
> I get from a twitter stream), and I am not experienced enough to be able to
> tell whether this worker-per-task approach is the right way to do it. I am
> afraid that my application will be flooded with workers and eventually can
> collapse.
>
> I was thinking that maybe I can use a worker process pool? A process pool
> seems unusual in an Erlang application, does it not? Although the "Learn
> yourself Erlang..." does have a chapter on building a pool like this.
>
> Or maybe rate-limit the number of incoming messages?
>
> I would really appreciate an advice or two.
>
> Thank you
> Konstantin
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20131116/1c46100f/attachment.htm>