[erlang-questions] Process swarm vs Queues
lud
ludovic@REDACTED
Thu Jul 20 18:49:58 CEST 2017
Hi,
I would like some advice regarding the design of my system.
I have a couple of processes monitoring data endpoints on the internet,
and sending events in my system when data changes.
Then, I have a swarm of processes dedicated to handle those events.
These processes are different, it's not the same code that handles
changes for different resources types (comments, feeds,
The specific things :
* Processes register themselves to an ETS table to tell which resources
they monitor.
* Processes states are very important and I can't afford to lose them.
So the state is saved to disk after every handled event, and retrieved
on restart.
* Events are quite rare, about 10 per second, whereas I could have 1000
or 10,000 monitored resources.
At this point, I was thinking : why use a process for each resource and
have so much hibernating processes. Why not just use a job queue with 10
workers, receive an event, load the data from disk, handle the event,
save to disk ?
It seems correct to me. I started with processes because it feels
natural, but now I'm quite lost, I don't need all those idling
processes. I feel like I was just thinking OOP (SHAME ! Just kidding …).
How do you choose, why would you choose one of these two designs, or one
another ?
Thank you for reading
Best regards
Ludovic
More information about the erlang-questions
mailing list