[erlang-questions] Process swarm vs Queues

lud ludovic@REDACTED
Thu Jul 20 18:49:58 CEST 2017


Hi,

I would like some advice regarding the design of my system.

I have a couple of processes monitoring data endpoints on the internet, 
and sending events in my system when data changes.
Then, I have a swarm of processes dedicated to handle those events. 
These processes are different, it's not the same code that handles 
changes for different resources types (comments, feeds,

The specific things :
  * Processes register themselves to an ETS table to tell which resources 
they monitor.
  * Processes states are very important and I can't afford to lose them. 
So the state is saved to disk after every handled event, and retrieved 
on restart.
  * Events are quite rare, about 10 per second, whereas I could have 1000 
or 10,000 monitored resources.

At this point, I was thinking : why use a process for each resource and 
have so much hibernating processes. Why not just use a job queue with 10 
workers, receive an event, load the data from disk, handle the event, 
save to disk ?

It seems correct to me. I started with processes because it feels 
natural, but now I'm quite lost, I don't need all those idling 
processes. I feel like I was just thinking OOP (SHAME ! Just kidding …).

How do you choose, why would you choose one of these two designs, or one 
another ?

Thank you for reading

Best regards

Ludovic





More information about the erlang-questions mailing list