[erlang-questions] Process pool map/3 implementation

Dmitrii Dimandt dmitrii@REDACTED
Wed Jul 20 10:07:32 CEST 2011

There's also RabbitMQ's worker_pool, http://www.lshift.net/blog/2010/03/29/on-the-limits-of-concurrency-worker-pools-in-erlang,  and my feeble attempt at somewhat extending it: https://github.com/dmitriid/worker_pool

> Hash: SHA1
> For a work project I have a large list (thousands of items) to process
> and at first built a "pmap" implementation as per Joe's book until I
> found the plists module (which is awesome btw).
> There is one glaring issue with the list -> subdivide -> spawn x
> processes for n sublist items strategy; if an item in the sublist takes
> longer than all the other items it blocks the entire resource allotment
> until it is done.
> In most cases, the plists/pmap implementation works just fine because
> the items in the list probably don't take more than a few milliseconds
> to map the fun over. However, it does become an issue when that is not
> the case.
> So, I figured the next best strategy would be to implement a process
> pool since it would allow for slow running processes to continue their
> work while finished processes can die and new processes spawned into the
> pool ready for work - so none of the resources are sitting idle.
> Right now, my module isn't nearly as feature-complete as the plists
> module is - this is only a drop in replacement for map. Please submit
> your criticisms and comments to me at this address.
> You may find the code on BitBucket: https://bitbucket.org/ixmatus/ppool
> - -- 
> Parnell "ixmatus" Springmeyer (http://ixmat.us)
> Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
> Comment: GPGTools - http://gpgtools.org
> 4+Zx9hx1V/+86OVtXcqcOY+yxNcHezNEKkw8z2XHmDAWbeOl3bbINFySRXbQVydV
> 854lArqCHRG+ZlJ6ZrgecXKf9mG8ldbK1InwEZWOVZBj63rhmloMaGiyTzmxA88S
> 7mDNS4uhhpvRT2znpnsWt1x12IAzeayV0hf5/BLjp+b5FMZPc9oSa4n5uzyA9AVW
> +av6hyuFfK32lhxUb4u3bVMaHOf2n/YwJexS25+NODcpkI3BLXNkrmKwgz8Lv/sA
> omKzKTiuhpa0vTM+TLI9pn82GCJLdD+ON9DDOFN4ww+BnmXjhykiicBQCg7yhtQ=
> =GP7K
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions

Dmitrii Dimandt

Erlang in Russian


LinkedIn: http://www.linkedin.com/in/dmitriid
GitHub: https://github.com/dmitriid

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20110720/ece5c3a6/attachment.htm>

More information about the erlang-questions mailing list