[erlang-questions] Process pool map/3 implementation

[Parnell Springmeyer]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Thanks for pointing that out - I tried looking for stuff that was
already out there but couldn't find anything. Suppose I should have
asked first!!

Dmitrii Dimandt <dmitrii@REDACTED> writes:

> There's also RabbitMQ's worker_pool, http://www.lshift.net/blog/2010/03
> /29/on-the-limits-of-concurrency-worker-pools-in-erlang,  and my feeble
> attempt at somewhat extending it: https://github.com/dmitriid/
> worker_pool
>
>     -----BEGIN PGP SIGNED MESSAGE-----
>     Hash: SHA1
>    
>     For a work project I have a large list (thousands of items) to
>     process
>     and at first built a "pmap" implementation as per Joe's book until
>     I
>     found the plists module (which is awesome btw).
>    
>     There is one glaring issue with the list -> subdivide -> spawn x
>     processes for n sublist items strategy; if an item in the sublist
>     takes
>     longer than all the other items it blocks the entire resource
>     allotment
>     until it is done.
>    
>     In most cases, the plists/pmap implementation works just fine
>     because
>     the items in the list probably don't take more than a few
>     milliseconds
>     to map the fun over. However, it does become an issue when that is
>     not
>     the case.
>    
>     So, I figured the next best strategy would be to implement a
>     process
>     pool since it would allow for slow running processes to continue
>     their
>     work while finished processes can die and new processes spawned
>     into the
>     pool ready for work - so none of the resources are sitting idle.
>    
>     Right now, my module isn't nearly as feature-complete as the plists
>     module is - this is only a drop in replacement for map. Please
>     submit
>     your criticisms and comments to me at this address.
>    
>     You may find the code on BitBucket: https://bitbucket.org/ixmatus/
>     ppool
>    
>     - --
>     Parnell "ixmatus" Springmeyer (http://ixmat.us)
>     -----BEGIN PGP SIGNATURE-----
>     Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
>     Comment: GPGTools - http://gpgtools.org
>    
>     iQEcBAEBAgAGBQJOJmKwAAoJEPvtlbpI1POL+asIAKPcR0SOw67hFwwIbmkf89sS
>     4+Zx9hx1V/+86OVtXcqcOY+yxNcHezNEKkw8z2XHmDAWbeOl3bbINFySRXbQVydV
>     854lArqCHRG+ZlJ6ZrgecXKf9mG8ldbK1InwEZWOVZBj63rhmloMaGiyTzmxA88S
>     7mDNS4uhhpvRT2znpnsWt1x12IAzeayV0hf5/BLjp+b5FMZPc9oSa4n5uzyA9AVW
>     +av6hyuFfK32lhxUb4u3bVMaHOf2n/YwJexS25+NODcpkI3BLXNkrmKwgz8Lv/sA
>     omKzKTiuhpa0vTM+TLI9pn82GCJLdD+ON9DDOFN4ww+BnmXjhykiicBQCg7yhtQ=
>     =GP7K
>     -----END PGP SIGNATURE-----
>     _______________________________________________
>     erlang-questions mailing list
>     erlang-questions@REDACTED
>     http://erlang.org/mailman/listinfo/erlang-questions
>
> ===================================
> Dmitrii Dimandt
> dmitrii@REDACTED
>
> ------------------------------------------------------------
> Erlang in Russian
> http://erlanger.ru/
>
> TurkeyTPS
> http://turkeytps.com/
> ------------------------------------------------------------
>
> LinkedIn: http://www.linkedin.com/in/dmitriid
> GitHub: https://github.com/dmitriid
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions

- -- 
Parnell "ixmatus" Springmeyer (http://ixmat.us)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org

iQEcBAEBAgAGBQJOJwh8AAoJEPvtlbpI1POLeEoIAJClEPTB6onqGk/1uPvZNW6M
KQNYEjFTr3gm5ZP7oiQNcdlLbjSoOXtHMok6eZxOz2sKzNOiIcDa+7pbEbVvybPD
8WncYqKMS65CQQNMyG5AXyIOtRuCygLQgbSeSUEjuAprGUhTid9aklIeCIDMY1d6
tmdleHZZhvBKtONxnFtKJ7u+a2CzywNRBZ4BaKB+ThfvMnjNSlEM1IswujV8T6by
vuld6eHtZu6410ksM05PK6FiDObdUa728/3E6BSz16ZSag0QYafC9BDHJ0S11eSS
uazgs8Au6VK94tKLm/XE8f1JSNBMRKknWtbfuAHKZd46TLtI0KCntaAaT5VLAWk=
=7RDO
-----END PGP SIGNATURE-----