[erlang-questions] Process pool map/3 implementation
Parnell Springmeyer
ixmatus@REDACTED
Wed Jul 20 07:08:00 CEST 2011
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
For a work project I have a large list (thousands of items) to process
and at first built a "pmap" implementation as per Joe's book until I
found the plists module (which is awesome btw).
There is one glaring issue with the list -> subdivide -> spawn x
processes for n sublist items strategy; if an item in the sublist takes
longer than all the other items it blocks the entire resource allotment
until it is done.
In most cases, the plists/pmap implementation works just fine because
the items in the list probably don't take more than a few milliseconds
to map the fun over. However, it does become an issue when that is not
the case.
So, I figured the next best strategy would be to implement a process
pool since it would allow for slow running processes to continue their
work while finished processes can die and new processes spawned into the
pool ready for work - so none of the resources are sitting idle.
Right now, my module isn't nearly as feature-complete as the plists
module is - this is only a drop in replacement for map. Please submit
your criticisms and comments to me at this address.
You may find the code on BitBucket: https://bitbucket.org/ixmatus/ppool
- --
Parnell "ixmatus" Springmeyer (http://ixmat.us)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
iQEcBAEBAgAGBQJOJmKwAAoJEPvtlbpI1POL+asIAKPcR0SOw67hFwwIbmkf89sS
4+Zx9hx1V/+86OVtXcqcOY+yxNcHezNEKkw8z2XHmDAWbeOl3bbINFySRXbQVydV
854lArqCHRG+ZlJ6ZrgecXKf9mG8ldbK1InwEZWOVZBj63rhmloMaGiyTzmxA88S
7mDNS4uhhpvRT2znpnsWt1x12IAzeayV0hf5/BLjp+b5FMZPc9oSa4n5uzyA9AVW
+av6hyuFfK32lhxUb4u3bVMaHOf2n/YwJexS25+NODcpkI3BLXNkrmKwgz8Lv/sA
omKzKTiuhpa0vTM+TLI9pn82GCJLdD+ON9DDOFN4ww+BnmXjhykiicBQCg7yhtQ=
=GP7K
-----END PGP SIGNATURE-----
More information about the erlang-questions
mailing list