[erlang-questions] Process pool map/3 implementation
Sat Jul 23 02:10:47 CEST 2011
But a pmap SHOULD start processes for all the elements in the list in parallel. It is after all a 'P'map. In which case all the processes will be running and processing in parallel as you want. The only reason I can see for using a worker pool is if you actually want to LIMIT the number of processes running at the same time.
IMAO in Erlang there are only two reasons for using worker/process pools:
- you want/need to limit the number of "things" running in parallel
- you actually do want to reuse a process for another computation, there is something in the application which mandates reusing processes.
Otherwise it is just extra work, process creation/termination is so fast that there is no real gain in keeping them around to reuse.
----- "Parnell Springmeyer" <ixmatus@REDACTED> wrote:
> Because the list has about 3000 items in it, and for each item about 20-50 HTTP requests are made; I needed a way of parallelizing the operations (instead of stepping through the list one by one) but in a controlled fashion and using a round robin strategy (worker pool).
> On Fri, Jul 22, 2011 at 6:10 AM, David Mercer < dmercer@REDACTED > wrote:
I was curious about that, too. Hoping you'll get a response...
> > -----Original Message-----
> > From: erlang-questions-bounces@REDACTED [mailto: erlang-questions-
> > bounces@REDACTED ] On Behalf Of Robert Virding
> > Sent: Wednesday, July 20, 2011 8:42 PM
> > To: Parnell Springmeyer
> > Cc: erlang-questions
> > Subject: Re: [erlang-questions] Process pool map/3 implementation
> > One quick question: what was wrong with the straightforward solution of
> > just spawning one process for each element in the list? Did this break
> > or do you actually need more control?
> > Robert
> > ----- "Parnell Springmeyer" < ixmatus@REDACTED > wrote:
> > > -----BEGIN PGP SIGNED MESSAGE-----
> > > Hash: SHA1
> > >
> > > For a work project I have a large list (thousands of items) to
> > > process
> > > and at first built a "pmap" implementation as per Joe's book until I
> > > found the plists module (which is awesome btw).
> > >
> > > There is one glaring issue with the list -> subdivide -> spawn x
> > > processes for n sublist items strategy; if an item in the sublist
> > > takes
> > > longer than all the other items it blocks the entire resource
> > > allotment
> > > until it is done.
> > >
> > > In most cases, the plists/pmap implementation works just fine because
> > > the items in the list probably don't take more than a few
> > > milliseconds
> > > to map the fun over. However, it does become an issue when that is
> > > not
> > > the case.
> > >
> > > So, I figured the next best strategy would be to implement a process
> > > pool since it would allow for slow running processes to continue
> > > their
> > > work while finished processes can die and new processes spawned into
> > > the
> > > pool ready for work - so none of the resources are sitting idle.
> > >
> > > Right now, my module isn't nearly as feature-complete as the plists
> > > module is - this is only a drop in replacement for map. Please submit
> > > your criticisms and comments to me at this address.
> > >
> > > You may find the code on BitBucket:
> > > https://bitbucket.org/ixmatus/ppool
> > >
> > > - --
> > > Parnell "ixmatus" Springmeyer ( http://ixmat.us )
> > > -----BEGIN PGP SIGNATURE-----
> > > Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
> > > Comment: GPGTools - http://gpgtools.org
> > >
> > > iQEcBAEBAgAGBQJOJmKwAAoJEPvtlbpI1POL+asIAKPcR0SOw67hFwwIbmkf89sS
> > > 4+Zx9hx1V/+86OVtXcqcOY+yxNcHezNEKkw8z2XHmDAWbeOl3bbINFySRXbQVydV
> > > 854lArqCHRG+ZlJ6ZrgecXKf9mG8ldbK1InwEZWOVZBj63rhmloMaGiyTzmxA88S
> > > 7mDNS4uhhpvRT2znpnsWt1x12IAzeayV0hf5/BLjp+b5FMZPc9oSa4n5uzyA9AVW
> > > +av6hyuFfK32lhxUb4u3bVMaHOf2n/YwJexS25+NODcpkI3BLXNkrmKwgz8Lv/sA
> > > omKzKTiuhpa0vTM+TLI9pn82GCJLdD+ON9DDOFN4ww+BnmXjhykiicBQCg7yhtQ=
> > > =GP7K
> > > -----END PGP SIGNATURE-----
> > > _______________________________________________
> > > erlang-questions mailing list
> > > erlang-questions@REDACTED
> > > http://erlang.org/mailman/listinfo/erlang-questions
> > _______________________________________________
> > erlang-questions mailing list
> > erlang-questions@REDACTED
> > http://erlang.org/mailman/listinfo/erlang-questions
> Parnell "ixmatus" Springmeyer ( http://ixmat.us )
> _______________________________________________ erlang-questions mailing list erlang-questions@REDACTED http://erlang.org/mailman/listinfo/erlang-questions
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the erlang-questions