[erlang-questions] Best approach to creating a process/connection pool

[Andrew Berman]

Thanks Ulf.  That seems to be way more complicated than what i
need....I think.  I actually created a pool application here:
https://github.com/aberman/pooly

Every pooling app I found or was suggested didn't really take into
account certain factors that most connection pools use, such as idle
timeout, maximum age of a connection, acquire increment, initial
number of connections to start, etc.  I wrote my pool with these
factors in mind and made them configurable in an external file.  It
was written for Riak, but it can easily be adapted to support any
gen_server.  Whether I did things properly, I don't know, but it seems
to work how I want it.

Thanks for your help,

Andrew

On Mon, Aug 8, 2011 at 11:32 PM, Ulf Wiger
<ulf.wiger@REDACTED> wrote:
>
> The Jobs load regulation system supports a pool concept. Actually, it allows
> you to combine concepts in a number of ways to tune characteristics.
> Here is one example, with a 'producer pool' and a passive queue. The
> producer pool is defined with a fun that is called to top up the pool. The
> producers are free to do pretty much what they want, and jobs will monitor
> them to ensure they are replaced if they die.
> https://gist.github.com/778319
> In this case, the producers fetch jobs from a passive queue. In Jobs, that's
> a queue where you have to explicitly dequeue them. Otherwise, you typically
> tell jobs at which rate it should draw from the queue.
> Rates and pool sizes can be made to adapt automatically to load, using load
> sampler plugins. Configuration/re-configuration can also be done at runtime
> through the Jobs API.
> http://github.com/esl/jobs
> BR,
> Ulf W
>
> On 8 Aug 2011, at 21:44, Joel Meyer wrote:
>
> If the things you are pooling are gen_servers, gen_server_pool may work for
> you (https://github.com/joelpm/gen_server_pool). That's a slight
> modification on a version I've used in production to pool thrift and
> protobuff conns to back-ends. To start a pool you'd simply do
> gen_server_pool:startlink( ..., GenServerPoolArgs). instead of doing
> gen_server:start_link(...). You would use the returned PID in the same way.
> I haven't had time to test that version, though, so YMMV.
> Joel
>
> 2011/8/4 Andrew Berman <rexxe98@REDACTED>
>>
>> Awesome!  This makes sense.  Thank you for the clarification on this.
>>
>> 2011/8/4 Frédéric Trottier-Hébert <fred.hebert@REDACTED>:
>> > Technically, the OTP supervision tree is also used for application
>> > upgrades (appups & relups). When sending messages to do the broad reloading
>> > of code, the code in charge of that just traverses the VM's supervision
>> > trees asking the supervisors what kind of modules they have and whether
>> > they're workers and supervisors (that's why the child specs contain details
>> > about workers, supervisors, [Modules] (and dynamic sending special messages
>> > to figure out the same info).
>> >
>> > If you want to use appups/relups, you have to use the OTP supervision
>> > trees otherwise you will have processes left behind to be purged rather than
>> > upgraded.
>> >
>> > --
>> > Fred Hébert
>> > http://www.erlang-solutions.com
>> >
>> >
>> >
>> > On 2011-08-04, at 13:39 PM, Hynek Vychodil wrote:
>> >
>> >> Using OTP supervisor tree is intended only for control of restarting
>> >> application parts. If you want anything else you have to use links or
>> >> monitors unless you write your own supervisor which is not recommended
>> >> way.
>> >>
>> >> On Thu, Aug 4, 2011 at 6:35 PM, Andrew Berman <rexxe98@REDACTED>
>> >> wrote:
>> >>> Thanks Tim.  So it looks like they are using erlang:monitor and
>> >>> demonitor, so is that the best approach in an OTP application?  How
>> >>> does it figure into the supervision tree?
>> >>>
>> >>>
>> >>>
>> >>> On Thu, Aug 4, 2011 at 3:22 AM, Tim Watson <watson.timothy@REDACTED>
>> >>> wrote:
>> >>>> Take a look at the sources for http://zotonic.com/ as they use a
>> >>>> decent connection pool.
>> >>>>
>> >>>> On 4 August 2011 08:01, Andrew Berman <rexxe98@REDACTED> wrote:
>> >>>>> I'm fairly new to Erlang and I am trying to write a process pool
>> >>>>> which
>> >>>>> essentially is a connection pool to a database.  I have designed the
>> >>>>> application like this:  The main pool gen_server has a queue of PIDs
>> >>>>> of gen_fsm's.  Each gen_fsm calls spawn_links a 3rd party gen_server
>> >>>>> and carries
>> >>>>> the state of the connection.  So the 3rd party gen_server and the
>> >>>>> gen_fsm should always live and die together.  I currently have set
>> >>>>> it
>> >>>>> up such that gen_fsm calls proc_lib:spawn_link() to start the 3rd
>> >>>>> party gen_server, but I'm wondering how this factors into the
>> >>>>> supervision tree.  I currently have two supervisors:
>> >>>>>
>> >>>>> Sup1 -> one_for_all -> 2 children: pool gen_server and Sup2
>> >>>>> Sup2 -> simple_one_for_one -> 1 child: gen_fsm
>> >>>>>
>> >>>>> So my question is: is it ok to use spawn_link instead of somehow
>> >>>>> trying to put the 3rd party gen_server into the supervision tree?
>> >>>>>  If
>> >>>>> it should go into the supervision tree, how would I organize the
>> >>>>> tree
>> >>>>> then such that the gen_fsm and 3rd party gen_server are reliant on
>> >>>>> each other?
>> >>>>>
>> >>>>> Thanks for any help!
>> >>>>> _______________________________________________
>> >>>>> erlang-questions mailing list
>> >>>>> erlang-questions@REDACTED
>> >>>>> http://erlang.org/mailman/listinfo/erlang-questions
>> >>>>>
>> >>>>
>> >>> _______________________________________________
>> >>> erlang-questions mailing list
>> >>> erlang-questions@REDACTED
>> >>> http://erlang.org/mailman/listinfo/erlang-questions
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> --Hynek (Pichi) Vychodil
>> >>
>> >> Analyze your data in minutes. Share your insights instantly. Thrill
>> >> your boss.  Be a data hero!
>> >> Try GoodData now for free: www.gooddata.com
>> >> _______________________________________________
>> >> erlang-questions mailing list
>> >> erlang-questions@REDACTED
>> >> http://erlang.org/mailman/listinfo/erlang-questions
>> >
>> >
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
> Ulf Wiger, CTO, Erlang Solutions, Ltd.
> http://erlang-solutions.com
>
>
>