[erlang-questions] [ANN] Asynchronous PostgreSQL driver

[Tim Watson]

On 8 November 2011 06:08, Anton Lebedevich <mabrek@REDACTED> wrote:

> On 11/02/2011 07:21 AM, Tim Watson wrote:
> > I've found the gen_servers that use cast heavily can easily get
> > overwhelmed without back pressure.
>
> I'm considering providing both kinds of API: call and cast.
> What's the better way to implement back pressure?
> Currently there are 3 queues: driver process inbox, driver internal
> queue of sent to backend requests, tcp send buffer.
> One kind of backpressure implementation might look like using
> gen_server:call that returns after calling gen_tcp:send. In this case
> slow database will eventually stop reading from tcp socket, then
> gen_tcp:send will hang, then gen_server:call will hang too.
>
>
Take a look at https://github.com/esl/jobs. Whilst you might not wish to
depend on it directly, it implements a pretty good model for wiring sensors
that can be used to regulate queue sizes. There are other solutions around
too, like pooling and so on. I think your idea of using the responsiveness
of the network might work quite well, although it's just gut feeling there
and I'm sure others on the list will have more practical experience to
share.


> > Now honestly, you're really missing a trick with this change. Does the
> > client (pid) get a message as soon as data comes off the wire and a row
> > has been successfully decoded? Because that is *far* more useful that
> > just happening not to block the client pid while you go off and fetch
> > everything in one go. If you either allow the client to pass a callback
> > or simply send a message for each DataRow, you will have provided a boon
> > for fast web applications.
>
> I've started experimenting with it in branch
> https://github.com/mabrek/epgsql/tree/row-callback
> Currently it's incomplete, passes only one test case.
>
>
Cool! I shall have a good play with this. Are you planning on benchmarking
your driver once it's more complete? IIRC the Tsung load testing tool
supports postgres and I wonder if perhaps one could tweak that to benchmark
the various postgres drivers based on a few stereotypical use cases?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20111108/954c32c0/attachment.htm>