[erlang-questions] PostgreSQL driver

[Paul Guyot]

Le 7 mars 2013 à 17:00, Max Lapshin a écrit :

> What should I do this weekend? Take a beer, write postgresql erlang driver
> or just write my own nosql database?
> 
> It is really interesting: why are there so many forks? Is it possible to
> consolidate all efforts? Maybe it is impossible at all because different
> drivers implement different approaches of querying async database from
> async erlang?

:)

Likewise, there are many HTTP servers, several XML and JSON parsers, etc. For example, we mostly use mochiweb here, but we built our own HTTP client and after having used mochiweb for JSON parsing, we transitioned to jiffy.
There even are several UUID modules written in Erlang!

The reasons are probably diverse, including:
- the ease to implement a working network tool, as Erlang is particularly good at protocols. Indeed, you can get a simple working PostgreSQL driver in a weekend;
- the existence of a built-in, but unsatisfying implementation (inets for httpd, odbc for PostgreSQL);
- the lack of support from first authors and the inactivity of their projects;
- the fact that some features may eventually require a rewrite.

To illustrate this point, I did some digging before open-sourcing our driver. Initially, we used ODBC, as we try to favor OTP built-in solutions. Then, because ODBC didn't satisfy our needs, in 2009, we went with ejabberd's version of Christian Sunesson's driver. I remember doing a diff with jungerl's version to find out bug fixes from ProcessOne. This driver was the most popular at the time, but it lacked OTP-supervision as we do releases and code_changes. So we implemented that, as well a thin-layer resembling ODBC API and the API that we had with other database drivers we used. At that point, we could still call our implementation a fork, with a couple of modules untouched.

I believe we fixed several bugs in this driver and I might have filed a request at ProcessOne (I surely did submit patches to ProcessOne for eunit). It didn't handle latest PostgreSQL protocol if I remember properly. It also didn't handle timestamp well, so we imported some timestamp code from Will Grozer's driver. Entirely switching to Will Grozer's driver was not so much an option as it didn't feature OTP compliancy either. We eventually had to rewrite all the code to support fold/map/foreach, asynchronous cancellation and timeouts. This was quite deep as these features are based on a different sequencing of PostgreSQL backend protocol.

At every conversations on postgresql drivers, I wondered if we shall open source our own (which cannot really be considered a fork as of today) and decided so after Anton's comments that timeouts are hard. I don't mean we solved the issue, but we actually use query timeouts in our code using our driver, and it required parting from Will Grozer's and Christian Sunesson's code, which Anton didn't do with his fork.

Paul
-- 
Semiocast            http://semiocast.com/
+33.183627948 - 20 rue Lacaze, 75014 Paris