[erlang-questions] trouble with erlang or erlang is a ghetto

[Tim Watson]

On 29 July 2011 22:27, Alex Arnon <alex.arnon@REDACTED> wrote:
> I had built a rather basic prototype that uses a jnode running a JDBC proxy.
> This approach would increase latency, but provide access to every RDBMS
> under the sun.

That really scares me, and I would hesitate to use it. If you planning
on conquering the enterprise, I suspect that jumping out of you main
language VM into another one just to do database access isn't going to
go down too well. Although I could be wrong, because honestly I've
seen big companies buy "All Kinds of Stuff" (TM).

Once I'd gone to that much trouble, I might as well write the whole
shebang in Java. In fact, I prefer picking "the right tool for the
job" for each part of a project and Erlang doesn't fit every niche so
sometimes turning to Java, Python and other things is the correct
choice. Erlang does have some decent open source libraries for doing
database access (to postgres at least) and I've seen a very good
closed source Oracle driver written as a port_driver using the OCL
libraries - quick and stable, but sadly not available to the general
public. The author claims he'll help do a rewrite when native
processes get added to Erlang. :)

>
> I think that what is truly missing is an agreed-upon, standard EDBC API
> definition. IMHO one that is based on JDBC could certainly be good enough.
> Once that is in place, implementation is just a detail :)
> Erlang Community, time to claim your place in the Enterprise!!!! :)
>

I kind of agree that a standard API would be good. Unless it comes
from Ericsson or gets into the OTP distro, it's not really going to be
a standard though. If someone starts one, they really need to get on
this list and push people for feedback. I'd also suggest making it as
thin a veneer as possible on top of the real library/application being
used, as this is less likely to introduce problems.

I think a big part of the reason why medium/large enterprises don't
tend to use Erlang as a core development language, actually has
nothing to do with technology. It's a resourcing issue. It's harder to
find real Erlang expertise (i.e., commercial experience) than it is to
find Java and/or .NET developers and that increases both cost and
risk. I certainly think that Erlang's productivity and other factors
(like its built-in support for making fault tolerant applications) can
actually outweigh the initial cost of hiring good developers, but
again this is a risk that many big businesses (which are mainly run by
accountants in my experience) won't take. Start-ups and niche sectors
are a completely different kettle of fish.

Finally, I agree that JDBC has some good patterns to follow, but it's
whole design is very Object Oriented (i.e., mutable state oriented)
and therefore not much of it is really a good fit for Erlang. Things
that might translate nicely as features are (IMHO):

1. Runtime discovery of drivers
2. Runtime discovery of data source(s) a la JNDI - this is very useful
in Java applications and something like it (maybe gproc based?) would
be nice
3. Standard API for dealing with connections (and possibly connection
pools, although I'm not sure I'd want to open that particular can of
worms myself)
4. Standard API for working with database metadata (e.g.,
INFORMATION_SCHEMA, system views and the like)
5. Standard API for working with result sets (e.g., moving/scrolling,
indexed access to columns in the current row, etc)
6. QLC queries for interacting with result sets?

Not sure if that last one is really a good idea - it's very late here
and I've had a busy week.

AFAICT good API design for Erlang doesn't usually involve exposing
internal data structures (records, etc) to the end user and any state
is usually hidden as much as possible. My assumption is that the
various API modules involved would usually just return an opaque
handle to the user, which is then used during further calls. I
certainly wouldn't go off using parameterised modules as they're not
supported. So you'd get something more like:

{ok, DSHandle} = edbc_datasource:get_named_datasource(postgres_test),
{ok, ConnHandle} = edbc_connection:open(DSHandle, [PerConnOptions]),
{ok, TxnHandle} = edbc_transaction:begin(ConnHandle, [{isolation,
read_committed]),
try
    %% fall back to normal (not prepared) statement when this doesn't work...
    {ok, StatementHandle} = ebdc_connection:prepare_statement(ConnHandle, SQL),
    {ok, StatementHandle} = edbc_statement:bind_named_params([{id,
12345}, {customer_name, "Mr Smith"}]),

    {ok, Cursor} = edbc_connection:execute_query(ConnHandle, StatementHandle),
    edbc_rowset:foreach(fun(Row, Col, Value) -> io:format("~p:~p =
~p~n", [Row, Col, Value]) end, Cursor),

    %% version of execute/2 that takes raw SQL instead of prepared statements
    edbc_connection:execute_update(ConnHandle, "INSERT INTO Food
VALUES ('Ice Cream', 'Vanilla')"),
    {ok, committed} = edbc_transaction:commit(TxnHandle)
catch
    _:_ -> {ok, aborted} = edbc_transaction:rollback(TxnHandle)
after
    {ok, closed} = edbc_connection:close(ConnHandle)


Probably a lot of these calls to specific modules could be hidden
behind a slightly higher level API:

{ok, Conn} = edbc:open_connection(postgres_test),
Results = edbc:map(Conn, fun do_with_each_cell/2, SqlQuery),
%% or
Tree = edbc:fold(Conn, Query,
    fun(Row, Col, Val, Acc) -> gb_trees:enter({Row, Col}, Val, Acc) end),
etc.....

You're right that having a good standard API against which all of the
various implementations could be used is a very nice idea. The
question is, how close are the existing libraries/applications to a
*nice* API (whatever that is) and how much work is involved in
bridging between them. This is why I'd like to see if the binding
between API and implementation can be generated - I'd like to avoid
having state in the API and let the underlying implementations deal
with their own states, servers, errors, etc.