[erlang-questions] trouble with erlang or erlang is a ghetto

[Alex Arnon]

On Sat, Jul 30, 2011 at 3:44 AM, Tim Watson <watson.timothy@REDACTED>wrote:

> On 29 July 2011 22:27, Alex Arnon <alex.arnon@REDACTED> wrote:
> > I had built a rather basic prototype that uses a jnode running a JDBC
> proxy.
> > This approach would increase latency, but provide access to every RDBMS
> > under the sun.
>
> That really scares me, and I would hesitate to use it. If you planning
> on conquering the enterprise, I suspect that jumping out of you main
> language VM into another one just to do database access isn't going to
> go down too well. Although I could be wrong, because honestly I've
> seen big companies buy "All Kinds of Stuff" (TM).
>
>
I assure you, I am not planning on conquering any Enterprises :)
However, I disagree with the above... please see below.


> Once I'd gone to that much trouble, I might as well write the whole
> shebang in Java. In fact, I prefer picking "the right tool for the
> job" for each part of a project and Erlang doesn't fit every niche so
> sometimes turning to Java, Python and other things is the correct
> choice. Erlang does have some decent open source libraries for doing
> database access (to postgres at least) and I've seen a very good
> closed source Oracle driver written as a port_driver using the OCL
> libraries - quick and stable, but sadly not available to the general
> public. The author claims he'll help do a rewrite when native
> processes get added to Erlang. :)
>
>
I agree with the sentiment. Java will indeed usually get the least friction
for any implementation, which for the Enterprise-minded is usually reason
enough to use it for everything.
However, in many places one could find instances where building at least a
prototype in Erlang might be the Right Thing. Several back-end or even
front-end services I have built I would certainly have knocked up in Erlang
very rapidly, just as an idea to Show The Boss or even as a real deployment
candidate. In all these instances, the database was either Oracle or MS-SQL,
with reasonably low DB access performance expectations.
Now, like you said above, just dropping into another VM for whatever service
is going to "look bad". Certainly if you've really built it from scratch for
your pet project, but what if we were talking about something that's been
battle-proven? That maybe speaks Enterprise lingo? Maybe put a nice face on
it, for instance - add a RESTful JSON interface or something, as a
diversion? :) Like you said, the Enterprise is often willing to take a lot
of crap just to have something that looks "standard" - and deploying a Lean
JDBC Proxy can go down relatively easily.


> >
> > I think that what is truly missing is an agreed-upon, standard EDBC API
> > definition. IMHO one that is based on JDBC could certainly be good
> enough.
> > Once that is in place, implementation is just a detail :)
> > Erlang Community, time to claim your place in the Enterprise!!!! :)
> >
>
> I kind of agree that a standard API would be good. Unless it comes
> from Ericsson or gets into the OTP distro, it's not really going to be
> a standard though. If someone starts one, they really need to get on
> this list and push people for feedback. I'd also suggest making it as
> thin a veneer as possible on top of the real library/application being
> used, as this is less likely to introduce problems.
>
>
Right.
Something that I'd use the JDBC Proxy Abomination suggested above is a
fallback. In that case, you're not expecting blazing performance - however,
you'll have a stable backend to bang your API out on. Gradually adding clean
Erlang backends can come later.


> I think a big part of the reason why medium/large enterprises don't
> tend to use Erlang as a core development language, actually has
> nothing to do with technology. It's a resourcing issue. It's harder to
> find real Erlang expertise (i.e., commercial experience) than it is to
> find Java and/or .NET developers and that increases both cost and
> risk. I certainly think that Erlang's productivity and other factors
> (like its built-in support for making fault tolerant applications) can
> actually outweigh the initial cost of hiring good developers, but
> again this is a risk that many big businesses (which are mainly run by
> accountants in my experience) won't take. Start-ups and niche sectors
> are a completely different kettle of fish.
>
>
I think that's spot on.
Neither should they change this approach.
If you want to leap forward in language/tool technology in the Enterprise,
there's always stuff like Scala (which is an excellent step up from Java,
and starting to get some Traction).


> Finally, I agree that JDBC has some good patterns to follow, but it's
> whole design is very Object Oriented (i.e., mutable state oriented)
> and therefore not much of it is really a good fit for Erlang. Things
> that might translate nicely as features are (IMHO):
>
> 1. Runtime discovery of drivers
> 2. Runtime discovery of data source(s) a la JNDI - this is very useful
> in Java applications and something like it (maybe gproc based?) would
> be nice
> 3. Standard API for dealing with connections (and possibly connection
> pools, although I'm not sure I'd want to open that particular can of
> worms myself)
> 4. Standard API for working with database metadata (e.g.,
> INFORMATION_SCHEMA, system views and the like)
> 5. Standard API for working with result sets (e.g., moving/scrolling,
> indexed access to columns in the current row, etc)
> 6. QLC queries for interacting with result sets?
>
> Not sure if that last one is really a good idea - it's very late here
> and I've had a busy week.
>
> AFAICT good API design for Erlang doesn't usually involve exposing
> internal data structures (records, etc) to the end user and any state
> is usually hidden as much as possible. My assumption is that the
> various API modules involved would usually just return an opaque
> handle to the user, which is then used during further calls. I
> certainly wouldn't go off using parameterised modules as they're not
> supported. So you'd get something more like:
>
> {ok, DSHandle} = edbc_datasource:get_named_datasource(postgres_test),
> {ok, ConnHandle} = edbc_connection:open(DSHandle, [PerConnOptions]),
> {ok, TxnHandle} = edbc_transaction:begin(ConnHandle, [{isolation,
> read_committed]),
> try
>    %% fall back to normal (not prepared) statement when this doesn't
> work...
>    {ok, StatementHandle} = ebdc_connection:prepare_statement(ConnHandle,
> SQL),
>    {ok, StatementHandle} = edbc_statement:bind_named_params([{id,
> 12345}, {customer_name, "Mr Smith"}]),
>
>    {ok, Cursor} = edbc_connection:execute_query(ConnHandle,
> StatementHandle),
>    edbc_rowset:foreach(fun(Row, Col, Value) -> io:format("~p:~p =
> ~p~n", [Row, Col, Value]) end, Cursor),
>
>    %% version of execute/2 that takes raw SQL instead of prepared
> statements
>    edbc_connection:execute_update(ConnHandle, "INSERT INTO Food
> VALUES ('Ice Cream', 'Vanilla')"),
>    {ok, committed} = edbc_transaction:commit(TxnHandle)
> catch
>    _:_ -> {ok, aborted} = edbc_transaction:rollback(TxnHandle)
> after
>    {ok, closed} = edbc_connection:close(ConnHandle)
>
>
> Probably a lot of these calls to specific modules could be hidden
> behind a slightly higher level API:
>
> {ok, Conn} = edbc:open_connection(postgres_test),
> Results = edbc:map(Conn, fun do_with_each_cell/2, SqlQuery),
> %% or
> Tree = edbc:fold(Conn, Query,
>    fun(Row, Col, Val, Acc) -> gb_trees:enter({Row, Col}, Val, Acc) end),
> etc.....
>
> You're right that having a good standard API against which all of the
> various implementations could be used is a very nice idea. The
> question is, how close are the existing libraries/applications to a
> *nice* API (whatever that is) and how much work is involved in
> bridging between them. This is why I'd like to see if the binding
> between API and implementation can be generated - I'd like to avoid
> having state in the API and let the underlying implementations deal
> with their own states, servers, errors, etc.
>

AFAICT, the existing APIs are similar to a severely cut-down JDBC API. So
nothing very surprising there - I'm guessing most (if not all) have been
built on a per-need basis.
I agree that one should try to veer as much away from heavily-stateful APIs,
and this can probably be avoided.
However, since we ARE dealing with RDBMS's, some of the flavour of using
them is bound to come through the API - connection pooling, transactions and
error handling/reporting should definitely be part of any API that expects
to be widely useful. I'd also add an Erlang-y flavour to some operations,
like streaming of result sets, to work with the platform's grain and not so
much against it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20110731/c2720c2a/attachment.htm>