[erlang-questions] trouble with erlang or erlang is a ghetto

[Alex Arnon]

On Sun, Jul 31, 2011 at 10:10 PM, Tim Watson <watson.timothy@REDACTED>wrote:

> >
> > I agree with the sentiment. Java will indeed usually get the least
> friction
> > for any implementation, which for the Enterprise-minded is usually reason
> > enough to use it for everything.
> > However, in many places one could find instances where building at least
> a
> > prototype in Erlang might be the Right Thing. Several back-end or even
> > front-end services I have built I would certainly have knocked up in
> Erlang
> > very rapidly, just as an idea to Show The Boss or even as a real
> deployment
> > candidate. In all these instances, the database was either Oracle or
> MS-SQL,
> > with reasonably low DB access performance expectations.
> > Now, like you said above, just dropping into another VM for whatever
> service
> > is going to "look bad". Certainly if you've really built it from scratch
> for
> > your pet project, but what if we were talking about something that's been
> > battle-proven? That maybe speaks Enterprise lingo? Maybe put a nice face
> on
> > it, for instance - add a RESTful JSON interface or something, as a
> > diversion? :) Like you said, the Enterprise is often willing to take a
> lot
> > of crap just to have something that looks "standard" - and deploying a
> Lean
> > JDBC Proxy can go down relatively easily.
> >
>
> I've written a few real production systems in Erlang, but they were
> all OSS applications and it was the right fit. Even in a telco, most
> BSS applications tend not to be written in Erlang. I'm not saying
> that's right or good - I personally would rather build something in
> Erlang than Java - but there we have it.
>
> >
> > Right.
> > Something that I'd use the JDBC Proxy Abomination suggested above is a
> > fallback. In that case, you're not expecting blazing performance -
> however,
> > you'll have a stable backend to bang your API out on. Gradually adding
> clean
> > Erlang backends can come later.
> >
>
> Yeah I get that it will work ok, it just feels a bit weird not to use
> existing native implementations that are known to work. ESL have a
> postgres driver which I'm assuming (?) is used in production
> applications. The database support for Zotonic is also very solid in
> my experience. As I said, a guy at work wrote a driver based Oracle
> back end and it's absolutely rock solid in production - the app quite
> data access centric and has never once failed in the last 3 years
> whilst it deals with a reasonable load (avg. few thousand
> regular/daily users with peak times dipping into five figures).
>
>
I agree, it does feel unnatural. However, in this case:
1. It's rather simple, easy to make consistent and stable.
2. It enables us to deliver immediate value to the customer... HELP I'VE GOT
ENTERPRISITIS!!!
But seriously:
2. It makes any and all RDBMS's available, immediately.
3. It's a fallback.



> >
> > I think that's spot on.
> > Neither should they change this approach.
> > If you want to leap forward in language/tool technology in the
> Enterprise,
> > there's always stuff like Scala (which is an excellent step up from Java,
> > and starting to get some Traction).
> >
>
> Like I said, we use Erlang at work where it's deemed appropriate.
> Nobody there thinks Erlang "is a ghetto" but they do see it as niche
> and the fact that they pay sub contractors a lot more for it than bog
> standard Java applications is something they think about a lot. Having
> said that IMHO the fact that these Erlang based systems never seem to
> go wrong is testament to what I call the "you get what you pay for"
> effect combined with "some tools are better than others". Just my
> opinion though.
>
> >
> > AFAICT, the existing APIs are similar to a severely cut-down JDBC API. So
> > nothing very surprising there - I'm guessing most (if not all) have been
> > built on a per-need basis.
>
> That's why I'd like to start by fronting a well established API - like
> the ESL postgres driver - just to see what's already there and build
> on it in the general case.
>
>
OK.


> > I agree that one should try to veer as much away from heavily-stateful
> APIs,
> > and this can probably be avoided.
>
> Yes - just avoiding exposing internal data structures by providing a
> handle is probably enough to do this. The various modules can actually
> return whatever (real state if required) but the API should just hide
> this by exporting opaque types only:
>
> -opaque connection_handle :: term().
> -export_type([connection_handle/0]).
>
> > However, since we ARE dealing with RDBMS's, some of the flavour of using
> > them is bound to come through the API - connection pooling, transactions
> and
> > error handling/reporting should definitely be part of any API that
> expects
> > to be widely useful. I'd also add an Erlang-y flavour to some operations,
> > like streaming of result sets, to work with the platform's grain and not
> so
> > much against it.
> >
>
> Yes all these things (pooling, transactions, etc) are necessary. I
> maintain that you can provide them to the user without the
> implementation details leaking though.
>
> I also agree with the notion of having operations that are Erlang-y.
> Not sure what the best thing to do with streaming result sets would
> be, but I suspect looking at the existing implementations would
> clarify what people expect to see. For sure you don't always want to
> have to wait until the entire result set has been processed before you
> can do anything. I do think having operations that convert result sets
> using some transformation fun/function would be useful though - you
> might think of there as the Erlang equivalent of the Spring Framework
> JdbcTemplate ResultSetConverter or RowMapper interfaces. Except in
> Erlag you just a function, plain and simple.
>
>
Right.
One beneficial side effect of streaming result sets is that it enables
processing enormous sets without blowing up your process/node.
Transformation/map/reduce/whichever is probably a good starting point.


> Is your JDBC thing open source? I might have a stab at this just for
> fun - maybe fronting your library and the ESL postgres API as the
> first cut.
>
>
It was a prototype, got to the
usable-but-hell-no-you're-not-seeing-this-pile stage. :)
I don't think I've even updated my main repo with the last changes in a few
months.

Basically what I did was:
1. register a global controller (edbc_master). This is optional, meant to be
used for general monitoring of the connection processes.
2. Each connection is an {Erlang process, JNode} pair, linked to its
creator/user and registered with 'edbc_master'.
    The connection is started with a configuration such as:
    { { proxy_dir, "c:/dev/edbc/java/deploy" },
      { driver_class, "com.oracle.xxx.yyy.OracleDriver" },
      { hostname, "localhost" },
      { port, 4444 },
      ...
      other connection-stringy stuff
    }
3. The Controller process (the linked Erlang process) gets a "unique" node
name for the JNode from the edbc_master, and spawns it.
4. A basic handshake with X-second timeout between the Controller process
and a counterpart on the JNode (registered as 'edbc_proxy' on the JNode I
think) is done, including adding monitors on success.
5. From this point on, operations are forwarded via the Controller to the
JNode, which executes them, packages up the results and sends them back.

What I've done is somewhat crude - a process per connection etc., no proper
support for most datatypes - but it was just a couple of days' work to get a
simple SELECT/UPDATE/INSERT working. And it would work on Oracle, MSSql,
Postgres, MySQL, Sybase, what have you.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20110801/ae5acabb/attachment.htm>