[erlang-questions] The future of Erlang and BEAM

[Tim Watson]

On 13 February 2012 17:53, Matthew Evans <mattevans123@REDACTED> wrote:

>  A well written email Tim,
>
> I would also add that with development packages such as Maven, building
> complex Java applications is a breeze. Now personally I think Maven is a
> royal pain in the ass (XML hell anyone), the ability to bring in
> dependancies in such a simple way is awesome for product packaging. I've
> even used it myself - since most of my team are Java developers - for
> Erlang deployments (see http://erlang-plugin.sourceforge.net/).
>
>
Yes indeed. Rebar does a reasonable job with dependencies, but having to
build them from source each time isn't ok from the perspective of a large
enterprise team with dozens of components making up each project. As we've
gradually started breaking up our libraries/applications to factor out
common code, this problem has grown. The team have adopted the maven erlang
plugin for exactly this reason, although personally whilst I don't mind
using maven/ivy for java projects, I'd much rather stick to rebar for
Erlang ones.

A binary artefact repository is the thing that seems to be missing, and
having it only available as a hosted thing (like Erlware or CEAN 1.0/2.0)
doesn't cut it for most shops with large development teams and
infrastructure to manage (we have a combination of in-house and out-sourced
engineers which I've heard (anecdotally) is around > 0.5 million
technologists of which about a quarter are developers - so I guess we
qualify as 'large').

Personally I think a good solution to this is to use the maven repository
structure to deploy artefacts into and fetch them from. That way you can
use a free (or commercial version) repository such as Sonatype Nexus for
such things internally and have public facing mirrors also. There are
complications (with having to use naming conventions to distinguish between
different target versions of beam and different target os/architectures
where drivers are present) but these aren't insurmountable. I'm quietly
working on a dependency manager (like ivy, not a package manager like
cean/agner) to do just this, as a rebar plugin.


> I think that what needs to be asked long term is where do we see Erlang
> been used? Will it just be a niche language, used in certain specific
> situations? Or do we see it becoming more of a general purpose language?
>

I hope the latter, although it's not fit for every domain and personally I
have no problem with multiple technologies mixed together, as long as the
architectural boundaries are cleanly defined (e.g., via messaging or
whatever integration approach fits the domain best).


>
> Of course, it's not ever going to beat Java, C++ or other
> OOP/imperative languages in terms of number of users, although it would be
> nice if it would. For one it's off to a bad start with it been functional
> language. However, unlike Haskell it's not a pure-functional language. Your
> code can contain little bits of imperative functionality; functions can
> have side effects for example. In other words it's not too hard to learn.
>
>
Coming from an Python + OCaml background, I found Java and .NET quite
awkward to learn! ;)


> I guess I can only speak from my own experience trying to get Erlang
> accepted into two businesses. Normally there is an existing code base, or a
> set of users who have been coding in language X all their career. These
> people are always very reluctant to embrace something new. Many of these
> people will think that it's going to threaten their future in the company.
> Many will actively lie and present massaged data to prove their point (I've
> seen this a few times - for example comparing a simple NGINX reverse proxy
> that is doing nothing to a full-blown Yaws web-server for example).
>
> Managers also have the same worry, but also need to look at what it means
> to retain staff and employ new staff.
>
> Erlang, unfortunately, often times faces an up-hill battle.
>

I've been through this too. I work in telecoms and unsurprisingly there is
a reasonable amount of Erlang based infrastructure, but absolutely nothing
existed in the (BSS) business applications space, where Java and .NET
dominate. That's gradually changing, but only because my team took the time
to build prototypes in our spare time and demonstrate the quality,
robustness, *performance* and maintenance benefits these
rewrites/alternatives delivered. We still run into a lot of road blocks,
and I think it'd be the same if we were pushing Ruby or Python as well.


>
> Something like Erjang can help here. I personally think it's non-ideal for
> most Erlang deployments, but can be used to get a foot in the door. Deploy
> Erjang on something that they are familiar with (JVM), have it "natively"
> talk to Java, and then introduce BEAM later.
>
>
IMO the bottom line is that a lot of people are loved-up with Java because
of the 'enterprise features' which is another way of saying 'all the stuff
that makes their life as a developer simple/easy.' Those of us who've used
Erlang in anger know that in reality, Java is far more complex than it
needs to be and Erlang is part of a group of much more productive
languages/platforms. But many more people would be willing to try if there
were

- standardised interfaces/APIs (a la Servlets, JDBC)
- better dependency management tools
- awesome build tools*

* IMO rebar is already *the* awesome build tool and I'm using it for all my
open source stuff. I think the other two points could (and maybe should??)
come from the community, but there's a lot of effort involved. Also Erlang
doesn't have a neat mechanism for defining an 'interface' that everyone can
follow. There's custom behaviours, which are great, but no simple way to
swap in and out of implementations cleanly and easily. Fixing that would be
*massively* beneficial to the platform IMHO, but I digress.


> I also agree with other people that Erlang's performance is in most cases
> more than "good enough" for what it's being used for. But when you have a
> manager who is already reluctant to use it, and employees who are even less
> willing to use it, they are going to be even less accepting when they
> *think* it's too slow (even though that might not actually be the case).
> Anything that can be done by the community to help here is invaluable.
>
> I would add that in my experience many issues of poor-performance are
> often a case of poor code. People not adhering to OTP principles, people
> not getting to grips with concurrent programming by doing things like
> implementing a single Erlang process that serializes requests for the whole
> system. Or in other cases people not using what's available in the language
> (regex on binary data using binary_to_list and and then the re module,
> instead of the binary module). In those cases it won't matter a squat what
> Ericsson do to improve BEAM. But perhaps we need to make an effort to
> hammer home good programming practice and be more consistent with core
> libraries (especially with lists vs binaries).
>
>
+1. Most performance difficulties are down to poorly written code - often
code that is not 'dirty' but simply not idiomatically correct. We have a
web based application (RESTful web service) written on Misultin that out
performs the Java application it replaced both in terms of concurrent
connections and its raw (round trip) response time (which includes
accessing a postgres database).

Incidentally, the key to our success was *not* copying the request
lifecycle that the java application had. Asynchronous communication with
the database is handled carefully so that each row returned from the
network driver is almost immediately made available to the response
generating code, which in turn streams its output back to the client
immediately. Not waiting until the whole response has been built up saved
3-4x memory use and increased the 'speed' of the application by a similar
factor. This is a much more Erlang-ish approach, and that really seems to
pay off.


> I guess I'm trying to say that everyone is correct, performance is usually
> more than good enough. But there is a perception among many non-Erlang
> developers that that is not the case.
>

Benchmarks will fix this. Problem is, they need to be based on the *full
request lifecycle* - e.g., for a bog standard web application/service.
Perhaps we need a proof of concept application that does all the various
stuff you'd expect to see happening on the server i.e., mapping the request
to a handler, validation of input data/parameters, performing some
application domain logic, talking to a database, serialising the response
back to the client. The Java and .NET crowds have a "Pet Store" application
that demonstrates this, attempting to point out good practises along the
way. Perhaps if we, as a community, were to produce the same thing, it
would not only benefit new users but also provide an opportunity to show
that Erlang can be a competitive solution for building such applications. I
for one would be willing to participate, although I'd be *much* more
inclined to do so if there was a standard API for interacting with web
servers. There are a couple of open source efforts, but they're still at
about 70 - 80% completeness from what I can tell, and personally I won't
back anything that is based on parameterised modules until the OTP team
decides once and for all if they're staying or going.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20120213/598deb94/attachment.htm>