[erlang-questions] Erlang is the best choice for building commercial application servers

[Tim Watson]

On 12 Mar 2012, at 03:00, Shahrdad Shadab wrote:
> 
> Thanks everybody for your valuable comments, What I got from your comments is basically comes down to the very point 
> that the decision of picking java/j2ee over Erlang is most business driven than computer science / technology driven.
> The similar reason is behind creation of C# and .net which is a close copy of java and J2ee (Just because of competition between Microsoft and Sun).  
> If I want to go a little bit deeper to the heart of the problem (at least in North America) big companies like Microsoft or Oracle kind of 
> derailed IT from its normal path. You would never see such negative influence of money and power in other fields like applied math or physics because the result of innovation in these fields isn't directly applicable in business problems. Here we are dealing with business man's decision override right technical decisions.This is just trouble. This causes many practitioners use their mind and time to understand and maintane systems that are implemented in technologies which were not the right solution since the day one. 
> Even business is currently paying for such bad decisions that they make in this filed by spending too much money for maintenance of the systems that are implemented with non-suitable technologies.
> Currently I don't see any solution to this problem unless IT's point of view respected by line of business and business decision makers don't cross their red lines and invade IT realm.
> 

There have been some interesting perspectives here and I have a few observations to share. Firstly, there are plenty of areas where computing science isn't constrained to the 'big corporate platforms' quite so much, for example in bioinformatics and other areas of science, niche areas (like telecoms, life critical systems, etc) and these are to name but a few. If you think that money isn't an influence in those other human endeavours though - the sciences are really no exception to this - you are quite mistaken. Even academics have to earn a living and competition over grants, certainly in the UK, is not for the faint hearted from what I hear.

My second observation comes from spending considerable effort helping a rather *large* IT unit (whose technical staff runs into tens of thousands) with solutions to varying challenges that rally under the banner of 'governance', many of which are about processes and people as well as technology. One thing is very clear to me from this experience. In the modern business technical landscape, Dijkstra's call for simplicity and elegance is pretty much lost in a sea of voices clamouring for cost/waste reduction, increased throughput, reduced turn around times and so on. Especially when financial hard times hit, the order of the day is to do 'more' with fewer people, less money, and in less time. And naturally nobody wants the quality to go down either. Frankly in the midst of that kind of storm, picking the right tool for the job has to account for a lot more than just the requisite technical merits of one platform versus another. The comments that have been made here about risk are absolutely spot on, and in practise this 'risk aversion' often has little or nothing to do with the fact that some unknown thing like Erlang is 'unusual' or 'unfamiliar'. The risk is far more practical than that, and it boils down to the exact questions you ask yourself in an architect or solution designer's role every time you have this kind of technology choice to make:

1. will Technology-X interface with any existing tech we already have on the ground?
2. is Technology-X sufficiently battle proven in the wild and with notable companies/organisations?
3. will I be able to hire developers to continue working on Technology-X if Joe and Rob (who are so keen to use it) decide to quit half way through the project?
4. if I do have to hire new developers, will they cost me twice as much because Technology-X represents a niche skill set?
5. will I ever be able to find an affordable second and/or third line support team for this?

It can be harder to do things at scale with java, distribution doesn't come so easily and getting concurrency right requires skill, experience and good analysis and measurement tools. However for most of points 1..5, java scores very highly. Erlang does great on 1 and 2. Being able to find developers is getting better, but it's still incomparably more difficult to find a good Erlang developer who is available for hire than it is to get one with Java and/or .NET experience. True, a really good developer will be able to pick up almost any language, and will even be able to adapt to new and unfamiliar programming paradigms. However 'really good' developers who can pick things up like that, actually costs a lot of money, and it's not particularly easy to find ready made teams of them that you can outsource to. An IT department with tens of thousands of people that supports a business of hundreds of thousands, simply *has* to outsource some of its engineering work. And as for finding an affordable ASG solution, just forget it. Go and compare the cost of setting up a second or third line support contract for some bog standard Java web application or Oracle suite or Sharepoint application or whatever. There's no comparison.

Despite the fact that in Erlang, doing those things that are 'hard work' in java is a lot easier (and more pleasant for the developer), it is not impossibly hard to do these things in java either. I have implemented java server applications that handle a very large amount of throughput, have to deal with issues of redundancy and failover correctly when problems occur - these were 'big' projects and it was hard work, especially to test. I am also able to write concurrent programs in java (and .NET for that matter) and whilst it is not my preference to do this, it is very much possible. I am not an 'average' programmer, but I'm not a superstar or a genius either. Ironically, Erlang was developed to make it easier for 'average' programmers to build safe and correct programs, yet because it is perceived as being a niche skill, this (simplicity making it easy to do the right thing) is lost of most people who're at decision making level in a company with > 10k IT staff.

This isn't to even start to mention industry standards, which again are a contributing factor. Sometimes the 'big players' have an advantage because they've ploughed millions into getting ahead of the game in some areas. Is it big conspiracy between businesses and Microsoft/Oracle do we think? No, it's much simpler than that. The large vendors get in the door and sell their story to the beleaguered IT managers, who sign on the dotted line whilst desperately hoping that with this new product/platform in place, they might at last get their weekends back.

Now you know what's influencing those decisions - it's absolutely not about business decision makers versus IT. It is IT who makes those decisions, and IT does tend to choose the popular, well known and 'safe' platforms and tool sets for these reasons (and many others besides). These are not bad decisions at all. An IT department does not exist in a vacuum separately from the business, and if we're talking about IT departments whose job it is to support and enable the business, doing what's *right* for the business means getting it right on the finances, accounting for business continuity and so on. So the point about all this is that Erlang might be the right tool for the job, but that doesn't necessarily equal the right thing for the business.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20120312/69c1165d/attachment.htm>