[erlang-questions] trouble with erlang or erlang is a ghetto

[Ulf Wiger]

On 27 Jul 2011, at 08:08, Michael Truog wrote:

> On 07/26/2011 09:42 PM, Richard O'Keefe wrote:
>> (3) There doesn't seem to be anything in the Erlang _approach_ that
>>    should interfere with scaling, but the _implementation_ appears
>>    not to scale well much past 16 cores.
>> 
>>    I don't know if that is current information.  If true, it's an
>>    important limitation of the implementation which I'm sure will be
>>    given a lot of attention.  I don't expect it to stay true.
> 
> Any problems scaling on systems with more than 16 cores just seems to be related to the current cost of those systems (would love to know what the tests currently show on the Tileras > 16 cores though). 


I guess it's fair to say that the Erlang/OTP doesn't have a "dangerously sexy" approach to SMP scalability. They recognise that their sponsors [1] would have their heads if BEAM started hanging or dumping core in the sole interest of scaling to ridiculously many cores on hardware that none of the sponsors are using. Instead, they try to deliver incremental improvements without ever sacrificing the rock-solid performance of BEAM that Erlang users have come to expect. They're not batting 100% in that regard, but close enough to be very impressive, IMHO.

They *have* run benchmarks on 64-core machines, but mainly to learn about what's around the corner, and understand what changes will be needed to get there. You will soon hear about some pretty exciting developments in the area of Erlang for massive scalability, but there is a time and a place for everything… ;-)

What is interesting for most Erlang users (I think), is not how well Erlang can scale with synthetic benchmarks, but on *real applications*, where lots of other factors affect the result, and scalability is only one parameter (albeit a very important one).

Having had the privilege of participating in key roles throughout the development lifecycle of a number of telecom-class Erlang-based systems, I've come to think that constant-factor nuisances (like syntax quibbles) are simply unimportant. What matters in these projects is that you get the fundamentals right, esp. in regard to messaging, fault-tolerance and provisioning. If you don't, you will never be able to control schedules and costs, and your product is unlikely to ever be profitable.

In this environment, very few programming languages are even considered: usually, it's either C/C++ or Java, and in some companies, Erlang as well. The .NET languages are usually ruled out because their commercial viability is largely tied to the Windows platform, and such a single-platform focus is an issue when you expect your product to serve for decades.  Ruby, Python etc. can be useful for writing test scripts, and perhaps some supporting tools, or perhaps even some peripheral component in the system, but (at this point in time) not much more than that.

I realise that most people today are not in this environment, but rather developing small [2] projects, often in areas where Ruby, Python, Clojure, Scala, Haskell et al are perfectly viable alternatives. I think it's great that the Erlang/OTP team is getting feedback from this crowd, and even pressure to compete with the "coolest" languages out there. Erlang will be better for it.

But let's do this with some mutual respect. I'm not even going to begin commenting on the "Erlang cargo culters" references, and find it curious that Erlang is so often said to "suck", even though the same people often admit that there is no viable replacement - isn't this much more of a criticism against those other languages, which apparently don't just suck enough to give discomfort, but to the point of being near unusable in this realm?

Having been in the position, some years ago, where I was thinking about changing jobs, and couldn't really find any jobs where I could make good use of my Erlang experience (even if not in Erlang), I find today's software market much more exciting - where other languages are drawing inspiration from Erlang, and in some cases really challenging it, even in the concurrency domain.

[1] Sponsors = commercial projects within Ericsson that use Erlang to build 5-nines-style complex messaging products for commercial gain. They typically need to use NEBS-compliant ATCA hardware from reputable vendors, which is standardised across the corporation to minimise supply chain costs. In these settings, I would expect most systems to be running on 4- or 8-core machines today, perhaps starting to move up to 16-core in a few places.

[2] To me, anything under 100 KLOC is small.

BR,
Ulf W

Ulf Wiger, CTO, Erlang Solutions, Ltd.
http://erlang-solutions.com