[erlang-questions] The future of Erlang and BEAM

[Ulf Wiger]

Radek,

My take on the LMAX Disruptor is that it would probably be hard to get the same raw throughput in Erlang. OTOH, for many products, this is not the clincher, and Erlang may still be the better option over time, since it tends to make it easier to make use of multicore or improvements in cluster technology in order to get the desired performance improvements.

In my own experience, this has been demonstrated many times. I have witnessed many different C++-based applications temporarily run circles around their Erlang-based counterparts, but eventually running into severe problems, either because they hit hard limits that stopped them from achieving the required performance on available hardware [1], or because changing requirements added a complexity that they were not designed to manage [2]. Another notable occasion was the shift to multicore, where most C++ and Java developers ended up facing a magnificent challenge, while we could ship Erlang-based systems on the new architectures with only minimal changes.

The thing that was missing from your list of things Erlang is that it shines for coordination problems. Erlang is sometimes called an *orchestration language*. This is where most of the competing solutions are often woefully inadequate. The key is where you get inputs from different sources, and must be able to handle events arriving in unexpected order. This is typical for control systems, and actually where the biggest complexity lies.

LMAX Disruptor, if I recall correctly, is essentially a processing pipeline, where each job is independent and processed to completion. Thus, there is no coordination complexity to speak of, and the main challenges are managing access to shared resources and achieving high processor utilization.

Granted, I have spent my whole career working with problems where Erlang is a near-perfect fit. YMMV depending on domain. :)

BR,
Ulf W

[1] In one case, the performance gains came in no small part from the fact that the C++ app was using static memory management. This worked fine, until the required capacity required more RAM than could be installed onto the available processor board. Rewriting the code to use dynamic memory management was deemed too risky and expensive, given the schedule.

[2] See e.g. my presentation, Death by Accidental Complexity, http://www.infoq.com/presentations/Death-by-Accidental-Complexity

On 11 Feb 2012, at 12:14, Max Bourinov wrote:

> Hi Radek,
> 
> Before answering your question, could you please clarify to the list what is so special in "LMAX Disruptor"? Why it is impossible to implement it in Erlang? Does it really makes sense to use this technology? I asked those questions because I read about "LMAX Disruptor" while ago, but I didn't find anything special in it. Moreover for me is seems as an obvious thing in a shiny package.
> 
> About Java vs Erlang: After many years in Java world (banking, investing, transportation) I started to work with Erlang (including all guys in our company). Now all our software is written in Erlang/C. It handles huge payload, very easy to scale, in case of problems it heals itself automatically. It is very easy to develop and maintain software in Erlang. This is a true dream technology. We are really happy now. I don't see any Java use cases in our company in the future.
> 
> p.s. Of course Erlang is not designed for UI.
> 
> ---
> 
> So, thanks again for Ericson company and all those great people that developed Erlang (I hope you read this mail. I am really proud to know you) and all of those people that contribute to Erlang/OTP. Thank you!
> 
> Best regards,
> Max
> 
> 
> On Sat, Feb 11, 2012 at 2:34 PM, Radek <poprosturadek@REDACTED> wrote:
> Hello Group,
> 
> it's my first post here, although I've been reading many interesting posts here for quite a long time.
> Anyway, I posted a question on StackOverflow today with the same title (here). As I've written there, I don't want to start any flame-war, particularly because I'm actually an Erlang fan-boy :) but I'd like to know your opinion on the subject. Here's the original question:
> 
> -------
> Some time ago I got seriously interested in Erlang (coming from C++/PHP/Java world) - and I've seen it has been successfuly used in the industry, by Ericsson, Facebook, Goldman Sachs, etc. So, I thought it would be a great platform to build high demanding apps, with low-latency profile, with a lot cleaner and nicer language than, for example, Java (for me).
> 
> But after "wow effect" has gone, I discovered that there are many high performance Java libraries that seem to resolve many problems that Erlang is theoretically best suited for (real-time, low-latency applications, concurrency, fault-tolerance, etc.). Moreover, it seems that there are things that, despite Erlang profile, are just not possible to achieve on BEAM (like LMAX Disruptor concurrent framework).
> 
> So the question arises: is Erlang still the best platform to build such demanding applications ? Wouldn't it be better if we stick to one, very mature (J)VM and try to make it even better than trying to achieve something similar with less resources available (size of OTP team vs. JVM team, supporters, etc) ? And is it possible at all to achieve this kind of performance and adoption with BEAM ?
> 
> And just to make things clear: I don't want to init a flame war here. I am just curious becouse I really like Erlang and I think it's a great platform and I'd like to invest time and effort to build real-life projects on it. But I'd just like to know what others might say about that and - if I'wrong - maybe someone could correct me.
> -------
> 
> I hope we can discuss it since I think it would be valuable not only for me.
> 
> Greetings,
> Radek
> 
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
> 
> 
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20120211/4cb95cdc/attachment.htm>