Why I chose erlang (very, very long story)

[Jay Nelson]

I've been following the discussions about OTP the past few days and found 
the comments by various participants particularly enlightening.  It's 
always nice to observe a lively discussion -- so much can be learned.  I 
thought I'd throw in my opinions as to why I am using erlang.  I hope they 
are useful to someone, especially the Ericsson Product Manager.

My background is rather varied having started with Basic, assembler and 
RPG, before moving on to APL, Prolog, Pascal, Forth, Smalltalk-80, Modula-2 
(on the Lilith), then those halcyon days of Symbolics LISP with Flavors & 
Z-macs (an editor without meta-, hyper-, and super- seems so limited), 
Expert Systems and other AI techniques, a rude introduction to C on Sun 
(why is this new concept of memory allocation necessary?), followed by C++, 
OODBs, CLOS with MOP (MOP being the main reason why OO was so interesting 
in the first place -- why, oh why, was it eliminated in future OO 
languages, just to allow mere mortals with no passion for programming to 
have access to the Gospel?), Perl, Javascript, Java, Python, Ruby (now that 
is a fine OO language for small problems) and now erlang.

I am first and foremost a language dilettante, but when I find one that is 
useful I work with it to discover what is different about it and in what 
situations it might apply.  I have written small things (compared neural 
net implementations in Pascal, spreadsheet macros and APL, a mini-SECD 
machine in assembler on TRS-80) to large things (traditional 
business/banking/manufacturing inventory/accounting/personnel systems, 
aerospace message-based OODB applications, a Perl program that 
automatically loaded a structured database from English text free-formatted 
OCRed resumes) to Internet things (webserver / gameserver, meta-search 
tools, html generators, data extraction systems).  Unfortunately, a job 
often dictates the language and the problem so my useful efforts have been 
limited.

My observations about languages:

1) Spend a little time evaluating and toss quickly those that you don't like
2) Toy problems are designed to make a language look good or to point out 
its strongest features
3) Fiddling with your own toy problems highlights issues of a language 
relevant to your use of it
4) Until you implement a real problem you can't understand the language
5) Until you hit the limits of applicability you aren't proficient in a 
language
6) A language will subliminimally and fundamentally alter your perception 
of a problem
7) Don't believe the hype
8) The proper or improper design of data structures can make a problem 
range from trivial to unsolvable; ease of expressing and redesigning data 
structures is essential to solving real problems

#6 is most apparent if you compare any combination of a functional 
language, imperative language, constraint-base language, rule-based 
language, OO language, dataflow lnaguage, pattern-based language or any 
other paradigm to the same problem.  Another easy way to see #6 is to 
discover the right language for a problem, solve it elegantly and then try 
to talk to a C, C++, or Java programmer about the problem.

I have always been partial to recursive languages, functional languages, 
and pattern-based languages (I even played around a little with REFAL on 
that last note).  A combination of these attributes are more natural and 
readable to my sensibilities.  Others may have their own conceptual and 
readability biases.

As an aside: I hate statically typed languages, and so-called OO languages 
where the data have no type, but the containers you put them in have type 
and forces same on its data.  I never understood the usefulness of a 
language that dictates if you take your laptop to the library it magically 
becomes a book and has no notion that it was ever anything else.


What attracted me to erlang:

I have been building a webserver / gameserver for the last 4 years on and 
off in my spare time (see www.5x5.net) out of general curiosity to 
understand all aspects of eCommerce from the technology underlying the 
gamesite, to the marketing, advertising, and business implications of an 
Internet only company, but also because I saw that the mega-media companies 
were taking over the Internet and service providers and I wanted to have my 
own capability that couldn't be usurped by the software vendor or service 
provider.  At the time I thought Java was mature enough to support what I 
wanted to do, but it did have a couple limitations, including the fact that 
each simultaneous player had to have a separate thread (no support for 
non-blocking threads).  I thought this would be challenging and teach me a 
little about threaded programming, but without the headaches of a C or C++ 
approach.

After about 30K lines of Java in both applets and Tomcat servlets I was 
nearly ready to launch the website.  I had built all the user account 
management, registration, demographics, automated email, MySQL web-based 
management tools (so I could manage the site from work or from home), and 
the gameserver (designed to deliver HTML pages, applets and ads all from 
the same IP address and port to tunnel through firewalls and deliver ads 
directly to the applet), and started an Alpha test.  After 3-4 days the 
thread that reports when people enter and leave game rooms stopped 
responding.  It didn't crash but it didn't do any more work.  All other 
threads worked fine with ads, webpages and HTML still being 
delivered.  Somehow it starved.   After 4 weeks or so, Tomcat stopped 
running servlets intermittently.  The logfiles had a message that the JVM 
could no longer spawn new threads (is that Java, Linux or Tomcat with the 
problem?).

I had built some kickstart scripts and cronjobs that would attempt to 
restore freezes, etc. (oooh, this sounds too much like Microsoft and IBM's 
"self-healing" software), but it all seemed so hokey and unnecessary -- and 
still didn't solve the problem of giving me a remotely administerable 
non-stop server.  Whenever the server came down all users in the gamerooms 
would freeze or get booted from their applets.  There was no redundancy, 
distributed processing or fault tolerance.  The latter was my mistake in 
the message protocol because it was possible to lose one message and not 
know the state of the game or be able to rejoin and reproduce without 
leaving the table and disrupting it for everyone else.  I learned quite a 
bit, including Visual Age Java, but only enough to believe that while Java 
could solve the problem, it was the wrong language -- I felt I had hit its 
limits for this application.  In desperation I abandoned the project and 
started browsing the web listlessly for several months.

The main issues with Java boiled down to:

1) Threads are much easier than C, but still a royal pain
2) There are no useful tools for administering a server that I built
3) The OO straight-jacket is a major hindrance to web-based computing
4) There is no fault tolerance or distributed computing, the VM is heavy 
and flaky
5) The use of Java applets put a heavy constraint on the server in 
delivering content
6) Performance was always a concern, especially handling thousands of threads
7) Embedded capabilities were just being developed and not stable or 
performing well

When I came across erlang it was obvious that #3 was solved straight 
away.  The biggest pain was in message passing.  Messages had to be 
received as a generic Msg type, a jump table based on the first byte was 
used to get a msg handler, which in turn parsed, converted and cast the 
message as the right object type (hopefully).  Fighting the static type 
checking of the compiler every step of the way, I successfully created 
type-less objects and a functional callback implementation that resulted in 
a remote object (which of course had no knowledge or state corresponding to 
the originating object's environment space).  All this immediately 
disappears and becomes natural with a functional approach, the first 
solution that you would attempt rather than the last futile try.

Threading seemed to be completely a non-issue in erlang, with the added 
benefit that every user could be a separate process in addition to the game 
table itself, leading to an automatic distributed, fault-tolerant approach 
of losing only one game room rather than the entire server in a failure 
situation (and even the possibility that a game room fails but the game may 
continue on a new instance of a game room).  I could freely add servers to 
allow more players without any more effort than the load-balancing 
aspect.  The lighter VM solved #4, and the lightweight processes solved #6 
beyond my wildest dreams (probably even beyond my ability to design a 
meaningful UI for 50,000 game tables per server).

#5 was now not as great a concern because the load-balancing reduces the 
strain created by the Java applets.  The trick is just faking them into 
believing everything is coming from the same server, or transparently 
relocating the user before delivering the applet.

I have been reading the docs, fiddling around and have started 
reimplementing the server.  I am getting 5-10 times code reduction with an 
incredible increase in code clarity.  HTML and the pattern-based functional 
style are the perfect marriage.  This brings back the happy days of LISP, 
but I find the code much more readable and the language concepts more 
succinct for this problem set than LISP was.  The beauty of erlang is that 
it is small, clear and succinct and therefore implements problems clearly 
and efficiently.

I've spent the last 4 months or so following the email list and reading 
docs.  The learning curve for the full system is steeper than most every 
language I've pursued except the most interesting/useful ones: 
Smalltalk-80, Common LISP Object System with the Meta Object Protocol, and 
Java.  Java may prove to be the largest and least useful of the set, its 
usefulness mainly is in improving on the C++ experiment.  (Ada has to be 
the absolutely least useful steep learning curve language because it was 
designed by committee and not to solve any particular problem).  The 
learning curve on the language itself is practically nil if you've ever 
used a functional language before, and if you haven't erlang promises the 
greatest hope that you ever will.

OTP.  The raging debate about this was the most informative thing I've read 
over the last few months, aside from those postings that corrected all my 
misperceptions of erlang the language.  I decided to invest the time in 
erlang rather than skip over it like I did Mozart OZ, precisely because OTP 
was present.  I have never seen anything like it, and it solved exactly the 
problem that I did not want to waste my time on -- developing another Java 
/ Linux application set for managing the server.  That task would have been 
bigger than the gameserver itself that had taken me way too long already.

I believe OTP provides me:

1) Embedded turnkey system using SASL, reboot and all that
2) Non-stop server using supervision trees
3) Failover and takeover for seamless, continuous gameplay during server 
maintenance
4) Distributed debugging and logging that can be turned on and off at will
5) A complicated build, but with the result of hotcode version upgrade
6) Mnesia distributed database, with in memory and on disk capabilities
7) Pre-built behaviours that match my problem set

Maybe I'm mistaken in evaluating based on the documents, and I'll find out 
that I'm not capable of making OTP work, but it has more promise than 
anything I've seen.  OTP solves problems that I've not seen examples of 
having been solved before.  I'm learning and the best way to learn is to 
take a real problem, push the limits and see where things break.  In 6 
months I should know.  When I'm done with all this, I expect to be able to 
have gained knowledge enough that when someone asks me "Can A or B be 
done?"  I can reply based on first-hand experience and have the confidence 
that if I say yes, I would be capable of delivering what was asked.  In 5 
years I may have to roll my own OTP; maybe I'll be deep in the source code 
6 months from now figuring out how to roll my own.  If I was still stuck 
with Java, I would be spending my time at the beach instead.

Besides all that, erlang has got me excited about programming again.  I'm 
trying to figure out how to sneak a prototype into my job to convert them 
as well.  Right now, my day job is programming stock trading systems for 
mutual fund administrators using a homebrew rule-based dataflow language 
that no one seems to want any more (the language was developed there long 
before I arrived).  Guess what, the problem requires non-stop, distributed, 
fault-tolerant, constraint-based, functional capabilites, with in memory 
DB, distributed logging, failover and realtime interactive monitoring, 
querying and progress reporting.  Sounds like a job for erlang and OTP 
(Online Trading Platform).  As a programmer, I really could care less what 
it is called, does it solve my problem in a way that I enjoy coding and 
maintaining?

P.S.  The Telephony of OTP did really put me off and delayed my looking 
into the capability until my problem made it apparent this was really 
useful.  From a marketing standpoint a name is needed, as well as a 
separation between the language and the platform.  More would download and 
play with the language.  Only those with a real need will download the 
platform.


Sorry for the loss of bandwidth.  Had to drop in my 20,000 drachmas worth.

jay


---------------------------------------------------
DuoMark International, Inc.
6523 Colgate Avenue, Suite 325
Los Angeles, CA  90048-4410 / USA
Voice: +1 323 381-0001
FAX: +1 323 549 0172
Email: jay@REDACTED
WWW: http://www.duomark.com/