[erlang-questions] Erlang newbie questions

[Sam Bobroff]

On 18 October 2011 08:14, Gerry Weaver <gerryw@REDACTED> wrote:
> Hello All,
>
> I am new to Erlang and I'm trying to make the case to continue learning this language. I have read all of the books available and seen several presentations on the language. I am interested in Erlang primarily for the distributed and fault tolerant features it offers. However, I have some reservations about how useful this really is. For example, if you have a network server and the listener process fails, how do you recover? It seems that the fault tolerance is only applicable in the VM context. This is, in my mind equivalent to any other application process. If I have a network server written in C, I can have a watchdog process to kill and/or restart it in the event of failure. The C based approach actually seems more robust. In this scenario, how would one restart the Erlang VM? It has been my experience that network servers most often fail due to OS or hardware limits rather than
> bugs in the code (at least for thoroughly tested production ready code). When you factor in the amount of baggage that comes with Erlang, or any other VM based language, it's difficult to justify. Now there is also the rapid development aspect, but I'm finding that at least for now, the time savings is being eaten up by looking for documentation. I understand that this situation will improve over time, but the various posts I've seen from folks with more experience seem to indicate that this will take quite a while. I like the language (except the , ; . thing). You can do some pretty cool things with it, but when I got over that initial gee whiz, I had some trouble seeing a real world production use case. I'm not trying to be critical, I just figure I must have missed something along the way. Any perspectives or advice on this would be greatly appreciated.
>
>
> Thanks,
> -G

Hi Gerry,

I agree with most of the (great) follow up posts to your initial
question: Erlang, in the right situation, is a powerful and very
useful language and platform. Personally I also really enjoy using it
because it's a language that is relaxing to program in: things are
generally sensible and it rarely bites you in the ass (contrast to
C++... *CHOMP* *CHOMP* *CHOMP*).

However when you start using Erlang you'll eventually need to build a
package of some kind, deploy it and then manage the service or program
while it runs on the OS. This is not so well handled.

It's also interesting that no-one has mentioned Erlang's ability to
upgrade code while the system is running ("hot code loading"). In
theory this is a great feature that is very hard to get in other
languages. Ericsson has used it to deliver amazing up-times but I've
never seen it used anywhere else. Ever. Not in yaws, not ejabberd or
rabbitMQ (someone please correct me if I'm wrong). The reason seems to
be that it's both tricky to actually do, just because when you get to
the details it really is a hard thing to do, but it's also tied into
the way you build and release packages and upgrades which brings me to
my next point.

Packaging: There is no standard packaging method for Erlang (there are
several good community projects like rebar but none are standard).
There seems to be, or to have been, some standard system because the
OTP documentation mentions it but the documentation is incomplete or
missing and the functions don't really seem to hang together or do
what you need them to do. See make_tar, create_release,
release_handler etc. Apparently some updates are coming (at least to
the documentation) soon, which is good news but at the moment this is
a big pain in the ass for anyone who wants to deploy any code. Every
package I've seen either uses a community project or rolls their own.

Unix service integration: The VM process itself (at least on Unix) is
not well behaved and causes problems for system administrators and
monitoring programs.

* The VM process detaches from the console immediately with a
successful exit code, even if it's unable to start your Erlang
applications or if it's going to crash right away. This means that
when you deploy it, you have to tell your administrators to run the
start script and then manually read some log file and try to see if
it's running successfully or if it even began to start up. It's
impossible to deliver any messages to the console during startup in
detached mode so even obvious things like a missing config file have
to be discovered by digging through a log file.

* The VM has no support for writing a PID file and it forks when
detaching so it's hard (or impossible) to externally monitor or stop.
The heart watchdog (see below) makes this even worse. If multiple
Erlang VMs are run on the same machine it can be hard to tell them
apart, especially if they are crashing or broken somehow... and that's
when it matters!

* If you use the built-in heart program to restart a crashed or hung
VM process you can get into the situation where you can't (easily)
stop a VM that's crashing on start-up. If you kill heart then the VM
restarts it, if you kill the VM then heart restarts it. This might be
OK on an embedded machine but it does not make a Unix admin happy when
a service needs to be stopped right away.

* All these problems make package upgrades (e.g. via RPM or deb)
rather unsafe: it's not easy to determine which services to stop, its
hard to stop them and be sure they've stopped. When restarting you
can't know if it's started! Gah! Rather than just "rpm --upgrade" you
will always have a sequence of manual steps and checks in order to
perform an upgrade.

* The log file written by the VM has a weird rotation strategy that
makes it difficult to see which file is the actual current log file.
This makes the start up issues worse; I suspect that most projects
replace the logging code.

In short, I've found that code running IN the VM is very reliable and
easy to manage (the OTP supervisor system is great) but managing the
OS VM process is hard and error prone.

If you're not using Erlang as a service then things aren't so bad but
you're still hit by some weirdness: the program "erl" takes command
line parameters in a non standard (non-gnu anyway!) way and because of
this either your program will also have to do that (and so you'll end
up having to explain that to every user of the program) or you'll have
to write some wrapper just to tidy them up.

Sam.