[erlang-questions] Erlang and the learning curve

[Steve Vinoski]

On Tue, Jan 4, 2011 at 4:07 PM, Morten Krogh <mk@REDACTED> wrote:
> On 1/4/11 8:59 PM, Steve Vinoski wrote:
>>
>> On Tue, Jan 4, 2011 at 1:35 PM, Morten Krogh<mk@REDACTED>  wrote:
>>>
>>> On 1/4/11 4:37 PM, Steve Vinoski wrote:
>>>>
>>>> On Tue, Jan 4, 2011 at 6:31 AM, Morten Krogh<mk@REDACTED>    wrote:
>>>>>
>>>>> Why are one fourth, or whatever, of the posts on this list about nif or
>>>>> c
>>>>> nodes or ports?
>>>>
>>>> Even if we assume your percentage is accurate (which I don't), simply
>>>> assuming these features are used only for performance reasons is a
>>>> case of premature optimization. ;-)
>>>>
>>>> While some uses of these features involve performance concerns, C
>>>> nodes, ports, drivers, and NIFs all serve as points for integrating
>>>> Erlang with other environments. NIFs have already proven themselves
>>>> particularly useful in this regard. Because of the continued growth in
>>>> the popularity and application of Erlang, the number of developers
>>>> interested in using these features to integrate with non-Erlang code
>>>> is also growing.
>>>>
>>>> These are areas of Erlang involving languages other than Erlang, such
>>>> as C, so it could be they're harder to understand than other parts of
>>>> the language. It also could be that even with the aforementioned
>>>> growing popularity of the language, the number of Erlang developers
>>>> who've ever written C nodes and drivers used in production remains
>>>> relatively small, and so finding good sources of detailed information
>>>> about how best to write them is challenging.
>>>>
>>>> But still, I'd argue there really aren't that many questions posted
>>>> here about these features.
>>>>
>>>> --steve
>>>
>>> My percentage is probably wrong. I didn't count. I just wanted to stress
>>> that I see quite a lot of performance optimization in Erlang.
>>
>> As do I, but not by jumping out of Erlang. Using Erlang's profilers
>> and other tools is very straightforward and can significantly help
>> improve performance.
>>
> Yes, and that is also my point. Because people have think more about
> performance in Erlang than in C or C++ say, profilers and other tools are
> needed
> earlier and should be included in a discussion of the difficulty of various
> languages.  Erlang with profilers and tools is a harder "language" than
> Erlang without.

So what you're claiming is the fact that Erlang provides profiling
tools proves that it performs poorly? No offense, Morten, but that
doesn't make sense. Name me a system that doesn't offer profiling
tools and I'll name you a system that isn't used for real-world
development.

Erlang provides these tools so that programmers can analyze their
programs at the level of abstraction of the language itself. I've
never tried the following, but maybe you could instead use a tool like
callgrind at the Erlang VM level to analyze your Erlang programs, but
I think it would probably be pretty challenging to match your findings
back to your Erlang source code.

As far as I've seen, I use profiling tools (regardless of language) a
lot more than most programmers, yet the amount of Erlang code I've
worked on that I've felt the need to actually profile and measure is
likely in the 2-3% range. I'm sure I'm not alone on this mailing list
in my opinion that idiomatic Erlang code generally performs very well
and needs no specific tuning or measurement.

Also, consider this: some of the first hard Erlang performance issues
I ever faced was when I tried tackling Tim Bray's original WideFinder
problem several years ago. I was still relatively new to Erlang then
and compared to what I know now, I barely knew anything, and yet the
Erlang solution that I submitted was one of the fastest solutions
among all languages for awhile. I'm sure if I looked at it now I'd
find that the code I submitted was actually fairly terrible, and I
knew nothing about Erlang's profilers then (I was just using the timer
module to measure how long various functions took). Eventually my
solution fell by the wayside in the contest because it had a bug
somewhere that I was too inexperienced too resolve, but IIRC correctly
the winner ended up being an OCaml program (and yes, it was faster
than any C or C++ that was submitted).

The point of all this, of course, is that all of this information runs
contrary to your claims that Erlang requires deep, continuous focus on
performance. In my experience, it doesn't.

>>> Why were NIFs introduced at all? Why new BIFs all the time? All of that
>>> could be done in Erlang.
>>
>> Only the OTP team can say for sure why they introduced NIFs, but my
>> understanding is that it allows for easier integration with and access
>> to C than drivers. I personally don't find drivers difficult, but I
>> know they can be overkill for simple integration needs.
>>
> Maybe, but why C to start with. Performance must be a big issue here, but
> anyway.

The Erlang runtime system is written in C, so NIFs and drivers are C.
What else would you write it in? C is a universal portable language,
so it's a logical choice, especially given the timeframe in which
Erlang development started. I don't know if Kresten's Erjang,
implemented in Java, supports NIFs yet, but they'd be written in Java
just as Erjang's drivers are written in Java.

>> Are new BIFs really being introduced "all the time" strictly for
>> performance reasons? If so can you cite some examples?
>
> I was thinking about the Binary module for example. Isn't that whole module
> just for performance reasons more or less.

I thought it provided a richer way of working with binaries. Are
binaries present in the language only for performance reasons? Klacke
can tell us for sure, but I don't think so; they're there to provide a
simple yet extremely powerful abstraction for manipulating bytes and
bits. When you're dealing with network protocols, like you do in the
telecoms world for which Erlang was originally created, the need to do
that easily and in a way that makes for readable and maintainable code
is critically important. The fact that binaries and bitstrings also
turn out to be useful in other app domains should come as no surprise.

> I don't have a list of all new BIFs. It was just my understanding that every
> new release would have some new BIFS, that could be
> implemented in pure Erlang. What about min and max? Aren't they BIFs?

I don't know off the top of my head, but I actually don't care because
to me BIFs look and act like regular Erlang.

> Anyway, my point in this thread is just that performance matters, and when
> you compare languages you need to include the whole package including
> that you some language needs more work on performance than others. I was
> citing NIFs and BIFs etc to claim that other people than me cares about
> performance
> in Erlang.

Sure, performance matters, but only in areas that are proven to be
deficient via actual measurements of actual running programs. Still,
performance is only one of a number of engineering trade-offs that
have to be properly balanced in real-world systems. I invite you to
read an article I wrote about this topic a few years ago, specifically
about the misguided focus on middleware performance that seemed to be
rampant in my world back then:

http://steve.vinoski.net/pdf/IEEE-The_Performance_Presumption.pdf

>>> Steve, I have never seen a NIF that was not performance motivated.
>>> But you have seen many more NIFs than me.
>>> Do you have any examples?
>>
>> I have two NIFs in the system I work on, for example, neither of which
>> is performance-oriented:
>>
>> * One NIF allows simple access the Linux UUID library so that my
>> Erlang code can generate UUIDs the same as the C++ components with
>> which it interoperates over the internal chassis network. This has
>> nothing to do with performance and everything to do with consistent
>> UUID generation across all the components in the chassis.
>>
>> * The other NIF enables control plane calls to some specialized
>> offload sockets on our video delivery hardware. The system components
>> providing these sockets are written in C, so the NIF imports their
>> header files to pick up all their #defines, constants, and function
>> prototypes, and then presents that functionality in Erlang in a way
>> that's natural and simple for the Erlang programmer to use. This NIF
>> has nothing to do with performance and everything to do with code
>> coherency, integration with the C code, and minimizing duplication
>> between Erlang and C.
>>
> Fine, so performance is not your reason to use more than pure Erlang. I will
> still claim that it is for many others.
> But it leads to the same conclusion, namely that functional languages are
> not so much easier.
> With such interfaces in your code, you are not supporting the notion of
> functional languages being easy, quite on the contrary.

I have to call BS on you there, since you can't logically reach your
conclusions from what's been discussed here. I wrote a relatively
small amount of C in NIF form to integrate Erlang with the rest of our
system, which means my fellow Erlang colleagues are able to use these
facilities from their regular Erlang code without having to think
about C. They're able to stay in the functional programming world and
still get our specialized hardware to do what they need it to do --
it's the best of both worlds.

I don't know how much actual real-world experience you have with
functional programming, but everything I've ever done with FP tells me
that yes, things actually *are* easier with FP -- the code is easier
to write, easier to read, smaller, more maintainable, easier to debug,
easier to extend, and it performs and scales reasonably well, and all
of this can be done in less time than with most other languages. I've
watched half a dozen of my colleagues learn Erlang from the start and
they were all, without exception, able to become very proficient in a
very short time, shorter than with any other language I've ever seen.
If you have different experiences, then fine -- all that means is that
you have different experiences with FP than many of us on this list.
It certainly doesn't mean the IMO faulty broad conclusions you've been
drawing trump our individual experiences.

I'm bowing out of this discussion now, too much work to do.

--steve