[erlang-questions] What problem are we trying to solve here? [was Erland users group [was re: languages in use? [was: Time for OTP to be Renamed?]]]

[Fred Hebert]

Answers inline.

On 02/16, Pieter Hintjens wrote:
> On Sat, Feb 15, 2014 at 6:08 PM, Fred Hebert <mononcqc@REDACTED> wrote:
> 
> > That makes for quite a steep curve, doesn't it?
> 
> This is a core problem. It is an uncanny valley; before you can get
> those theoretical productivity gains you have to make large
> investments.
> 
> We had this experience in the ZeroMQ community (where I take many of
> my lessons from), and the principal lesson was that although there are
> sufficient very smart people to help with the hardest problems, the
> real life comes from passers-by who learn rapidly in an easy
> environment, and get benefit from day one. Decreasing slices of these
> people then stick around and learn more and more.
> 

This is fortunately an area people are actively working on. I know I've
worked on Recon (http://ferd.github.io/recon/) to help debugging
production systems. Tuncer Ayaz recently decumented most of rebar
commands (`rebar -help') and worked speeding it up, and Basho has
allocated people to work on it. An entire team of people have added work
in Relx (https://github.com/erlware/relx) to make project building
simpler. Kostis has been working on concurrency bug detection tools.
Garrett has worked on e2 to make the learning curve overall smaller.
Loïc Hoguin has recently reworked all of his cowboy documentation, build
tools, and examples to be simpler to used. That's just a few.

The OTP team has been working to add maps in the language to make it
more approachable, Steve Vinoski has put in work for dirty schedulers,
making it safer to run C code in Erlang. The Industrial Erlang user
group has been negotiating a new license, other than the EPL, to make
adoption simpler.

Erlang Solutions has started building and offering up-to-date packages
for many platforms to make installation simpler:
https://www.erlang-solutions.com/downloads

You'll notice that all these contributions (most of which are not done
by the core OTP team) aim to help existing Erlang developers, or ease
adoption. There is work being done.

It's quite frustrating to be part of people trying to make it better and
seeing that it's always 'too hard' for beginners. If the problem is with
the tooling, this is getting worked on.

If the problem is with the model (functional, message passing), then
that won't change without changing the definition of Erlang itself and
removing most of the actual benefits the language offers.

> If you scare off the tourists, you cut the number of solid
> contributors sharply. This has killed many technically brilliant
> projects. Indeed, technical brilliance is one of the best ways to
> scare away contributors. Bumbling sincerity beats code quality any
> time of day.
> 

It beats them for contributions, but the last thing I want is bumbling
sincerity messing up the schedulers, causing deadlocks and segfaults in
production code.

The OTP team block many contributions, but they also make sure a certain
quality exists with code. Like anything it sometimes blocks or slows
down valid contribution, but long term, the results haven't been too
bad.

> It's not snowing in Belgium. However, community building is the same
> problem in any country. Remove barriers, remove confusion, create
> competition, encourage people to work together instead of reinventing
> the same answers over and over. Create space for specialization and
> profit. Open up communications. Standardize obsessively. Spread
> ownership and decision making as widely as possible.
> 
> The ex-cathedra model can't compete with the rabble in the
> marketplace. So Erlang's greatest challenge IMO is simply shifting
> away from the top down "we decide for you" model to one that is driven
> by users.
> 

In the last few years:

- Erlang development moved to Git and Github (from e-mails!)
- Moved patch discussions to Github pull requests (while still
  supporting erlang-patches emails)
- Erlang moved its pre-release views from a once-released preview to a
  nightly build equivalent anybody can check out
- Added Unicode support to source files for a more international
  community
- Consistently had more and more user contributions (averaging 50 user
  contribs per minor releases in R16, outside of the OTP team). Still
  rather low, but always better.
- Modernized EEPs and have actually implemented a few (EEP-43 was
  OTP-submitted and OTP-implemented. EEP-37 was user-submitted and
  user-implemented. They'll make it in 17.0.
  See http://www.erlang.org/eeps/eep-0000.html)
- Documented everything you need to do to help
  (https://github.com/erlang/otp/wiki#wiki-development)
- Planned license changes from EPL to something more standard like
  MPL 2.0.

> If there's no shift in power by Erlang's current owners to a real
> community driven thinking process, the other options would be to fork
> the entire language and open it up, or allow it to die.
> 

The steps are being taken as shown above. You're still free to take it
and fork it if you want: https://github.com/erlang/otp
It's all open, except having a public-facing issue tracker, which would
be fairly important.

> To present this choice as (a) Ericsson or (b) Random Stupid Committee
> is a false dichotomy. The correct answer is (c) collective ownership
> and decision making with no single points of failure.
> 
> It's no coincidence that the only sustainable organizational model for
> building distributed systems is itself a distributed system. Which the
> Erlang community isn't, today.
> 

It is, partially. The OTP team often has a say in decisions because
they're the ones with the highest concentration of know-how about system
internals, and equipment to run tests to safeguard the platform's
quality.

The trend, though, is for a more and more open development. It's *not*
moving towards a more closed model. It's in fact doing the opposite, and
on a very encouraging motion.

> IMO Erlang is trying to deny Conway's law. It's like trying to deny
> gravity. Face, let me introduce you to Ground. Let's see who wins.
> 
> I'd love to learn Erlang, mainly because of the people who use it.
> However I'm lazy and modest in my investments and can't spend six
> months to see profit in a project. I won't use tools I can't improve
> myself. I won't join communities where I depend on others for decision
> making.
> 
> No complaints here. Just observation and comment based on watching a
> tragic number of good projects die because people didn't raise their
> heads up and ask some basic questions about relevance to the market.
> 
> -Pieter

In that context, I think the biggest problem may be the model. Are you
experienced with functional code? Are you familiar with share-nothing
concurrency? If not, there *will* be a required investment no matter
what the language is.

As mentioned in my previous post, the community could work and try to
make the learning curve shorter, but it won't go away entirely. Things
are incredibly nicer than a few years before (we had few tutorials, no
support outside of Emacs, a closed development process, far less
tooling, no way to pull in dependencies, no build tools outside of
rolling your own makefiles, etc.). It's just that Erlang came from so
far behind that even with years of work, it's still behind competition
on many issues.

It keeps moving forward, however, and seems to be catching up in speed.
I'm optimistic about Erlang's future, because I have been here long
enough (although I'm not in the old guard just yet), I can see a
positive trend. Someone who's starting today will see lonely data
points, where Erlang's is falling behind. In fact, it's catching up,
even though plenty of work and help is required.

Regards,
Fred.