[erlang-questions] Where is the Golden Path?

Sun Jul 14 18:04:44 CEST 2019

From http://zwrob.com/posts/erlang-01/

I'm slowly learning [Erlang][0]. It's taken me more than 10 years to get to
where I am now...which, frankly, isn't very far. I thought I'd share some newbie
thoughts from a veteran software developer with over 25 years of experience in
the industry.

# History

From 2009 to 2019...

* read books [(Joe, Fred, etc.)][4]
* write little code snippets
* post little [how-to-tutorials][3] showing how little I understand about Erlang
* wander away as I lose focus and time
* repeat.

This year, after Joe died, I thought...I'm not getting any younger either. I
decided to take the summer off and really study Erlang.  So far the process
has been:

* read the [same books][4], but only as references
* have a real problem to solve (I'm basing a business on it)
* write little code snippets (escripts) to POC business components
* document what I am learning on my [personal website][5]
* design the full scale app
* implement ... oops hitting the road blocks...

## Road blocks

### Erlang installation and tools

I wish I'd known about [asdf][6] much earlier. That's really simplified my dev box
configuration.  I wish that [rebar3][7] and [kerl][8] and other docs just pointed this
out.  In fact, I wish there was a `golden` path published that just says, look,
here's the common accepted tool chain you are going to want to use. Stay on
this path, and life will be easy. [rebar3][7] is a big part of this story, but
not the whole thing.

### Migrating from escripts to `real` apps

    $ rebar3 new escript myfooscript

I started with escripts because I didn't want the hassle of understanding OTP
or anything else as I was writing POC snippets to understand the basic Erlang
libraries. So I'd write an escript to shell out and curl some resource and
gradually build up an application that ingests data and writes into a database,
blah blah. But along the way I could learn about `xmerl` and other useful
libraries. Oh, and I could run the app and see STDOUT. Cool! Just like any
other language I'm familiar with.

So I roll up my sleeves and decide to create an app...

### Naming things is hard

    $ rebar3 new release|umbrella myfooapp

Um, why does 'umbrella' exist as an option here? Is it the same thing as a
'release'? Is it necessary? Why is 'release' called 'release'? This is an
overloaded term, as there are 'releases' which is a thing, and there's a
'release' command, and there's a 'release' target in the rebar3 command. And
there's an 'umbrella' in there, too. OK, release == umbrella, so I pick one.

    $ rebar3 new release myfooapp

### Where is STDOUT?

So, let me say, `rebar3` is great. I barely remember what it was like to use
`sinan` or `faxien`. (And while Loic's Makefiles look amazing, I'm nowhere near
smart enough to use them.) But here's the thing, I had to watch a number of videos
before I found out what the `shell` command was. This is important to me because
the next block I had was... how do I start my application _and_ see the output
from it? I want to watch STDOUT, right? I was watching some conference video
when someone mentioned that 'shell' was the coolest part of rebar3 and I'm
like, 'what is that'? And voila! It's what I needed.

This should be on the golden path.

    $ cd myfooapp
    $ rebar3 shell myfooapp

### Where is the golden path?

It's weird that there is no golden path. [In this talk][2], at 14:50, Fred talks
about how it's only far into various Erlang books that authors start talking
about running code outside the shell, in contrast to K&R C where it's discussed
on page 6. In fact, the entire (excellent) presentation is on the terrible state
of tooling in the Erlang ecosystem.

*begin rant*

#### Here's what's weird about Erlang : Books are out of date

* The books are already out of date. Even [LYSEFGG][9] still talks about
  [systools][11] and [reltool][12] even though [rebar3][7] and [relx][10] have
  supplanted them?  What's weird about this is to be reading the rebar3
  documentation, then cross-referencing with relx, and reading the various books
  that describe the old ways of building releases and relups, etc.

#### Docs are hard to follow

* [relx][10] documentation is pretty hard to follow
* Community support seems fragmented
  * [Github shows relx][14] has 41 open issues and 6 open pull requests, with
    89 contributors
  * [Github shows rebar3][13] has 107 open issues and 11 open pull requests,
    with 239 contributors

#### Who's maintaining these tools?

If I were Tristan or Fred, boy I'd be tired of supporting this stuff.

#### Where are the videos?

Here's another thing that's weird...

I cannot find videos on how to use the Erlang shell. So on top of that weird
fact, I'm finding out about erlang by watching Elixir videos. I was watching an
Elixir video on 'wobserver' and realized, 'Hey! I bet that's just a web
interface on some Erlang thingy called "observer"'.  So then I find out about
'observer' and how useful that is. Great!  But things like, how to I do basic
stuff like navigate from one process to another, what do I do when everything
seems hung and broken, what do real live seasoned Erlang devs do when '_stuff_
hit's the fan'? How do they go about examining a running system? The web seems
to have nothing on this. But that's just weird, b/c Erlang has been around for
over 25 years, and any other language / framework/ runtime whatever has zillions
of youtube videos.

*end rant*

## Culture of Quick Fixes

OK, so enough ranting. Let me get into [Fred's mention of the 'Culture of Quick
Fixes'][2] (at about 21:00) by introducing my own steaming pile of a quick fix.

So here's the deal. I'm trying to build a [company][15]. I have limited $$$, I
can't screw around learning stuff forever. At some point I have to build a real
live app and try to generate some buzz. I'll roughly describe my first
app/service and run through my decision points...

### App

The app basically pulls data from publicly available but disparate data sets,
ingests them, normalizes them into a common data store, and then surfaces the
data store for analysis, reporting, AI/ML, etc, etc. In a nutshell, take data,
add magic, produce pretty pictures.

### Process

* prototype in bash (or python, or whatever...)

Seriously, with just `bash` and I can query for my data, extract in, play
around with it, etc. If I wanted I could hook it up to `cron` and probably move on
to actually using the data rather than learning Erlang and how to build 'proper'
infrastructure.

### Prototype in escript

Here's where I take small tasks and learn the Erlang libraries to accomplish
what I'm trying to do. It takes about as much time to do this as the previous
prototyping step. I find this to be a great way to both decompose a problem and
learn a new language.

### Start building industrial grade services

So now, rather than focusing on my business, finding customers, analyzing the
data I've collected, or in any other way providing value that could bring in
revenue and allow me to purchase more Erlang books... I'm veering off into the
weeds and learning how to:

  * build erlang apps for real
  * build erlang releases
  * wire all this into docker containers
  * wire in monitoring and all the 'industrial grade goodies'

Along the way I tend to post about what I learn [here][5].

* Is any of this worthwhile?
* Am I just engaged in endless pencil sharpening?
* I dunno. I'm too close to the fire.

I'm having fun though. And it's incredibly useful to my brain in some way I cannot describe.

### The Problem

So back to "The Culture of Quick Fixes". As I'm building my first set of
components and learning about rebar.config and config/sys.config and more groovy
stuff, I run into the issue of how to deploy Erlang to docker containers. The
answer is, run the release with the `foreground` command. But this wasn't
obvious to me at first and took a lot of digging around the 'net, [posting a
question on the rebar3 discussion list][16], and just general noodling around.

But, if I'm building industrial grade applications, then I need to solve for
issues such as:

* How do I emit metrics from my app?
* How do I monitor my metrics (e.g. Prometheus)?
* How do I hook up a visualization tool (e.g. Grafana)? Alarms?
* And how do I wire all this up into a containerized solution so that it's
  re-usable and shareable?
* Don't forget my "awesome" Makefiles! I like to have a common control plane for
  all my services, and common Makefiles are great for this.

Now, if you watched the end of Fred's video, he was talking about many of these
same things.

*And I'll suggest that what I've built is an example of exactly what
shouldn't be done:*

* I ignored everything on hex. Why? Um, I don't really understand hex that well
  yet, I'd have to go read a bunch of code to understand what folks have done,
  and I'm still learning Erlang by building things myself. None of these is a
  good reason, BTW.
* I only published my stuff to github, and not to hex. This may be a good thing
  as I'd just be duplicating other, probably better, efforts.
* What I built should probably be a rebar3 plugin, and not a Makefile with
  dependencies on rebar3 etc.

However, I'll show you what this is because I think it satisfies a need and
perhaps we can all come up with a better implementation strategy. Or learn
something.

## Erlang-Seed

So, here we have 'yet-another-quick-fix' called [erlang-seed][17]. Let me be
honest, this is a 'quick-fix' to satisfy my own personal needs...

### The Good

What this does is create an erlang application, inject it into a Docker
container, and then spin up a local Docker cluster that monitors the Erlang app
and surfaces the metrics via Prometheus and Grafana. So with one command I can
generate an OTP application, all the infrastructure necessary to monitor it with
common tooling, _and_ spin it up and run it on my dev box. I find this to be
really useful. YMMV.

### The Bad

This is really crufty. It calls into rebar3 to generate the app. It should
probably be a rebar3 plugin so that it can better evolve with the tooling. It
could then more intelligently inject the necessary dependencies, instrument the
code with metrics, etc. Right now it's gross, as it just overwrites files as
necessary using mustache templates.

### The Ugly

* Do I have time to maintain this? No.
* Do I have time to make this a rebar3 plugin? Not right now.
* Will this probably break over time due to either rebar3, docker, or some other dependency changing? Certainly, eventually, yes.

## Summary

The reason I'm sharing this is to just widen the discussion on what the Erlang
education and tooling could look like. I think that generating full stacks that
allow developers to focus more on solving business domain issues rather than
forcing them to understand all the arcane details of whatever is under the
covers is the 'way to go (TM)'. There's got to be a way to allow developers to
maintain velocity while increasing their depth and scope of knowledge as
necessary. I've been studying Erlang intensely for at least a month now, read
the books _countless_ times, and read the online docs and I still don't quite
understand how this all fits together. I mean I get the big picture. But there's
a _lot_ of details.

Contrast this with Go. I was writing production code in 2 days. The quality was
production quality after about a week. Back in the day it took me about a month
to become proficient in Java, though it required reading Bloch's book "Java
Concurrency In Practice" to be 'good'. Python was similar, it took about a week
to ramp up, about a month to become comfortable. In all of these languages it
was a relatively short and easy ramp up and then smooth sailing for years and
years.

With Erlang it's an incredibly steep ramp that goes on and on. I guess if I
remember that I'm learning the equivalent of a language, an orchestration
framework (like kubernetes), a database system, etc., then it starts to make
sense.

Anyway, for my part, I'll keep posting things as I figure them out.

-Todd

[0]:https://www.erlang.org/
[1]:https://ferd.ca/ten-years-of-erlang.html
[2]:https://youtu.be/Z28SDd9bXcE?t=1260
[3]:https://github.com/ToddG/experimental/tree/master/erlang/wilderness
[4]:https://github.com/ToddG/experimental/blob/master/erlang/resources/books.md
[5]:http://zwrob.com/posts
[6]:https://github.com/asdf-vm/asdf
[7]:https://www.rebar3.org
[8]:https://github.com/kerl/kerl
[9]:https://learnyousomeerlang.com/
[10]:https://github.com/erlware/relx
[11]:http://erlang.org/doc/man/systools.html
[12]:http://erlang.org/doc/man/reltool.html
[13]:https://github.com/erlang/rebar3
[14]:https://github.com/erlware/relx
[15]:http://envirosoftwaresolutions.com
[16]:https://www.rebar3.org/discuss/5d29172873e1ed00447cb362
[17]:https://github.com/ToddG/erlang-seed
[18]:http://zwrob.com/posts/developer-automation/

-Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20190714/271e7975/attachment.htm>