[erlang-questions] Where is the Golden Path?

[Richard O'Keefe]

A former manager used to say "You hero, sir" when someone did something
good.
Imagine me saying it.


On Mon, 15 Jul 2019 at 19:02, Todd <toddg@REDACTED> wrote:

> From http://zwrob.com/posts/erlang-01/
>
> I'm slowly learning [Erlang][0]. It's taken me more than 10 years to get to
> where I am now...which, frankly, isn't very far. I thought I'd share some
> newbie
> thoughts from a veteran software developer with over 25 years of
> experience in
> the industry.
>
> # History
>
> From 2009 to 2019...
>
> * read books [(Joe, Fred, etc.)][4]
> * write little code snippets
> * post little [how-to-tutorials][3] showing how little I understand about
> Erlang
> * wander away as I lose focus and time
> * repeat.
>
>
> This year, after Joe died, I thought...I'm not getting any younger either.
> I
> decided to take the summer off and really study Erlang.  So far the process
> has been:
>
> * read the [same books][4], but only as references
> * have a real problem to solve (I'm basing a business on it)
> * write little code snippets (escripts) to POC business components
> * document what I am learning on my [personal website][5]
> * design the full scale app
> * implement ... oops hitting the road blocks...
>
>
> ## Road blocks
>
> ### Erlang installation and tools
>
> I wish I'd known about [asdf][6] much earlier. That's really simplified my
> dev box
> configuration.  I wish that [rebar3][7] and [kerl][8] and other docs just
> pointed this
> out.  In fact, I wish there was a `golden` path published that just says,
> look,
> here's the common accepted tool chain you are going to want to use. Stay on
> this path, and life will be easy. [rebar3][7] is a big part of this story,
> but
> not the whole thing.
>
>
> ### Migrating from escripts to `real` apps
>
>     $ rebar3 new escript myfooscript
>
> I started with escripts because I didn't want the hassle of understanding
> OTP
> or anything else as I was writing POC snippets to understand the basic
> Erlang
> libraries. So I'd write an escript to shell out and curl some resource and
> gradually build up an application that ingests data and writes into a
> database,
> blah blah. But along the way I could learn about `xmerl` and other useful
> libraries. Oh, and I could run the app and see STDOUT. Cool! Just like any
> other language I'm familiar with.
>
> So I roll up my sleeves and decide to create an app...
>
> ### Naming things is hard
>
>     $ rebar3 new release|umbrella myfooapp
>
> Um, why does 'umbrella' exist as an option here? Is it the same thing as a
> 'release'? Is it necessary? Why is 'release' called 'release'? This is an
> overloaded term, as there are 'releases' which is a thing, and there's a
> 'release' command, and there's a 'release' target in the rebar3 command.
> And
> there's an 'umbrella' in there, too. OK, release == umbrella, so I pick
> one.
>
>     $ rebar3 new release myfooapp
>
> ### Where is STDOUT?
>
> So, let me say, `rebar3` is great. I barely remember what it was like to
> use
> `sinan` or `faxien`. (And while Loic's Makefiles look amazing, I'm nowhere
> near
> smart enough to use them.) But here's the thing, I had to watch a number
> of videos
> before I found out what the `shell` command was. This is important to me
> because
> the next block I had was... how do I start my application _and_ see the
> output
> from it? I want to watch STDOUT, right? I was watching some conference
> video
> when someone mentioned that 'shell' was the coolest part of rebar3 and I'm
> like, 'what is that'? And voila! It's what I needed.
>
> This should be on the golden path.
>
>     $ cd myfooapp
>     $ rebar3 shell myfooapp
>
>
> ### Where is the golden path?
>
> It's weird that there is no golden path. [In this talk][2], at 14:50, Fred
> talks
> about how it's only far into various Erlang books that authors start
> talking
> about running code outside the shell, in contrast to K&R C where it's
> discussed
> on page 6. In fact, the entire (excellent) presentation is on the terrible
> state
> of tooling in the Erlang ecosystem.
>
> *begin rant*
>
> #### Here's what's weird about Erlang : Books are out of date
>
> * The books are already out of date. Even [LYSEFGG][9] still talks about
>   [systools][11] and [reltool][12] even though [rebar3][7] and [relx][10]
> have
>   supplanted them?  What's weird about this is to be reading the rebar3
>   documentation, then cross-referencing with relx, and reading the various
> books
>   that describe the old ways of building releases and relups, etc.
>
> #### Docs are hard to follow
>
> * [relx][10] documentation is pretty hard to follow
> * Community support seems fragmented
>   * [Github shows relx][14] has 41 open issues and 6 open pull requests,
> with
>     89 contributors
>   * [Github shows rebar3][13] has 107 open issues and 11 open pull
> requests,
>     with 239 contributors
>
> #### Who's maintaining these tools?
>
> If I were Tristan or Fred, boy I'd be tired of supporting this stuff.
>
> #### Where are the videos?
>
> Here's another thing that's weird...
>
> I cannot find videos on how to use the Erlang shell. So on top of that
> weird
> fact, I'm finding out about erlang by watching Elixir videos. I was
> watching an
> Elixir video on 'wobserver' and realized, 'Hey! I bet that's just a web
> interface on some Erlang thingy called "observer"'.  So then I find out
> about
> 'observer' and how useful that is. Great!  But things like, how to I do
> basic
> stuff like navigate from one process to another, what do I do when
> everything
> seems hung and broken, what do real live seasoned Erlang devs do when
> '_stuff_
> hit's the fan'? How do they go about examining a running system? The web
> seems
> to have nothing on this. But that's just weird, b/c Erlang has been around
> for
> over 25 years, and any other language / framework/ runtime whatever has
> zillions
> of youtube videos.
>
> *end rant*
>
> ## Culture of Quick Fixes
>
> OK, so enough ranting. Let me get into [Fred's mention of the 'Culture of
> Quick
> Fixes'][2] (at about 21:00) by introducing my own steaming pile of a quick
> fix.
>
> So here's the deal. I'm trying to build a [company][15]. I have limited
> $$$, I
> can't screw around learning stuff forever. At some point I have to build a
> real
> live app and try to generate some buzz. I'll roughly describe my first
> app/service and run through my decision points...
>
> ### App
>
> The app basically pulls data from publicly available but disparate data
> sets,
> ingests them, normalizes them into a common data store, and then surfaces
> the
> data store for analysis, reporting, AI/ML, etc, etc. In a nutshell, take
> data,
> add magic, produce pretty pictures.
>
> ### Process
>
> * prototype in bash (or python, or whatever...)
>
> Seriously, with just `bash` and I can query for my data, extract in, play
> around with it, etc. If I wanted I could hook it up to `cron` and probably
> move on
> to actually using the data rather than learning Erlang and how to build
> 'proper'
> infrastructure.
>
> ### Prototype in escript
>
> Here's where I take small tasks and learn the Erlang libraries to
> accomplish
> what I'm trying to do. It takes about as much time to do this as the
> previous
> prototyping step. I find this to be a great way to both decompose a
> problem and
> learn a new language.
>
> ### Start building industrial grade services
>
> So now, rather than focusing on my business, finding customers, analyzing
> the
> data I've collected, or in any other way providing value that could bring
> in
> revenue and allow me to purchase more Erlang books... I'm veering off into
> the
> weeds and learning how to:
>
>   * build erlang apps for real
>   * build erlang releases
>   * wire all this into docker containers
>   * wire in monitoring and all the 'industrial grade goodies'
>
> Along the way I tend to post about what I learn [here][5].
>
> * Is any of this worthwhile?
> * Am I just engaged in endless pencil sharpening?
> * I dunno. I'm too close to the fire.
>
> I'm having fun though. And it's incredibly useful to my brain in some way
> I cannot describe.
>
> ### The Problem
>
> So back to "The Culture of Quick Fixes". As I'm building my first set of
> components and learning about rebar.config and config/sys.config and more
> groovy
> stuff, I run into the issue of how to deploy Erlang to docker containers.
> The
> answer is, run the release with the `foreground` command. But this wasn't
> obvious to me at first and took a lot of digging around the 'net, [posting
> a
> question on the rebar3 discussion list][16], and just general noodling
> around.
>
> But, if I'm building industrial grade applications, then I need to solve
> for
> issues such as:
>
> * How do I emit metrics from my app?
> * How do I monitor my metrics (e.g. Prometheus)?
> * How do I hook up a visualization tool (e.g. Grafana)? Alarms?
> * And how do I wire all this up into a containerized solution so that it's
>   re-usable and shareable?
> * Don't forget my "awesome" Makefiles! I like to have a common control
> plane for
>   all my services, and common Makefiles are great for this.
>
> Now, if you watched the end of Fred's video, he was talking about many of
> these
> same things.
>
> *And I'll suggest that what I've built is an example of exactly what
> shouldn't be done:*
>
> * I ignored everything on hex. Why? Um, I don't really understand hex that
> well
>   yet, I'd have to go read a bunch of code to understand what folks have
> done,
>   and I'm still learning Erlang by building things myself. None of these
> is a
>   good reason, BTW.
> * I only published my stuff to github, and not to hex. This may be a good
> thing
>   as I'd just be duplicating other, probably better, efforts.
> * What I built should probably be a rebar3 plugin, and not a Makefile with
>   dependencies on rebar3 etc.
>
> However, I'll show you what this is because I think it satisfies a need and
> perhaps we can all come up with a better implementation strategy. Or learn
> something.
>
> ## Erlang-Seed
>
> So, here we have 'yet-another-quick-fix' called [erlang-seed][17]. Let me
> be
> honest, this is a 'quick-fix' to satisfy my own personal needs...
>
> ### The Good
>
> What this does is create an erlang application, inject it into a Docker
> container, and then spin up a local Docker cluster that monitors the
> Erlang app
> and surfaces the metrics via Prometheus and Grafana. So with one command I
> can
> generate an OTP application, all the infrastructure necessary to monitor
> it with
> common tooling, _and_ spin it up and run it on my dev box. I find this to
> be
> really useful. YMMV.
>
> ### The Bad
>
> This is really crufty. It calls into rebar3 to generate the app. It should
> probably be a rebar3 plugin so that it can better evolve with the tooling.
> It
> could then more intelligently inject the necessary dependencies,
> instrument the
> code with metrics, etc. Right now it's gross, as it just overwrites files
> as
> necessary using mustache templates.
>
> ### The Ugly
>
> * Do I have time to maintain this? No.
> * Do I have time to make this a rebar3 plugin? Not right now.
> * Will this probably break over time due to either rebar3, docker, or some
> other dependency changing? Certainly, eventually, yes.
>
>
> ## Summary
>
> The reason I'm sharing this is to just widen the discussion on what the
> Erlang
> education and tooling could look like. I think that generating full stacks
> that
> allow developers to focus more on solving business domain issues rather
> than
> forcing them to understand all the arcane details of whatever is under the
> covers is the 'way to go (TM)'. There's got to be a way to allow
> developers to
> maintain velocity while increasing their depth and scope of knowledge as
> necessary. I've been studying Erlang intensely for at least a month now,
> read
> the books _countless_ times, and read the online docs and I still don't
> quite
> understand how this all fits together. I mean I get the big picture. But
> there's
> a _lot_ of details.
>
> Contrast this with Go. I was writing production code in 2 days. The
> quality was
> production quality after about a week. Back in the day it took me about a
> month
> to become proficient in Java, though it required reading Bloch's book "Java
> Concurrency In Practice" to be 'good'. Python was similar, it took about a
> week
> to ramp up, about a month to become comfortable. In all of these languages
> it
> was a relatively short and easy ramp up and then smooth sailing for years
> and
> years.
>
> With Erlang it's an incredibly steep ramp that goes on and on. I guess if I
> remember that I'm learning the equivalent of a language, an orchestration
> framework (like kubernetes), a database system, etc., then it starts to
> make
> sense.
>
> Anyway, for my part, I'll keep posting things as I figure them out.
>
> -Todd
>
> [0]:https://www.erlang.org/
> [1]:https://ferd.ca/ten-years-of-erlang.html
> [2]:https://youtu.be/Z28SDd9bXcE?t=1260
> [3]:https://github.com/ToddG/experimental/tree/master/erlang/wilderness
> [4]:
> https://github.com/ToddG/experimental/blob/master/erlang/resources/books.md
> [5]:http://zwrob.com/posts
> [6]:https://github.com/asdf-vm/asdf
> [7]:https://www.rebar3.org
> [8]:https://github.com/kerl/kerl
> [9]:https://learnyousomeerlang.com/
> [10]:https://github.com/erlware/relx
> [11]:http://erlang.org/doc/man/systools.html
> [12]:http://erlang.org/doc/man/reltool.html
> [13]:https://github.com/erlang/rebar3
> [14]:https://github.com/erlware/relx
> [15]:http://envirosoftwaresolutions.com
> [16]:https://www.rebar3.org/discuss/5d29172873e1ed00447cb362
> [17]:https://github.com/ToddG/erlang-seed
> [18]:http://zwrob.com/posts/developer-automation/
>
> -Todd
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20190717/7154451f/attachment.htm>