[erlang-questions] Energy Analytics Project : Post 01

[Todd]

Hi everyone!

First, thank you to everyone that responded to me both on-list and off-list
about my 'Where is the Erlang Golden Path?' (see
http://zwrob.com/posts/erlang-01/).

It made my year to get a compliment from ROK :-).

I'd like to follow up with what I've been working on all summer. I have a lot
of thoughts about Erlang and such... but (spoiler) I'm not going to talk about
any of that. Instead I'm going to invite you all to help me combat climate
change with my new GPL3 project, the "Energy Analytics Project".

Here's the first in a series of posts on it: http://zwrob.com/posts/eap01/
Here's the actual project (EAP): https://github.com/energy-analytics-project/energy-dashboard

And here's my pitch...

I emailed the Erlang list a few years ago and tried to get a thread going about
how to deal with climate change. But let's face it. It's huge and demoralizing
and just talking about this is depressing. Only a few people joined the
conversation and it fizzled quickly. Fast forward to this spring.

I quit my job and decided to learn Erlang. That was one of two goals. The other
goal was to create a company focused on addressing climate change, and the
first product was the EAP. BTW - that company exists as an LLC, but in reality,
I haven't made a dime. I've been happily unemployed, holed up in my house,
writing code for 10 hours a day all summer... at first working on Goal #1,
learn Erlang.

Along the way I discovered how much I *love* Erlang. I really do. But, I also
realized it was the wrong language for this project. In fact, once I really
learned Erlang, I realized that this was the wrong goal anyway. Goal #1 should
have been, combat climate change...write the EAP.

Timeline:

* The first versions of EAP were bash scripts.
* The next versions were in Erlang. I'll write about how cool this was soon.
* The latest versions have all been in Python3

The problem I ran into is that my envisioned clients are data scientists not
Erlang developers. I want them to be self sufficient, able to create and make
changes to the pipelines etc. If everything was in Erlang, that would never
happen and I would be stuck managing this project forever. Python, however, is
the LinguaFranca of data science. And I know Python reasonably well. So I
pivoted to Python3. Again, I have lots of observations from implementing the
EAP in both languages, and I promise to write them up soon.

In summary, even though my climate change email thread fizzled out a few years
ago, now I have a project that has clear technical and domain needs.

Here's what I'm looking for:

* concept validation
    * are these useful data-sets?
    * is it useful to expose data as Sqlite3 databases?
* notebooks
    * we need actual notebooks that answer real questions
    * we need real data scientists to help make solid Jupyter Notebook
      prototypes to base the notebooks to follow (I'm not a data scientist, I
      know only the most basic things about Pandas, and it's been a full-time
      effort getting all this put together... this is an area I definitely need
      help in)
* data-sets
    * what are the most useful data sets that we currently have?
    * where can we source similar data from all the other Western States?
    * what additional data do we need in order to answer these regional questions?
* domain knowledge
    * If you, or someone you know, is an energy sector domain expert, then
      please join (I'm also not an energy sector specialist)
* documentation
* tests

At this point, I have 20 out of the 90+ data feeds published. I'm fully
committed to getting the remaining 80 CAISO/OASIS feeds published and available.
But I'm at 100% capacity just getting this project off the ground. All the
stuff mentioned above, and anything I've forgotten about... well, that's why I
need help. I also need to get back to paid work soon, so I won't be able to
devote my full attention to this.

-----------------------------------------------------------------------------
Anyone that would like to contribute to this project, please jump in:

https://github.com/energy-analytics-project/energy-dashboard
-----------------------------------------------------------------------------

Note: if any of you are in other countries and want to adopt this project for
your own needs. Go for it. It's all GPL3.

-Todd

Todd Greenwood-Geer
http://zwrob.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20190924/db180b1b/attachment.htm>