<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div><span style="background-color: rgba(255, 255, 255, 0);">Thanks Gerhard,</span></div><br><blockquote type="cite"><div dir="ltr"><div dir="ltr"><div dir="ltr"><font color="#000000"><span style="caret-color: rgb(0, 0, 0); background-color: rgba(255, 255, 255, 0);">Anyone on the team can have a local setup as close to production as possible with a single command: make deploy-docker-stack-local . Yes, that's right: make is the secret sauce that holds everything together. </span></font></div></div></div></blockquote><div><br></div>Music to my ears! Since my work is self-funded, minimizing overhead through testing, beta, and start-up phases is very high on my list of essential goals.<div><br></div><div>This detailed overview gives me greater insight and confidence into what’s possible.</div><div><br></div><div>All the best,</div><div><br></div><div>Lloyd<br><br><div id="AppleMailSignature" dir="ltr">Sent from my iPad</div><div dir="ltr"><br>On Apr 17, 2019, at 5:11 AM, Gerhard Lazu <<a href="mailto:gerhard@lazu.co.uk">gerhard@lazu.co.uk</a>> wrote:<br><br></div><blockquote type="cite"><div dir="ltr"><div dir="ltr"><div dir="ltr">We have been running Phoenix in production since 2016. The app serves a few TBs of traffic every month.<div><br></div><div>We used to manage the entire infrastructure using Ansible, including app deploys as Docker containers. We used Concourse for the CI, this is what the entire infrastructure pipeline looked like: <a href="http://pipeline.gerhard.io/#/57" target="_blank">http://pipeline.gerhard.io/#/57</a> . The next couple of slides capture some of the tasks that used to run in various CI jobs. A peek into what used to happen on a new git commit to the app repository: <a href="http://pipeline.gerhard.io/#/60" target="_blank">http://pipeline.gerhard.io/#/60</a> . The entire infrastructure codebase is public: <a href="https://github.com/thechangelog/infrastructure/tree/0fb90109da8ce6f6f263df6b00aaeae56b74e51c" target="_blank">thechangelog/infrastructure</a><div><br></div><div>We have learned so much from this initial way of doing things that we revamped the entire setup earlier this year. The new setup is captured in <a href="https://changelog.com/posts/the-new-changelog-setup-for-2019" target="_blank">The new changelog.com setup for 2019</a>. This is what the current development workflow looks like:</div></div><div><div><br></div><div><image.png><br></div></div><div><br></div><div>My favourite part from the new setup is how <a href="https://github.com/thechangelog/changelog.com/blob/31506f1f6e5ff59f9379cf4519d78ed90ce331bb/docker/changelog.stack.yml" target="_blank">the core config is captured in a Docker stack</a>. This includes monitoring, logging, backups & app updater, as well as the regular 3-tier suspects: proxy, app & db. Because of this approach, <a href="https://github.com/thechangelog/changelog.com/blob/31506f1f6e5ff59f9379cf4519d78ed90ce331bb/docker/changelog.stack.local.yml" target="_blank">the core setup can be reproduced locally</a>, since all IaaS-specific gubbins are isolated in a separate layer. Anyone on the team can have a local setup as close to production as possible with a single command: <font face="monospace, monospace">make deploy-docker-stack-local</font> . Yes, that's right: <font face="monospace, monospace">make</font> is the secret sauce that holds everything together. It's always the simple things that have the potential of <font face="monospace, monospace">making</font> the biggest difference (pun intended):</div><div><br></div><div><div><image.png><br></div></div><div><br></div><div>This is what production looks like right now - <font face="monospace, monospace">make ctop</font></div><div><br></div><div><div><image.png><br></div></div><div><br></div><div>Cheers, Gerhard.</div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 16, 2019 at 11:15 PM Lloyd R. Prentice <<a href="mailto:lloyd@writersglen.com" target="_blank">lloyd@writersglen.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">Hi Dieter,<div><br></div><div>Really helpful! </div><div><br></div><div>From my poking around, Erlang deployment gets too little attention on the web or in the literature— particularly in this age of cloud deployment, containers, and distributed systems. So thanks again for your nuts-and-bolts overview.</div><div><br></div><div>I’d love to read how others have brought Erlang apps and websites into production and lessons learned.</div><div><br></div><div>Best wishes, </div><div><br></div><div>Lloyd</div><div><br><br><div id="gmail-m_-4013064507377907539gmail-m_-2609031999904128258AppleMailSignature" dir="ltr">Sent from my iPad</div><div dir="ltr"><br>On Apr 16, 2019, at 4:24 PM, Dieter Schön <<a href="mailto:dieter@schoen.or.at" target="_blank">dieter@schoen.or.at</a>> wrote:<br><br></div><blockquote type="cite"><div dir="ltr">
<p>Hi Lloyd,</p>
<p>this week I am not in the office, so I can just try to do a brain
dump.</p>
<p>The installation was quite simple, I installed a plain Ubuntu
desktop,</p>
<p>and then extracted the erlang tar file into /opt. This is done by
one or two shell scripts.</p>
<p><br>
</p>
<p>The erlang app is started on demand by another application, which
is launched by the <br>
</p>
<p>operator.</p>
<p>So I do not have to deal with systemd (which I also do not like
very much) or the like.</p>
<p>My application could run as a server, but in the current use case
there is no need for that.</p>
<p><br>
</p>
<p>For the footprint of the docker container I will have to come
back next week.</p>
<p>I really have no idea how big it is. In former times I have used
VirtualBoxes with snapshots</p>
<p>for similar tasks, but if the focus is quite narrow, a docker
container is far more easier and quicker</p>
<p>than a complete VM.</p>
<p><br>
</p>
<p>Regards,</p>
<p>Dieter</p>
<p><br>
</p>
<div class="gmail-m_-4013064507377907539gmail-m_-2609031999904128258moz-cite-prefix">On 16.04.19 21:54,
<a class="gmail-m_-4013064507377907539gmail-m_-2609031999904128258moz-txt-link-abbreviated" href="mailto:lloyd@writersglen.com" target="_blank">lloyd@writersglen.com</a> wrote:<br>
</div>
<blockquote type="cite">
<font size="2" face="arial">
<p style="margin:0px;padding:0px;font-family:arial;font-size:10pt">Hi Dieter,</p>
<p style="margin:0px;padding:0px;font-family:arial;font-size:10pt"> </p>
<p style="margin:0px;padding:0px;font-family:arial;font-size:10pt">Many thanks for sharing your
experience.</p>
<p style="margin:0px;padding:0px;font-family:arial;font-size:10pt"> </p>
<p style="margin:0px;padding:0px;font-family:arial;font-size:10pt">Could you please outline
your installation procedure? Also, the size of your Docker
application image (RAM footprint) ? I've been considering LXC
containers, but I'd like to test with Docker.</p>
<p style="margin:0px;padding:0px;font-family:arial;font-size:10pt"> </p>
<p style="margin:0px;padding:0px;font-family:arial;font-size:10pt">All the best,</p>
<p style="margin:0px;padding:0px;font-family:arial;font-size:10pt"> </p>
<p style="margin:0px;padding:0px;font-family:arial;font-size:10pt">Lloyd</p>
<p style="margin:0px;padding:0px;font-family:arial;font-size:10pt"> </p>
<p style="margin:0px;padding:0px;font-family:arial;font-size:10pt"> </p>
<p style="margin:0px;padding:0px;font-family:arial;font-size:10pt"> </p>
<p style="margin:0px;padding:0px;font-family:arial;font-size:10pt">-----Original Message-----<br>
From: "Dieter Schön" <a class="gmail-m_-4013064507377907539gmail-m_-2609031999904128258moz-txt-link-rfc2396E" href="mailto:dieter@schoen.or.at" target="_blank"><dieter@schoen.or.at></a><br>
Sent: Tuesday, April 16, 2019 3:46pm<br>
To: "Erlang-Questions Questions"
<a class="gmail-m_-4013064507377907539gmail-m_-2609031999904128258moz-txt-link-rfc2396E" href="mailto:erlang-questions@erlang.org" target="_blank"><erlang-questions@erlang.org></a><br>
Subject: Re: [erlang-questions] How do you go to production?<br>
<br>
</p>
<div id="gmail-m_-4013064507377907539gmail-m_-2609031999904128258SafeStyles1555444185">
<div style="font-family:Tahoma;font-size:16px;direction:ltr"><br>
Hi,
<div>here is a case from the other end of the spectrum. </div>
<div>I developed a protocol converter for a satellite test
system, it is installed on 2 (in words: two) PCs.</div>
<div>For development, I used one application and rebar3.
Production is just "rebar3 as prod tar" (From a labelled
git commit).</div>
<div>Testing was done on the two PCs, where I used one as
the system under test and the other as test harness/data
generator.<br>
I was also using wireshark and other third party tools, to
prevent incestuous behaviour.</div>
<div>To test the installation procedure I used docker, which
was really nice. I had a blank machine in two seconds, </div>
<div>where I could load and execute the installation script
from the host. I had turnaround cycles from several
seconds.</div>
<div>What else.. the application is quite small. It fits
into one erlang application. Apart from it, the release
only contains the observer and sasl.</div>
<div>Unit testing was done in EUnit.</div>
<br>
<div>This was my first project where I used erlang, and the
learning curve was quite gentle. </div>
<div>Kind regards,</div>
<div>Dieter</div>
<div><br>
<div>Am So., Apr. 14, 2019 11:20
schrieb Max Lapshin <a class="gmail-m_-4013064507377907539gmail-m_-2609031999904128258moz-txt-link-rfc2396E" href="mailto:max.lapshin@gmail.com" target="_blank"><max.lapshin@gmail.com></a>:</div>
<blockquote>
<div>
<div>
<div dir="ltr">
<div dir="ltr">Hi.
<div>I'm developing Flussonic for almost 10
years and we have some practices for
packaging, deploying, running, maintainig that
are not well known, however they are rather
good for us.</div>
<div>It is interesting how do other people solve
this.</div>
<div>At first, we do not use releases. It is
because when we have started our long path,
using of releases was not very easy. Couple
of days ago I've tested it with rebar3 and it
is really easier to use. Perhaps we would use
releases today.</div>
<div>Next: we use our own fpm script replacement
for packaging. I can boast that it seems to be
the only existing implementation of rpm
outside of original library, however I'd
better never pass this path again. I've
written it in pre-docker era and frankly
speaking it is a traumatic experience.
However, we are using it for debian and it is
really convenient:</div>
<div><a rel="external" href="https://github.com/flussonic/epm" target="_blank">https://github.com/flussonic/epm</a></div>
<div>Some time ago we have switched to systemd.
I personally consider systemd a very badly
designed thing that was created without any
discussions with existing system
adminstrators. For example, systemd doesn't
offer config validation before launch. Another
brilliant idea is to offer libsystemd for
linking into application. Unknown library with
unknown quality. What can go wrong if you link
it into your erlang or java application?</div>
<div>Sorry, but no. We have systemd.erl: <a rel="external" href="https://gist.github.com/maxlapshin/01773f0fca706acdcb4acb77d91d78bb" target="_blank">https://gist.github.com/maxlapshin/01773f0fca706acdcb4acb77d91d78bb</a></div>
<div>
<div>Use Type=notify in youdaemon.service</div>
</div>
<div>After you manage to launch your erlang
daemon, you need to collect statistics. We had
to add some more linux-related tools to fetch:
cpu usage, disk I/O usage, system ram usage
(swap, etc), per-interface network statistics,
udp errors count, nvidia card usage, etc. </div>
<div>If this is worthy outsource, I think we can
extract it. </div>
<div>Our os_stat library is linked with our
in-erlang pulsedb library. We try to maintain
as less dependencies as possible, so we
collect all ticks from monitoring tools inside
erlang library pulsedb: <a rel="external" href="https://github.com/pulsedb/pulsedb" target="_blank">https://github.com/pulsedb/pulsedb</a>
(maybe should update public branch)</div>
<div>It can save several thousands metrics with
one tick per 1-3 seconds.</div>
<div>Support is an important part of our
business, because customers cannot just launch
software, they often need help. We have many
people in support staff and I do not want to
manage their own public ssh keys on customer's
servers. So we have written an ssh proxy:
system that login to customer server with one
private key and allow support guy to use his
own key: <a rel="external" href="https://github.com/flussonic/ssh-proxy" target="_blank">https://github.com/flussonic/ssh-proxy</a></div>
<div>All these things are rather useless for
development and many of them are not required
for in-house development, however it is hard
to live without them when you sell software.</div>
<div>What is your experience with such things
that standard erlang lacks?</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</font>
<br>
<fieldset class="gmail-m_-4013064507377907539gmail-m_-2609031999904128258mimeAttachmentHeader"></fieldset>
<pre class="gmail-m_-4013064507377907539gmail-m_-2609031999904128258moz-quote-pre">_______________________________________________
erlang-questions mailing list
<a class="gmail-m_-4013064507377907539gmail-m_-2609031999904128258moz-txt-link-abbreviated" href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a>
<a class="gmail-m_-4013064507377907539gmail-m_-2609031999904128258moz-txt-link-freetext" href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a>
</pre>
</blockquote>
</div></blockquote><blockquote type="cite"><div dir="ltr"><span>_______________________________________________</span><br><span>erlang-questions mailing list</span><br><span><a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a></span><br><span><a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a></span><br></div></blockquote></div></div>_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" rel="noreferrer" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
</blockquote></div>
</div></blockquote></div></body></html>