<br><br><div class="gmail_quote">On Wed, Feb 16, 2011 at 5:58 AM, Gregory Haskins <span dir="ltr"><<a href="mailto:gregory.haskins@gmail.com">gregory.haskins@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
Hi All,<br>
<br>
I have been struggling trying to wrap my head around what might be the<br>
best-practice for distributing a clustered erlang-application to a<br>
plurality of erlang nodes, and keeping the cluster up to date from a<br>
central location.<br>
<br>
Ideally, the subordinate erlang nodes that have just-enough logic to<br>
connect to one another (e.g. cookies, ssl-keys, gossip-controller<br>
addresses, etc are pre-distributed) but otherwise no specific<br>
personality is known apriori. I can then implement some kind of central<br>
service to remotely load/run a specific application (and any of its<br>
dependencies). IOW: This is really about bootstrap for the time being<br>
(e.g. Hot update is nice, but not a short term requirement at the<br>
moment). The right conceptual vehicle for the payload would be a<br>
standard erlang-release.<br>
<br>
So I have given this a lot of thought and read a ton of the books/doc on<br>
the various options here. For instance, I have seen Joe Armstrong's<br>
examples of sending a lambda down as payload in a message, using<br>
dist_ac, boot-server, release_handler, etc. Every solution seems to<br>
present its own set of problems/limitations, so I am unclear as to what<br>
might be my best option here.<br></blockquote><div><br>Oh dear - confusion reigns.<br><br>There is a fundamental problem - no two distributed systems have the same <br>requirements as regards fault-tolerance, behavior under load, latency, scalability etc.<br>
there are just two many variables.<br><br>So what do we do in practice? - one common solution is the BFI (Brute force and Ignorance) <br>solution. Assume a system of K physically separated nodes. K being say 5..25 (this is all very <br>
hand waving for K < 5 a different approach is probably needed) - for K < 1000 (ish) then<br>full replication of all the code on a local file system is perfectly feasible. Forget everything<br>about hot code upgrades etc. Simple take an entire node out of service, change all the code on the<br>
node, put it back into service. This implies that you implement some kind of data exchange protocol<br>to get all the active data you need from a node before closing it down and to restore the data<br>before you restart it.<br>
<br>This is for planned outages - you can actually do this pretty quickly so it's not so bad - if it takes more<br>than a few seconds then look at your algorithms, something is wrong.<br><br>For non-planned outages (crashes) your design should put the system into a stable state and<br>
try to recover.<br><br>You can of course, make things far more fine grained than this, using load_binary() etc.<br>In my experience is used in development to get a quick turn-round times when developing.<br>It's very convenient to just throw new code at the system to see what will happen, but this is<br>
very different to a planned roll-out of new code in a production system.<br><br>I think you need to think about "what code replacement do I need in development" (my suggestion would<br>be to just slurp new code over without worrying, if it crashes who cares, restart everything) and "how do<br>
I upgrade the code in a production system" (with great care node-at-a-time round-robbin fashion).<br><br>It occurs to me that there is a good opportunity here for a complete framework to support<br>common generic cases - ie think "gen_cluster" rather than gen_supervisor/server. I have seen such beasts<br>
but they have a heck of a lot more callbacks than gen_server :-)<br><br><br><br> </div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<br>
For instance, sending functions as payload only seems to work as long as<br>
that function only invokes modules that already loaded/available on the<br>
remote node. I can, of course, ensure that all required modules are<br>
loaded on the remote node using things like code:load_binary(), but this<br>
would mean I would need to a) figure out what modules are needed as part<br>
of the release I want to distribute, and b) manually load each and every<br>
module to the remote node, right?<br>
<br></blockquote><div><br>Well manually isn't so bad - a program does if for you - Its just a matter of sending a few<br>Kbytes .. MBytes down a wire, hardly worth optimising ...<br> </div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
As far as dist_ac, this seems to be more geared towards creating<br>
active/standby replicas of an app, rather than a process-peer model, so<br>
that's out.<br>
<br>
boot-server is getting warmer, though I really want my grid to use<br>
SSL-mutualauth + cookies, and I am not sure if the erl_prim_loader can<br>
work over SSL inets.<br>
<br>
release_handler is getting warmer, though I still need to address the<br>
bootstrap problem (either pre-load the initial release or use the<br>
erl_prim_loader, as above). The other problem here is its focus is<br>
really on hot-update, and at this juncture I really only care about<br>
remote-update (e.g. cold restart is probably ok for now).<br></blockquote><div> </div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<br>
One solution I thought of is a play on the boot-server model, where I<br>
can have one erlang application that connects in whatever ssl/cookie<br>
protocol I want, contacts the service, downloads the release.tar.gz, and<br>
then fires up a new beam instance with the release...hot updates, if<br>
desired, could just use standard release-handler calls direct to the<br>
subordinate vm. The problem with this solution is then I get into "well<br>
how do I keep my loader-app up to date?" and my head explodes in a<br>
circular reference ;).<br></blockquote><div><br>Why not just have one simple roll-your-own boot-loader that you *never*<br>change. If you make it very simple you'll never have to change it.<br><br>There are bits of the system you can change later and bits you cannot change later.<br>
Make you mind up. Remember that what you can and cannot change at run-time is a design decision.<br>Just make you mind up and don't change it. Your brain will go into a loop if you start thinking<br>about how to change the boot loader which you had previously decided that by design<br>
"could not be changed". This is a common mistake in system design - first you say<br>"X is fixed" you design the system with this in mind. Some person says "what happens if we change X"<br>and you brain goes into a loop - "correct answer - we can't X is fixed"<br>
<br>Having fix points in a design make life really really easy <br><br>Decide that "I will never change the boot loader" and write your own boot loader<br>so that you completely understand it and never change it - think of it as hardware.<br>
<br>I've given an example below<br> </div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<br>
Long story short(er): I suspect I am just making all of this harder<br>
than it needs to be, and there must be a ton of you who are just doing<br>
similar operations all the time. Any help/pointers/guidance/suggestions<br>
appreciated.<br>
<br></blockquote><div><br>It's actually not that difficult. You need to be able to remote stop a system<br>send a file to the stopped system, restart the the stopped system all in a controlled manner.<br><br>I did this on planet lab a few years ago. I made a simple socket server with a layer of encryption.<br>
the bootstrap opened a single port and listened a term-to-binary encoded message.<br>Something like:<br><br>(this the boot loader - never change it - it is so simple you will never have to change it)<br><br>boot(Socket) -><br>
receive<br> {tcp, Socket, Bin} -><br> {M,F,A} = binary_to_term(Bin),<br> apply(M,F,A),<br> boot(Socket);<br> end<br><br>Then I send tuples like {file,write_file,[<<Bin>>]}, etc. to the remote nodes from a single master node.<br>
You can just ping the remote nodes or keep a socket open to detect failure.<br><br>You don't have to use *any* of the release stuff nor the kernel distribution stuff if you don't want to.<br><br>/Joe <br><br><br><br>
<br><br><br><br> </div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
Kind Regards,<br>
-Greg<br>
<br>
</blockquote></div><br>