Announce: tcpbalance 1.0 for distcc

[Scott Lystig Fritchie]

Attached is a message I sent to the <distcc@REDACTED> mailing
list yesterday about a TCP load-balancing proxy I wrote in Erlang.  Of
the load-balancing TCP proxies I'd found before, none were ideally
suited for use with distcc.  So I wrote one I wanted in about an
afternoon.  :-)

Of interest to erlang-questions readers?  Perhaps.  It's an example of
an application, complete with supervisor.  The overall packaging isn't
complete, (intentionally) nowhere near Ulf Wiger's recently-posted
example ... but perhaps the README, which describes a bit of fun to be
had with hot code loading and using "appmon" to simulate fault
injection, may be a useful example for introducing the language to
newcomers?

-Scott

P.S.  Distcc is a wonderful utility for helping distribute C & C++
compilation to remote machines ... without having to worry (much)
about compiler versions, file system layout mismatches, header file
differences, etc.  See http://distcc.samba.org/.  I've used it with
Make and SCons (http://www.scons.org/) with happy results.  (Much
happier with SCons, BTW.)

------- Forwarded Message

From: Scott Lystig Fritchie <fritchie@REDACTED>
To: distcc@REDACTED
Subject: Announce: tcpbalance 1.0 for distcc
Date: Thu, 16 Jan 2003 17:51:24 -0600

Here's the beginning of the README file I've created for tcpbalance
version 1.0.  Nice, original name, huh?  The full README file, as well
as the source code, is available at
http://www.snookles.com/erlang/tcpbalance/.  Please email me with
comments, bug fixes, critique, etc.

I hope that my proxy is useful in either of two ways:

    1. Used as-is to help share distcc back-end hosts for multiple
       developers working on multiple workstations.

    2. Used as a model for a "real", full-featured proxy that
       otherwise does the same as #1.

Special thanks to Martin Pool for his work in bringing distcc version
1.0 to the world.

- -Scott

- --- snip --- snip --- snip --- snip --- snip --- snip --- snip --- 


TCPBalance, a load-balancing TCP proxy for distcc
=================================================

There are dozens of Open Source TCP proxies available, written in
close to a dozen languages, many of them capable of load balancing.
Many of them would work with "distcc".  Why write yet another TCP
proxy?  Why do it in Erlang?

All of the TCP proxies I found, none appeared to have the following
combination of features:

    1. Not be too HTTP-centric to not be able to work with "distcc".

    2. Be aware that some back-end hosts may be faster than other
       hosts.  For each client connection, the proxy should choose the
       fastest back-end host that is currently idle.

    3. Be aware of back-end hosts with multiple CPUs.

    4. When all back-end hosts are busy, make the client wait for the
       next available back-end host when it is available, rather than
       giving a back-end host more work than it is configured to
       handle.

    5. Detect when a back-end host is down and do something sane (like
       avoid giving future jobs to the dead machine).  

    6. Permit an administrator to put back-end hosts back in service,
       take them out of service, as well as add and remove hosts from
       the pool without adversely affecting clients using the proxy.

    7. Keep basic statistics about back-end hosts and make them
       available via HTTP or Telnet.

Features 1-5 were mandatory.  Features 6-7 would be nice.  In a couple
of hours of Web surfing, I didn't find a TCP proxy that was capable of
doing 1-5, so I decided to write my own.

I knew it would be fairly easy to implement features 6-7 as well as
1-5 in Erlang (see http://www.erlang.org/), so that's what I used.

This proxy has been in use at Caspian Networks for over two months.
It's pretty solid.

This README file is quite long.  Sorry about that.  However, much of
it is a tutorial for Erlang newbies ... and perhaps a bit of
evangelism.  :-)  I'll try to keep things straightforward, but I will
also demonstrate some of the nifty communication, fault-tolerance, and
hot code upgrade features of Erlang.

------- End of Forwarded Message