Announce: tcpbalance 1.0 for distcc
Scott Lystig Fritchie
Fri Jan 17 08:30:03 CET 2003
Attached is a message I sent to the <> mailing
list yesterday about a TCP load-balancing proxy I wrote in Erlang. Of
the load-balancing TCP proxies I'd found before, none were ideally
suited for use with distcc. So I wrote one I wanted in about an
Of interest to erlang-questions readers? Perhaps. It's an example of
an application, complete with supervisor. The overall packaging isn't
complete, (intentionally) nowhere near Ulf Wiger's recently-posted
example ... but perhaps the README, which describes a bit of fun to be
had with hot code loading and using "appmon" to simulate fault
injection, may be a useful example for introducing the language to
P.S. Distcc is a wonderful utility for helping distribute C & C++
compilation to remote machines ... without having to worry (much)
about compiler versions, file system layout mismatches, header file
differences, etc. See http://distcc.samba.org/. I've used it with
Make and SCons (http://www.scons.org/) with happy results. (Much
happier with SCons, BTW.)
------- Forwarded Message
From: Scott Lystig Fritchie <>
Subject: Announce: tcpbalance 1.0 for distcc
Date: Thu, 16 Jan 2003 17:51:24 -0600
Here's the beginning of the README file I've created for tcpbalance
version 1.0. Nice, original name, huh? The full README file, as well
as the source code, is available at
http://www.snookles.com/erlang/tcpbalance/. Please email me with
comments, bug fixes, critique, etc.
I hope that my proxy is useful in either of two ways:
1. Used as-is to help share distcc back-end hosts for multiple
developers working on multiple workstations.
2. Used as a model for a "real", full-featured proxy that
otherwise does the same as #1.
Special thanks to Martin Pool for his work in bringing distcc version
1.0 to the world.
- --- snip --- snip --- snip --- snip --- snip --- snip --- snip ---
TCPBalance, a load-balancing TCP proxy for distcc
There are dozens of Open Source TCP proxies available, written in
close to a dozen languages, many of them capable of load balancing.
Many of them would work with "distcc". Why write yet another TCP
proxy? Why do it in Erlang?
All of the TCP proxies I found, none appeared to have the following
combination of features:
1. Not be too HTTP-centric to not be able to work with "distcc".
2. Be aware that some back-end hosts may be faster than other
hosts. For each client connection, the proxy should choose the
fastest back-end host that is currently idle.
3. Be aware of back-end hosts with multiple CPUs.
4. When all back-end hosts are busy, make the client wait for the
next available back-end host when it is available, rather than
giving a back-end host more work than it is configured to
5. Detect when a back-end host is down and do something sane (like
avoid giving future jobs to the dead machine).
6. Permit an administrator to put back-end hosts back in service,
take them out of service, as well as add and remove hosts from
the pool without adversely affecting clients using the proxy.
7. Keep basic statistics about back-end hosts and make them
available via HTTP or Telnet.
Features 1-5 were mandatory. Features 6-7 would be nice. In a couple
of hours of Web surfing, I didn't find a TCP proxy that was capable of
doing 1-5, so I decided to write my own.
I knew it would be fairly easy to implement features 6-7 as well as
1-5 in Erlang (see http://www.erlang.org/), so that's what I used.
This proxy has been in use at Caspian Networks for over two months.
It's pretty solid.
This README file is quite long. Sorry about that. However, much of
it is a tutorial for Erlang newbies ... and perhaps a bit of
evangelism. :-) I'll try to keep things straightforward, but I will
also demonstrate some of the nifty communication, fault-tolerance, and
hot code upgrade features of Erlang.
------- End of Forwarded Message
More information about the erlang-questions