[erlang-questions] Developing killer / open source apps

[Shaun Kruger]

Unfortunately I'm guilty of writing a cool, but under documented open source program.  

What Surrogate does is provide forward proxy (HTTP and SOCKS) or reverse proxy with load balancing.  The HTTP proxy is meant to allow developers to create custom stream modification handlers.  These handlers can change various configuration settings.  For instance, mod_host_pool reads the hostname from the Host: header and routes the traffic to a different load balancer pool based on an mnesia lookup.  

I also implemented a raw proxy handler which allows me to load balance anything that does not need to be protocol aware.  This may be useful for load balancing mysql database connections in some cases.

Surrogate does a lot, but it is to be viewed as a platform for creating other things as is evidenced by all of the behaviours that I have specified (and need to better document).

filter_stream - This is the behaviour for inspecting and modifying the HTTP proxy behaviour based on what data passes through.

gen_balancer - This behaviour allows a developer to implement additional load balance policies.  I only wrote a round robin balancer thus far, but I would like to build others like least connections or response time.

healthcheck - This behaviour allows for creating modules that work with mod_balancer.  healthchecks return status information about a host so it can be automatically removed from a pool.

proxy_mod - This behaviour is used for automatically starting and stopping modules.  This facility is not unlike gen_mod in ejabberd.

proxy_protocol - I recently add this one to allow surrogate to implement multiple protocol handlers.  Before I did this it was just an HTTP/HTTPS/SOCKS proxy.  Now I have been able to add a raw handler, and an http_admin handler.  

There is a lot there.  filter_stream was the first big thing I wanted to do with Surrogate because I thought it would be great if open source load balancers could integrate better with web applications.  To me it is the difference between load balancing and application delivery and I wanted to have more options than F5 when it comes to that.  gen_balancer and heathcheck flowed from that based on the fact that without balance pools that are smart enough to drop nodes then you won't get very far.  

Once I had those basics down I really wanted to add clustering.  This is erlang after all and load balancing really does need to be redundant in any kind of serious environment.  That lead me to start developing some clustering features that I eventually split into their own project: https://github.com/skruger/ClusterSupervisor

Cluster supervisor is integrated through gen_cluster which starts the cluster supervisor application and subscribes to vip up and vip down notification from cluster supervisor.  Some of this is still a bit clunky.  Last I tested it things worked pretty OK, but this is clustering and clustering with service takeover on failure is hard by nature.

Now that I've talked a lot about what is in Surrogate I guess I need to explain how to get started.  That I'm not sure how to do.  I think I suffer an affliction common to many developers; I'm so close to my code that I don't know what isn't obvious to others.  That being said, I would love feedback about what I need to clarify in order to help others get started.  I think I'm going to make myself sit down and write some step by step instructions for installing Surrogate and configuring a forward and reverse HTTP proxy.  Check the wiki tomorrow, I'll see how far I can get on that tonight.  Once I finish that I would appreciate some feedback about what the document is missing.

Shaun

----- Original Message -----
> From: "David Mercer" <dmercer@REDACTED>
> To: "Shaun Kruger" <skruger@REDACTED>, "Nick S" <nick.sfx.1@REDACTED>
> Cc: erlang-questions@REDACTED
> Sent: Wednesday, August 10, 2011 2:49:50 PM
> Subject: RE: [erlang-questions] Developing killer / open source apps
> 
> On Tuesday, August 02, 2011, Shaun Kruger wrote:
> 
> > I could really use some contributions for my proxy server/load
> > balancer
> > project. :)
> > 
> > https://github.com/skruger/Surrogate
> > 
> > I recently modified it to allow for writing multiple protocol
> > handlers
> > (not just HTTP proxy) and I'm working on the admin interface
> > finally.
> > I know it's a shameless plug, but the project really is cool.
> >  Though,
> > I might just think that because I wrote it. :)
> 
> Is there any introduction to this project, a quick-start guide or
> something
> that briefly but specifically explains what it does/what it is
> intended to
> do?
> 
>