Managing Experiments

[Tim Fletcher]

> Basically I need to be able to encapsulate and dynamically swap out
> different bits of functionality throughout the application based upon
> who is the current user. Anyone done something similar to this or have
> any insights into how they might approach such a problem?

A system for "enabling features for a subset of users" is more general
than A/B testing. Here are some relevant slides:

    http://www.paulhammond.org/2010/06/trunk/

As with your example, the approach described basically amounts to
branching within the application code using guard conditions.


> - case statements littered throughout the code

This is my preference, because it's simple. As the aforementioned
slides quote: "There is one consumer of the software: you". Adding
this test code is adding a feature that delivers value to you, so
instead of litter just think of it as part of the application.


If you want to "hot swap" the guard conditions without having to re-
deploy your entire application then you could store the code for the
guard functions external to the app (file system, mnesia, redis etc)
and evaluate them on the fly.


As for A/B testing specifically, it may be useful to look at existing
frameworks for ideas:

    http://vanity.labnotes.org/ab_testing.html

    http://www.bingocardcreator.com/abingo/usage


Hope that helps.

Tim