[erlang-questions] Managing Experiments

[Jesper Louis Andersen]

On Sun, Jun 27, 2010 at 5:58 AM, Jeremy Raymond <jeraymond@REDACTED> wrote:

> case user_in_experiment(UserId, new_feature_x) of
>   true ->
>      % do experiment functionality for new feature 'x'
>   false ->
>      % do current functionality
> end

I'd do something like:

ets:new(ab_test_user, [named_table, public]),
NewFeature = fun() -> ... end,
ets:insert(ab_test, {{UserId, new_feature_x}, NewFeature}),
...

case ets:lookup(ab_test, {UserId, Feature}) of
  [] ->
     CallNormalFunctionality();
  [{K, F}] ->
     F()
end,
...

That is, wrap up the functionality for the test-group into a function
and store those usernames which should get treatment (the non-matching
users are the control group). Once in place, you can avoid having to
change the code anymore as it is now essentially a question of
manipulating data (and loading new test functions). The only caveat is
that I don't know what happens in the ERTS when you do this. How are
the function closures stored? (shared?) and what happens when you
hot-load new modules onto the system from which you drew the functions
(i.e., can you upgrade your ab_test by hot-loading).
-- 
J.