At least I've ended with such solution: https://github.com/maxlapshin/elixir/blob/master/src/elixir_tracker.erl it is just like mochiweb reloader, that looks for changed scripts and create proxy modules for them.