[erlang-questions] Best practices for QAing distribution?

[Christopher Phillips]

Thanks, I'm actually familiar with netem, and assumed it could play a piece
in any solution people had. I more meant, what are your actual testing
mechanisms in a distributed environment? Do you just start up a couple of
nodes by hand on separate VMs and then manually fire off commands to netem,
run a test suite, then set the interface back to normal? As that seems
inefficient.

I want something that I can write some config, some tests, and reproducibly
have my code exercise the tests, with netsplits, packet delays/loss, etc,
and confirm that my expectations still work. It doesn't seem like it would
be that hard to do, if I assume I'm running on a *nix, as between netem,
iptables, and ifconfig, I can create new interfaces, start my apps
listening on them, and proceed to mess with traffic in all manner of
interesting ways, creating partially disconnected clusters, partitioned
clusters, partitioned nodes, delayed traffic, spotty traffic, etc, and see
what happens. Maybe even a long running, random process that introduces
these in different places in my cluster, a sort of chaos monkey for the
network. Etc.

I figure other people have probably wanted to test things in a similar
manner, so I wanted to see what methods others used.


On Tue, Nov 25, 2014 at 1:54 PM, Felix Gallo <felixgallo@REDACTED> wrote:

> http://www.linuxfoundation.org/collaborate/workgroups/networking/netem
>
> On Tue, Nov 25, 2014 at 9:18 AM, Christopher Phillips <
> lostcolony@REDACTED> wrote:
>
>>
>> Does anyone have a library or a set of best practices for testing
>> netsplits? I.e., given a couple nodes, create a netsplit between them and
>> see what one's distributed application ~really~ does in such a situation?
>> I've looked around a bit online but my google-fu has failed me, as I wasn't
>> able to find anything that seemed to do what I'm looking for, either in
>> Erlang specifically, or just in a general application sense (there's
>> Jepsen, but that seems geared for testing generic distribution libraries in
>> a basic sense, rather than domain specific code, and I'm not very good with
>> Clojure) which seems...odd. I've got some ideas,  but figured I'd ask
>> around to see if anyone has done anything similar first.
>>
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20141125/4be359e7/attachment.htm>