[erlang-questions] [ANN]: Damocles, a library for testing distribution scenarios on a single machine

Wed Jan 7 19:28:45 CET 2015

Yeah, I've got a few ideas on how I could do it. Just gonna take a lot of
playing and testing. Thanks again for bringing up tuntap though (and for
writing a library to make using them easier ), it's opened up some new
possibilities for when I've got time (one more feature against the current
code base and it does everything I want it to for my current project, but
rewriting to leverage tuns or taps would let me subsume all the related
functionalities I've seen/thought of in some manner or other, in a portable
manner, which is...really, really cool).

On Wed, Jan 7, 2015 at 12:42 PM, Michael Santos <michael.santos@REDACTED>
wrote:

> On Wed, Jan 07, 2015 at 11:00:42AM -0500, Christopher Phillips wrote:
> >   Thanks for that. Yeah, I had a look at your tuntap library, and it
> looks
> > like both a portable way to reimplement the existing functionality, but
> > also some ways to add some features more easily that I intended anyway,
> and
> > some I had mentally dismissed as too unpleasant. I just need to find time
> > to play with it and confirm/evolve my mental model. I'm thinking I'm
> going
> > to try dropping one more feature on the existing code base, then start
> > experimenting on a branch.
>
> Yeah, just a suggestion, not sure it will work out for you. By the way,
> very cool project!
>
> > There's some complexity in managing multiple
> > destinations from one source, as well as the actual implementation of
> > drop/delay (I need to make it so queued packets don't chain their delay.
> > I.e., if I changed the code in that example to have a solid 1 second
> delay,
> > for every frame coming in, and I got 5 frames in rapid succession, I'd
> have
> > one received at the destination after one second, the next after two, the
> > next after three, etc. I need to track when I received them and apply
> rules
> > appropriately, across multiple nodes)
>
> One way would be to have a number of buckets for holding frames. Each
> bucket is an erlang process. Frames are distributed across the processes
> and if the mailbox queue exceeds some size, new frames are discarded.
>
> > On Wed, Jan 7, 2015 at 10:20 AM, Michael Santos <
> michael.santos@REDACTED>
> > wrote:
> >
> > > On Mon, Jan 05, 2015 at 08:55:06PM -0500, Christopher Phillips wrote:
> > > >   Hmm, I've had so little experience with tuntap interfaces I didn't
> > > think
> > > > of them. That sounds like the way to go to get things working on OSX,
> > > > though at a much larger amount of effort (both the learning curve on
> my
> > > > part, and re-implementing the degradation behaviors I get for free
> from
> > > the
> > > > kernel). Thanks for suggesting it; once I get things to where it's
> usable
> > > > on Linux in all the ways I want, some research and a rewrite may be
> in
> > > > order to get it fully portable.
> > >
> > > Sure, that is definitely reasonable.
> > >
> > > Working with tuntap devices isn't too difficult. For comparison, here
> is
> > > a very simple example that forwards ethernet frames over erlang
> > > distribution and randomly drops or delays a percentage of the frames:
> > >
> > > https://gist.github.com/msantos/f2823fcba40975003dc3
> > >
> > > Set up 2 distributed nodes:
> > >
> > >     erl -name n@REDACTED -setcookie CHOCOLATEBUTTERSCOTCH
> > >     erl -name n@REDACTED -setcookie CHOCOLATEBUTTERSCOTCH
> > >
> > > Then create the tunnel:
> > >
> > >     % Drop 10% of the frames, randomly delay 40% up to 1 second
> > >     wastrel:start('n@REDACTED', "10.1.1.1", "10.1.1.2", 0.1, 0.4).
> > >
> > > Try to login using ssh (or whatever)!
> > >
> > >     ssh 10.1.1.2
> > >
> > > Meh, I've had to use worse ...
> > >
> > > Notes:
> > > * relies on this erlang tuntap driver
> > >
> > >     https://github.com/msantos/tunctl
> > >
> > > * beam needs to be running as root or have CAP_NET_ADMIN set:
> > >
> > >     sudo setcap cap_net_admin=ep /usr/local/lib/erlang/erts-*/bin/beam*
> > >
> > > > On Mon, Jan 5, 2015 at 8:11 PM, Michael Santos <
> michael.santos@REDACTED
> > > >
> > > > wrote:
> > > >
> > > > > On Mon, Jan 05, 2015 at 09:16:42AM -0500, Christopher Phillips
> wrote:
> > > > > > For OSX ipfw was deprecated in Lion and removed in Yosemite. I've
> > > done a
> > > > > > bit of looking at the replacement, pf, and it looks like dropping
> > > packets
> > > > > > based on percentage is doable, as is bandwidth throttling
> (something
> > > I'd
> > > > > > like to add, in general), but I don't see any way to induce a
> delay,
> > > > > beyond
> > > > > > an implicit one based on tos prioritization. If someone knows how
> > > and can
> > > > > > point me in the right direction I'd appreciate it.
> > > > >
> > > > > The portable way is to use a tuntap device. Then you can
> arbitrarily
> > > > > drop packets, throttle bandwidth, introduce latency, whatever, from
> > > your
> > > > > code. Sort of like quickcheck for networks :)
> > > > >
> > > > > > On Mon, Jan 5, 2015 at 6:52 AM, Sergej Jurečko <
> > > sergej.jurecko@REDACTED
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > This looks like a great tool and something that could easily be
> > > added
> > > > > to
> > > > > > > unit tests.
> > > > > > > Anyone with ipfw skills to add bsd/osx support?
> > > > > > >
> > > > > > > Sergej
> > > > > > > On Jan 5, 2015 1:41 AM, "Christopher Phillips" <
> > > lostcolony@REDACTED>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> https://github.com/lostcolony/damocles
> > > > > > >>
> > > > > > >> I asked a while back on this mailing list if anyone had any
> useful
> > > > > > >> libraries or similar for testing distribution scenarios. I
> only
> > > got
> > > > > back a
> > > > > > >> few responses (maybe co-op riak_test? Maybe make use of the
> > > underlying
> > > > > > >> Linux traffic control and network emulation apps?), and my own
> > > > > searches,
> > > > > > >> while finding a few libraries, didn't find anything I could
> easily
> > > > > co-op
> > > > > > >> for my purposes.
> > > > > > >>
> > > > > > >> To that end, I went ahead and spent part of my break on this,
> and
> > > it
> > > > > just
> > > > > > >> got sufficiently feature complete to throw out there. I
> haven't
> > > had a
> > > > > > >> chance to really start using it heavily, and I've only been
> > > testing
> > > > > it on
> > > > > > >> my dev box, but a basic run through of the functionality as I
> > > typed
> > > > > up the
> > > > > > >> readme worked (so any issues being pointed out would be
> > > appreciated).
> > > > > > >> Essentially, it allows you to create and manipulate local
> > > interfaces
> > > > > on a
> > > > > > >> Linux machine to emulate packet delay and loss (using the
> > > underlying
> > > > > > >> traffic control and network emulation mechanisms), with a
> number
> > > of
> > > > > > >> convenience methods to (hopefully) easily describe fairly
> > > intricate
> > > > > > >> distribution scenarios.
> > > > > > >>
> > > > > > >> Things like "create these 5 interfaces, (now from my test
> code,
> > > > > launch a
> > > > > > >> copy of my app on each one, or even a different app on one of
> > > them,
> > > > > to see
> > > > > > >> what happens when that resource is flaky); now make it so 1
> and 2
> > > > > can't
> > > > > > >> talk to 3 and 4, and vice versa, but everyone can still talk
> to
> > > 5, but
> > > > > > >> replies have a 50% chance of being dropped from 5 when
> responding
> > > to
> > > > > 1 and
> > > > > > >> 2, and there's a 300ms delay between 3 and 4; (now, let's run
> > > more of
> > > > > our
> > > > > > >> test code to assert that trying to write to any node still
> > > succeeds);
> > > > > okay,
> > > > > > >> now let's restore the network back to normal (and have our
> test
> > > code
> > > > > make
> > > > > > >> sure the write was retained)", or whatever, can be set up in a
> > > > > > >> straightforward, automated manner as part of a common test
> run,
> > > and
> > > > > not be
> > > > > > >> reliant on certain VMs being up, nor the tests being run on a
> > > specific
> > > > > > >> network. The tradeoff, obviously, being that you can't really
> load
> > > > > test
> > > > > > >> things with it. Still, it fits my basic needs, and I figured
> it
> > > might
> > > > > be of
> > > > > > >> use to others.
> > > > > > >>
> > > > > > >> I'll be adding some simple examples when I next get free time
> (I
> > > ran
> > > > > out
> > > > > > >> of it from the holiday break without getting to them; dunno
> when I
> > > > > will),
> > > > > > >> and will try and get to any bugs or simple suggestions in a
> timely
> > > > > manner,
> > > > > > >> but hopefully it's fairly straightforward and useful as is.
> > > > > > >>
> > > > > > >> _______________________________________________
> > > > > > >> erlang-questions mailing list
> > > > > > >> erlang-questions@REDACTED
> > > > > > >> http://erlang.org/mailman/listinfo/erlang-questions
> > > > > > >>
> > > > > > >>
> > > > >
> > > > > > _______________________________________________
> > > > > > erlang-questions mailing list
> > > > > > erlang-questions@REDACTED
> > > > > > http://erlang.org/mailman/listinfo/erlang-questions
> > > > >
> > > > >
> > >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20150107/02aa20dd/attachment.htm>