[erlang-questions] Testing of nifs
Wed Oct 5 11:08:34 CEST 2011
On 10/03/11 10:51, Tim Watson wrote:
> On 2 October 2011 22:27, Dmitry Groshev <lambdadmitry@REDACTED
> <mailto:lambdadmitry@REDACTED>> wrote:
> Well, it's about hidden details. Erlang is pretty straightforward
> language; C is a lot more tricky to make right. So there is a lot of
> non obvious bugs that I can't foresee when I write unit tests;
> PropEr's *random* tests are way better in this sense, because you
> just describe the domain of your task, not particular cases. You can
> also test your NIF incrementally, writing property after property
> (or command after command if there is some state involved).
> Speaking about my experience — I'm using PropEr to test this library
> https://github.com/band115/ecirca , that implements mutable circular
> array in Erlang (all calls to this array are constrained to it's
> creator, so I hope this mutability can't be a problem). There are no
> docs right now other than code; I will definitely fix it in a week
> or two when I'll have more time. I've tried to test it with EUnit
> when it was just plain array; but then I've added different types of
> array and realized that I should add like 5 times more code to cover
> all cases, and there was a thought about adding different types of
> values (int16/32/64), so I realized than EUnit is definitely not the
> best tool, just because the space of inputs is so multidimensional
> (and I really wanted to cover as much of it as I can with tests —
> Erlang's VM is very fragile to NIFs after all). So I started to dig
> into PropEr's docs, wrote first commands/checks for statem, compiled
> a debug version of BEAM, made a habit to attach GDB to the process
> running tests (this way in case of segfault, and I had a lot of
> them, I can quickly find where I broke the code) — and right now I
> can be sure to some degree (which is proportional to amount of tests
> that you've asked PropEr to do) that this NIFs won't explode in my
> face tomorrow.
> That makes perfect sense. I've been struggling in my mind with the best
> ways to test linked-in driver code, which is a little different. ...
> At the moment I'm relying on traditional unit testing of the C code
> along with static analysis and integration testing using common_test.
IMO, you should really use PropEr instead of the above. This will take
you a long way into what you want to eventually achieve.
> I guess what I'd really like is something like PULSE (the randomising
> scheduler employed by QuickCheck) to help me identify where my driver
> code might be locking up the emulator or whatever, but for open source
> projects that's not an option. Currently I rely on load tests in a
> mirror of the target environment*s(, but this isn't really good enough
> because if I move from an 8 core to a 16 core machine, problems can
> mysteriously appear that just didn't show up and they're really hard to
> figure out.
There are better techniques than random-based testing for the types of
bugs you want to eliminate. One such tool is Concuerror (available from
https://github.com/mariachris/Concuerror). This is currently work in
progress and we had not had the time to create a proper homepage for it.
However, you can take a look at its (really really basic at this point
-- more to come, hopefully soon) documentation and the examples that its
distribution contains and also get a good idea of the concurrency errors
Concuerror can detect and how to use the tool by reading the paper:
Test-Driven Development of Concurrent Programs using Concuerror
The paper is very comprehensive and I strongly recommend it to anyone
who takes elimination of concurrency errors in Erlang seriously.
PS. If your application is open-source, I'll gladly help you in using
Concuerror, but I suggest we take this off this list, if interested.
More information about the erlang-questions