[erlang-questions] Testing of nifs

[Tim Watson]

On 2 October 2011 22:27, Dmitry Groshev <lambdadmitry@REDACTED> wrote:

> Well, it's about hidden details. Erlang is pretty straightforward language;
> C is a lot more tricky to make right. So there is a lot of non obvious bugs
> that I can't foresee when I write unit tests; PropEr's *random* tests are
> way better in this sense, because you just describe the domain of your task,
> not particular cases. You can also test your NIF incrementally, writing
> property after property (or command after command if there is some state
> involved).
> Speaking about my experience — I'm using PropEr to test this library
> https://github.com/band115/ecirca , that implements mutable circular array
> in Erlang (all calls to this array are constrained to it's creator, so I
> hope this mutability can't be a problem). There are no docs right now other
> than code; I will definitely fix it in a week or two when I'll have more
> time. I've tried to test it with EUnit when it was just plain array; but
> then I've added different types of array and realized that I should add like
> 5 times more code to cover all cases, and there was a thought about adding
> different types of values (int16/32/64), so I realized than EUnit is
> definitely not the best tool, just because the space of inputs is so
> multidimensional (and I really wanted to cover as much of it as I can with
> tests — Erlang's VM is very fragile to NIFs after all). So I started to dig
> into PropEr's docs, wrote first commands/checks for statem, compiled a debug
> version of BEAM, made a habit to attach GDB to the process running tests
> (this way in case of segfault, and I had a lot of them, I can quickly find
> where I broke the code) — and right now I can be sure to some degree (which
> is proportional to amount of tests that you've asked PropEr to do) that this
> NIFs won't explode in my face tomorrow.
>
>
That makes perfect sense. I've been struggling in my mind with the best ways
to test linked-in driver code, which is a little different. The main reason
I reach for a driver is that I don't want to block the VM and will probably
use the asyc driver APIs or a custom thread pool. Separating the driver
interface code from the work it is meant to do seems useful. I can provide a
stub implementation for testing that will, for example, fake running out of
memory at certain points or return "unexpected" errors. But most often these
drivers are calling into some C/C++ library to do something like run as XSLT
transformation or talk to a database, etc. In these cases, stubbing out the
implementation isn't enough to know that the API calls I'm make are going to
work as invalid XML or nonsensical SQL will just puke in a real scenario.

At the moment I'm relying on traditional unit testing of the C code along
with static analysis and integration testing using common_test. I guess what
I'd really like is something like PULSE (the randomising scheduler employed
by QuickCheck) to help me identify where my driver code might be locking up
the emulator or whatever, but for open source projects that's not an option.
Currently I rely on load tests in a mirror of the target environment*s(, but
this isn't really good enough because if I move from an 8 core to a 16 core
machine, problems can mysteriously appear that just didn't show up and
they're really hard to figure out.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20111003/e762e0d7/attachment.htm>