more real-world static analysis results

[Tobias Lindahl]

Hello,

During EUC I got the question if Dialyzer would find the things that xref
does, and the answer is (as Matthias found out) that although there can be
a certain overlap it is generally a good idea to run both tools. There is
no reason for Dialyzer to try to capture all things that xref does,
basically since xref is there and does a good job.

As for the spurious warnings... Well, list_to_binary is a bit special and
I will look into having a special case for it. I want to stress that since
the aim is to have no spurious warnings we are really interested in
knowing these things. Please send us all warnings that you find strange so
that we can have a look at them. Also, we are really interested in knowing
about bugs you find that is _not_ found by Dialyzer, but that you think
should have been, so that we can make an effort to find them.

Tobias

PS Thanks for the compliment, Matthias :)


On Sun, 24 Oct 2004, Matthias Lang wrote:

> Hi,
>
> inspired by Tobias' excellent presentation (best of the conference,
> IMO) at the EUC, I ran a couple of static analysis tools on a subset
> of Corelatus' production code. This code has an extensive test-suite
> and much of it has been in production use for years.
>
> Presenting the results in the same format as the paper, as run on
> about 20kloc of code
>
>   Tool     |  Explosives    Camouflages    Cemeteries    Spurious
>  ----------+-----------------------------------------------------------
>  Dialyzer  |  2             1              3             2
>  xref      |  1             0              3             0
>
> The 'Dialyzer' paper observes that the bugs found by static analysis
> in general, and dialyzer in particular, tend to be shallow. The bugs
> found above are indeed all simple bugs---easy to fix and not causing
> any real trouble.
>
> Breaking the bugs down:
>
> True bugs (code shown is simplified for clarity)
> ---------
>
>  1.  String = "string" ++ 5 + (integer_to_list(N - 5) div 2) ++ "A".
>
>  2.  case is_ip4_addr(Addr) of              %% returns {ok, Tuple} | error
>        {ok, _} -> E#eth_params{gw = Addr};
>        {error, Reason} -> {error, Reason}
>      end;
>
>  3.  Call to a module which doesn't exist (typo in module name)
>
>  The first two were found by dialyzer, the last by xref.
>
>  These bugs were missed by our test suite because the offending lines
>  of code were not exercised (covered) by any test.
>
>  These bugs were "missed" by live operation for various reasons.
>  #1 only happens in a hardware configuration which isn't currently used.
>  #2 only happens if an external system does the wrong thing first.
>  #3 only happens in a feature which isn't currently used.
>
>  Dialyzer also found this code:
>
>      Bin = list_to_binary([1|<<2>>])
>
>  The list is improper, but it doesn't matter because list_to_binary/1
>  works as expected on that type of improper list, which is also why
>  it passes the test suite.
>
> Non-bugs
> --------
>
>   The remaining eight problems found by dialyzer and xref were
>   non-bugs. In all cases the warnings can be eliminated by changing
>   the code slightly, e.g. I have to stop using this idiom to mark
>   work-in-progress:
>
>      die = nyi,   % nyi = not-yet-implemented
>
> I'm currently playing around trying to get 'cover' to work on my
> target system. It's not entirely trivial because it requires a fairly
> complete erlang environment and is relatively memory hungry.
>
> Matt
>