[erlang-bugs] Test suite status for "master"/R15A?

[Björn-Egil Dahlberg]

The statement regarding stability are for a released and tagged Erlang/OTP,
i.e. OTP_R14B04, OTP_R14B03.

You can expect less stability during development and especially between
major versions, e.g. R14 -> R15. That said, we try to make things in master
as stable as possible. It must compile and known issues must be resolved
before merged to our development branches, but things do slip between the
cracks.

The number of failures of an entire run usually varies between hardware,
os, environment etc. But the most I have seen have been nowhere near 292
failures. I think the highest i've ever seen have been around 200 and that
was with native compiled stdlib and kernel and a faulty openssl. I don't
have the figures in front of me but emulator, kernel and stdlib runs more
or less clean. System has some undefined functions and MIB compiler error
(faulty testcase). Corba and cos* has some issues in current master if I
remember correctly. If inets, ssl or ssh are bonkers that usually mean a
bad environment.

// Björn-Egil


2011/11/15 Scott Lystig Fritchie <fritchie@REDACTED>

> Hi, all.  I'm not quite certain if this is the best forum for this
> question, but I'll try anyway.  What should I be expecting the status of
> the full Erlang/OTP test suite to be for the "master" branch
> a.k.a. R15A?
>
> I just ran a complete set of tests, per the instructions over at
> https://github.com/erlang/otp/wiki/Running-tests.  I checked out the
> "master" branch of the OTP repo, with this as the last commit:
>
>    # git log | head -6
>    commit 9169a0322066614373ef2d602325e02e90ffc597
>    Merge: 6acdb64 5fbadf9
>    Author: Micael Karlberg <bmk@REDACTED>
>    Date:   Mon Nov 14 17:06:17 2011 +0100
>
>        Merge branch 'bmk/megaco/r15_integration'
>
> ... and built on a 64-bit CentOS 5 machine (well, a virtual machine)
> with 2GB of RAM and 2 virtual i686 CPUs.  After about 7 hours of
> runtime, I see that test_server/index.html reports:
>
>    OK: 9780
>    Failed: 292
>    Skipped (User/Auto): 1228 (544/684)
>    Missing suites: 12
>
> According to the wiki:
>
>    "There will currently be 15 or so failed test cases in the kernel
>    application."
>
> Have I managed to do something bad/stupid with my test environment, or
> are the 292 failures about right for the current R15A code?
>
> -Scott
> _______________________________________________
> erlang-bugs mailing list
> erlang-bugs@REDACTED
> http://erlang.org/mailman/listinfo/erlang-bugs
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-bugs/attachments/20111115/0b107c02/attachment.htm>