[erlang-bugs] Test suite status for "master"/R15A?

[Henrik Nord]

Now that the ssl fixes where merged to 'master' the test fails for 'pu' 
on my local build dropped "abit"
Latest Test Result 
<http://bilbo.otp.ericsson.se:8080/job/otp_test_pu/lastCompletedBuild/testReport/> (56 
failures / -171).

So if you are using the new version of OpenSSL you might see a drop 
after yesterdays push

On 11/15/2011 10:56 PM, Björn-Egil Dahlberg wrote:
> The statement regarding stability are for a released and tagged 
> Erlang/OTP, i.e. OTP_R14B04, OTP_R14B03.
>
> You can expect less stability during development and especially 
> between major versions, e.g. R14 -> R15. That said, we try to make 
> things in master as stable as possible. It must compile and known 
> issues must be resolved before merged to our development branches, but 
> things do slip between the cracks.
>
> The number of failures of an entire run usually varies between 
> hardware, os, environment etc. But the most I have seen have been 
> nowhere near 292 failures. I think the highest i've ever seen have 
> been around 200 and that was with native compiled stdlib and kernel 
> and a faulty openssl. I don't have the figures in front of me but 
> emulator, kernel and stdlib runs more or less clean. System has some 
> undefined functions and MIB compiler error (faulty testcase). Corba 
> and cos* has some issues in current master if I remember correctly. If 
> inets, ssl or ssh are bonkers that usually mean a bad environment.
>
> // Björn-Egil
>
>
> 2011/11/15 Scott Lystig Fritchie <fritchie@REDACTED 
> <mailto:fritchie@REDACTED>>
>
>     Hi, all.  I'm not quite certain if this is the best forum for this
>     question, but I'll try anyway.  What should I be expecting the
>     status of
>     the full Erlang/OTP test suite to be for the "master" branch
>     a.k.a. R15A?
>
>     I just ran a complete set of tests, per the instructions over at
>     https://github.com/erlang/otp/wiki/Running-tests.  I checked out the
>     "master" branch of the OTP repo, with this as the last commit:
>
>        # git log | head -6
>        commit 9169a0322066614373ef2d602325e02e90ffc597
>        Merge: 6acdb64 5fbadf9
>        Author: Micael Karlberg <bmk@REDACTED <mailto:bmk@REDACTED>>
>        Date:   Mon Nov 14 17:06:17 2011 +0100
>
>            Merge branch 'bmk/megaco/r15_integration'
>
>     ... and built on a 64-bit CentOS 5 machine (well, a virtual machine)
>     with 2GB of RAM and 2 virtual i686 CPUs.  After about 7 hours of
>     runtime, I see that test_server/index.html reports:
>
>        OK: 9780
>        Failed: 292
>        Skipped (User/Auto): 1228 (544/684)
>        Missing suites: 12
>
>     According to the wiki:
>
>        "There will currently be 15 or so failed test cases in the kernel
>        application."
>
>     Have I managed to do something bad/stupid with my test environment, or
>     are the 292 failures about right for the current R15A code?
>
>     -Scott
>     _______________________________________________
>     erlang-bugs mailing list
>     erlang-bugs@REDACTED <mailto:erlang-bugs@REDACTED>
>     http://erlang.org/mailman/listinfo/erlang-bugs
>
>
>
>
> _______________________________________________
> erlang-bugs mailing list
> erlang-bugs@REDACTED
> http://erlang.org/mailman/listinfo/erlang-bugs


-- 
/Henrik Nord Erlang/OTP

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-bugs/attachments/20111116/e93bcc6d/attachment.htm>