[erlang-questions] Desperately looking for official Erlang tests results

[REIX, Tony]

Resending this email WITHOUT the attachment since it is too big to be sent without some approval.


Cordialement,

Tony Reix

Bull - ATOS
IBM Coop Architect & Technical Leader
Office : +33 (0) 4 76 29 72 67
1 rue de Provence - 38432 Échirolles - France
www.atos.net<http://www.atos.net/>
________________________________
De : REIX, Tony
Envoyé : jeudi 2 mars 2017 16:19
À : Björn Gustavsson
Cc : erlang-questions@REDACTED; GIRARDET, JEAN
Objet : RE:[erlang-questions] Desperately looking for official Erlang tests results

Hi Björn

Compressed test results file from FC25/x86_64 is attached to this email: /tmp/erlang-19.2.3-1.fc25.tests.sh.res2.xz .

About kernel_test, there are failed tests common to both FC25/x86_64 and AIX, and failed tests specific to each of the 2 OS.

1) On FC25/x86_64 :

# grep "Testing tests.kernel_test" *tests*res2 | grep FAILED
1> Testing tests.kernel_test: *** FAILED test case 125 of 750 ***
1> Testing tests.kernel_test: *** FAILED test case 157 of 750 ***
1> Testing tests.kernel_test: *** FAILED test case 158 of 750 ***
1> Testing tests.kernel_test: *** FAILED test case 160 of 750 ***
1> Testing tests.kernel_test: *** FAILED test case 161 of 750 ***
1> Testing tests.kernel_test: *** FAILED test case 241 of 750 ***
1> Testing tests.kernel_test: *** FAILED test case 244 of 750 ***
1> Testing tests.kernel_test: *** FAILED test case 259 of 750 ***
1> Testing tests.kernel_test: *** FAILED test case 261 of 750 ***
1> Testing tests.kernel_test: *** FAILED test case 262 of 750 ***
1> Testing tests.kernel_test: *** FAILED test case 294 of 750 *** x86_64 only
1> Testing tests.kernel_test: *** FAILED test case 307 of 750 *** x86_64 only
1> Testing tests.kernel_test: *** FAILED {gen_tcp_api_SUITE,init_per_suite} ***
1> Testing tests.kernel_test: *** FAILED test case 384 of 750 *** x86_64 only
1> Testing tests.kernel_test: *** FAILED test case 417 of 750 ***
1> Testing tests.kernel_test: *** FAILED test case 561 of 750 *** x86_64 only
1> Testing tests.kernel_test: *** FAILED test case 562 of 750 *** x86_64 only
1> Testing tests.kernel_test: *** FAILED test case 606 of 750 ***
1> Testing tests.kernel_test: *** FAILED test case 607 of 750 ***
1> Testing tests.kernel_test: *** FAILED test case 612 of 750 ***
1> Testing tests.kernel_test: *** FAILED test case 613 of 750 ***
1> Testing tests.kernel_test: *** FAILED test case 627 of 750 ***
1> Testing tests.kernel_test: *** FAILED test case 629 of 750 ***
1> Testing tests.kernel_test: *** FAILED test case 630 of 750 ***
1> Testing tests.kernel_test: *** FAILED {seq_trace_SUITE,init_per_suite} ***



Case 125:
disk_log_SUITE:open_overwrite failed on line 1865
Reason: {badmatch,{ok,n}}

157,158:
erl_boot_server_SUITE:start failed on line 83
Reason: {badmatch,{error,{badarg,["dorado-vm1","gandalf"]}}}

160:
erl_boot_server_SUITE:add failed on line 153
Reason: {badmatch,{error,nxdomain}}

161:
erl_boot_server_SUITE:delete failed on line 186
Reason: {badmatch,{error,nxdomain}}

241:
file_SUITE:test_directory failed on line 1376
Reason: {badmatch,read}

244:
file_SUITE:file_write_file_info failed on line 1524
Reason: {badmatch,ok}

259:
file_SUITE:e_delete failed on line 2109
Reason: {badmatch,ok}

261:
file_SUITE:e_make_dir failed on line 2240
Reason: {badmatch,ok}

262:
file_SUITE:e_del_dir failed on line 2287
Reason: {badmatch,ok}

294:
file_name_SUITE:make_icky_dir failed on line 642
Reason: {badmatch,{error,no_translation}}

307:
gen_sctp_SUITE:ok failed on line 544
Reason: function_clause

384:
gen_tcp_misc_SUITE:test_prio_fail failed on line 1871
Reason: {badmatch,ok}

417:
gen_udp_SUITE:implicit_inet6 failed on line 733
Reason: not_ok

561:
multi_load_SUITE:do_ensure_modules_loaded failed on line 282
Reason: {badmatch,false}

562:
multi_load_SUITE:make_and_load failed on line 330
Reason: {badmatch,ok}

606:
prim_file_SUITE:test_directory failed on line 801
Reason: {badmatch,read}

607:
prim_file_SUITE:test_directory failed on line 801
Reason: {badmatch,read}

612:
prim_file_SUITE:file_write_file_info failed on line 968
Reason: {badmatch,ok}

613:
prim_file_SUITE:file_write_file_info failed on line 968
Reason: {badmatch,ok}

627:
prim_file_SUITE:e_delete failed on line 1455
Reason: {badmatch,ok}

629:
prim_file_SUITE:e_make_dir failed on line 1591
Reason: {badmatch,ok}

630:
prim_file_SUITE:e_del_dir failed on line 1641
Reason: {badmatch,ok}



2) On AIX, I have these AIX-specific tests that fail:

1> Testing tests.kernel_test: *** FAILED test case 88 of 750 ****
code_SUITE:big_boot_embedded failed on line 1119
Reason: {badmatch,{error,timeout}}

1> Testing tests.kernel_test: *** FAILED test case 220 of 750 ***
file_SUITE:expect failed on line 217
Reason: function_clause

1> Testing tests.kernel_test: *** FAILED test case 284 of 750 ***
erlang:hd failed
Reason: badarg

1> Testing tests.kernel_test: *** FAILED test case 285 of 750 ***
rlang:hd failed
Reason: badarg

1> Testing tests.kernel_test: *** FAILED test case 499 of 750 ***
inet_SUITE:do_getif failed on line 952
Reason: {test_case_failed,no_HWAs}

1> Testing tests.kernel_test: *** FAILED test case 502 of 750 ***
inet_SUITE:getifaddrs failed on line 999
Reason: {test_case_failed,{should_have_hwaddr,{unix,aix}}}

1> Testing tests.kernel_test: *** FAILED test case 590 of 750 ***
prim_file_SUITE:expect failed on line 157
Reason: function_clause

1> Testing tests.kernel_test: *** FAILED test case 591 of 750 ***
prim_file_SUITE:expect failed on line 157
Reason: function_clause


Cordialement,

Tony Reix

Bull - ATOS
IBM Coop Architect & Technical Leader
Office : +33 (0) 4 76 29 72 67
1 rue de Provence - 38432 Échirolles - France
www.atos.net

________________________________________
De : bgustavsson@REDACTED [bgustavsson@REDACTED] de la part de Björn Gustavsson [bjorn@REDACTED]
Envoyé : jeudi 2 mars 2017 13:37
À : REIX, Tony
Cc : erlang-questions@REDACTED; GIRARDET, JEAN
Objet : Re: [erlang-questions] Desperately looking for official Erlang tests results

Did you run Erlang from the git repository/source
tree or did you do "make release" and run it from
there?

A few tests cases will fail if you have not done
"make release". (One example is
xref_SUITE:md/1 in tools.)

I think that the best way for us to help you is
if you can share the names of the failed
test cases in the following test suites:

emulator_test
stdlib_test
kernel_test

We should be able to tell you which ones
that are most likely real bugs, and which
ones that fail because of envioronmental
issues.

/Björn

On Thu, Mar 2, 2017 at 12:23 PM, REIX, Tony <tony.reix@REDACTED> wrote:
> Hi,
>
> In order to understand which Erlang tests and features do not work fine
> within our port of version 19.2 on AIX, we need some "official" Erlang tests
> results for current version (19.2.3).
> However, searching on Erlang web-site, I was unable to find such tests
> results.
>
> So, I have built and tested Erlang on 3 environments: AIX 6.1,
> Fedora25/PPC64, and Fedora25/x86_64.
> I hope I've correctly ran the tests, but I'm not sure.
>
> In short, using nearly the same procedure (use Fedora .spec file), but
> without HiPE-SMP-Threads for AIX, I've got:
>
> OK FAIL SKIP TOTAL
> ------------------------------------------
> AIX 6.1 10,838 204 2,991 14,033 (no HiPE-SMP-Threads !)
> FC25/PPC64 12,376 193 3,608 16,177
> FC25/x86_64 12,407 288 3,475 16,170
>
> I am really surprised by:
> - the BIG amount of SKIPped tests on FedoraCore 25 on x86_64 : 23% of all
> tests !!
> - how few failures I have on AIX, comparable to the amount of failures on
> FC25/x86_64.
> (However: - when trying with SMP and/or HiPE, I had hangs, so I'm
> first building/testing without them
> - we applied several AIX-specific patches)
> - having LESS failures on Fedora25/PPC64 than on Fedora25/x86_64 .
> (However: there are some more SKIPped tests: 133)
>
> Since result files contains about 600,000 lines, it is very difficult for me
> to understand where/what are the issues.
> I think we have understood that, in many cases, some environment is
> missing/failing and thus the test is skipped.
>
> I'd like to get some help about:
> - checking that I'm doing the Erlang configuration, build, and tests
> correctly,
> - finding some "official" Erlang tests results on Fedora / x86_64
> - how to analyze these results and find the real issues.
>
> I have attached to this email the index.html file produced on
> Fedora25/x86_64 :
>
> /root/rpmbuild/BUILD/otp-OTP-19.2.3/release/tests/test_server/index.html
>
> Thanks
>
> Cordialement,
>
> Tony Reix
>
> Bull - ATOS
> IBM Coop Architect & Technical Leader
> Office : +33 (0) 4 76 29 72 67
> 1 rue de Provence - 38432 Échirolles - France
> www.atos.net
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>



--
Björn Gustavsson, Erlang/OTP, Ericsson AB
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20170302/d1d0bcdd/attachment.htm>