AssertionError: failed to join the thread in 30.0 seconds

https://jenkins-continuous-infra.apps.ci.centos.org/job/fedora-rawhide-pr-pipeline/501/artifact/package-tests/logs/FAIL_selftest.log

[citest]

@psss Any idea how would I reproduce this locally?

@psss I am trying to help out (FYI)

So locally I get:

+ python3.7 -m test -j0 -x test_socket
/usr/bin/python3.7: No module named test.__main__; 'test' is a package and cannot be directly executed

And that's OK, as I forgot to specify that python3-test should get installed.

Why the test actually run on the CI is a mystery for me.

Verified: On the CI all noarch and x86_64 packages from the Koji scratch build are installed.

The Ci build has the virtual OS image in Artifacts. When I downloaded it, I could get in:

$ virt-customize -a test_subject.qcow2 --root-password password:toor
[   0,0] Examining the guest ...
[  20,1] Setting a random seed
[  20,1] Setting the machine ID in /etc/machine-id
[  20,1] Setting passwords
[  22,1] Finishing off
$ guestfish -a test_subject.qcow2 -i touch /etc/cloud/cloud-init.disabled  # disbale cloud-init
$ virt-install --connect qemu:///system -n python-test -r 1024 --network=default --import --disk path=./test_subject.qcow2 --nographics
...
Fedora 30 (Cloud Edition)
Kernel 4.20.0-0.rc2.git0.1.fc30.x86_64 on an x86_64 (ttyS0)

localhost login: root
Password: toor
[root@localhost ~]# python3 --version
Python 3.7.1
[root@localhost /]# python3.7 -m test -j0 -x test_socket

I haven't found the tests in the image, so i had to reconstruct the actual invocation by hand, but the tests run. I'll see whether I can reproduce the failure.

So this I get locally:

== Tests result: FAILURE ==

390 tests OK.

2 tests failed:
    test_asyncio test_ssl

22 tests skipped:
    test_asdl_parser test_clinic test_curses test_devpoll test_gdb
    test_kqueue test_msilib test_ossaudiodev test_smtpnet
    test_socketserver test_startfile test_timeout test_tix test_tk
    test_ttk_guionly test_urllib2net test_urllibnet test_winconsoleio
    test_winreg test_winsound test_xmlrpc_net test_zipfile64

Total duration: 8 min 29 sec
Tests result: FAILURE

This I get on the CI:

== Tests result: FAILURE ==

385 tests OK.

7 tests failed:
    test_asyncgen test_distutils test_multiprocessing_forkserver
    test_shutil test_ssl test_threading test_time

22 tests skipped:
    test_asdl_parser test_clinic test_curses test_devpoll test_gdb
    test_kqueue test_msilib test_ossaudiodev test_smtpnet
    test_socketserver test_startfile test_timeout test_tix test_tk
    test_ttk_guionly test_urllib2net test_urllibnet test_winconsoleio
    test_winreg test_winsound test_xmlrpc_net test_zipfile64

Total duration: 12 min 17 sec
Tests result: FAILURE

I'll rerun the tests couple times to see if they are flaky.

[citest]

First Ci result:

== Tests result: FAILURE ==

382 tests OK.

10 tests failed:
    test_asyncgen test_asyncio test_compile test_concurrent_futures
    test_itertools test_multiprocessing_fork
    test_multiprocessing_forkserver test_shutil test_ssl test_time

22 tests skipped:
    test_asdl_parser test_clinic test_curses test_devpoll test_gdb
    test_kqueue test_msilib test_ossaudiodev test_smtpnet
    test_socketserver test_startfile test_timeout test_tix test_tk
    test_ttk_guionly test_urllib2net test_urllibnet test_winconsoleio
    test_winreg test_winsound test_xmlrpc_net test_zipfile64

Total duration: 42 min 9 sec
Tests result: FAILURE

Local runs however seem to deliver the same results over and over (2 failures: test_asyncio test_ssl).

Ha! A third local failure:

== Tests result: FAILURE ==

389 tests OK.

3 tests failed:
    test_asyncio test_ssl test_time

22 tests skipped:
    test_asdl_parser test_clinic test_curses test_devpoll test_gdb
    test_kqueue test_msilib test_ossaudiodev test_smtpnet
    test_socketserver test_startfile test_timeout test_tix test_tk
    test_ttk_guionly test_urllib2net test_urllibnet test_winconsoleio
    test_winreg test_winsound test_xmlrpc_net test_zipfile64

Total duration: 7 min 27 sec
Tests result: FAILURE

This is so unreliable. Not sure whether we should:

• spend extraordinary time and effort to figure out why
• get a list of flaky tests and disable them for now (most likely for ever)
• abandon the effort

Thoughts?

@pviktori @torsava @cstratak @lbalhar @vstinner

Instead of abandoning the effort, I'd disable the flaky tests. Running most of the tests is better than none.

Is it possible to disable a group of tests just for the CI, but still run them during build?

Sure, see https://src.fedoraproject.org/tests/python/blob/master/f/selftest/parallel.sh

Disable flaky tests – but not forever :)
Fixing flaky tests will join "CI on all arches" on the list of things needed before we can disable the rpmbuild %check.

[citest]

Ready for review.

Please add glibc-all-langpacks explicitly as well. With that, merge if it passes.