rpms / buildstream

Clone
Source Code
GIT

#2 Enable most of the tests
Merged 2 years ago by atim. Opened 2 years ago by music.
rpms/ music/buildstream run-tests  into  rawhide

file modified
+48 -20 
@@ -1,7 +1,6 @@ 
 

- # Disable the tests for now, too many unavailable dependencies.
 

  # Tests works with enabled network. Also possible to run tests in COPR:
 

  # https://download.copr.fedorainfracloud.org/results/atim/buildstream/fedora-35-x86_64/02660103-buildstream/builder-live.log.gz
 

- %bcond_with test
 

+ %bcond_without test
 

  

  Name:          buildstream
 

  Summary:       Build/integrate software stacks
 
@@ -36,27 +35,27 @@ 
 

  BuildRequires: python3-ujson
 

  

  %if %{with test}
 

- # These from cov-requirements.in + dev-requirements.in
 

- BuildRequires: python3-coverage >= 4.4
 

- BuildRequires: python3-pylint >= 2.10
 

+ BuildRequires: zip
 

+ BuildRequires: tar
 

+ BuildRequires: lzip
 

+ BuildRequires: git-core
 

+ BuildRequires: bzr
 

+ BuildRequires: ostree
 

+ BuildRequires: python3-pip
 

+ # These from dev-requirements.in
 

  BuildRequires: python3-pytest >= 3.7
 

- BuildRequires: python3-pytest-cov >= 2.5
 

- BuildRequires: python3-pytest-timeout
 

- BuildRequires: python3-pytest-xdist
 

- BuildRequires: python3-pep8
 

  BuildRequires: python3-pytest-datafiles
 

  BuildRequires: python3-pytest-env
 

- %endif
 

- 

- # These aren't in Fedora any more, preventing us from running the tests.
 

- # Buildstream master moved to pycodestyle though, so this will sort itself out.
 

- #   * python3-pep8
 

- #   * python3-pytest-pep8
 

+ BuildRequires: python3-pytest-timeout
 

+ BuildRequires: python3-pytest-xdist
 

+ # Not available: pyftpdlib (we skip the associated tests)
 

  

- # These aren't in Fedora yet, preventing us from running the tests
 

- #   * python3-pytest-datafiles
 

- #   * python3-pytest-env
 

- #   * python3-pytest-pylint
 

+ # We do not need linters (nor omitted coverage from cov-requirements.in); see:
 

+ # https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_linters
 

+ #
 

+ # BuildRequires: python3-pep8
 

+ # BuildRequires: python3-pylint >= 2.10
 

+ %endif
 

  

  Requires:      bubblewrap >= 0.1.2
 

  Requires:      fuse
 
@@ -130,7 +129,36 @@ 
 

  

  %if %{with test}
 

  %check
 

- %{__python3} setup.py test
 

+ # The following fail in koji but not in a local mock build; this seems to have
 

+ # something to do with sandboxing.
 

+ k="${k-}${k+ and }not test_push_pull"
 

+ k="${k-}${k+ and }not test_push_pull_all"
 

+ k="${k-}${k+ and }not test_pull_secondary_cache"
 

+ k="${k-}${k+ and }not test_push_pull_specific_remote"
 

+ k="${k-}${k+ and }not test_push_pull_non_strict"
 

+ k="${k-}${k+ and }not test_push_pull_track_non_strict"
 

+ k="${k-}${k+ and }not test_push_pull_cross_junction"
 

+ k="${k-}${k+ and }not test_pull_missing_blob"
 

+ k="${k-}${k+ and }not test_pull_access_rights"
 

+ k="${k-}${k+ and }not test_push"
 

+ k="${k-}${k+ and }not test_push_all"
 

+ k="${k-}${k+ and }not test_push_after_pull"
 

+ k="${k-}${k+ and }not test_artifact_expires"
 

+ k="${k-}${k+ and }not test_artifact_too_large"
 

+ k="${k-}${k+ and }not test_recently_pulled_artifact_does_not_expire"
 

+ k="${k-}${k+ and }not test_push_cross_junction"
 

+ k="${k-}${k+ and }not test_push_already_cached"
 

+ # This test expects 'arch' to be 'x86_64' or 'x86_32', so it fails when the
 

+ # builder is anything but x86_64. Even 32-bit x86 fails, as 'arch' is 'i686'.
 

+ # Since the package is noarch, we just skip it unconditionally.
 

+ k="${k-}${k+ and }not test_project_error"
 

+ # Ignored tests would require pyftpdlib, which is not packaged
 

+ %pytest -vv -k "${k-}" \
 

+     --ignore=tests/testutils/file_server.py \
 

+     --ignore=tests/testutils/ftp_server.py \
 

+     --ignore=tests/sources/remote.py \
 

+     --ignore=tests/sources/tar.py \
 

+     --ignore=tests/sources/zip.py
 

  %endif

I just packaged python-pytest-env, and had a look around to see which packages might have had to patch out a pytest-env dependency while it was not available.

This PR demonstrates that it is now possible to run most of buildstream’s tests.

Thanks a lot. I've built it successfully with patch and tests passed for f37.

Let's hold it for one day to so others can review also this change.

cc: @jjardon @gasinvein

I want to ascertain why the CI scratch build failed. I suspect there may be architecture-dependent issues in this noarch package, so I kicked off a scratch build for which I manually selected all arches (fedpkg --release rawhide scratch-build --srpm --arches x86_64 i686 ppc64le aarch64 s390x): https://koji.fedoraproject.org/koji/taskinfo?taskID=84691828

It seems like the same 18 tests are failing on all architectures:

=========================== short test summary info ============================
FAILED tests/artifactcache/junctions.py::test_push_pull - AssertionError: Art...
FAILED tests/format/listdirectiveerrors.py::test_project_error - AssertionErr...
FAILED tests/frontend/pull.py::test_push_pull_all - AssertionError: Artifact ...
FAILED tests/frontend/pull.py::test_pull_secondary_cache - AssertionError: Ar...
FAILED tests/frontend/pull.py::test_push_pull_specific_remote - AssertionErro...
FAILED tests/frontend/pull.py::test_push_pull_non_strict - AssertionError: Ar...
FAILED tests/frontend/pull.py::test_push_pull_track_non_strict - AssertionErr...
FAILED tests/frontend/pull.py::test_push_pull_cross_junction - AssertionError...
FAILED tests/frontend/pull.py::test_pull_missing_blob - AssertionError: Artif...
FAILED tests/frontend/pull.py::test_pull_access_rights - AssertionError: 
FAILED tests/frontend/push.py::test_push - AssertionError: Artifact share at ...
FAILED tests/frontend/push.py::test_push_all - AssertionError: Artifact share...
FAILED tests/frontend/push.py::test_push_after_pull - AssertionError: Artifac...
FAILED tests/frontend/push.py::test_artifact_expires - AssertionError: Artifa...
FAILED tests/frontend/push.py::test_artifact_too_large - AssertionError: Arti...
FAILED tests/frontend/push.py::test_recently_pulled_artifact_does_not_expire
FAILED tests/frontend/push.py::test_push_cross_junction - AssertionError: ass...
FAILED tests/frontend/push.py::test_push_already_cached - AssertionError: ass...
=========== 18 failed, 918 passed, 131 skipped in 209.05s (0:03:29) ============

except for tests/format/listdirectiveerrors.py::test_project_error, which does pass on x86_64.

I am not sure why these are failing in koji but not in a local mock build, even with --enablerepo=local.

I think that, except for tests/format/listdirectiveerrors.py::test_project_error, these may have to do with messages like:

[--:--:--][7c822fb0][build:import-dev.bst                ] WARNING Sandbox configuration is not supported by the platform.
    Falling back to UID 0 GID 0. Artifact will not be pushed.

I still can’t explain why this only happens in koji, but I suspect these tests may just need to be skipped.

I wonder if whatever sandboxing approach bubblewrap/bwrap is doing can’t be nested with some sandboxing the koji builders are doing.

tests/format/listdirectiveerrors.py::test_project_error is supposed to work only on x86_64 and x86_32. It fails on 32-bit x86, too, because it gets i686 as the arch instead of x86_32.

Program stderr was:
Error loading project: Invalid value for arch option 'arch': i686
Valid values: x86_32, x86_64

1 new commit added

  • Document and skip tests that fail in koji
2 years ago

Pull-Request has been merged by atim
2 years ago
Metadata
None
No Tags
Changes Summary 1
+48 -20
file changed
buildstream.spec
Powered by Pagure 5.13.3
Documentation • File an Issue • About this Instance • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.