PR#1: Add CI tests using the standard test interface - rpms/pth

rpms / pth

#1 Add CI tests using the standard test interface

Closed 5 years ago by rdieter. Opened 6 years ago by sturivny.

git://fedorapeople.org/~sturivny/pth add_tests into master

Add CI tests using the standard test interface

Serhii Turivny • 6 years ago

d03298d

tests/selftest/Makefile

file added

+63

		`@@ -0,0 +1,63 @@`
		`+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`
		`+ #`
		`+ # Makefile of /CoreOS/pth/Sanity/selftest`
		`+ # Description: Runs make tests on build`
		`+ # Author: Zbysek MRAZ <zmraz@redhat.com>`
		`+ #`
		`+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`
		`+ #`
		`+ # Copyright (c) 2010 Red Hat, Inc.`
		`+ #`
		`+ # This copyrighted material is made available to anyone wishing`
		`+ # to use, modify, copy, or redistribute it subject to the terms`
		`+ # and conditions of the GNU General Public License version 2.`
		`+ #`
		`+ # This program is distributed in the hope that it will be`
		`+ # useful, but WITHOUT ANY WARRANTY; without even the implied`
		`+ # warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR`
		`+ # PURPOSE. See the GNU General Public License for more details.`
		`+ #`
		`+ # You should have received a copy of the GNU General Public`
		`+ # License along with this program; if not, write to the Free`
		`+ # Software Foundation, Inc., 51 Franklin Street, Fifth Floor,`
		`+ # Boston, MA 02110-1301, USA.`
		`+ #`
		`+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`
		`+`
		`+ export TEST=/CoreOS/pth/Sanity/selftest`
		`+ export TESTVERSION=1.1`
		`+`
		`+ BUILT_FILES=`
		`+`
		`+ FILES=$(METADATA) runtest.sh Makefile PURPOSE`
		`+`
		`+ .PHONY: all install download clean`
		`+`
		`+ run: $(FILES) build`
		`+ ./runtest.sh`
		`+`
		`+ build: $(BUILT_FILES)`
		`+ chmod a+x runtest.sh`
		`+`
		`+ clean:`
		`+ rm -f *~ $(BUILT_FILES)`
		`+`
		`+`
		`+ include /usr/share/rhts/lib/rhts-make.include`
		`+`
		`+ $(METADATA): Makefile`
		`+ @echo "Owner: Zbysek MRAZ <zmraz@redhat.com>" > $(METADATA)`
		`+ @echo "Name: $(TEST)" >> $(METADATA)`
		`+ @echo "TestVersion: $(TESTVERSION)" >> $(METADATA)`
		`+ @echo "Path: $(TEST_DIR)" >> $(METADATA)`
		`+ @echo "Description: Runs make tests on build" >> $(METADATA)`
		`+ @echo "Type: Sanity" >> $(METADATA)`
		`+ @echo "TestTime: 5m" >> $(METADATA)`
		`+ @echo "RunFor: pth" >> $(METADATA)`
		`+ @echo "Requires: pth" >> $(METADATA)`
		`+ @echo "Priority: Normal" >> $(METADATA)`
		`+ @echo "License: GPLv2" >> $(METADATA)`
		`+ @echo "Confidential: no" >> $(METADATA)`
		`+ @echo "Destructive: no" >> $(METADATA)`
		`+`
		`+ rhts-lint $(METADATA)`

tests/selftest/PURPOSE

file added

		`@@ -0,0 +1,5 @@`
		`+ PURPOSE of /CoreOS/pth/Sanity/selftest`
		`+ Description: Runs make tests on build`
		`+ Author: Zbysek MRAZ <zmraz@redhat.com>`
		`+`
		`+ Runst the upstream test suite from SRPM file`

tests/selftest/runtest.sh

file added

+70

		`@@ -0,0 +1,70 @@`
		`+ #!/bin/bash`
		`+ # vim: dict=/usr/share/beakerlib/dictionary.vim cpt=.,w,b,u,t,i,k`
		`+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`
		`+ #`
		`+ # runtest.sh of /CoreOS/pth/Sanity/selftest`
		`+ # Description: Runs make tests on build`
		`+ # Author: Zbysek MRAZ <zmraz@redhat.com>`
		`+ #`
		`+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`
		`+ #`
		`+ # Copyright (c) 2010 Red Hat, Inc.`
		`+ #`
		`+ # This copyrighted material is made available to anyone wishing`
		`+ # to use, modify, copy, or redistribute it subject to the terms`
		`+ # and conditions of the GNU General Public License version 2.`
		`+ #`
		`+ # This program is distributed in the hope that it will be`
		`+ # useful, but WITHOUT ANY WARRANTY; without even the implied`
		`+ # warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR`
		`+ # PURPOSE. See the GNU General Public License for more details.`
		`+ #`
		`+ # You should have received a copy of the GNU General Public`
		`+ # License along with this program; if not, write to the Free`
		`+ # Software Foundation, Inc., 51 Franklin Street, Fifth Floor,`
		`+ # Boston, MA 02110-1301, USA.`
		`+ #`
		`+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`
		`+`
		`+ # Include rhts environment`
		`+ . /usr/bin/rhts-environment.sh`
		`+ . /usr/share/beakerlib/beakerlib.sh`
		`+`
		`+ PACKAGE="pth"`
		+ TARGET=$(echo `uname -m` \| egrep ppc)
		`+ if [[ $TARGET != "" ]]; then`
		`+ if rlIsRHEL 4 5; then`
		`+ TARGET="--target ppc"`
		`+ else`
		+ TARGET="--target `uname -m`";
		`+ fi`
		`+ fi`
		`+`
		`+ rlJournalStart`
		`+ rlPhaseStartSetup`
		`+ rlAssertRpm $PACKAGE`
		`+ if rlIsRHEL 4 5; then`
		`+ BDIR='/usr/src/redhat'`
		`+ else`
		`+ BDIR='/root/rpmbuild'`
		`+ fi`
		+ rlRun "TmpDir=\`mktemp -d\`" 0 "Creating tmp directory"
		`+ rlRun "pushd $TmpDir"`
		`+ rlFetchSrcForInstalled $PACKAGE`
		`+ rlRun "rpm -ivh $PACKAGE*.src.rpm" 0 "Installing source RPM"`
		`+ rlRun "rpmbuild -bc $BDIR/SPECS/${PACKAGE}.spec $TARGET" 0 "Building package"`
		`+ rlRun "cd $BDIR/BUILD/${PACKAGE}*"`
		`+ rlPhaseEnd`
		`+`
		`+ rlPhaseStartTest`
		`+ rlRun "make test > $TmpDir/tests.log 2>&1" 0 "Testing"`
		`+ rlAssertGrep "OK - ALL TESTS SUCCESSFULLY PASSED" $TmpDir/tests.log`
		`+ rlPhaseEnd`
		`+`
		`+ rlPhaseStartCleanup`
		`+ rlRun "popd"`
		`+ rlRun "rm -r $TmpDir" 0 "Removing tmp directory"`
		`+ rlIsRHEL 6 7 && rlRun "rm -r $BDIR" 0 "Removing rpmbuild directory"`
		`+ rlPhaseEnd`
		`+ rlJournalPrintText`
		`+ rlJournalEnd`

tests/tests.yml

file added

+16

		`@@ -0,0 +1,16 @@`
		`+ ---`
		`+ # Tests for docker/local`
		`+ - hosts: localhost`
		`+ tags:`
		`+ - container`
		`+ - classic`
		`+ roles:`
		`+ - role: standard-test-beakerlib`
		`+ tests:`
		`+ - selftest`
		`+ required_packages:`
		`+ - pth # selftest needs pth RPM installed`
		`+ - wget # selftest needs wget command`
		`+ - gcc # selftest needs gcc command`
		`+ - rpm-build # selftest needs rpmbuild command`
		`+ - make # selftest needs make command`

no initial comment

mschwendt commented 6 years ago

Odd request. You ought to submit that patch upstream, since it doesn't modify the packaging.

sturivny commented 6 years ago

Justification

Adds tests according to the CI wiki specifically the standard test interface in the spec.

The playbook includes Tier1 level test cases that have been tested in the following contexts and is passing reliably: Classic and Container. Test logs are stored in the artifacts directory.

The following steps are used to execute the tests using the standard test interface:

Test environment

Make sure you have installed packages from the spec

$ rpm -q ansible python2-dnf libselinux-python standard-test-roles
ansible-2.3.2.0-1.fc26.noarch
python2-dnf-2.6.3-11.fc26.noarch
libselinux-python-2.6-7.fc26.x86_64
standard-test-roles-2.4-1.fc26.noarch

Run tests for Classic

$ export TEST_SUBJECTS=
$ sudo ansible-playbook --tags classic tests.yml

Snip of the example test run:

TASK [standard-test-beakerlib : Check the results] ************************************
changed: [localhost]
PLAY RECAP ****************************************************************************
localhost                  : ok=15   changed=9    unreachable=0    failed=0

Run tests for Container

$ export TEST_SUBJECTS=docker:docker.io/library/fedora:26
$ sudo ansible-playbook --tags=container tests.yml

Snip of the example test run:

TASK [standard-test-beakerlib : Check the results] ****************************************************************************************
changed: [ed021fcff70ff478a2bc76f8d7ca81405b234dd81ade8105d0c4b60928e0164d]

PLAY RECAP *****************************************************************************
ed021fcff70ff478a2bc76f8d7ca81405b234dd81ade8105d0c4b60928e0164d : ok=15   changed=11   unreachable=0    failed=0

Notes

Tests will be enabled in CI, yet gating is currently disabled, so nothing will change. Tests will run on each dist-git commit, they are not triggered on koji builds and if you are using FMN, it should notify you of failures normally.

The RH QE maintainer contact in case you have questions: ksrot @redhat.com
The idea is that these tests become yours just as you're maintaining the package, there will of course be people around if you have questions or troubles.

Edited 6 years ago by sturivny

sturivny commented 6 years ago

Odd request. You ought to submit that patch upstream, since it doesn't modify the packaging.

Those tests are not part of the package.
It's infrastructure for Fedora rpm.

mschwendt commented 6 years ago

https://fedoraproject.org/wiki/Changes/InvokingTests

Red Hat would like to bring integration tests upstream.

no impact on the packagers' workflow and tooling.

Consider getting access to dist-git, so you can commit and maintain such extra files yourself.

ignatenkobrain commented on line 37 of tests/selftest/runtest.sh 6 years ago

I have no idea why I see it in Fedora...

ignatenkobrain commented 6 years ago

This test seems to be useless and unreadable to me honestly.

dperpeet commented 6 years ago

Consider getting access to dist-git, so you can commit and maintain such extra files yourself.

While that would be possible, I don't think it would be a good idea. I believe that the package maintainers should have control over dist-git, without others meddling by adding tests that would eventually be a factor in deciding whether the built package is used or not (gating), e.g. in Fedora Atomic Host.

As @sturivny outlined, this test is part of what's currently being run downstream. For sure, these can be adapted to be of more use in Fedora. This test is a sample, and I'd be happy to help you add your own. Do you have other tests that you think are useful and you would be willing to gate on, eventually?

I believe a proper CI workflow in Fedora dist-git will be a benefit to everyone. I invite you to criticize the tests and this pull-request. You, the maintainers, know your package best and I appreciate your feedback on the tests. Let's work together to add tests here that will ensure eventually no packages will ship in Fedora that don't pass these tests. Ultimately, it's your decision which tests you want to use for that - and you can always modify them just as you would anything else in dist-git.

ignatenkobrain commented 6 years ago

@dperpeet I believe that pth is old, unsupported and should be retired. Same as @mschwendt pointed on devel@lists.fp.o...

And I'm not sure if you really expect package maintainers to maintain such unreadable tests. Syntax is horrible, there are also some license header in tests which do not make any sense to me..

mschwendt commented 6 years ago

There is a big risk of alienating Fedora packagers.

The people, who are behind this PR, you need to start a public campaign on devel@ list and/or packagers@ list and advertize what you're trying to achieve, give examples of test coverage and give examples of what defects have been found in packages. So far, simply adding the files to dist git outside the RPM package would be a no-op. It would be a fire-and-forget commit to dist-git, with the files likely getting out-of-date and requiring future updates. If you want Fedora packagers to raise interest in your work, much more is needed. Even if you turned on some magic switches at Fedora infrastructure somewhere to trigger automatic running of such tests and sending test results to packagers, without packager commitment, nothing would happen.