PR#1: Add CI tests using the standard test interface - rpms/coreutils

rpms / coreutils

#1 Add CI tests using the standard test interface

Closed 6 years ago by kdudka. Opened 6 years ago by rasibley.

rpms/ rasibley/coreutils coreutils-tests into master

add reference link to fedora ci wiki

Rachel Sibley • 6 years ago

ab7465c

Initial commit for Atomic Host Tests

Rachel Sibley • 6 years ago

bb681e6

tests/readlink-cannot-handle-recursive-symlinks/Makefile

file added

+63

		`@@ -0,0 +1,63 @@`
		`+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`
		`+ #`
		`+ # Makefile of /CoreOS/coreutils/readlink-cannot-handle-recursive-symlink s`
		`+ # Description: Test for readlink cannot handle recursive symlinks`
		`+ # Author: Jan Scotka <jscotka@redhat.com>`
		`+ #`
		`+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`
		`+ #`
		`+ # Copyright (c) 2010 Red Hat, Inc. All rights reserved.`
		`+ #`
		`+ # This copyrighted material is made available to anyone wishing`
		`+ # to use, modify, copy, or redistribute it subject to the terms`
		`+ # and conditions of the GNU General Public License version 2.`
		`+ #`
		`+ # This program is distributed in the hope that it will be`
		`+ # useful, but WITHOUT ANY WARRANTY; without even the implied`
		`+ # warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR`
		`+ # PURPOSE. See the GNU General Public License for more details.`
		`+ #`
		`+ # You should have received a copy of the GNU General Public`
		`+ # License along with this program; if not, write to the Free`
		`+ # Software Foundation, Inc., 51 Franklin Street, Fifth Floor,`
		`+ # Boston, MA 02110-1301, USA.`
		`+ #`
		`+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`
		`+`
		`+ export TEST=/CoreOS/coreutils/readlink-cannot-handle-recursive-symlinks`
		`+ export TESTVERSION=1.0`
		`+`
		`+ BUILT_FILES=`
		`+`
		`+ FILES=$(METADATA) runtest.sh Makefile PURPOSE`
		`+`
		`+ .PHONY: all install download clean`
		`+`
		`+ run: $(FILES) build`
		`+ ./runtest.sh`
		`+`
		`+ build: $(BUILT_FILES)`
		`+ chmod a+x runtest.sh`
		`+`
		`+ clean:`
		`+ rm -f *~ $(BUILT_FILES)`
		`+`
		`+`
		`+ include /usr/share/rhts/lib/rhts-make.include`
		`+`
		`+ $(METADATA): Makefile`
		`+ @echo "Owner: Jan Scotka <jscotka@redhat.com>" > $(METADATA)`
		`+ @echo "Name: $(TEST)" >> $(METADATA)`
		`+ @echo "TestVersion: $(TESTVERSION)" >> $(METADATA)`
		`+ @echo "Path: $(TEST_DIR)" >> $(METADATA)`
		`+ @echo "Description: Test for readlink cannot handle recursive symlinks" >> $(METADATA)`
		`+ @echo "Type: Sanity" >> $(METADATA)`
		`+ @echo "TestTime: 5m" >> $(METADATA)`
		`+ @echo "RunFor: coreutils" >> $(METADATA)`
		`+ @echo "Requires: coreutils" >> $(METADATA)`
		`+ @echo "Priority: Normal" >> $(METADATA)`
		`+ @echo "License: GPLv2" >> $(METADATA)`
		`+ @echo "Confidential: no" >> $(METADATA)`
		`+ @echo "Destructive: no" >> $(METADATA)`
		`+`
		`+ rhts-lint $(METADATA)`

tests/readlink-cannot-handle-recursive-symlinks/PURPOSE

file added

+54

		`@@ -0,0 +1,54 @@`
		`+ PURPOSE of /CoreOS/coreutils/readlink-cannot-handle-recursive-symlinks`
		`+ Description: Test for readlink cannot handle recursive symlinks`
		`+ Author: Jan Scotka <jscotka@redhat.com>`
		`+ Bug summary: readlink cannot handle recursive symlinks`
		`+`
		`+ Description:`
		`+`
		`+ Description of problem:`
		`+ The readlink command fails with an error "Too many levels of symbolic links", even if there are only 2 levels.`
		`+`
		`+ The readlink command from RHEL 3 and RHEL 4 and from Fedora 11 all work fine.`
		`+`
		`+ Among other changes between RHEL 4 and RHEL 5, RHEL 5's coreutils added calls to cycle_check() in lib/canonicalize.c, but in upstream canonicalize.c (now in gnulib instead of coreutils), we see the comment:`
		`+ /* Detect loops. We cannot use the cycle-check module here,`
		`+ since it's actually possible to encounter the same symlink`
		`+ more than once in a given traversal. However, encountering`
		`+ the same symlink,NAME pair twice does indicate a loop. */`
		`+`
		`+ http://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=lib/canonicalize.c;h=4f348398fd69ae516396313d18ac294a4ca3dde3;hb=b653eda3ac4864de205419d9f41eec267cb89eeb#l262`
		`+`
		`+ The latest canonicalize.c uses seen_triple() instead of cycle_check().`
		`+`
		`+`
		`+ Version-Release number of selected component (if applicable):`
		`+ coreutils-5.97-19.el5`
		`+`
		`+ How reproducible:`
		`+ every time`
		`+`
		`+ Steps to Reproduce:`
		`+ 1. Create a directory with a symlink to itself`
		`+ mkdir /tmp/dir`
		`+ cd /tmp/dir`
		`+ ln -s ../dir dirlink`
		`+`
		`+ 2. Run readlink using the 'dirlink' symlink recursively`
		`+ readlink -v -f dirlink`
		`+ readlink -v -f dirlink/dirlink`
		`+ readlink -v -f dirlink/dirlink/dirlink`
		`+`
		`+ Actual results:`
		`+ The first readlink command on just dirlink succeeds, but the 2nd and 3rd commands fail with`
		`+ readlink: dirlink/dirlink: Too many levels of symbolic links`
		`+`
		`+ Expected results:`
		`+ $ readlink -v -f dirlink`
		`+ /tmp/dir`
		`+ $ readlink -v -f dirlink/dirlink`
		`+ /tmp/dir`
		`+ $ readlink -v -f dirlink/dirlink/dirlink`
		`+ /tmp/dir`
		`+`
		`+`
		`+ Additional info:`

tests/readlink-cannot-handle-recursive-symlinks/runtest.sh

file added

+60

		`@@ -0,0 +1,60 @@`
		`+ #!/bin/bash`
		`+ # vim: dict=/usr/share/beakerlib/dictionary.vim cpt=.,w,b,u,t,i,k`
		`+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`
		`+ #`
		`+ # runtest.sh of /CoreOS/coreutils/readlink-cannot-handle-recursive-symlinks`
		`+ # Description: Test for readlink cannot handle recursive symlinks`
		`+ # Author: Jan Scotka <jscotka@redhat.com>`
		`+ #`
		`+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`
		`+ #`
		`+ # Copyright (c) 2010 Red Hat, Inc. All rights reserved.`
		`+ #`
		`+ # This copyrighted material is made available to anyone wishing`
		`+ # to use, modify, copy, or redistribute it subject to the terms`
		`+ # and conditions of the GNU General Public License version 2.`
		`+ #`
		`+ # This program is distributed in the hope that it will be`
		`+ # useful, but WITHOUT ANY WARRANTY; without even the implied`
		`+ # warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR`
		`+ # PURPOSE. See the GNU General Public License for more details.`
		`+ #`
		`+ # You should have received a copy of the GNU General Public`
		`+ # License along with this program; if not, write to the Free`
		`+ # Software Foundation, Inc., 51 Franklin Street, Fifth Floor,`
		`+ # Boston, MA 02110-1301, USA.`
		`+ #`
		`+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`
		`+`
		`+ # Include rhts environment`
		`+ . /usr/bin/rhts-environment.sh`
		`+ . /usr/lib/beakerlib/beakerlib.sh`
		`+`
		`+ PACKAGE="coreutils"`
		`+`
		`+ rlJournalStart`
		`+ rlPhaseStartSetup`
		`+ rlAssertRpm $PACKAGE`
		+ rlRun "TmpDir=\`mktemp -d\`" 0 "Creating tmp directory"
		`+ rlRun "pushd $TmpDir"`
		`+ rlRun "mkdir link" 0 "Creating test directory"`
		`+ rlRun "cd link" 0 "cd to this dir"`
		`+ rlRun "ln -s ../link link" 0 "creating symlink to ../link -> link"`
		`+ rlPhaseEnd`
		`+`
		`+ rlPhaseStartTest`
		`+ rlLog "Test of readlink up to 20 iteration"`
		`+ export TMPVAR="link"`
		+ for foo in `seq 20`
		`+ do echo $TMPVAR`
		`+ rlRun "readlink -v -f $TMPVAR" 0 "readlink of $TMPVAR"`
		`+ TMPVAR="$TMPVAR/link"`
		`+ done`
		`+ rlPhaseEnd`
		`+`
		`+ rlPhaseStartCleanup`
		`+ rlRun "popd"`
		`+ rlRun "rm -r $TmpDir" 0 "Removing tmp directory"`
		`+ rlPhaseEnd`
		`+ rlJournalPrintText`
		`+ rlJournalEnd`

tests/test-basics

file added

+39

		`@@ -0,0 +1,39 @@`
		`+ #!/bin/sh`
		`+`
		`+ # Checks that touch ls rm and foo work`
		`+ # https://www.mankier.com/1/beakerlib#Examples`
		`+ . /usr/share/beakerlib/beakerlib.sh`
		`+`
		`+ # Set the full test name`
		`+ TEST="/examples/beakerlib/Sanity/phases"`
		`+`
		`+ # Package being tested`
		`+ PACKAGE="coreutils"`
		`+`
		`+ rlJournalStart`
		`+ # Setup phase: Prepare test directory`
		`+ rlPhaseStartSetup`
		`+ rlAssertRpm $PACKAGE`
		`+ rlRun 'TmpDir=$(mktemp -d)' 0 'Creating tmp directory' # no-reboot`
		`+ rlRun "pushd $TmpDir"`
		`+ rlPhaseEnd`
		`+`
		`+ # Test phase: Testing touch, ls and rm commands`
		`+ rlPhaseStartTest`
		`+ rlRun "touch foo" 0 "Creating the foo test file"`
		`+ rlAssertExists "foo"`
		`+ rlRun "ls -l foo" 0 "Listing the foo test file"`
		`+ rlRun "rm foo" 0 "Removing the foo test file"`
		`+ rlAssertNotExists "foo"`
		`+ rlRun "ls -l foo" 2 "Listing foo should now report an error"`
		`+ rlPhaseEnd`
		`+`
		`+ # Cleanup phase: Remove test directory`
		`+ rlPhaseStartCleanup`
		`+ rlRun "popd"`
		`+ rlRun "rm -r $TmpDir" 0 "Removing tmp directory"`
		`+ rlPhaseEnd`
		`+ rlJournalEnd`
		`+`
		`+ # Print the test report`
		`+ rlJournalPrintText`

tests/test_basics.yml

file added

		`@@ -0,0 +1,9 @@`
		`+ ---`
		`+ # This first play always runs on the local staging system`
		`+ - hosts: localhost`
		`+ tags:`
		`+ - atomic`
		`+ - classic`
		`+ - container`
		`+ roles:`
		`+ - { role: standard-test-beakerlib, tests: [ test-basics, readlink-cannot-handle-recursive-symlinks ] }`

tests/tests.yml

file added

		`@@ -0,0 +1,2 @@`
		`+ # Fedora Continuous Integration: https://fedoraproject.org/wiki/CI`
		`+ - include: test_basics.yml`

rasibley commented 6 years ago

Adds tests according to the CI wiki [0] specifically the standard test interface in the spec [1].

The playbook includes Tier1 level test cases that have been tested in the following contexts and is passing reliably: Atomic Host, Docker, and Classic.
Test logs are stored in the Artifacts directory.

The following steps are used to execute the tests using the standard test interface:

Atomic
sudo ANSIBLE_INVENTORY=$(test -e inventory && echo inventory || echo /usr/share/ansible/inventory) TEST_SUBJECTS=../atomic.qcow2 TEST_ARTIFACTS=$PWD/artifacts ansible-playbook --tags atomic tests.yml
Docker
sudo ANSIBLE_INVENTORY=$(test -e inventory && echo inventory || echo /usr/share/ansible/inventory) TEST_DEBUG=1 TEST_SUBJECTS=docker:docker.io/library/fedora:26 TEST_ARTIFACTS=$PWD/artifacts ansible-playbook --tags container tests.yml
Classic
sudo ANSIBLE_INVENTORY=$(test -e inventory && echo inventory || echo /usr/share/ansible/inventory) TEST_SUBJECTS="" TEST_ARTIFACTS=$PWD/artifacts ansible-playbook --tags classic tests.yml

Test Logs: (If you you would like a pointer to the complete log, I can include that as well)

Atomic
<snip>
03:19:46 TASK [standard-test-beakerlib : Execute beakerlib tests]
03:19:52 changed: [../atomic.qcow2] => (item=test-basics)
03:20:00 changed: [../atomic.qcow2] => (item=readlink-cannot-handle-recursive-symlinks)
03:20:00
03:20:00 TASK [standard-test-beakerlib : Make the master test summary log artifact]
03:20:01 changed: [../atomic.qcow2] => (item=test-basics)
03:20:01 changed: [../atomic.qcow2] => (item=readlink-cannot-handle-recursive-symlinks)
03:20:01
03:20:01 TASK [standard-test-beakerlib : Pull out the logs]
03:20:01 changed: [../atomic.qcow2]
03:20:01
03:20:01 TASK [standard-test-beakerlib : Check the results]
03:20:02 changed: [../atomic.qcow2]
03:20:02
03:20:02 PLAY RECAP
03:20:02 ../atomic.qcow2 : ok=13 changed=9 unreachable=0 failed=0
03:20:02
03:20:02 ###################################
03:20:02 Test results:
03:20:02 ###################################
03:20:02 PASS test-basics
03:20:02 PASS readlink-cannot-handle-recursive-symlinks
03:20:02 ###################################
Docker
<snip>
03:23:08 TASK [standard-test-beakerlib : Execute beakerlib tests]
03:23:11 changed: [2e310b9a86988556529f8e7ce4cb1c5c73f2201ea067642776534ee641ac4677] => (item=test-basics)
03:23:14 changed: [2e310b9a86988556529f8e7ce4cb1c5c73f2201ea067642776534ee641ac4677] => (item=readlink-cannot-handle-recursive-symlinks)
03:23:14
03:23:14 TASK [standard-test-beakerlib : Make the master test summary log artifact]
03:23:14 changed: [2e310b9a86988556529f8e7ce4cb1c5c73f2201ea067642776534ee641ac4677] => (item=test-basics)
03:23:15 changed: [2e310b9a86988556529f8e7ce4cb1c5c73f2201ea067642776534ee641ac4677] => (item=readlink-cannot-handle-recursive-symlinks)
03:23:15
03:23:15 TASK [standard-test-beakerlib : Pull out the logs]
03:23:16 changed: [2e310b9a86988556529f8e7ce4cb1c5c73f2201ea067642776534ee641ac4677]
03:23:16
03:23:16 TASK [standard-test-beakerlib : Check the results]
03:23:17 changed: [2e310b9a86988556529f8e7ce4cb1c5c73f2201ea067642776534ee641ac4677]
03:23:17
03:23:17 PLAY RECAP
03:23:17 2e310b9a86988556529f8e7ce4cb1c5c73f2201ea067642776534ee641ac4677 : ok=14 changed=10 unreachable=0 failed=0
03:23:17
03:23:17 ###################################
03:23:17 Test results:
03:23:17 ###################################
03:23:17 PASS test-basics
03:23:17 PASS readlink-cannot-handle-recursive-symlinks
Classic
<snip>
03:16:37 ==> default: TASK [standard-test-beakerlib : Execute beakerlib tests]
03:16:39 ==> default: changed: [localhost] => (item=test-basics)
03:16:41 ==> default: changed: [localhost] => (item=readlink-cannot-handle-recursive-symlinks)
03:16:41 ==> default:
03:16:41 ==> default: TASK [standard-test-beakerlib : Make the master test summary log artifact]
03:16:42 ==> default: changed: [localhost] => (item=test-basics)
03:16:42 ==> default: changed: [localhost] => (item=readlink-cannot-handle-recursive-symlinks)
03:16:42 ==> default:
03:16:42 ==> default: TASK [standard-test-beakerlib : Pull out the logs]
03:16:42 ==> default: changed: [localhost]
03:16:42 ==> default:
03:16:42 ==> default: TASK [standard-test-beakerlib : Check the results]
03:16:42 ==> default: changed: [localhost]
03:16:42 ==> default:
03:16:42 ==> default: PLAY RECAP
03:16:42 ==> default: localhost : ok=14 changed=11 unreachable=0 failed=0
03:16:42 ==> default: ++ '[' 0 -ne 0 ']'
03:16:42 ==> default: ++ cat /home/vagrant/coreutils/artifacts/test.log
03:16:42 ==> default: PASS test-basics
03:16:42 ==> default: PASS readlink-cannot-handle-recursive-symlinks

[0] https://fedoraproject.org/wiki/CI
[1] https://fedoraproject.org/wiki/Changes/InvokingTests

kdudka commented 6 years ago

Thank you for working on this! Do I understand it correctly that the tests you prepared work already and just merging this PR will enable the CI?

rasibley commented 6 years ago

Your welcome! Yes this will enable the CI, yet gating is currently disabled, so nothing will change. Also just want to point out the tests will run on each dist-git commit, they are not triggered on koji builds and if you are using FMN, it should notify you of failures normally.

The RH QE maintainer contact in case you have questions: rskvaril@redhat.com
The idea is that these tests become yours just as you're maintaining the package, there will of course be people around if you have questions or troubles.

kdudka commented 6 years ago

I am not sure how to use this interface for pull requests but your commits are now pushed.