PR#9: Add CI tests using the standard test interface - rpms/cups

rpms / cups

#9 Add CI tests using the standard test interface

Closed 5 years ago by sturivny. Opened 5 years ago by sturivny.

git://fedorapeople.org/~sturivny/cups upstream_tests into master

Add CI tests using the standard test interface

sturivny • 5 years ago

e42dc1c

tests/basic-commands/Makefile

file added

+59

		`@@ -0,0 +1,59 @@`
		`+ # Makefile - basic commands`
		`+ # Author: Petr Sklenar <psklenar@redhat.com>`
		`+ # Location: /CoreOS/cups/Sanity/basic_commands/Makefile`
		`+`
		`+ # Description: Test for basic CUPS commands`
		`+`
		`+ # Copyright (c) 2008 Red Hat, Inc. This copyrighted material`
		`+ # is made available to anyone wishing to use, modify, copy, or`
		`+ # redistribute it subject to the terms and conditions of the GNU General`
		`+ # Public License v.2.`
		`+ #`
		`+ # This program is distributed in the hope that it will be useful, but WITHOUT ANY`
		`+ # WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A`
		`+ # PARTICULAR PURPOSE. See the GNU General Public License for more details.`
		`+ #`
		`+ # You should have received a copy of the GNU General Public License`
		`+ # along with this program; if not, write to the Free Software`
		`+ # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.`
		`+`
		`+`
		`+`
		`+ TOPLEVEL_NAMESPACE=/CoreOS`
		`+ PACKAGE_NAME=cups`
		`+ RELATIVE_PATH=Sanity/basic-commands`
		`+`
		`+ export TEST=$(TOPLEVEL_NAMESPACE)/$(PACKAGE_NAME)/$(RELATIVE_PATH)`
		`+ export TESTVERSION=1.0`
		`+`
		`+ BUILT_FILES=`
		`+`
		`+ FILES=$(METADATA) runtest.sh testfile.jpg testfile.pdf Makefile PURPOSE`
		`+`
		`+ .PHONY: all install download clean`
		`+`
		`+ run: $(FILES) build`
		`+ ./runtest.sh`
		`+`
		`+ build: $(BUILT_FILES)`
		`+ chmod a+x ./runtest.sh`
		`+`
		`+ clean:`
		`+ rm -f *~ $(BUILT_FILES)`
		`+`
		`+`
		`+ $(METADATA): Makefile`
		`+ @touch $(METADATA)`
		`+ @echo "Owner: Petr Sklenar <psklenar@redhat.com>" > $(METADATA)`
		`+ @echo "Name: $(TEST)" >> $(METADATA)`
		`+ @echo "Path: $(TEST_DIR)" >> $(METADATA)`
		`+ @echo "TestVersion: $(TESTVERSION)" >> $(METADATA)`
		`+ @echo "Description: Test for basic CUPS commands" >> $(METADATA)`
		`+ @echo "Type: Sanity" >> $(METADATA)`
		`+ @echo "TestTime: 15m" >> $(METADATA)`
		`+ @echo "RunFor: $(PACKAGE_NAME)" >> $(METADATA)`
		`+ @echo "Requires: $(PACKAGE_NAME)" >> $(METADATA)`
		`+ @echo "Requires: ghostscript libjpeg-turbo" >> $(METADATA)`
		`+ @echo "License: GPLv2" >> $(METADATA)`
		`+`
		`+ rhts-lint $(METADATA)`

tests/basic-commands/PURPOSE

file added

+11

		`@@ -0,0 +1,11 @@`
		`+ Test Name: basic commands`
		`+ Author: Yulia Kopkova <ykopkova@redhat.com>`
		`+ Location: /CoreOS/cups/Sanity/basic_commands/Makefile`
		`+`
		`+ Short Description: Test for basic CUPS commands`
		`+`
		`+`
		`+ Long Description:`
		`+`
		`+ Test covers basic commands such as: lpadmin, lp, lpr, lpstat, cancel, lpinfo, lpq, lpc, lprm`
		`+ Test based on CUPS test plan from http://www.cups.org`

tests/basic-commands/runtest.sh

file added

+144

		`@@ -0,0 +1,144 @@`
		`+ #!/bin/bash`
		`+ # runtest.sh - basic commands`
		`+ # Author: Yulia Kopkova <ykopkova@redhat.com>`
		`+ # Location: /CoreOS/cups/Sanity/basic_commands/Makefile`
		`+`
		`+ # Description: Test for basic CUPS commands`
		`+`
		`+ # Copyright (c) 2008 Red Hat, Inc. This copyrighted material`
		`+ # is made available to anyone wishing to use, modify, copy, or`
		`+ # redistribute it subject to the terms and conditions of the GNU General`
		`+`
		`+ # Public License v.2.`
		`+ #`
		`+ # This program is distributed in the hope that it will be useful, but WITHOUT ANY`
		`+ # WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A`
		`+ # PARTICULAR PURPOSE. See the GNU General Public License for more details.`
		`+ #`
		`+ # You should have received a copy of the GNU General Public License`
		`+ # along with this program; if not, write to the Free Software`
		`+ # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.`
		`+`
		`+`
		`+ PACKAGE=cups`
		`+`
bgoncalv commented 5 years ago You need to load beakerlib functions. Try adding: . /usr/share/beakerlib/beakerlib.sh \|\| exit 1
		`+ FILEPDF=testfile.pdf`
		`+ FILEJPG=testfile.jpg`
		`+ NUMJOBS=50`
		`+ TIMEOUT=30`
		`+ TPRN1="testprinter1"`
		`+ TPRN2="testprinter2"`
		`+ FILEDEV="FileDevice YES"`
		`+ CUPSCONF=/etc/cups/cupsd.conf`
		+ if `rlIsRHEL 5` ; then
		`+ ML="postscript.ppd.gz"`
		`+ else`
		`+ ML="drv:///sample.drv/deskjet.ppd"`
		`+ fi`
		`+`
		`+ function setCupsCtlFileDeviceConf {`
		+ if `rlIsRHEL 5 6`;then
		`+ echo 'FileDevice YES' >> /etc/cups/cupsd.conf`
		`+ else`
		`+ echo 'FileDevice YES' >> /etc/cups/cups-files.conf`
		`+ fi`
		`+ }`
		`+`
		`+ rlJournalStart`
		`+ rlPhaseStartSetup`
		`+`
		`+ rlAssertRpm $PACKAGE`
		`+ rlFileBackup --clean "/etc/cups" "/etc/printcap"`
		`+`
		`+ rlServiceStop cups`
		`+ rlFileBackup $CUPSCONF /etc/cups/cups-files.conf`
		`+ rlRun "setCupsCtlFileDeviceConf"`
		`+ rlServiceStart cups`
		`+`
		`+ rlPhaseEnd`
		`+`
		`+ rlPhaseStartTest`
		`+`
		`+ rlRun "lpadmin -p $TPRN1 -v file:/dev/null -E -m $ML" 0 "Create printer $TPRN1"`
		`+ rlRun "lpadmin -d $TPRN1" 0 "Set $TPRN1 default"`
		`+ rlRun "lpadmin -p $TPRN2 -v file:/dev/null -E -m $ML" 0 "Create printer $TPRN2"`
		`+`
		`+ rlRun "lp $FILEPDF" 0 "Print $FILEPDF with lp and default printer"`
		`+ rlRun "lp -d $TPRN2 $FILEJPG" 0 "Print $FILEJPG with lp and $TPRN2"`
		`+ rlRun "lp -d $TPRN1 -P 1-2 -o job-sheets=classified,classified $FILEPDF" 0 "Print $FILEPDF with lp and $TPRN1"`
		`+`
		`+ for ((i=0; i < $NUMJOBS; i++)); do`
		`+ lp -d $TPRN1 $FILEJPG &`
		`+ sleep 0.02s`
		`+ lp -d $TPRN2 $FILEJPG &`
		`+ sleep 0.02s`
		`+ lppid=$!`
		`+ done`
		`+`
		`+ rlRun "wait $lppid" 0 "$NUMJOBS jobs queued"`
		`+`
		`+ for ((i=$TIMEOUT; i>0; i-=5 )); do`
		`+ jobs=$(lpstat)`
		`+ [ "x$jobs" = "x" ] && break`
		`+ sleep 5`
		`+ done`
		`+`
		`+ rlRun "lpr $FILEPDF" 0 "Print $FILEPDF with lpr and default printer"`
		`+ rlRun "lpr -P $TPRN2 $FILEJPG" 0 "Print $FILEJPG with lpr and $TPRN2"`
		`+ rlRun "lpr -P $TPRN1 -o number-up=2 -o job-sheets=standard,none $FILEPDF" 0 "Print $FILEPDF with lpr and $TPRN1"`
		`+`
		`+ for ((i=0; i < $NUMJOBS ; i++)); do`
		`+ lpr -P $TPRN1 $FILEJPG &`
		`+ sleep 0.02s`
		`+ lpr -P $TPRN2 $FILEJPG &`
		`+ sleep 0.02s`
		`+ lprpid=$!`
		`+ done`
		`+`
		`+ rlRun "wait $lprpid" 0 "$NUMJOBS jobs queued"`
		`+`
		`+ for ((i=$TIMEOUT; i>0 ; i-=5)); do`
		`+ jobs=$(lpstat)`
		`+ [ "x$jobs" = "x" ] && break`
		`+ sleep 5`
		`+ done`
		`+`
		`+ job_id=$(lp -d $TPRN1 -o job-hold-until=indefinite $FILEJPG \| awk '{ match($0, /testprinter1-([0-9]+)/, arr); print arr[1] }')`
		`+ rlRun "cancel $TPRN1-$job_id" 0 "Cancel $TPRN1-$job_id job"`
		`+ rlRun "lp -d $TPRN1 -o job-hold-until=indefinite $FILEJPG" 0 "Hold job on $TPRN1"`
		`+ rlRun "lp -d $TPRN2 -o job-hold-until=indefinite $FILEJPG" 0 "Hold job on $TPRN2"`
		`+ rlRun "cancel -a" 0 "Cancel all jobs"`
		`+`
		`+ rlRun "lpc status" 0 "Show printers status"`
		`+`
		`+ rlRun "lpq" 0 "Show printer queue status"`
		`+`
		`+ rlRun "lpstat -t" 0 "Show all status information"`
		`+`
		`+ rlRun "lpinfo -m" 0 "Show list of available drivers"`
		`+`
		`+ rlRun "lpinfo -v" 0 "Show list of available devices"`
		`+`
		`+ rlRun "lpr -o job-hold-until=indefinite $FILEJPG" 0 "Hold test file"`
		`+ rlRun "lprm" 0 "Cancel current job on default printer"`
		`+`
		`+ rlRun "lpr -P $TPRN1 -o job-hold-until=indefinite $FILEJPG" 0 "Hold test file"`
		`+ rlRun "lprm $TPRN1" 0 "Cancel current job on $TPRN1"`
		`+`
		`+ rlRun "lpadmin -p $TPRN1 -v file:/tmp/$TPRN1 -o PageSize=A4" 0 "Modify $TPRN1"`
		`+`
		`+ rlRun "lpadmin -x $TPRN1" 0 "Delete $TPRN1"`
		`+ rlRun "lpadmin -x $TPRN2" 0 "Delete $TPRN2"`
		`+`
		`+`
		`+ rlPhaseEnd`
		`+`
		`+ rlPhaseStartCleanup`
		`+ rlServiceStop cups`
		`+ rlRun "rm -f $CUPSCONF" 0 "Remove modified cupsd.conf"`
		`+ rlFileRestore`
		`+ rlServiceRestore cups`
		`+`
		`+ rlPhaseEnd`
		`+ rlJournalPrintText`
		`+ rlJournalEnd`

tests/basic-commands/testfile.jpg

file added

empty or binary file added

tests/basic-commands/testfile.pdf

file added

empty or binary file added

tests/tests.yml

file added

+11

		`@@ -0,0 +1,11 @@`
		`+ ---`
		`+ # This first play always runs on the local staging system`
		`+ - hosts: localhost`
		`+ roles:`
		`+ - role: standard-test-beakerlib`
		`+ tags:`
		`+ - classic`
		`+ tests:`
		`+ - basic-commands`
		`+ required_packages:`
		`+ - cups`

no initial comment

sturivny commented 5 years ago

Justification

Adds tests according to the CI wiki specifically the standard test interface in the spec.

The playbook includes Tier1 level test cases that have been tested in the following contexts and is passing reliably: Classic and Container. Test logs are stored in the artifacts directory.

The following steps are used to execute the tests using the standard test interface:

Test environment

Make sure you have installed packages from the spec

$ rpm -q ansible python2-dnf libselinux-python standard-test-roles
ansible-2.3.2.0-1.fc26.noarch
python2-dnf-2.6.3-11.fc26.noarch
libselinux-python-2.6-7.fc26.x86_64
standard-test-roles-2.4-1.fc26.noarch

Clone tests to your PC

$ git clone git://fedorapeople.org/~sturivny/cups
$ cd cups/

Run tests for Classic

All test should be run under the root

# tests/
# sudo ansible-playbook --tags=classic tests.yml

Snip of the example test run:

TASK [str-common-final : Pull out the logs from test environment to test runner] **************************************************************************************************************
changed: [localhost]

TASK [str-common-final : Report role result] **************************************************************************************************************************************************
ok: [localhost] => {
    "failed_when_result": false, 
    "msg": [
        "Tests failed: False", 
        "Tests msg: ", 
        ""
    ]
}

PLAY RECAP ************************************************************************************************************************************************************************************
localhost                  : ok=33   changed=9    unreachable=0    failed=0

Run tests for Atomic

All test should be run under the root

# export ANSIBLE_INVENTORY=$(test -e inventory && echo inventory || echo /usr/share/ansible/inventory)
# curl -Lo /tmp/atomic.qcow2 https://getfedora.org/atomic_qcow2_latest
# export TEST_SUBJECTS=/tmp/atomic.qcow2
# ansible-playbook --tags=atomic tests.yml

Snip of the example test run:

TASK [str-common-final : Pull out the logs from test environment to test runner] **************************************************************************************************************
changed: [/tmp/atomic.qcow2]

TASK [str-common-final : Report role result] **************************************************************************************************************************************************
ok: [/tmp/atomic.qcow2] => {
    "failed_when_result": false, 
    "msg": [
        "Tests failed: False", 
        "Tests msg: ", 
        ""
    ]
}

PLAY RECAP ************************************************************************************************************************************************************************************
/tmp/atomic.qcow2          : ok=35   changed=18   unreachable=0    failed=0

Notes

Tests will be enabled in CI, yet gating is currently disabled, so nothing will change. Tests will run on each dist-git commit, they are not triggered by koji builds and if you are using FMN, it should notify you of failures normally.

The RH QE maintainer contact in case you have questions: Petr Sklenar psklenar@redhat.com
The idea is that these tests become yours just as you're maintaining the package, there will, of course, be people around if you have questions or troubles.

sturivny commented 5 years ago

@zdohnal: Created new updated PR.

zdohnal commented 5 years ago

Hi Serhii,

when I run the tests on F28 machine, I got this error:

TASK [standard-test-beakerlib : Run beakerlib tests] ******************
failed: [localhost] (item=basic-commands) => {"changed": true, "cmd": "export OUTPUTFILE=/dev/stdout TEST=basic-commands\n export PATH=\"$PATH:/var/str/\"\n logfile=/tmp/artifacts//$(echo basic-commands | sed -e 's/\//-/g').log\n exec 2>>$logfile 1>>$logfile\n cd /var/str/\n if [ -f basic-commands ]; then\n cd $(dirname basic-commands)\n /bin/sh -e ./$(basename basic-commands)\n elif [ -d basic-commands ]; then\n cd basic-commands\n if [ -f Makefile ] && command -p -v make >/dev/null 2>&1; then\n make run\n elif [ -f runtest.sh ]; then\n /bin/sh -e ./runtest.sh\n else\n echo \"FAIL don't know how to run test basic-commands\"\n fi\n else\n echo \"FAIL test basic-commands does not appear to be a file or directory\"\n fi", "delta": "0:00:04.295463", "end": "2018-07-09 06:40:12.931638", "item": "basic-commands", "msg": "non-zero return code", "rc": 2, "start": "2018-07-09 06:40:08.636175", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

zdohnal commented 5 years ago

@sturivny Same on rawhide machine.

zdohnal commented 5 years ago

[citest]

sturivny commented 5 years ago

@zdohnal: Trying to reproduce

zdohnal commented 5 years ago

@sturivny It seems like beakerlib and cups itself (at least cups-client) is missing when I try to run the tests. Sample from FAIL_basic-commands.log file from artifacts directory:

chmod a+x ./runtest.sh
./runtest.sh
./runtest.sh: line 33: rlIsRHEL: command not found
./runtest.sh: line 47: rlJournalStart: command not found
./runtest.sh: line 48: rlPhaseStartSetup: command not found
./runtest.sh: line 50: rlAssertRpm: command not found
./runtest.sh: line 51: rlFileBackup: command not found
./runtest.sh: line 53: rlServiceStop: command not found
./runtest.sh: line 54: rlFileBackup: command not found
./runtest.sh: line 55: rlRun: command not found
./runtest.sh: line 56: rlServiceStart: command not found
./runtest.sh: line 58: rlPhaseEnd: command not found
./runtest.sh: line 60: rlPhaseStartTest: command not found
./runtest.sh: line 62: rlRun: command not found
./runtest.sh: line 63: rlRun: command not found
./runtest.sh: line 64: rlRun: command not found
./runtest.sh: line 66: rlRun: command not found
./runtest.sh: line 67: rlRun: command not found
./runtest.sh: line 68: rlRun: command not found
lp: No such file or directory
lp: No such file or directory
lp: No such file or directory
lp: No such file or directory
lp: No such file or directory
lp: No such file or directory
lp: No such file or directory
lp: No such file or directory
lp: No such file or directory
lp: No such file or directory
lp: No such file or directory
lp: No such file or directory
lp: No such file or directory