PR#1: Add CI tests using the standard test interface - rpms/gdb

rpms / gdb

#1 Add CI tests using the standard test interface

Closed 6 years ago by sergiodj. Opened 6 years ago by esakaiev.

git://fedorapeople.org/~esakaiev/gdb new_tests into master

Adding tests to the new_tests branch

esakaiev@redhat.com • 6 years ago

ccb743e

tests/README

file added

+28

		`@@ -0,0 +1,28 @@`
		`+ Justification`
		`+`
		`+ Adds tests according to the CI wiki specifically the standard test interface in the spec.`
		`+`
		`+ The playbook includes Tier1 level test cases that have been tested in the following contexts and is passing reliably: Classic and Container. Test logs are stored in the artifacts directory.`
		`+`
		`+ The following steps are used to execute the tests using the standard test interface:`
		`+ Test environment`
		`+`
		`+ Make sure you have installed packages from the spec`
		`+`
		`+ # rpm -q ansible python2-dnf libselinux-python standard-test-roles \`
		`+ ansible-2.3.2.0-1.fc26.noarch \`
		`+ python2-dnf-2.6.3-11.fc26.noarch \`
		`+ libselinux-python-2.6-7.fc26.x86_64 \`
		`+ standard-test-roles-2.4-1.fc26.noarch`
		`+`
		`+ Run tests for Classic`
		`+`
		`+ # export ANSIBLE_INVENTORY=$(test -e inventory && echo inventory \|\| echo /usr/share/ansible/inventory)`
		`+ # ansible-playbook --tags=classic tests.yml`
		`+`
		`+ Run tests for Container`
		`+`
		`+ # export ANSIBLE_INVENTORY=$(test -e inventory && echo inventory \|\| echo /usr/share/ansible/inventory)`
		`+ # export TEST_SUBJECTS=docker:docker.io/library/fedora:26`
		`+ # ansible-playbook --tags=container tests.yml`
		`+`

tests/debug-system-binary/Makefile

file added

+64

		`@@ -0,0 +1,64 @@`
		`+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`
		`+ #`
		`+ # Makefile of /tools/gdb/Sanity/debug-system-binary`
		`+ # Description: Debug a system binary.`
		`+ # Author: Marek Polacek <polacek@redhat.com>`
		`+ #`
		`+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`
		`+ #`
		`+ # Copyright (c) 2012 Red Hat, Inc. All rights reserved.`
		`+ #`
		`+ # This copyrighted material is made available to anyone wishing`
		`+ # to use, modify, copy, or redistribute it subject to the terms`
		`+ # and conditions of the GNU General Public License version 2.`
		`+ #`
		`+ # This program is distributed in the hope that it will be`
		`+ # useful, but WITHOUT ANY WARRANTY; without even the implied`
		`+ # warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR`
		`+ # PURPOSE. See the GNU General Public License for more details.`
		`+ #`
		`+ # You should have received a copy of the GNU General Public`
		`+ # License along with this program; if not, write to the Free`
		`+ # Software Foundation, Inc., 51 Franklin Street, Fifth Floor,`
		`+ # Boston, MA 02110-1301, USA.`
		`+ #`
		`+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`
		`+`
		`+ export TEST=/tools/gdb/Sanity/debug-system-binary`
		`+ export TESTVERSION=1.0`
		`+`
		`+ BUILT_FILES=`
		`+`
		`+ FILES=$(METADATA) runtest.sh Makefile PURPOSE cmds`
		`+`
		`+ .PHONY: all install download clean`
		`+`
		`+ run: $(FILES) build`
		`+ ./runtest.sh`
		`+`
		`+ build: $(BUILT_FILES)`
		`+ test -x runtest.sh \|\| chmod a+x runtest.sh`
		`+`
		`+ clean:`
		`+ rm -f *~ $(BUILT_FILES)`
		`+`
		`+`
		`+ include /usr/share/rhts/lib/rhts-make.include`
		`+`
		`+ $(METADATA): Makefile`
		`+ @echo "Owner: Marek Polacek <mpolacek@redhat.com>" > $(METADATA)`
		`+ @echo "Name: $(TEST)" >> $(METADATA)`
		`+ @echo "TestVersion: $(TESTVERSION)" >> $(METADATA)`
		`+ @echo "Path: $(TEST_DIR)" >> $(METADATA)`
		`+ @echo "Description: Debug a system binary." >> $(METADATA)`
		`+ @echo "Type: Sanity" >> $(METADATA)`
		`+ @echo "TestTime: 5m" >> $(METADATA)`
		`+ @echo "RunFor: gdb" >> $(METADATA)`
		`+ @echo "Requires: gdb coreutils coreutils-debuginfo glibc glibc-debuginfo" >> $(METADATA)`
		`+ @echo "Priority: Normal" >> $(METADATA)`
		`+ @echo "License: GPLv2" >> $(METADATA)`
		`+ @echo "Confidential: no" >> $(METADATA)`
		`+ @echo "Destructive: no" >> $(METADATA)`
		`+ @echo "Releases: -RHEL4 -RHELClient5 -RHELServer5" >> $(METADATA)`
		`+`
		`+ rhts-lint $(METADATA)`

tests/debug-system-binary/PURPOSE

file added

		`@@ -0,0 +1,3 @@`
		`+ PURPOSE of /tools/gdb/Sanity/debug-system-binary`
		`+ Description: Debug a system binary.`
		`+ Author: Marek Polacek <polacek@redhat.com>`

tests/debug-system-binary/cmds

file added

+16

		`@@ -0,0 +1,16 @@`
		`+ sta`
		`+ si 30`
		`+ n 10`
		`+ up`
		`+ n 15`
		`+ up`
		`+ p/x &hextobin`
		`+ si 10`
		`+ up`
		`+ down`
		`+ bt`
		`+ frame 1`
		`+ b exit`
		`+ c`
		`+ c`
		`+ q`

tests/debug-system-binary/runtest.sh

file added

+60

		`@@ -0,0 +1,60 @@`
		`+ #!/bin/bash`
		`+ # vim: dict=/usr/share/beakerlib/dictionary.vim cpt=.,w,b,u,t,i,k`
		`+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`
		`+ #`
		`+ # runtest.sh of /tools/gdb/Sanity/debug-system-binary`
		`+ # Description: Debug a system binary.`
		`+ # Author: Marek Polacek <polacek@redhat.com>`
		`+ #`
		`+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`
		`+ #`
		`+ # Copyright (c) 2012 Red Hat, Inc. All rights reserved.`
		`+ #`
		`+ # This copyrighted material is made available to anyone wishing`
		`+ # to use, modify, copy, or redistribute it subject to the terms`
		`+ # and conditions of the GNU General Public License version 2.`
		`+ #`
		`+ # This program is distributed in the hope that it will be`
		`+ # useful, but WITHOUT ANY WARRANTY; without even the implied`
		`+ # warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR`
		`+ # PURPOSE. See the GNU General Public License for more details.`
		`+ #`
		`+ # You should have received a copy of the GNU General Public`
		`+ # License along with this program; if not, write to the Free`
		`+ # Software Foundation, Inc., 51 Franklin Street, Fifth Floor,`
		`+ # Boston, MA 02110-1301, USA.`
		`+ #`
		`+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`
		`+`
		`+ # Include Beaker environment`
		`+ . /usr/bin/rhts-environment.sh \|\| exit 1`
		`+ . /usr/lib/beakerlib/beakerlib.sh \|\| exit 1`
		`+`
		`+ CMD="gdb"`
		`+ BIN="$(which $CMD)"`
		`+ export PACKAGE="${PACKAGE:-$(rpm -qf --qf='%{name}\n' $BIN)}"`
		`+`
		`+ PACKAGES="coreutils coreutils-debuginfo glibc glibc-debuginfo"`
		`+`
		`+ rlJournalStart`
		`+ rlPhaseStartSetup`
		`+ rlShowRunningKernel`
		`+ rlAssertRpm $PACKAGE`
		`+ for pkg in $PACKAGES; do`
		`+ rlAssertRpm $pkg`
		`+ done`
		`+ rlRun "TmpDir=\$(mktemp -d)"`
		`+ rlRun "cp cmds $TmpDir"`
		`+ rlRun "pushd $TmpDir"`
		`+ rlPhaseEnd`
		`+`
		`+ rlPhaseStartTest`
		`+ rlRun "gdb -nx -q -batch -x cmds /bin/echo"`
		`+ rlPhaseEnd`
		`+`
		`+ rlPhaseStartCleanup`
		`+ rlRun "popd"`
		`+ rlRun "rm -r $TmpDir"`
		`+ rlPhaseEnd`
		`+ rlJournalPrintText`
		`+ rlJournalEnd`

tests/debug-toolset-binary/Makefile

file added

+64

		`@@ -0,0 +1,64 @@`
		`+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`
		`+ #`
		`+ # Makefile of /tools/gdb/Sanity/debug-toolset-binary`
		`+ # Description: We debug a binary that is delivered via the Toolset.`
		`+ # Author: Marek Polacek <polacek@redhat.com>`
		`+ #`
		`+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`
		`+ #`
		`+ # Copyright (c) 2012 Red Hat, Inc. All rights reserved.`
		`+ #`
		`+ # This copyrighted material is made available to anyone wishing`
		`+ # to use, modify, copy, or redistribute it subject to the terms`
		`+ # and conditions of the GNU General Public License version 2.`
		`+ #`
		`+ # This program is distributed in the hope that it will be`
		`+ # useful, but WITHOUT ANY WARRANTY; without even the implied`
		`+ # warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR`
		`+ # PURPOSE. See the GNU General Public License for more details.`
		`+ #`
		`+ # You should have received a copy of the GNU General Public`
		`+ # License along with this program; if not, write to the Free`
		`+ # Software Foundation, Inc., 51 Franklin Street, Fifth Floor,`
		`+ # Boston, MA 02110-1301, USA.`
		`+ #`
		`+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`
		`+`
		`+ export TEST=/tools/gdb/Sanity/debug-toolset-binary`
		`+ export TESTVERSION=1.0`
		`+`
		`+ BUILT_FILES=`
		`+`
		`+ FILES=$(METADATA) runtest.sh Makefile PURPOSE cmds`
		`+`
		`+ .PHONY: all install download clean`
		`+`
		`+ run: $(FILES) build`
		`+ ./runtest.sh`
		`+`
		`+ build: $(BUILT_FILES)`
		`+ test -x runtest.sh \|\| chmod a+x runtest.sh`
		`+`
		`+ clean:`
		`+ rm -f *~ $(BUILT_FILES)`
		`+`
		`+`
		`+ include /usr/share/rhts/lib/rhts-make.include`
		`+`
		`+ $(METADATA): Makefile`
		`+ @echo "Owner: Marek Polacek <mpolacek@redhat.com>" > $(METADATA)`
		`+ @echo "Name: $(TEST)" >> $(METADATA)`
		`+ @echo "TestVersion: $(TESTVERSION)" >> $(METADATA)`
		`+ @echo "Path: $(TEST_DIR)" >> $(METADATA)`
		`+ @echo "Description: We debug a binary that is delivered via the Toolset." >> $(METADATA)`
		`+ @echo "Type: Sanity" >> $(METADATA)`
		`+ @echo "TestTime: 5m" >> $(METADATA)`
		`+ @echo "RunFor: gdb" >> $(METADATA)`
		`+ @echo "Requires: gdb coreutils binutils binutils-debuginfo glibc glibc-debuginfo gcc gcc-debuginfo zlib zlib-debuginfo" >> $(METADATA)`
		`+ @echo "Priority: Normal" >> $(METADATA)`
		`+ @echo "License: GPLv2" >> $(METADATA)`
		`+ @echo "Confidential: no" >> $(METADATA)`
		`+ @echo "Destructive: no" >> $(METADATA)`
		`+ @echo "Releases: -RHEL4 -RHELClient5 -RHELServer5" >> $(METADATA)`
		`+`
		`+ rhts-lint $(METADATA)`

tests/debug-toolset-binary/PURPOSE

file added

		`@@ -0,0 +1,3 @@`
		`+ PURPOSE of /tools/gdb/Sanity/debug-toolset-binary`
		`+ Description: We debug a binary that is delivered via the Toolset.`
		`+ Author: Marek Polacek <polacek@redhat.com>`

tests/debug-toolset-binary/cmds

file added

+18

		`@@ -0,0 +1,18 @@`
		`+ b main`
		`+ r`
		`+ n 10`
		`+ si 30`
		`+ up`
		`+ n 15`
		`+ watch myname`
		`+ p/s myname`
		`+ si 10`
		`+ up`
		`+ down`
		`+ bt`
		`+ frame 1`
		`+ b xexit`
		`+ c`
		`+ c`
		`+ c`
		`+ q`

tests/debug-toolset-binary/runtest.sh

file added

+71

		`@@ -0,0 +1,71 @@`
		`+ #!/bin/bash`
		`+ # vim: dict=/usr/share/beakerlib/dictionary.vim cpt=.,w,b,u,t,i,k`
		`+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`
		`+ #`
		`+ # runtest.sh of /tools/gdb/Sanity/debug-toolset-binary`
		`+ # Description: We debug a binary that is delivered via the Toolset.`
		`+ # Author: Marek Polacek <polacek@redhat.com>`
		`+ #`
		`+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`
		`+ #`
		`+ # Copyright (c) 2012 Red Hat, Inc. All rights reserved.`
		`+ #`
		`+ # This copyrighted material is made available to anyone wishing`
		`+ # to use, modify, copy, or redistribute it subject to the terms`
		`+ # and conditions of the GNU General Public License version 2.`
		`+ #`
		`+ # This program is distributed in the hope that it will be`
		`+ # useful, but WITHOUT ANY WARRANTY; without even the implied`
		`+ # warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR`
		`+ # PURPOSE. See the GNU General Public License for more details.`
		`+ #`
		`+ # You should have received a copy of the GNU General Public`
		`+ # License along with this program; if not, write to the Free`
		`+ # Software Foundation, Inc., 51 Franklin Street, Fifth Floor,`
		`+ # Boston, MA 02110-1301, USA.`
		`+ #`
		`+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`
		`+`
		+ # We create an empty file `empty.S'. The we run the as(1) to produce
		+ # the `a.out' file and at the same time, we debug the as(1).
		`+ # Note that as(1) should come from the Developer Toolset (TM),`
		`+ # but in fact, it doesn't have to. It should not fail either way.`
		`+`
		`+ # Include Beaker environment`
		`+ . /usr/bin/rhts-environment.sh \|\| exit 1`
		`+ . /usr/lib/beakerlib/beakerlib.sh \|\| exit 1`
		`+`
		`+ CMD="gdb"`
		`+ BIN="$(which $CMD)"`
		`+ export PACKAGE="${PACKAGE:-$(rpm -qf --qf='%{name}\n' $BIN)}"`
		`+`
		`+ PACKAGES="coreutils binutils binutils-debuginfo glibc glibc-debuginfo gcc gcc-debuginfo zlib zlib-debuginfo"`
		`+`
		`+ rlJournalStart`
		`+ rlPhaseStartSetup`
		`+ rlShowRunningKernel`
		`+ rlAssertRpm $PACKAGE`
		`+ for pkg in $PACKAGES; do`
		`+ rlAssertRpm $pkg`
		`+ done`
		`+ rlRun "TmpDir=\$(mktemp -d)"`
		`+ rlRun "cp cmds $TmpDir"`
		`+ rlRun "pushd $TmpDir"`
		`+ rlRun ": >empty.s"`
		`+ rlPhaseEnd`
		`+`
		`+ rlPhaseStartTest`
		`+ rlRun "gdb -nx -q -batch -x cmds --args as empty.s"`
		`+ rlRun "nm >symbs"`
		`+ if [ -s symbs ]; then`
		`+ rlFail "nm should't output any symbols"`
		`+ rlRun "cat symbs"`
		`+ fi`
		`+ rlPhaseEnd`
		`+`
		`+ rlPhaseStartCleanup`
		`+ rlRun "popd"`
		`+ rlRun "rm -r $TmpDir"`
		`+ rlPhaseEnd`
		`+ rlJournalPrintText`
		`+ rlJournalEnd`

tests/inventory

file added

		`@@ -0,0 +1,3 @@`
		`+ #!/bin/bash`
		`+ export TEST_DOCKER_EXTRA_ARGS="--privileged"`
		`+ exec merge-standard-inventory "$@"`

tests/tests.yml

file added

+24

		`@@ -0,0 +1,24 @@`
		`+ ---`
		`+ # This first play always runs on the local staging system`
		`+ - hosts: localhost`
		`+ roles:`
		`+ - role: standard-test-beakerlib`
		`+ tags:`
		`+ - classic`
		`+ - container`
		`+ tests:`
		`+ - debug-toolset-binary`
		`+ - debug-system-binary`
		`+ required_packages:`
		`+ - gdb`
		`+ - coreutils`
		`+ - binutils`
		`+ - binutils-debuginfo`
		`+ - glibc`
		`+ - glibc-debuginfo`
		`+ - gcc`
		`+ - gcc-debuginfo`
		`+ - zlib`
		`+ - zlib-debuginfo`
		`+ - coreutils-debuginfo`
		`+ - which`

no initial comment

esakaiev commented 6 years ago

Justification

Adds tests according to the CI wiki specifically the standard test interface in the spec.

The playbook includes Tier1 level test cases that have been tested in the following contexts and is passing reliably: Classic and Container. Test logs are stored in the artifacts directory.

The following steps are used to execute the tests using the standard test interface:

Test environment

Make sure you have installed packages from the spec

    # rpm -q ansible python2-dnf libselinux-python standard-test-roles \
    ansible-2.3.2.0-1.fc26.noarch \
    python2-dnf-2.6.3-11.fc26.noarch \
    libselinux-python-2.6-7.fc26.x86_64 \
    standard-test-roles-2.4-1.fc26.noarch

Tests should be started under root user.

Run tests for Classic

       sudo -s
    # export ANSIBLE_INVENTORY=$(test -e inventory && echo inventory || echo /usr/share/ansible/inventory)
        # ansible-playbook --tags=classic tests.yml

Run tests for Container

        sudo -s
    # export ANSIBLE_INVENTORY=$(test -e inventory && echo inventory || echo /usr/share/ansible/inventory)
    # export TEST_SUBJECTS=docker:docker.io/library/fedora:26
        # ansible-playbook --tags=container tests.yml

Snip of the example test run for Classic tests:

PLAY RECAP ********************************************************************************************************************************************************************************************************
localhost : ok=20   changed=14   unreachable=0    failed=0   

PASS debug-toolset-binary
PASS debug-system-binary

Snip of the example test run for Container tests:

PLAY RECAP ********************************************************************************************************************************************************************************************************
3d520de4a0e87c62529d2a320e2e7b4bc1a02ee93f6b57700a8297662ccf020f : ok=20   changed=14   unreachable=0    failed=0   

PASS debug-toolset-binary
PASS debug-system-binary

Notes

Tests will be enabled in CI, yet gating is currently disabled, so nothing will change. Tests will run on each dist-git commit, they are not triggered on koji builds and if you are using FMN, it should notify you of failures normally.

The RH QE maintainer contact in case you have questions: bgoncalv @redhat.com
The idea is that these tests become yours just as you're maintaining the package, there will of course be people around if you have questions or troubles.

Edited 6 years ago by esakaiev