#1 Add CI tests using the standard test interface
Closed a year ago by sergiodj. Opened a year ago by esakaiev.
git://fedorapeople.org/~esakaiev/gdb new_tests  into  master

Adding tests to the new_tests branch
esakaiev@redhat.com • a year ago  
tests/README
file added
+28

@@ -0,0 +1,28 @@

+ Justification

+ 

+ Adds tests according to the CI wiki specifically the standard test interface in the spec.

+ 

+ The playbook includes Tier1 level test cases that have been tested in the following contexts and is passing reliably: Classic and Container. Test logs are stored in the artifacts directory.

+ 

+ The following steps are used to execute the tests using the standard test interface:

+ Test environment

+ 

+ Make sure you have installed packages from the spec

+ 

+     # rpm -q ansible python2-dnf libselinux-python standard-test-roles \

+     ansible-2.3.2.0-1.fc26.noarch \

+     python2-dnf-2.6.3-11.fc26.noarch \

+     libselinux-python-2.6-7.fc26.x86_64 \

+     standard-test-roles-2.4-1.fc26.noarch

+ 

+ Run tests for Classic

+ 

+     # export ANSIBLE_INVENTORY=$(test -e inventory && echo inventory || echo /usr/share/ansible/inventory)

+         # ansible-playbook --tags=classic tests.yml

+ 

+ Run tests for Container

+ 

+     # export ANSIBLE_INVENTORY=$(test -e inventory && echo inventory || echo /usr/share/ansible/inventory)

+     # export TEST_SUBJECTS=docker:docker.io/library/fedora:26

+         # ansible-playbook --tags=container tests.yml

+ 

tests/debug-system-binary/Makefile
file added
+64

@@ -0,0 +1,64 @@

+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

+ #

+ #   Makefile of /tools/gdb/Sanity/debug-system-binary

+ #   Description: Debug a system binary.

+ #   Author: Marek Polacek <polacek@redhat.com>

+ #

+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

+ #

+ #   Copyright (c) 2012 Red Hat, Inc. All rights reserved.

+ #

+ #   This copyrighted material is made available to anyone wishing

+ #   to use, modify, copy, or redistribute it subject to the terms

+ #   and conditions of the GNU General Public License version 2.

+ #

+ #   This program is distributed in the hope that it will be

+ #   useful, but WITHOUT ANY WARRANTY; without even the implied

+ #   warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR

+ #   PURPOSE. See the GNU General Public License for more details.

+ #

+ #   You should have received a copy of the GNU General Public

+ #   License along with this program; if not, write to the Free

+ #   Software Foundation, Inc., 51 Franklin Street, Fifth Floor,

+ #   Boston, MA 02110-1301, USA.

+ #

+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

+ 

+ export TEST=/tools/gdb/Sanity/debug-system-binary

+ export TESTVERSION=1.0

+ 

+ BUILT_FILES=

+ 

+ FILES=$(METADATA) runtest.sh Makefile PURPOSE cmds

+ 

+ .PHONY: all install download clean

+ 

+ run: $(FILES) build

+ 	./runtest.sh

+ 

+ build: $(BUILT_FILES)

+ 	test -x runtest.sh || chmod a+x runtest.sh

+ 

+ clean:

+ 	rm -f *~ $(BUILT_FILES)

+ 

+ 

+ include /usr/share/rhts/lib/rhts-make.include

+ 

+ $(METADATA): Makefile

+ 	@echo "Owner:           Marek Polacek <mpolacek@redhat.com>" > $(METADATA)

+ 	@echo "Name:            $(TEST)" >> $(METADATA)

+ 	@echo "TestVersion:     $(TESTVERSION)" >> $(METADATA)

+ 	@echo "Path:            $(TEST_DIR)" >> $(METADATA)

+ 	@echo "Description:     Debug a system binary." >> $(METADATA)

+ 	@echo "Type:            Sanity" >> $(METADATA)

+ 	@echo "TestTime:        5m" >> $(METADATA)

+ 	@echo "RunFor:          gdb" >> $(METADATA)

+ 	@echo "Requires:        gdb coreutils coreutils-debuginfo glibc glibc-debuginfo" >> $(METADATA)

+ 	@echo "Priority:        Normal" >> $(METADATA)

+ 	@echo "License:         GPLv2" >> $(METADATA)

+ 	@echo "Confidential:    no" >> $(METADATA)

+ 	@echo "Destructive:     no" >> $(METADATA)

+ 	@echo "Releases:        -RHEL4 -RHELClient5 -RHELServer5" >> $(METADATA)

+ 

+ 	rhts-lint $(METADATA)

tests/debug-system-binary/PURPOSE
file added
+3

@@ -0,0 +1,3 @@

+ PURPOSE of /tools/gdb/Sanity/debug-system-binary

+ Description: Debug a system binary.

+ Author: Marek Polacek <polacek@redhat.com>

tests/debug-system-binary/cmds
file added
+16

@@ -0,0 +1,16 @@

+ sta

+ si 30

+ n 10

+ up

+ n 15

+ up

+ p/x &hextobin

+ si 10

+ up

+ down

+ bt

+ frame 1

+ b exit

+ c

+ c

+ q

tests/debug-system-binary/runtest.sh
file added
+60

@@ -0,0 +1,60 @@

+ #!/bin/bash

+ # vim: dict=/usr/share/beakerlib/dictionary.vim cpt=.,w,b,u,t,i,k

+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

+ #

+ #   runtest.sh of /tools/gdb/Sanity/debug-system-binary

+ #   Description: Debug a system binary.

+ #   Author: Marek Polacek <polacek@redhat.com>

+ #

+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

+ #

+ #   Copyright (c) 2012 Red Hat, Inc. All rights reserved.

+ #

+ #   This copyrighted material is made available to anyone wishing

+ #   to use, modify, copy, or redistribute it subject to the terms

+ #   and conditions of the GNU General Public License version 2.

+ #

+ #   This program is distributed in the hope that it will be

+ #   useful, but WITHOUT ANY WARRANTY; without even the implied

+ #   warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR

+ #   PURPOSE. See the GNU General Public License for more details.

+ #

+ #   You should have received a copy of the GNU General Public

+ #   License along with this program; if not, write to the Free

+ #   Software Foundation, Inc., 51 Franklin Street, Fifth Floor,

+ #   Boston, MA 02110-1301, USA.

+ #

+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

+ 

+ # Include Beaker environment

+ . /usr/bin/rhts-environment.sh || exit 1

+ . /usr/lib/beakerlib/beakerlib.sh || exit 1

+ 

+ CMD="gdb"

+ BIN="$(which $CMD)"

+ export PACKAGE="${PACKAGE:-$(rpm -qf --qf='%{name}\n' $BIN)}"

+ 

+ PACKAGES="coreutils coreutils-debuginfo glibc glibc-debuginfo"

+ 

+ rlJournalStart

+   rlPhaseStartSetup

+     rlShowRunningKernel

+     rlAssertRpm $PACKAGE

+     for pkg in $PACKAGES; do

+       rlAssertRpm $pkg

+     done

+     rlRun "TmpDir=\$(mktemp -d)"

+     rlRun "cp cmds $TmpDir"

+     rlRun "pushd $TmpDir"

+   rlPhaseEnd

+ 

+   rlPhaseStartTest

+     rlRun "gdb -nx -q -batch -x cmds /bin/echo"

+   rlPhaseEnd

+ 

+   rlPhaseStartCleanup

+     rlRun "popd"

+     rlRun "rm -r $TmpDir"

+   rlPhaseEnd

+ rlJournalPrintText

+ rlJournalEnd

tests/debug-toolset-binary/Makefile
file added
+64

@@ -0,0 +1,64 @@

+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

+ #

+ #   Makefile of /tools/gdb/Sanity/debug-toolset-binary

+ #   Description: We debug a binary that is delivered via the Toolset.

+ #   Author: Marek Polacek <polacek@redhat.com>

+ #

+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

+ #

+ #   Copyright (c) 2012 Red Hat, Inc. All rights reserved.

+ #

+ #   This copyrighted material is made available to anyone wishing

+ #   to use, modify, copy, or redistribute it subject to the terms

+ #   and conditions of the GNU General Public License version 2.

+ #

+ #   This program is distributed in the hope that it will be

+ #   useful, but WITHOUT ANY WARRANTY; without even the implied

+ #   warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR

+ #   PURPOSE. See the GNU General Public License for more details.

+ #

+ #   You should have received a copy of the GNU General Public

+ #   License along with this program; if not, write to the Free

+ #   Software Foundation, Inc., 51 Franklin Street, Fifth Floor,

+ #   Boston, MA 02110-1301, USA.

+ #

+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

+ 

+ export TEST=/tools/gdb/Sanity/debug-toolset-binary

+ export TESTVERSION=1.0

+ 

+ BUILT_FILES=

+ 

+ FILES=$(METADATA) runtest.sh Makefile PURPOSE cmds

+ 

+ .PHONY: all install download clean

+ 

+ run: $(FILES) build

+ 	./runtest.sh

+ 

+ build: $(BUILT_FILES)

+ 	test -x runtest.sh || chmod a+x runtest.sh

+ 

+ clean:

+ 	rm -f *~ $(BUILT_FILES)

+ 

+ 

+ include /usr/share/rhts/lib/rhts-make.include

+ 

+ $(METADATA): Makefile

+ 	@echo "Owner:           Marek Polacek <mpolacek@redhat.com>" > $(METADATA)

+ 	@echo "Name:            $(TEST)" >> $(METADATA)

+ 	@echo "TestVersion:     $(TESTVERSION)" >> $(METADATA)

+ 	@echo "Path:            $(TEST_DIR)" >> $(METADATA)

+ 	@echo "Description:     We debug a binary that is delivered via the Toolset." >> $(METADATA)

+ 	@echo "Type:            Sanity" >> $(METADATA)

+ 	@echo "TestTime:        5m" >> $(METADATA)

+ 	@echo "RunFor:          gdb" >> $(METADATA)

+ 	@echo "Requires:        gdb coreutils binutils binutils-debuginfo glibc glibc-debuginfo gcc gcc-debuginfo zlib zlib-debuginfo" >> $(METADATA)

+ 	@echo "Priority:        Normal" >> $(METADATA)

+ 	@echo "License:         GPLv2" >> $(METADATA)

+ 	@echo "Confidential:    no" >> $(METADATA)

+ 	@echo "Destructive:     no" >> $(METADATA)

+ 	@echo "Releases:        -RHEL4 -RHELClient5 -RHELServer5" >> $(METADATA)

+ 

+ 	rhts-lint $(METADATA)

tests/debug-toolset-binary/PURPOSE
file added
+3

@@ -0,0 +1,3 @@

+ PURPOSE of /tools/gdb/Sanity/debug-toolset-binary

+ Description: We debug a binary that is delivered via the Toolset.

+ Author: Marek Polacek <polacek@redhat.com>

tests/debug-toolset-binary/cmds
file added
+18

@@ -0,0 +1,18 @@

+ b main

+ r

+ n 10

+ si 30

+ up

+ n 15

+ watch myname

+ p/s myname

+ si 10

+ up

+ down

+ bt

+ frame 1

+ b xexit

+ c

+ c

+ c

+ q

tests/debug-toolset-binary/runtest.sh
file added
+71

@@ -0,0 +1,71 @@

+ #!/bin/bash

+ # vim: dict=/usr/share/beakerlib/dictionary.vim cpt=.,w,b,u,t,i,k

+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

+ #

+ #   runtest.sh of /tools/gdb/Sanity/debug-toolset-binary

+ #   Description: We debug a binary that is delivered via the Toolset.

+ #   Author: Marek Polacek <polacek@redhat.com>

+ #

+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

+ #

+ #   Copyright (c) 2012 Red Hat, Inc. All rights reserved.

+ #

+ #   This copyrighted material is made available to anyone wishing

+ #   to use, modify, copy, or redistribute it subject to the terms

+ #   and conditions of the GNU General Public License version 2.

+ #

+ #   This program is distributed in the hope that it will be

+ #   useful, but WITHOUT ANY WARRANTY; without even the implied

+ #   warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR

+ #   PURPOSE. See the GNU General Public License for more details.

+ #

+ #   You should have received a copy of the GNU General Public

+ #   License along with this program; if not, write to the Free

+ #   Software Foundation, Inc., 51 Franklin Street, Fifth Floor,

+ #   Boston, MA 02110-1301, USA.

+ #

+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

+ 

+ # We create an empty file `empty.S'.  The we run the as(1) to produce

+ # the `a.out' file and at the same time, we debug the as(1).

+ # Note that as(1) should come from the Developer Toolset (TM),

+ # but in fact, it doesn't have to.  It should not fail either way.

+ 

+ # Include Beaker environment

+ . /usr/bin/rhts-environment.sh || exit 1

+ . /usr/lib/beakerlib/beakerlib.sh || exit 1

+ 

+ CMD="gdb"

+ BIN="$(which $CMD)"

+ export PACKAGE="${PACKAGE:-$(rpm -qf --qf='%{name}\n' $BIN)}"

+ 

+ PACKAGES="coreutils binutils binutils-debuginfo glibc glibc-debuginfo gcc gcc-debuginfo zlib zlib-debuginfo"

+ 

+ rlJournalStart

+   rlPhaseStartSetup

+     rlShowRunningKernel

+     rlAssertRpm $PACKAGE

+     for pkg in $PACKAGES; do

+        rlAssertRpm $pkg

+     done

+     rlRun "TmpDir=\$(mktemp -d)"

+     rlRun "cp cmds $TmpDir"

+     rlRun "pushd $TmpDir"

+     rlRun ": >empty.s"

+   rlPhaseEnd

+ 

+   rlPhaseStartTest

+     rlRun "gdb -nx -q -batch -x cmds --args as empty.s"

+     rlRun "nm >symbs"

+     if [ -s symbs ]; then

+       rlFail "nm should't output any symbols"

+       rlRun "cat symbs"

+     fi

+   rlPhaseEnd

+ 

+   rlPhaseStartCleanup

+     rlRun "popd"

+     rlRun "rm -r $TmpDir"

+   rlPhaseEnd

+ rlJournalPrintText

+ rlJournalEnd

tests/inventory
file added
+3

@@ -0,0 +1,3 @@

+ #!/bin/bash

+ export TEST_DOCKER_EXTRA_ARGS="--privileged"

+ exec merge-standard-inventory "$@"

tests/tests.yml
file added
+24

@@ -0,0 +1,24 @@

+ ---

+ # This first play always runs on the local staging system

+ - hosts: localhost

+   roles:

+   - role: standard-test-beakerlib

+     tags:

+     - classic

+     - container

+     tests:

+     - debug-toolset-binary

+     - debug-system-binary

+     required_packages:

+     - gdb

+     - coreutils

+     - binutils

+     - binutils-debuginfo

+     - glibc

+     - glibc-debuginfo

+     - gcc

+     - gcc-debuginfo

+     - zlib

+     - zlib-debuginfo

+     - coreutils-debuginfo

+     - which

no initial comment

Justification

Adds tests according to the CI wiki specifically the standard test interface in the spec.

The playbook includes Tier1 level test cases that have been tested in the following contexts and is passing reliably: Classic and Container. Test logs are stored in the artifacts directory.

The following steps are used to execute the tests using the standard test interface:

Test environment

Make sure you have installed packages from the spec

    # rpm -q ansible python2-dnf libselinux-python standard-test-roles \
    ansible-2.3.2.0-1.fc26.noarch \
    python2-dnf-2.6.3-11.fc26.noarch \
    libselinux-python-2.6-7.fc26.x86_64 \
    standard-test-roles-2.4-1.fc26.noarch

Tests should be started under root user.

Run tests for Classic

       sudo -s
    # export ANSIBLE_INVENTORY=$(test -e inventory && echo inventory || echo /usr/share/ansible/inventory)
        # ansible-playbook --tags=classic tests.yml

Run tests for Container

        sudo -s
    # export ANSIBLE_INVENTORY=$(test -e inventory && echo inventory || echo /usr/share/ansible/inventory)
    # export TEST_SUBJECTS=docker:docker.io/library/fedora:26
        # ansible-playbook --tags=container tests.yml

Snip of the example test run for Classic tests:

PLAY RECAP ********************************************************************************************************************************************************************************************************
localhost : ok=20   changed=14   unreachable=0    failed=0   

PASS debug-toolset-binary
PASS debug-system-binary

Snip of the example test run for Container tests:

PLAY RECAP ********************************************************************************************************************************************************************************************************
3d520de4a0e87c62529d2a320e2e7b4bc1a02ee93f6b57700a8297662ccf020f : ok=20   changed=14   unreachable=0    failed=0   

PASS debug-toolset-binary
PASS debug-system-binary

Notes

Tests will be enabled in CI, yet gating is currently disabled, so nothing will change. Tests will run on each dist-git commit, they are not triggered on koji builds and if you are using FMN, it should notify you of failures normally.

The RH QE maintainer contact in case you have questions: bgoncalv @redhat.com
The idea is that these tests become yours just as you're maintaining the package, there will of course be people around if you have questions or troubles.

rebased onto ccb743e

a year ago

This has been merged manually, but Pagure cannot detect it. Therefore I am closing the PR by hand.

Pull-Request has been closed by sergiodj

a year ago