PR#4: Adding cassandane test - tests/cyrus-imapd

tests / cyrus-imapd

#4 Adding cassandane test

Merged a year ago by fhrdina. Opened a year ago by fhrdina.

tests/ fhrdina/cyrus-imapd main into main

Adding cassandane test

Frantisek Hrdina • a year ago

a569c28

Sanity/cassandane/cassandane.ini

file added

+56

		`@@ -0,0 +1,56 @@`
		`+ # A basic cassandane.ini file for running cassandane as part of the Fedora`
		`+ # package build process.`
		`+`
		`+ # The idea here is to run tests on the just-compiled version of cyrus-imapd.`
		`+ # However, many of the build locations are just random temporary directories, and`
		`+ # so this requires some finesse.`
		`+`
		`+ [cassandane]`
		`+ rootdir = CASSDIR/work`
		`+ pwcheck = alwaystrue # This is enabled in Fedora builds`
		`+ cleanup = no`
		`+ maxworkers = 1`
		`+ base_port = 19100`
		`+`
		`+ #[valgrind]`
		`+ #enabled = no`
		`+`
		`+ # The installed copy`
		`+ [cyrus default]`
		`+ prefix = /usr`
		`+ destdir = BUILDROOT`
		`+ quota = cyr_quota`
		`+`
		`+ # Replication testing disabled`
		`+ # [cyrus replica]`
		`+ # [cyrus murder]`
		`+`
		`+ # Don't enable any of the gdb options but leave them here in case someone ever`
		`+ # needs to do so`
		`+ #[gdb]`
		`+ # imapd = yes`
		`+ # sync_server = yes`
		`+ # lntpd = yes`
		`+ # timsieved = yes`
		`+ # backupd = yes`
		`+`
		`+ [config]`
		`+ altnamespace = no`
		`+ unixhierarchysep = no`
		`+ client_timeout = 60`
		`+`
		`+ #[caldavtalk]`
		`+ #basedir = CASSDIR/cassandane/testdata`
		`+`
		`+ [imaptest]`
		`+ # Cassandane wants this to not be installed. Don't know why. To use it we`
		`+ # have to make a directory and link things into it.`
		`+ basedir = imaptest`
		`+`
		`+ # [jmaptester]`
		`+ # basedir = JMAP-Tester`
		`+ # The JMAP modules end up needing JSON-Typist (which I could bundle) and CryptX (which is a bit too much to bundle)`
		`+`
		`+ # [caldavtester]`
		`+ # XXX Would need to include the source in the cyrus package just as cassandane is, and get it built before running tests`
		`+ # basedir = ...`

Sanity/cassandane/main.fmf

file added

+25

		`@@ -0,0 +1,25 @@`
		`+ summary: Cassandane is a Perl-based integration test suite for Cyrus.`
		`+ description: ''`
		`+ contact: fhrdina@redhat.com`
		`+ component:`
		`+ - cyrus-imapd`
		`+ test: ./runtest.sh`
		`+ framework: beakerlib`
		`+ recommend:`
		`+ - cyrus-imapd`
		`+ - cyrus-imapd-virusscan`
		`+ - imaptest`
		`+ - rpm-build`
		`+ - patch`
		`+ duration: 1h`
		`+ enabled: true`
		`+ tag:`
		`+ - Tier1`
		`+ tier: '1'`
		`+ adjust:`
		`+ - enabled: false`
		`+ when: distro < rhel-7`
		`+ continue: false`
		`+ extra-nitrate: TC#0613627`
		`+ extra-summary: /CoreOS/cyrus-imapd/Sanity/cassandane`
		`+ extra-task: /CoreOS/cyrus-imapd/Sanity/cassandane`

Sanity/cassandane/patch-cassandane-fix-annotator

file added

+14

		`@@ -0,0 +1,14 @@`
		`+ diff --git a/utils/annotator.pl b/utils/annotator.pl`
		`+ index 265c73f..8af3d58 100755`
		`+ --- a/utils/annotator.pl`
		`+ +++ b/utils/annotator.pl`
		`+ @@ -140,6 +140,8 @@ GetOptions(`
		`+ xlog "annotator $$ starting";`
		`+ Cassandane::AnnotatorDaemon->run(`
		`+ pid_file => $pidfile,`
		`+ - port => $port`
		`+ + port => $port,`
		`+ + user => (getpwuid($<))[0],`
		`+ + group => (getgrgid($())[0],`
		`+ );`
		`+ xlog "annotator $$ exiting";`

Sanity/cassandane/patch-cassandane-no-syslog

file added

+41

		`@@ -0,0 +1,41 @@`
		`+ diff --git a/Cassandane/Instance.pm b/Cassandane/Instance.pm`
		`+ index fb36b8f..78e7100 100644`
		`+ --- a/Cassandane/Instance.pm`
		`+ +++ b/Cassandane/Instance.pm`
		`+ @@ -2146,12 +2146,8 @@ sub setup_syslog_replacement`
		`+ {`
		`+ my ($self) = @_;`
		`+`
		`+ - if (not(-e 'utils/syslog.so') \|\| not(-e 'utils/syslog_probe')) {`
		`+ - xlog "utils/syslog.so not found (do you need to run 'make'?)";`
		`+ - xlog "tests will not examine syslog output";`
		`+ - $self->{have_syslog_replacement} = 0;`
		`+ - return;`
		`+ - }`
		`+ + $self->{have_syslog_replacement} = 0;`
		`+ + return;`
		`+`
		`+ $self->{syslog_fname} = "$self->{basedir}/conf/log/syslog";`
		`+ $self->{have_syslog_replacement} = 1;`
		`+ diff --git a/Cassandane/Util/Log.pm b/Cassandane/Util/Log.pm`
		`+ index 7c17f0c..9eb07ad 100644`
		`+ --- a/Cassandane/Util/Log.pm`
		`+ +++ b/Cassandane/Util/Log.pm`
		`+ @@ -51,9 +51,6 @@ our @EXPORT = qw(`
		`+`
		`+ my $verbose = 0;`
		`+`
		`+ -openlog('cassandane', '', LOG_LOCAL6)`
		`+ - or die "Cannot openlog";`
		`+ -`
		`+ sub xlog`
		`+ {`
		`+ my $id;`
		`+ @@ -74,7 +71,6 @@ sub xlog`
		`+ $msg .= "($id) " if $id;`
		`+ $msg .= join(' ', @_);`
		`+ print STDERR "$msg\n";`
		`+ - syslog(LOG_ERR, "$msg");`
		`+ }`
		`+`
		`+ sub set_verbose`

Sanity/cassandane/patch-cassandane-xapian-delve-path

file added

+15

		`@@ -0,0 +1,15 @@`
		`+ diff --git a/Cassandane/Instance.pm b/Cassandane/Instance.pm`
		`+ index 78e7100..edf6f5d 100644`
		`+ --- a/Cassandane/Instance.pm`
		`+ +++ b/Cassandane/Instance.pm`
		`+ @@ -492,9 +492,7 @@ sub _find_binary`
		`+ my $base = $self->{cyrus_destdir} . $self->{cyrus_prefix};`
		`+`
		`+ if ($name eq 'delve') {`
		+ - my $lib = `ldd $base/libexec/imapd` \|\| die "can't ldd imapd";
		`+ - $lib =~ m{(/\S+)/lib/libxapian-([0-9.]+)\.so};`
		`+ - return "$1/bin/xapian-delve-$2";`
		`+ + return "/bin/xapian-delve";`
		`+ }`
		`+`
		`+ foreach (qw( bin sbin libexec libexec/cyrus-imapd lib cyrus/bin ))`

Sanity/cassandane/patch-cyrus-testsuite-timeout

file added

+13

		`@@ -0,0 +1,13 @@`
		`+ diff --git a/cunit/unit.c b/cunit/unit.c`
		`+ index 46dc358..ca37f22 100644`
		`+ --- a/cunit/unit.c`
		`+ +++ b/cunit/unit.c`
		`+ @@ -97,7 +97,7 @@ EXPORTED void fatal(const char *s, int code)`
		`+ }`
		`+`
		`+ /* Each test gets a maximum of 20 seconds. */`
		`+ -#define TEST_TIMEOUT_MS (20*1000)`
		`+ +#define TEST_TIMEOUT_MS (300*1000)`
		`+`
		`+ static jmp_buf jbuf;`
		`+ static const char *code;`

Sanity/cassandane/runtest.sh

file added

+197

		`@@ -0,0 +1,197 @@`
		`+ #!/bin/bash`
		`+ # vim: dict+=/usr/share/beakerlib/dictionary.vim cpt=.,w,b,u,t,i,k`
		`+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`
		`+ #`
		`+ # runtest.sh of /CoreOS/cyrus-imapd/Sanity/cassandane`
		`+ # Description: Cassandane is a Perl-based integration test suite for Cyrus.`
		`+ # Author: Martin Osvald <mosvald@redhat.com>`
		`+ #`
		`+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`
		`+ #`
		`+ # Copyright (c) 2022 Red Hat, Inc.`
		`+ #`
		`+ # This program is free software: you can redistribute it and/or`
		`+ # modify it under the terms of the GNU General Public License as`
		`+ # published by the Free Software Foundation, either version 2 of`
		`+ # the License, or (at your option) any later version.`
		`+ #`
		`+ # This program is distributed in the hope that it will be`
		`+ # useful, but WITHOUT ANY WARRANTY; without even the implied`
		`+ # warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR`
		`+ # PURPOSE. See the GNU General Public License for more details.`
		`+ #`
		`+ # You should have received a copy of the GNU General Public License`
		`+ # along with this program. If not, see http://www.gnu.org/licenses/.`
		`+ #`
		`+ # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`
		`+`
		`+ # Include Beaker environment`
		`+ . /usr/share/beakerlib/beakerlib.sh \|\| exit 1`
		`+`
		`+ PACKAGE="cyrus-imapd"`
		`+`
		`+ export CYRUS_USER=cyrus`
		`+ CYRUS_SRC_PATH=""`
		`+`
		`+ #`
		`+ # Variables that can be exported through 1minutetip -p parameter, i.e.: -p "VAR1=foo" -p "VAR2=bar" -p "VAR3=..."`
		`+ #`
		`+ # Download cyrus-imapd (and acompanying cassandane tests) from upstream git repo (if set to 1):`
		`+ #`
		`+ if [ -z "$UPSTREAM" ]; then`
		`+ UPSTREAM=0`
		`+ fi`
		`+ #`
		`+ # You can set location of *.rpm packages on koji by hand:`
		`+ #UPDATERPMS=https://kojipkgs.fedoraproject.org//work/tasks/8286/87158286/`
		`+ #`
		`+ # You can set location of .src.rpm on koji by hand (.src.rpm's koji task id is different from the rest of .rpm packages):`
		`+ #UPDATESRPM=https://kojipkgs.fedoraproject.org//work/tasks/8281/87158281/`
		`+ #`
		`+ # Don't keep the downloaded files between consecutive test runs (usefull when debugging some problem to conserve time and bandwidth):`
		`+ if [ -z "$KEEP_TMP_FILES" ]; then`
		`+ KEEP_TMP_FILES=0`
		`+ fi`
		`+ #`
		`+ # Example usage:`
		`+ #`
		`+ # Extract cassandane tests from provided .src.rpm and test binary packages from scratch build on it:`
		`+ #`
		`+ # $ 1minutetip -p "UPSTREAM=0" -p "KEEP_TMP_FILES=1" \`
		`+ # -p "UPDATESRPM=https://kojipkgs.fedoraproject.org//work/tasks/8281/87158281/" \`
		`+ # -p "UPDATERPMS=https://kojipkgs.fedoraproject.org//work/tasks/8286/87158286/" \`
		`+ # 1MT-Fedora-Rawhide 2>&1 \| tee test.log`
		`+ #`
		`+`
		`+ rlJournalStart`
		`+ rlPhaseStartSetup`
		`+ rlAssertRpm $PACKAGE`
		+ TESTDIR=`pwd`
		`+ if [ $KEEP_TMP_FILES -eq 1 ] && [ -d /cymap* ]; then`
		+ TmpDir=`echo /cymap*`
		`+ else`
		`+ rlRun "TmpDir=\$(mktemp -d -p / cymapXXX)" 0 "Creating tmp directory"`
		`+ fi`
		`+ rlRun "pushd $TmpDir"`
		`+ if [ ! -f cassandane-testdata-ca669d4b.tar.gz ]; then`
		`+ rlRun "wget https://github.com/brong/Net-CalDAVTalk/archive/ca669d4b76c71cbeb4fa840e263e2c031e19ea88/cassandane-testdata-ca669d4b.tar.gz" 0 "Fetching testdata"`
		`+ fi`
		`+ if [ $UPSTREAM -eq 0 ]; then`
		`+ if [ ! -f $TmpDir/rpms/cyrus-imapd*.src.rpm ]; then`
		`+ if [ -z "$UPDATESRPM" ]; then`
		`+ rlRun "mkdir -p $TmpDir/rpms"`
		`+ rlRun "pushd $TmpDir/rpms"`
		`+ rlRun "rlFetchSrcForInstalled --quiet $PACKAGE" 0 "Downloading cyrus-imapd srpm"`
		`+ rlRun "popd"`
		`+ else`
		`+ rlRun "wget -r -l1 --no-parent --directory-prefix $TmpDir/rpms --no-directories -A '.src.rpm' \"$UPDATESRPM/\"" 0 "Downloading provided cyrus-imapd srpm"`
		`+ fi`
		`+ # Cassandane uses unix sockets to test the behaviour, but PATH to that sockets`
		`+ # is maximally 108bytes long (man unix(7)), thus restraining the length by`
		`+ # installing the source code close to '/' to prevent tests failure.`
		`+ rlRun "rpm -i --define \"_topdir $TmpDir\" $TmpDir/rpms/cyrus-imapd-*.src.rpm" 0 "Installing srpm"`
		`+ rlRun "rpmbuild -bp --define \"_topdir $TmpDir\" $TmpDir/SPECS/cyrus-imapd.spec" 0 "Preprocessing source code"`
		`+ # Dnf doesn't support specifying rpm macros:`
		`+ # https://bugzilla.redhat.com/show_bug.cgi?id=1673333`
		`+ # This activates a build conditional for cassandane's BuildRequires`
		`+ rlRun "echo '%_with_cassandane 1' > ~/.rpmmacros"`
		`+ rlRun "dnf -q builddep \"$TmpDir/SPECS/cyrus-imapd.spec\" -y" 0 "Installing build dependencies"`
		`+ fi`
		+ CYRUS_SRC_PATH=`echo /$TmpDir/BUILD/cyrus-imapd-*/`
		`+ else`
		`+ if [ ! -f $TmpDir/rpms/cyrus-imapd-3.4.3.tar.gz ]; then`
		`+ rlRun "wget --directory-prefix $TmpDir/rpms https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.4.3/cyrus-imapd-3.4.3.tar.gz" 0 "Downoading upstream cyrus-imapd source code"`
		`+ # Cassandane uses unix sockets to test the behaviour, but PATH to that sockets`
		`+ # is maximally 108bytes long (man unix(7)), thus restraining the length by`
		`+ # installing the source code close to '/' to prevent tests failure.`
		`+ rlRun "tar -xf $TmpDir/rpms/cyrus-imapd-3.4.3.tar.gz -C $TmpDir" 0 "Decompressing upstream cyrus-imapd source code"`
		`+ rlRun "dnf -q builddep https://src.fedoraproject.org/rpms/cyrus-imapd/raw/rawhide/f/cyrus-imapd.spec -y" 0 "Installing build dependencies"`
		`+ fi`
		+ CYRUS_SRC_PATH=`echo $TmpDir/cyrus-imapd-*/`
		`+ fi`
		`+ if [ ! -f "$CYRUS_SRC_PATH/initialized" ]; then`
		`+ rlRun "pushd \"/$CYRUS_SRC_PATH/cassandane/\""`
		`+ rlRun "mkdir -p work" 0 "Creating cassandane working directory"`
		`+ rlRun "tar -xf \"$TmpDir/cassandane-testdata-ca669d4b.tar.gz\"" 0 "Decompressing testdata"`
		`+ # --forward is for when .src.rpm/UPSTREAM=0 gets used and it still calls Cassandane by default (it applies these patches)`
		`+ patch -p1 --forward < "$TESTDIR/patch-cassandane-no-syslog"`
		`+ patch -p1 --forward < "$TESTDIR/patch-cassandane-fix-annotator"`
		`+ patch -p1 --forward < "$TESTDIR/patch-cassandane-xapian-delve-path"`
		`+ rlRun "cp \"$TESTDIR/cassandane.ini\" ." 0 "Installing cassandane config template"`
		`+ rlRun "sed -i -e \"s#CASSDIR#$(pwd)#\" -e \"s#BUILDROOT#/#\" cassandane.ini" 0 "Patching cassandane config template"`
		`+ rlRun "mkdir -p imaptest/src" 0 "Creating imaptest/src"`
		`+ rlRun "ln -s /usr/bin/imaptest imaptest/src" 0 "Creating necessary symbolic links 1"`
		`+ rlRun "ln -s /usr/share/imaptest/tests imaptest/src" 0 "Creating necessary symbolic links 2"`
		`+ #rlRun "export NOCYRUS=1"`
		`+ rlRun "make" 0 "Building cassandane"`
		`+ if [ ! -z "$UPDATERPMS" ]; then`
		`+ rlRun "wget -r -l1 --no-parent --directory-prefix $TmpDir/rpms --no-directories -A \".rpm\" \"$UPDATERPMS/\"" 0 "Downloading specified cyrus-imapd version"`
		`+ rlRun "dnf -q update $TmpDir/rpms/*.rpm -y" 0 "Installing specified cyrus-imapd version"`
		`+ fi`
		`+ rlRun "touch \"$CYRUS_SRC_PATH/initialized\"" 0 "Cassandane initialization finished"`
		`+ # The whole $TmpDir needs to be owned by cyrus:mail otherwise Perl's abs_path() fails to locate cassandane.ini`
		`+ rlRun "chown -R cyrus:mail \"$TmpDir\"" 0 "Fixing \"$TmpDir\" permissions"`
		`+ else`
		`+ rlRun "pushd \"/$CYRUS_SRC_PATH/cassandane/\""`
		`+ fi`
		`+ rlPhaseEnd`
		`+`
		`+ rlPhaseStartTest`
		`+ # Construct the set of excluded tests to pass to Cassandane`
		`+ # ---------------------------------------------------------`
		`+ exclude=()`
		`+ tests=(`
		`+ # This exclusion list was verified on 2021-08-11.`
		`+`
		`+ # This tests coredumping and won't work on a machine where systemd`
		`+ # intercepts coredumps, which includes our builders.`
		`+ Cassandane::Test::Core`
		`+`
		`+ # Can't currently be run at build time because of compiled-in paths. See`
		`+ # https://github.com/cyrusimap/cyrus-imapd/issues/2386`
		`+ Admin.imap_admins`
		`+`
		`+ Rename.intermediate_cleanup`
		`+`
		`+ # TODO check this one, fails with:`
		`+ # expected '16', got undef at Cassandane/Cyrus/List.pm line 2188.`
		`+ # Cassandane::Cyrus::List::test_no_tombstones(Cassandane::Cyrus::List=HASH(0x55f00fa1e408)) called at /usr/share/perl5/vendor_perl/Test/Unit/TestCase.pm line 75`
		`+ Cyrus::List.no_tombstones`
		`+ # TODO run tests outside the build process:`
		`+ # https://bugzilla.redhat.com/show_bug.cgi?id=1887674`
		`+ # The below tests try to search in syslog file which we don't allow due to mock,`
		`+ # happenes for:`
		`+ # $ grep -R "assert.@lines" cyrus-imapd-/cassandane/`
		`+ Reconstruct.reconstruct_snoozed`
		`+ SearchSquat.simple`
		`+ SearchSquat.skip_unmodified`
		`+ )`
		`+ for i in ${tests[@]}; do exclude+=("!$i"); done`
		`+`
		`+ # If you interrupt the tests (by ictrl+c) while they are running (e.g. you notice some failing tests)`
		`+ # there might remain some processes that could influence the next re-run, so kill them first:`
		`+ killall master imapd fakesaslauthd &> /dev/null`
		`+`
		`+ # Set minimal number of tests that can run concurrently:`
		+ J_OPTION="`rpm --eval %{_smp_mflags}`"
		`+ if [ ${J_OPTION:2} -lt 4 ]; then`
		`+ J_OPTION=-j4`
		`+ fi`
		`+`
		`+ # Add -vvv for too much output`
		`+ time sudo -u cyrus ./testrunner.pl "$J_OPTION" -v -f pretty ${exclude[@]} 2>&1 \| tee "$TmpDir/cassandane.log"`
		`+`
		`+ if [ -s "/$CYRUS_SRC_PATH/cassandane/work/failed" ]; then`
		`+ rlFail "FAIL: Some tests failed"`
		`+ rlRun "cat \"/$CYRUS_SRC_PATH/cassandane/work/failed\""`
		`+ fi`
		`+ rlPhaseEnd`
		`+`
		`+ rlPhaseStartCleanup`
		`+ if [ $KEEP_TMP_FILES -eq 0 ]; then`
		`+ rlRun "popd; popd"`
		`+ rlRun "rm -rf $TmpDir ~/.rpmmacros" 0 "Removing tmp directory"`
		`+ fi`
		`+ rlPhaseEnd`
		`+ rlJournalPrintText`
		`+ rlJournalEnd`

no initial comment

1 new commit added

Updating cassandane test

a year ago

rebased onto 43811a9

a year ago

mosvald commented a year ago

The failure is ok (very likely caused by a slow machine), it simply fails from time to time on some of the tests "randomly", this is the reason we separated it.

This time only one test failed:

Cyrus::Master.sighup_recycling                                          [ERROR ]
...
There was 1 error:
1) Master.sighup_recycling
 Perl exception: Timed out waiting for lemmings to reach the expected census
 at Cassandane/Util/Wait.pm line 66.
Cassandane::Util::Wait::timed_wait(CODE(0x55680d5d0c70), "description", "lemmings to reach the expected census") called at Cassandane/Cyrus/Master.pm line 224
Cassandane::Cyrus::Master::lemming_wait(Cassandane::Cyrus::Master=HASH(0x55680c0900c0), "foo", HASH(0x55680d5cd0c8)) called at Cassandane/Cyrus/Master.pm line 1056
Cassandane::Cyrus::Master::test_sighup_recycling(Cassandane::Cyrus::Master=HASH(0x55680c0900c0)) called at /usr/share/perl5/vendor_perl/Test/Unit/TestCase.pm line 75
[...framework calls elided...]

It passes under 1minutetip. I would waive it if it was in the CI dashboard.

mosvald commented a year ago

Also, you can remove the spec file. I removed it in the cgit test with my last commit.

1 new commit added