llvm Tests

This repository contains tests for llvm.

Usage

Tests are executed using TMT (Test Management Tool). The repository includes a TMT plan that runs all the tests by default. All the tests support the following context dimensions:

• distro
• arch
• collection (only applicable to RHEL 7)

To run the tests:

# Run in the default virtual environment
tmt -c distro=<distro> -c arch=<arch> run -avv

# Run on a scheduled system
tmt -c distro=<distro> -c arch=<arch> run -avv provision -h connect -g <ip_or_hostname>

# Run specific tests
tmt -c distro=<distro> -c arch=<arch> run -avv provision -h connect -g <ip_or_hostname> test -n <regex>

Adding New Tests

Please consider the following when adding new tests:

• Test scripts must be named test.sh. If a different test is needed, you need to overwrite the test: key in the test main.fmf.
• Tests must be placed under tests/<test_name>. Additionally they can be placed into subdirectories if grouping make sense. E.g. tests/regression/<test_name>.
• This repository has a common configuration under tests/main.fmf, inherited by all tests. When adding new ones, be careful not to overwrite existing keys from the inherited tests/main.fmf.
• When setting FMF metadata keys, do merge values instead overwriting them. E.g. use require+: instead require:. For advanced use and more examples check FMF documentation.

After creating the new test, consider running tmt <context> tests show <test-name> to review that the resulting metadata such as requires, enable or test keys are as expected. For example:

$ tmt -c distro=fedora-38 -c arch=aarch64 tests show binutils-plugin-ar
/tests/binutils-plugin-ar
                 summary Make sure the the LLVM plugin for binutils' ar works
             description Test that ar picks the correct plugin when archiving
                         LLVM bitcode.
                 contact Jesus Checa Hidalgo <jcheca@redhat.com>
               component 'llvm-toolset' and 'llvm'
                    test ./test.sh
                    path /tests/binutils-plugin-ar
               framework shell
                  manual false
                     tty false
                 require binutils
                         llvm
                         llvm-libs
                         llvm-devel
                         clang
                duration 1m
                 enabled true
                  result respect
                    tier 1

Templates

Use the following templates when creating new tests.

FMF template. Uncomment needed fields, remove the rest.

summary: Descriptive summary for the test
# Maximum duration for the test
duration: 2m
# Short tests should be tier <= 1 so they're used for gating.
tier: 1
# Add an additional contact(s) if applicable
# contact+:
#   - Somebody <somebody@redhat.com>
# Additional requires
# require+:
#   - rpm-build
# Adjustments based on context dimensions
# adjust+:
#   - when: collection is defined
#     enabled: false
#     because: Test not supported in collections
# link+:
#   - verifies: https://issues.redhat.com/browse/...
# tag+:
#   - sometag