#5 Add %pyproject_buildrequires
Merged 2 months ago by churchyard. Opened 2 months ago by churchyard.
rpms/ churchyard/pyproject-rpm-macros dynamic_brs  into  master

file modified
+17 -7

@@ -5,8 +5,6 @@ 

  

  These macros are useful for packaging Python projects that use the [PEP 517] `pyproject.toml` file, which specifies the package's build dependencies (including the build system, such as setuptools, flit or poetry).

  

- [PEP 517]: https://www.python.org/dev/peps/pep-0517/

- 

  

  Usage

  -----

@@ -17,12 +15,20 @@ 

  

  This will bring in python3-devel, so you don't need to require python3-devel explicitly.

  

- Then, build a wheel in %build:

+ In order to get automatic build dependencies on Fedora 31+, run `%pyproject_buildrequires` in the `%generate_buildrequires` section:

+ 

+     %generate_buildrequires

+     %pyproject_buildrequires

+ 

+ Only build dependencies according to [PEP 517] and [PEP 518] will be added.

+ All other build dependencies (such as non-Python libraries or test dependencies) still need to be specified manually.

+ 

+ Then, build a wheel in `%build` with `%pyproject_wheel`:

  

      %build

      %pyproject_wheel

  

- And install the wheel in %install:

+ And install the wheel in `%install` with `%pyproject_install`:

  

      %install

      %pyproject_install

@@ -33,8 +39,12 @@ 

  

  `%pyproject_install` currently installs all wheels in `$PWD`. We are working on a more explicit solution.

  

- This macro changes shebang lines of every Python script in `%{buildroot}%{_bindir}` to `#! %{__python3} %{py3_shbang_opt}` (`#! /usr/bin/python -s`).

- We plan to preserve exisiting Python flags in shebangs, but the work is not yet finished.

+ This macro changes shebang lines of every Python script in `%{buildroot}%{_bindir}` to `#! %{__python3} %{py3_shbang_opt}` (`#! /usr/bin/python3 -s`).

+ We plan to preserve existing Python flags in shebangs, but the work is not yet finished.

+ 

+ The PEPs don't (yet) define a way to specify test dependencies and test runners.

+ That means you still need to handle test dependencies and `%check` on your own.

  

- Currently, the macros do not automatically generate BuildRequires. We are working on that as well.

  

+ [PEP 517]: https://www.python.org/dev/peps/pep-0517/

+ [PEP 518]: https://www.python.org/dev/peps/pep-0518/

file modified
+17 -7

@@ -1,18 +1,28 @@ 

  %pyproject_wheel() %{expand:\\\

- 	CFLAGS="${CFLAGS:-${RPM_OPT_FLAGS}}" LDFLAGS="${LDFLAGS:-${RPM_LD_FLAGS}}" \\\

- 	%{__python3} -m pip wheel --no-deps --use-pep517 --no-build-isolation --disable-pip-version-check --progress-bar off --verbose .

+ CFLAGS="${CFLAGS:-${RPM_OPT_FLAGS}}" LDFLAGS="${LDFLAGS:-${RPM_LD_FLAGS}}" \\\

+ %{__python3} -m pip wheel --no-deps --use-pep517 --no-build-isolation --disable-pip-version-check --progress-bar off --verbose .

  }

  

  

  %pyproject_install() %{expand:\\\

  %{__python3} -m pip install   --root %{buildroot}  --strip-file-prefix %{buildroot} --no-deps  --disable-pip-version-check --progress-bar off --verbose --ignore-installed --no-warn-script-location ./*.whl

- if [ -e %{buildroot}%{_bindir} ]; then

+ if [ -d %{buildroot}%{_bindir} ]; then

    pathfix.py -pni "%{__python3} %{py3_shbang_opts}" %{buildroot}%{_bindir}/*

  fi

- if [ -e %{buildroot}%{python3_sitelib}]; then

-   sed -i -e 's/pip/rpm/g' %{buildroot}%{python3_sitelib}/*.dist-info/INSTALLER

+ if [ -d %{buildroot}%{python3_sitelib} ]; then

+   sed -i 's/pip/rpm/' %{buildroot}%{python3_sitelib}/*.dist-info/INSTALLER

  fi

- if [ -e %{buildroot}{python3_sitearch}]; then

-   sed -i -e 's/pip/rpm/g' %{buildroot}%{python3_sitearch}/*.dist-info/INSTALLER

+ if [ -d %{buildroot}%{python3_sitearch} ]; then

+   sed -i 's/pip/rpm/' %{buildroot}%{python3_sitearch}/*.dist-info/INSTALLER

+ fi

+ }

+ 

+ %pyproject_buildrequires() %{expand:\\\

+ echo 'python3-devel'

+ echo 'python3dist(packaging)'

+ echo 'python3dist(pip) >= 19'

+ echo 'python3dist(pytoml)'

+ if [ -f %{__python3} ]; then

+   %{__python3} -I %{_rpmconfigdir}/redhat/pyproject_buildrequires.py

  fi

  }

file modified
+25 -8

@@ -1,24 +1,35 @@ 

  Name:           pyproject-rpm-macros

- Version:        0

- Release:        2%{?dist}

+ Summary:        RPM macros for PEP 517 Python packages

  License:        MIT

+ 

+ # Keep the version at zero and increment only release

+ Version:        0

+ Release:        3%{?dist}

+ 

  Source0:        macros.pyproject

- Source1:        README.md

- Source2:        LICENSE

+ Source1:        pyproject_buildrequires.py

+ 

+ Source8:        README.md

+ Source9:        LICENSE

+ 

  URL:            https://src.fedoraproject.org/rpms/pyproject-rpm-macros

+ 

  BuildArch:      noarch

- Summary:        RPM macros for PEP 517 Python packages

  

+ # We keep them here for now to avoid one loop of %%generate_buildrequires

+ # And to allow the other macros without %%pyproject_buildrequires (e.g. on Fedora 30)

+ # But those are also always in the output of %%generate_buildrequires

+ # in order to be removable in the future

  Requires: python3-pip >= 19

  Requires: python3-devel

  

- 

  %description

  This is a provisional implementation of pyproject RPM macros for Fedora 30+.

  These macros are useful for packaging Python projects that use the PEP 517

  pyproject.toml file, which specifies the package's build dependencies

  (including the build system, such as setuptools, flit or poetry).

  

+ 

  %prep

  # Not strictly necessary but allows working on file names instead

  # of source numbers in install section

@@ -29,16 +40,22 @@ 

  # nothing to do, sources are not buildable

  

  %install

- mkdir -p %{buildroot}/%{_rpmmacrodir}

- install -m 644 macros.pyproject %{buildroot}/%{_rpmmacrodir}/

+ mkdir -p %{buildroot}%{_rpmmacrodir}

+ mkdir -p %{buildroot}%{_rpmconfigdir}/redhat

+ install -m 644 macros.pyproject %{buildroot}%{_rpmmacrodir}/

+ install -m 644 pyproject_buildrequires.py %{buildroot}%{_rpmconfigdir}/redhat/

  

  %files

  %{_rpmmacrodir}/macros.pyproject

+ %{_rpmconfigdir}/redhat/pyproject_buildrequires.py

  

  %doc README.md

  %license LICENSE

  

  %changelog

+ * Tue Jul 02 2019 Miro Hrončok <mhroncok@redhat.com> - 0-3

+ - Add %%pyproject_buildrequires

+ 

  * Tue Jul 02 2019 Miro Hrončok <mhroncok@redhat.com> - 0-2

  - Fix shell syntax errors in %%pyproject_install

  - Drop PATH warning in %%pyproject_install

@@ -0,0 +1,89 @@ 

+ import sys

+ 

+ try:

+     import pytoml

+     from packaging.requirements import Requirement, InvalidRequirement

+     from packaging.utils import canonicalize_name, canonicalize_version

+ except ImportError:

+     # already echoed by the %pyproject_buildrequires macro

+     sys.exit(0)

+ 

+ 

+ try:

+     with open("pyproject.toml") as f:

+         pyproject_data = pytoml.load(f)

+ except FileNotFoundError:

+     pyproject_data = {}

+ except Exception as e:

+     sys.exit(e)

+ else:

+     import importlib

+ 

+     try:

+         backend = importlib.import_module(

+             pyproject_data["build-system"]["build-backend"]

+         )

+     except KeyError:

+         try:

+             import setuptools.build_meta

+         except ImportError:

+             print("python3dist(setuptools) >= 40.8")

+             print("python3dist(wheel)")

+             sys.exit(0)

+ 

+         backend = setuptools.build_meta

+     except ImportError:

+         backend = None

+ 

+ 

+ requirements = set()

+ rpm_requirements = set()

+ 

+ 

+ def add_requirement(requirement):

+     try:

+         requirements.add(Requirement(requirement))

+     except InvalidRequirement as e:

+         print(

+             f"WARNING: Skipping invalid requirement: {requirement}\n         {e}",

+             file=sys.stderr,

+         )

+ 

+ 

+ if "requires" in pyproject_data.get("build-system", {}):

+     try:

+         for requirement in pyproject_data["build-system"]["requires"]:

+             add_requirement(requirement)

+     except Exception as e:

+         sys.exit(e)

+ 

+ 

+ if hasattr(backend, "get_requires_for_build_wheel"):

+     try:

+         for requirement in backend.get_requires_for_build_wheel():

+             add_requirement(requirement)

+     except Exception as e:

+         sys.exit(e)

+ 

+ for requirement in requirements:

+     name = canonicalize_name(requirement.name)

+     if requirement.marker is not None and not requirement.marker.evaluate():

+         continue

+     together = []

+     for specifier in requirement.specifier:

+         version = canonicalize_version(specifier.version)

+         if specifier.operator == "!=":

+             together.append(

+                 f"(python3dist({name}) < {version} or python3dist({name}) >= {version}.0)"

+             )

+         else:

+             together.append(f"python3dist({name}) {specifier.operator} {version}")

+     if len(together) == 0:

+         rpm_requirements.add(f"python3dist({name})")

+     if len(together) == 1:

+         rpm_requirements.add(together[0])

+     elif len(together) > 1:

+         rpm_requirements.add(f"({' and '.join(together)})")

+ 

+ 

+ print(*sorted(rpm_requirements), sep="\n")

file added
+36

@@ -0,0 +1,36 @@ 

+ #!/usr/bin/bash -eux

+ 

+ config="/tmp/fedora-rawhide-x86_64-ci.cfg"

+ 

+ # create mock config if not present

+ # this makes sure tested version of pyproject-rpm-macros is available

+ # TODO: check if it has precedence if the release was not bumped in tested PR

+ if [ ! -f $config ]; then

+   original="/etc/mock/fedora-rawhide-x86_64.cfg"

+   split=$(sed -n '/\[fedora\]/=' $original | head -n1)

+   head -n$(($split-1)) $original > $config

+   cat /etc/yum.repos.d/test-pyproject-rpm-macros.repo >> $config

+   echo >> $config

+   tail -n +$split $original >> $config

+ fi

+ 

+ # prepare the rpmbuild folders, make sure nothing relevant is there

+ mkdir -p ~/rpmbuild/{SOURCES,SRPMS}

+ rm -f ~/rpmbuild/SRPMS/${1}-*.src.rpm

+ 

+ # download the sources and create SRPM

+ spectool -g -R ${1}.spec

+ rpmbuild -bs ${1}.spec

+ 

+ # build the SRPM in mock

+ mock -r $config --enablerepo=local init

+ mock -r $config --enablerepo=local ~/rpmbuild/SRPMS/${1}-*.src.rpm

+ 

+ # move the results to the artifacts directory, so we can examine them

+ artifacts=${TEST_ARTIFACTS:-/tmp/artifacts}

+ pushd /var/lib/mock/fedora-rawhide-x86_64/result

+ mv *.rpm ${artifacts}/

+ for log in *.log; do

+  mv ${log} ${artifacts}/${1}-${log}

+ done

+ popd

@@ -0,0 +1,46 @@ 

+ %global pypi_name entrypoints

+ Name:           python-%{pypi_name}

+ Version:        0.3

+ Release:        0%{?dist}

+ Summary:        Discover and load entry points from installed packages

+ License:        MIT

+ URL:            https://entrypoints.readthedocs.io/

+ Source0:        %{pypi_source}

+ 

+ BuildArch:      noarch

+ BuildRequires:  pyproject-rpm-macros

+ 

+ %description

+ Discover and load entry points from installed packages.

+ 

+ 

+ %package -n python3-%{pypi_name}

+ Summary:        %{summary}

+ %{?python_provide:%python_provide python3-%{pypi_name}}

+ 

+ %description -n python3-%{pypi_name}

+ Discover and load entry points from installed packages.

+ 

+ 

+ %prep

+ %autosetup -p1 -n %{pypi_name}-%{version}

+ 

+ 

+ %generate_buildrequires

+ %pyproject_buildrequires

+ 

+ 

+ %build

+ %pyproject_wheel

+ 

+ 

+ %install

+ %pyproject_install

+ 

+ 

+ %files -n python3-%{pypi_name}

+ %doc README.rst

+ %license LICENSE

+ %{python3_sitelib}/entrypoints-*.dist-info/

+ %{python3_sitelib}/entrypoints.py

+ %{python3_sitelib}/__pycache__/entrypoints.*

file modified
+1 -1

@@ -1,6 +1,6 @@ 

  %global pypi_name pytest

  Name:           python-%{pypi_name}

- Version:        5.0.0

+ Version:        4.4.2

  Release:        0%{?dist}

  Summary:        Simple powerful testing with Python

  License:        MIT

file added
+25

@@ -0,0 +1,25 @@ 

+ ---

+ - hosts: localhost

+   tags:

+     - classic

+   tasks:

+     - dnf:

+         name: "*"

+         state: latest

+ 

+ - hosts: localhost

+   roles:

+   - role: standard-test-basic

+     tags:

+     - classic

+     tests:

+     - pytest:

+         dir: .

+         run: ./mocktest.sh python-pytest

+     - entrypoints:

+         dir: .

+         run: ./mocktest.sh python-entrypoints

+     required_packages:

+     - mock

+     - rpmdevtools

+     - rpm-build

no initial comment

Shouldn't these be required by pyproject-rpm-macros, then?

The get_requires_for_build_wheel hook is optional, so an AttributeError on that . means no extra requirements.
See ("Optional hooks" in PEP 517)[https://www.python.org/dev/peps/pep-0517/#optional-hooks]

rebased onto 66ee0e6

2 months ago

they are generated by %pyproject_buildrequires - this allows us to later require pyproject-rpm-macros from python-srpm-macros etc. if we want. not sure if best approach, but testing it

rebased onto 0092eb5

2 months ago

this works (with patched mock)

to optimize stuff, we should probably just echo packaging toml and pip here

rebased onto 63532aa

2 months ago

So if I hit the scenario when build backend doesn't have the optional hook I need to add additional build requires manually, right? In that case I think it would be nice to get some output why that happened or have it in documentation.

So if I hit the scenario when build backend doesn't have the optional hook I need to add additional build requires manually, right?

I can imagine that I would be really confused why it is not working, checked documentation and then probably file a issue.

Define "not working" here.

Define "not working" here.
If I dont get all buildrequires my build would fail. In documentation I understood I 'll get them all.

I see. What about:

Only build dependencies according to PEP 517 and PEP 518 will be added. All other build dependencies (such as non-Python libraries or test dependencies) still need to be specified manually.

rebased onto 400b959

2 months ago

this shall not happen, right?

however this can fail if pyproject_data is {}

rebased onto 9adaf88

2 months ago

this adds wrong requires such as "configparser (>=3.5); python_version == '2.7'" on Python 3

rebased onto 2092c09

2 months ago

amended again. ready for review and testing

1 new commit added

  • WIP (experimantal): Build packages on CI
2 months ago

I'm trying to use the CI and will push force a lot. Please don't merge.

2 new commits added

  • WIP (experimantal): Build packages on CI
  • Add %pyproject_buildrequires
2 months ago

2 new commits added

  • WIP (experimantal): Build packages on CI
  • Add %pyproject_buildrequires
2 months ago

the build on the CI runs, but we don't see mock logs. maybe we can cp them to the artifacts directory, but I need to consult the documentation, as I don't remember how.

2 new commits added

  • WIP (experimantal): Build packages on CI
  • Add %pyproject_buildrequires
2 months ago

2 new commits added

  • WIP (experimantal): Build packages on CI
  • Add %pyproject_buildrequires
2 months ago

2 new commits added

  • WIP (experimantal): Build packages on CI
  • Add %pyproject_buildrequires
2 months ago

The CI part is ready for review as well, there is a temporary workaround (not to be merged) for the missing mock release.

2 new commits added

  • Build packages on Fedora CI
  • Add %pyproject_buildrequires
2 months ago

https://docs.fedoraproject.org/en-US/ci/standard-test-interface/#_invocation
To invoke the tests, the testing system must perform the following tasks for each test suite playbook:
MUST execute the playbook with the following operating system environment variables:
TEST_SUBJECTS: The test subjects string as described above
TEST_ARTIFACTS: The full path of an empty folder for test artifacts

maybe we should use $TEST_ARTIFACTS instead of /tmp/artifacts/

2 new commits added

  • Build packages on Fedora CI
  • Add %pyproject_buildrequires
2 months ago

https://docs.fedoraproject.org/en-US/ci/manifesto/#_the_manifesto
Packagers should be able to run individual tests on their own machines

I think it would be nice and simple to have some --local option.

It seems that we need to output results.yml file with this format.

BTW $TEST_ARTIFACTS is not set, going back to hardcoded /tmp/artifacts/

2 new commits added

  • Build packages on Fedora CI
  • Add %pyproject_buildrequires
2 months ago

I read it here
Each test suite playbook or test framework contained therein:
MUST create a results.yml file in the artifacts directory with test results in the results format defined above.

I suppose the standard test interface is not much enforced when the $TEST_ARTIFACTS is not set

I have no idea. The TEST_ARTIFACTS is probably set on the ansible host, not guest.

We don't need to follow any interface, we jut need this tested.

This needs to loose the trailing zero.

rebased onto fdf5116

2 months ago

I don't have any complaints then. Feel free to merge this.

2 new commits added

  • Build packages on Fedora CI
  • Add %pyproject_buildrequires
2 months ago

I've removed the custom mock build form tests.yaml. I think we can have this merged and ready, even when it does not yet work with current mock.

Pull-Request has been merged by churchyard

2 months ago