#44 Always pull in correct python3X-rpm-macros package
Merged 2 years ago by kevin. Opened 2 years ago by gotmax23.
rpms/ gotmax23/epel-rpm-macros epel8  into  epel8

file modified
+19 -6
@@ -1,6 +1,6 @@ 

  Name:           epel-rpm-macros

  Version:        8

- Release:        29

+ Release:        30

  Summary:        Extra Packages for Enterprise Linux RPM macros

  

  License:        GPLv2
@@ -33,11 +33,16 @@ 

  Requires:       python-srpm-macros

  # Since the RHEL python-devel package doesn't require them, bring them in here

  Requires:       python-rpm-macros

- Requires:       python2-rpm-macros

- # Make sure EPEL8's Python 3 packages use /usr/bin/python3.6 or python3.8

- #   instead of /usr/libexec/platform-python

- Requires:       (python-modular-rpm-macros if python3-devel)

- Suggests:       python36-rpm-macros

+ Requires:       (python2-rpm-macros if python2-devel)

+ # Make sure EPEL8's Python 3 packages use /usr/bin/python3.{6,8,9} instead of /usr/libexec/platform-python.

+ # This will install the python3X-rpm-macros version corresponding to the python3X-devel version installed.

+ # If multiple versions are installed together, it will prefer python36-rpm-macros, the default Python version.

+ Requires:       (python-modular-rpm-macros if (python36-devel or python38-devel or python39-devel))

+ # We use weak dependencies here in order to avoid creating conflicts between

+ # the different python3X-devel packages.

+ Recommends:     (python36-rpm-macros if python36-devel)

+ Suggests:       (python38-rpm-macros if python38-devel)

+ Suggests:       (python39-rpm-macros if python39-devel)

  #systemd additional macros https://pagure.io/epel/issue/77

  Requires:       (epel-rpm-macros-systemd if systemd-rpm-macros)

  Requires:       ansible-srpm-macros
@@ -103,6 +108,14 @@ 

  

  

  %changelog

+ * Tue May 24 2022 Maxwell G <gotmax@e.email> - 8-30

+ - Pull in appropriate python3X-rpm-macros package for non-default Python

+   versions

+ - Ensure the python macros work properly when building against non-default

+   python interpreters.

+ - Only pull in python2-rpm-macros when python2-devel is present.

+ - Remove macros that have been added to RHEL itself.

+ 

  * Tue May 17 2022 Maxwell G <gotmax@e.email> - 8-29

  - Add `Requires: ansible-srpm-macros`

  - Backport @churchyard's packaging nitpicks from epel9

file modified
+24 -60
@@ -7,29 +7,34 @@ 

  %_monodir %{_prefix}/lib/mono

  %_monogacdir %{_monodir}/gac

  

- # A directory for rpm macros

- %rpmmacrodir /usr/lib/rpm/macros.d

- 

- # Upstream RPM has this in Fedora, obsoleting the one above.

- %_rpmmacrodir /usr/lib/rpm/macros.d

- 

  # Bash completions; not in bash to not intefere with install ordering

- %bash_completion_dir /usr/share/bash-completion/completions/

+ %bash_completion_dir %{_datadir}/bash-completion/completions/

  

  # Use the non-underscored Python macros to refer to Python in spec, etc.

  %python2 %__python2

- %python3 %__python3

  

- # Simplified version backported from Fedora

- %py_shebang_fix %{expand:/usr/bin/pathfix.py -pni "%{__python} %{py_shbang_opts}"}

- %py2_shebang_fix %{expand:/usr/bin/pathfix.py -pni "%{__python2} %{py2_shbang_opts}"}

- %py3_shebang_fix %{expand:/usr/bin/pathfix.py -pni "%{__python3} %{py3_shbang_opts}"}

+ # This macro is copied here from https://git.centos.org/rpms/python-rpm-macros/pull-request/1

+ # and is needed for the overrides in macros.zzz-epel-override.

+ # It will be removed once it lands in RHEL.

+ # https://bugzilla.redhat.com/show_bug.cgi?id=2090007

+ #

+ # RHEL 9+ and Fedora compatibility macro

+ # Only use in macro backports, not intended to be used in spec files!

+ # In the future, the %%python3_pkgversion macro has a dot, e.g. 3.9 or 3.11

+ # However, in RHEL 8 at least, it does not, e.g. 38, 39

+ # This is a helpful macro that determines the proper "Python version" string with dot

+ # from %%python3_pkgversion without actually having Python installed.

+ # For values other than 3X, it should expand to %%python3_pkgversion unchanged.

+ # Examples of %%python3_pkgversion -> %%_python3_pkgversion_with_dot:

+ #   3    -> 3

+ #   38   -> 3.8

+ #   39   -> 3.9

+ #   310  -> 3.10

+ #   3.12 -> 3.12

+ #   4    -> 4

+ #   412  -> 412

+ %_python3_pkgversion_with_dot %{lua:print((rpm.expand("%python3_pkgversion"):gsub('^3(%d)', '3.%1')))}

  

- # https://fedoraproject.org/wiki/Changes/Python_Upstream_Architecture_Names

- %python_platform_triplet %(%{__python} -Esc "import sysconfig; print(sysconfig.get_config_var('MULTIARCH'))")

- %python_ext_suffix %(%{__python} -Esc "import sysconfig; print(sysconfig.get_config_var('EXT_SUFFIX'))")

- %python3_platform_triplet %(%{__python3} -Ic "import sysconfig; print(sysconfig.get_config_var('MULTIARCH'))")

- %python3_ext_suffix %(%{__python3} -Ic "import sysconfig; print(sysconfig.get_config_var('EXT_SUFFIX'))")

  

  # Users can use %%python only if they redefined %%__python (e.g. to %%__python3)

  %python() %{lua:\
@@ -41,47 +46,6 @@ 

      end\

  }

  

- # This only supports Python 3.5+ and will never work with Python 2.

- # Hence, it has no Python version in the name.

- %pycached() %{lua:

-   path = rpm.expand("%{?*}")

-   if (string.sub(path, "-3") ~= ".py") then

-     rpm.expand("%{error:%%pycached can only be used with paths explicitly ending with .py}")

-   else

-     print(path)

-     pyminor = path:match("/python3.(%d+)/") or "*"

-     dirname = path:match("(.*/)")

-     modulename = path:match(".*/([^/]+).py")

-     print("\\n" .. dirname .. "__pycache__/" .. modulename .. ".cpython-3" .. pyminor .. "{,.opt-?}.pyc")

-   end

- }

- 

- # This is intended for Python 3 only, hence also no Python version in the name.

- %__pytest /usr/bin/pytest-3

- %pytest %{expand:\\\

-   CFLAGS="${CFLAGS:-${RPM_OPT_FLAGS}}" LDFLAGS="${LDFLAGS:-${RPM_LD_FLAGS}}"\\\

-   PATH="%{buildroot}%{_bindir}:$PATH"\\\

-   PYTHONPATH="${PYTHONPATH:-%{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib}}"\\\

-   PYTHONDONTWRITEBYTECODE=1\\\

-   %__pytest}

- 

- %py_provides() %{lua:

-     local name = rpm.expand('%1')

-     if name == '%1' then

-         rpm.expand('%{error:%%py_provides requires at least 1 argument, the name to provide}')

-     end

-     local evr = rpm.expand('%2')

-     if evr == '%2' then

-         evr = rpm.expand('%{?epoch:%{epoch}:}%{version}-%{release}')

-     end

-     print('Provides: ' .. name .. ' = ' .. evr .. '\\n')

-     -- NB: dash needs to be escaped!

-     if name:match('^python3%-') then

-       replaced = name:gsub('^python3%-', 'python-')

-       print('Provides: ' .. replaced .. ' = ' .. evr .. '\\n')

-     end

- }

- 

  # With $PATH and $PYTHONPATH set to the %%buildroot,

  # try to import the given Python module(s).

  # Useful as a smoke test in %%check when running tests is not feasible.
@@ -138,6 +102,6 @@ 

  %python_wheel_pkg_prefix python%{?rhel:%{!?eln:%{python3_pkgversion}}}

  %python_wheel_dir %{_datadir}/%{python_wheel_pkg_prefix}-wheels

  

- # qt5 macro removed from RHEL8 but needed to ensure qtwebengine, and 

- #  it's dependencies build on supported arches.

+ # qt5 macro removed from RHEL8 but needed to ensure qtwebengine, and

+ #  its dependencies build on supported arches.

  %qt5_qtwebengine_arches %{ix86} x86_64 %{arm} aarch64 mips mipsel mips64el

file modified
+40
@@ -1,3 +1,43 @@ 

  # This file is used _only_ to override macros which exist in RHEL.

  # It should be used very sparingly.

  

+ # Override %__python3 here based on %%python3_pkgversion to avoid relying on the finicky

+ # python3X-rpm-macros packages.

+ # Backported from https://src.fedoraproject.org/rpms/python-rpm-macros/c/a8b26546eb699afe0dbfcef913a2aa7085fc5afb?branch=rawhide

+ #

+ # EPEL specific, RHEL defines this in the python3X-rpm-macros packages

+ # and the default is /usr/libexec/platform-python.

+ # Usually, we don't want EPEL packages to use platform-python.

+ # Packages that want to do that can explicitly override this.

+ %__python3 /usr/bin/python%{_python3_pkgversion_with_dot}

+ 

+ # RHEL 8's %%pytest does not work properly on non-default Python versions.

+ # Backported from https://src.fedoraproject.org/rpms/python-rpm-macros/c/02536540768555dde6d675935db45df81bcab7bf?branch=rawhide

+ # https://bugzilla.redhat.com/show_bug.cgi?id=2091462

+ %__pytest /usr/bin/pytest%(test %{python3_pkgversion} == 3 || echo -%{python3_version})

+ 

+ # RHEL 8's %%py3_dist does not work properly on non-default Python versions.

+ # This is copied here from https://git.centos.org/rpms/python-rpm-macros/pull-request/1

+ # until that lands in RHEL.

+ # https://bugzilla.redhat.com/show_bug.cgi?id=2090007

+ 

+ # Creates Python 3 dist tag(s) after converting names to canonical format

+ #   Needs to first put all arguments into a list, because invoking a different

+ #   macro (%py_dist_name) overwrites them

+ %py3_dist() %{lua:\

+         python3_pkgversion_with_dot = rpm.expand("%_python3_pkgversion_with_dot")\

+         args = {}\

+         arg = 1\

+         while (true) do\

+                 name = rpm.expand("%{?" .. arg .. ":%{" .. arg .. "}}");\

+                 if (name == nil or name == '') then\

+                         break\

+                 end\

+                 args[arg] = name\

+                 arg = arg + 1\

+         end\

+         for arg, name in ipairs(args) do\

+                 canonical = rpm.expand("%py_dist_name " .. name);\

+                 print("python" .. python3_pkgversion_with_dot .. "dist(" .. canonical .. ") ");\

+         end\

+ }

When building EPEL packages for non-default Python versions, it is necessary to manually add Requires: python%{python3_pkgversion}-rpm-macros or %global __python3 /usr/bin/python3.X. Doing so is not necessary with the default Python version, because python3-devel pulls in python36-rpm-macros. This PR makes doing this unnecessary for all versions of Python 3.

I was going to add

Requires:       (python36-rpm-macros if python36-devel)
Requires:       (python38-rpm-macros if python38-devel)
Requires:       (python39-rpm-macros if python39-devel)

instead, but then I realized that the python3X-rpm-macros packages conflict with each other, and I didn't want to create implicit conflicts between the python3X-devel packages. However, what I'm proposing may lead to confusing behavior if multiple python3X-devel packages are installed. Your feedback is appreciated.

CC: @churchyard and @orion

However, what I'm proposing may lead to confusing behavior if multiple python3X-devel packages are installed.

I've installed this to mock. Then installed various combinations of python3X-devel:

  1. python36-devel python38-devel python39-devel -> python39-rpm-macros
  2. python36-devel python38-devel -> python38-rpm-macros
  3. python36-devel python39-devel -> python39-rpm-macros
  4. python38-devel python39-devel -> python39-rpm-macros

Seems like the latest version wins. Can we add a stronger suggest to python36-rpm-macros somehow? Recommends maybe?

I'm going to keep the original Suggests and see if it makes any difference.

1 new commit added

  • fixup: Use a bigger hammer
2 years ago

I'm going to keep the original Suggests and see if it makes any difference.

No difference. Recommending works thou. See my fixup.

1 new commit added

  • Update python-rpm-macros specfile comments
2 years ago

I added some explanatory comments to the specfile. Please let me know if I should squash the commits.

I also noticed that some of the Python macros need to be removed from this package, because they have since been backported to RHEL.

Build succeeded.

1 new commit added

  • Remove macros that have been added to RHEL itself.
2 years ago

Build succeeded.

Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci

2 new commits added

  • Make %pytest and %py3_dist compatible with alt. pythons
  • Override %__python3 based on %python3_pkgversion
2 years ago

Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci

This should be ready for review. I would like to get this merged so I can build python38 packages more easily. @churchyard, can you please review the new changes when you have the chance?

The changes appear reasonable. For each Python macro that is defined here or redefined here and isn't EPEL specific, I would appreciate if there was a link to bugizlla. E.g.:

# This macro is copied here from https://git.centos.org/rpms/python-rpm-macros/pull-request/1
# and is needed for the overrides in macros.zzz-epel-override.
# It will be removed once it lands in RHEL.
# https://bugzilla.redhat.com/show_bug.cgi?id=2090007

Or:

# RHEL 8's %%pytest does not work properly on non-default Python versions.
# Backported from https://src.fedoraproject.org/rpms/python-rpm-macros/c/02536540768555dde6d675935db45df81bcab7bf?branch=rawhide
# https://bugzilla.redhat.com/show_bug.cgi?id=2091462

And for changes that are EPEL only, add an explanation.

# Override %__python3 here based on %%python3_pkgversion to avoid relying on the finicky
# python3X-rpm-macros packages.
# Backported from https://src.fedoraproject.org/rpms/python-rpm-macros/c/a8b26546eb699afe0dbfcef913a2aa7085fc5afb?branch=rawhide
# EPEL specific, RHEL defines this in the python3X-rpm-macros packages
# and the default is /usr/libexec/platform-python
# Usually, we don't want EPEL packages to use platform-python
# Packages that want to do that can override this implicitly back

1 new commit added

  • Reference bugzillas for modified macros
2 years ago

Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci

Thanks. Looking good from the Python perspective.

%rpmmacrodir is already defined in RHEL8's redhat-rpm-config, so we don't need it.

7 new commits added

  • Reference bugzillas for modified macros
  • Make %pytest and %py3_dist compatible with alt. pythons
  • Override %__python3 based on %python3_pkgversion
  • Remove macros that have been added to RHEL itself.
  • Update python-rpm-macros specfile comments
  • fixup: Use a bigger hammer
  • Always pull in correct python3X-rpm-macros package
2 years ago

%rpmmacrodir is already defined in RHEL8's redhat-rpm-config, so we don't need it.

@carlwgeorge, I've fixed this.

Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci

Please let me know if anything else needs to be fixed or if this needs more explanation. I'd really like to get this in.

Looking at https://pagure.io/releng/issue/10871 -- could you also please change:

Requires:       python2-rpm-macros

To:

Requires:       (python2-rpm-macros if python2-devel)

?

2 new commits added

  • Update changelog
  • Only pull in python2-rpm-macros when python2-devel is present.
2 years ago

I've added that. Is it worthwhile to change Requires: python-rpm-macros to Requires: (python-rpm-macros if (python2-rpm-macros or python-modular-rpm-macros)), as well? I believe that only python-srpm-macros needs to be in the default buildroot.

Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci

python-rpm-macros are 1) small 2) in the AppStream repo (not in a module) 3) required by python3-rpm-macros. IMHO not worth it.

Pull-Request has been merged by kevin

2 years ago

There might be a regression here somewhere wrt %pytest:

https://koschei.fedoraproject.org/affected-by/epel-rpm-macros?epoch1=0&version1=8&release1=29&epoch2=0&version2=8&release2=30&collection=epel8

e.g. https://koschei.fedoraproject.org/package/python-utils?collection=epel8 now has:

+ /usr/bin/pytest
RPM build errors:
/var/tmp/rpm-tmp.eak0FO: line 35: /usr/bin/pytest: No such file or directory

While previously, it ran /usr/bin/pytest-3.

yep, %__pytest /usr/bin/pytest%(test %{python3_pkgversion} == 3 || echo -%{python3_version}) is broken with 3.6.

Could be %__pytest /usr/bin/pytest-%{python3_version} all the time, or if we are aiming at bit-by-bit backwards-compatibility: %__pytest /usr/bin/pytest%(test %{python3_pkgversion} == 3 && echo -3 || echo -%{python3_version})

Thanks for catching this, and sorry that I didn't before :(. Just for reference:

[gotmax@toolbox] ~ >>> cat /etc/redhat-release
CentOS Stream release 8
[gotmax@toolbox] ~ >>> sudo dnf repoquery -q --list python3-pytest | grep '/usr/bin'
/usr/bin/py.test-3
/usr/bin/py.test-3.6
/usr/bin/pytest-3
/usr/bin/pytest-3.6
[gotmax@toolbox] ~ >>> sudo dnf repoquery -q --list python38-pytest | grep '/usr/bin'
/usr/bin/py.test-3.8
/usr/bin/pytest-3.8
[gotmax@toolbox] ~ >>> sudo dnf repoquery -q --list python39-pytest | grep '/usr/bin'
/usr/bin/py.test-3.9
/usr/bin/pytest-3.9
/usr/bin/pytest-3.9

Could be %__pytest /usr/bin/pytest-%{python3_version} all the time, or if we are aiming at bit-by-bit backwards-compatibility: %__pytest /usr/bin/pytest%(test %{python3_pkgversion} == 3 && echo -3 || echo -%{python3_version})

I don't think bit-for-bit compatibility is necessary here. python3-pytest in RHEL 8 provides both. If we're changing this here, we should also change it in Fedora.

I would prefer %__pytest /usr/bin/pytest-%{_python3_pkgversion_with_dot}, though. The current implementation prints sh: line 1: /usr/bin/python3X: No such file or directory several times when building an SRPM or otherwise parsing the specfile without the dependencies installed. It doesn't actually affect the build, but it's rather annoying. %{__python3} -m pytest would be a lot nicer, but you told me that's impossible so...

The current implementation prints sh: line 1: /usr/bin/python3X: No such file or directory several times when building an SRPM...

So does %{python3_stielib} etc.