#37 Add the %pycached macro
Merged 10 months ago by churchyard. Opened 10 months ago by churchyard.
rpms/ churchyard/python-rpm-macros pycached  into  master

file modified
+15
@@ -37,3 +37,18 @@ 

  %py3_install_wheel() %{expand:\\\

    pip%{python3_version} install -I dist/%{1} --root %{buildroot} --no-deps

  }

+ 

+ # This only supports Python 3.5+ and will never work with Python 2.

+ # Hence, it has no Python version in the name.

+ %pycached() %{lua:

+   path = rpm.expand("%{?1}")

+   if (string.sub(path, "-3") ~= ".py") then

+     rpm.expand("%{error:%%pycached can only be used with paths explicitly ending with .py}")

+   else

+     print(path)

+     pyminor = path:match("/python3.(%d+)/") or "*"

+     dirname = path:match("(.*/)")

+     modulename = path:match(".*/([^/]+).py")

+     print("\\n" .. dirname .. "__pycache__/" .. modulename .. ".cpython-3" .. pyminor .. "{,.opt-?}.pyc")

+   end

+ }

file modified
+1
@@ -80,6 +80,7 @@ 

  %changelog

  * Sat Dec 28 2019 Miro Hrončok <mhroncok@redhat.com> - 3-51

  - Define %%python, but make it work only if %%__python is redefined

+ - Add the %%pycached macro

  

  * Tue Nov 26 2019 Lumír Balhar <lbalhar@redhat.com> - 3-50

  - Update of bundled compileall2 module

Usage:

%files
...
%pycached %{python3_sitelib}/foo.py

This will list:

/usr/lib/python3.8/site-packages/foo.py
/usr/lib/python3.8/site-packages/__pycache__/foo.cpython-38{,.opt-?}.pyc

Assuming the Python 3 version is 3.8.
The bytecode files are globbed, their presence is not checked.

This will fail:

%pycached %{python3_sitelib}/foo

error: %pycached can only be used with paths explicitly ending with .py

And so will any of this:

%pycached %{python3_sitelib}/*
%pycached %{python3_sitelib}/foo.*
%pycached %{python3_sitelib}/foo.p?
%pycached %{python3_sitelib}/foo.?y
%pycached %{python3_sitelib}/foo.??

But this will work:

%pycached %{python3_sitelib}/foo*.py

Will generate the following globs:

/usr/lib/python3.8/site-packages/foo*.py
/usr/lib/python3.8/site-packages/__pycache__/foo*.cpython-38{,.opt-?}.pyc

Hey, nice documentation! It would be good to put it in a more permanent place than a PR description.

This explicitly doesn't list the __pycache__ directory, as that is supposed to be owned by python3-libs if directly in site-packages. Adding a check for that seemed like an overkill, if a package should own /usr/lib/python3.8/site-packages/foo/__pycache__ it is more appropriate to list %{python3_sitelib}/foo/ entirely.

nice documentation! It would be good to put it in a more permanent place than a PR description.

Since none of the other macros are documented within this package, my plan is to document the macro in the Fedora's Python packaging guidelines.

Side note: The PR description is also in the commit message - permanent, but not very discoverable.

The macro looks good to me.
IMO it needs an announcement on the python mailing list, though. I don't think there was any community input going into it.
It might be necessary to change it for EPEL, which has more python3 versions.

We could decouple the macro from python version and do the following:

  • glob python version based on the version already included in the path (/usr/lib/python3.8/site-packages/ -> .cpython-38{,.opt-?}.pyc)
  • glob any python version, when the path doesn't include it (/custom/or/globbed/path/ -> .cpython-*{,.opt-?}.pyc)

Will share this on packaging and python-devel.

rebased onto 2cad02feef219eed2ec00c0479f872f431b70b17

10 months ago

Amended. Added this to the commit message:

When used with paths that include Python 3 version, it globs with the version:

%pycached /opt/python3.10/foo.py

Generates:

/opt/python3.10/foo.py
/opt/python3.10/__pycache__/foo.cpython-310{,.opt-?}.pyc

While paths without version have less strict globs:

%pycached /custom/foo.py
/custom/foo.py
/custom/__pycache__/foo.cpython-3*{,.opt-?}.pyc

This will generate a warning in RPM build:

warning: File listed twice: /custom/__pycache__/foo.cpython-38.opt-1.pyc

However it ensures the optimized bytecode is there.

It might be necessary to change it for EPEL, which has more python3 versions.

This solution works generally, but will still fail with 3.4. For EPEL, we can make the code a bit more complicated, to handle 3.4 globs differently, or relax the globs.

I'm not that much concerned about Python 3.4.

OK, we can only make it work on EPEL with 3.6.

Sent to packaging and python-devel.

Reasonable suggestions:

  1. rename to %pyfile

I've decided that I prefer %pycached over %pyfile and since there was no further feedback, I will merge as is.

rebased onto 6c63a5b

10 months ago

Pull-Request has been merged by churchyard

10 months ago