PR#1: Platform-python bootstrap: step 1 - build without platform_python - rpms/rpm

rpms / rpm

#1 Platform-python bootstrap: step 1 - build without platform_python

Merged 6 years ago by churchyard. Opened 6 years ago by ishcherb.

rpms/ ishcherb/rpm master into master

Platform-python bootstrap: step 1 - build without platform_python

Iryna Shcherbina • 6 years ago

412abe2

platform-python-abi.patch

file added

+39

		`@@ -0,0 +1,39 @@`
		`+ From fef3b646f3facd26dc04cfccdc27c061cfe0ee37 Mon Sep 17 00:00:00 2001`
		`+ From: =?UTF-8?q?Miro=20Hron=C4=8Dok?= <miro@hroncok.cz>`
		`+ Date: Mon, 7 Aug 2017 16:28:59 +0200`
		`+ Subject: [PATCH] Generate requires and provides for platform-python(abi)`
		`+`
		`+ See https://fedoraproject.org/wiki/Changes/Platform_Python_Stack`
		`+ ---`
		`+ scripts/pythondeps.sh \| 8 ++++----`
		`+ 1 file changed, 4 insertions(+), 4 deletions(-)`
		`+`
		`+ diff --git a/scripts/pythondeps.sh b/scripts/pythondeps.sh`
		`+ index 10a060aac..a6d97ef4f 100755`
		`+ --- a/scripts/pythondeps.sh`
		`+ +++ b/scripts/pythondeps.sh`
		`+ @@ -13,8 +13,8 @@ case $1 in`
		`+ # generating a line of the form`
		`+ # python(abi) = MAJOR.MINOR`
		`+ # (Don't match against -config tools e.g. /usr/bin/python2.6-config)`
		`+ - grep "/usr/bin/python.\..$" \`
		`+ - \| sed -e "s\|.*/usr/bin/python$.\..$\|python(abi) = \1\|"`
		`+ + egrep '/usr/(bin/\|libexec/platform-)python.\..$' \`
		`+ + \| sed -r -e "s@.*/usr/(bin/\|libexec/(platform-))python(.\..)@\2python(abi) = \3@"`
		`+ ;;`
		`+ -R\|--requires)`
		`+ shift`
		`+ @@ -23,8 +23,8 @@ case $1 in`
		`+ # /PATH/OF/BUILDROOT/usr/lib64/pythonMAJOR.MINOR/`
		`+ # generating (uniqely) lines of the form:`
		`+ # python(abi) = MAJOR.MINOR`
		`+ - grep "/usr/lib[^/]/python.\../." \`
		`+ - \| sed -e "s\|./usr/lib[^/]/python$.\..$/.*\|python(abi) = \1\|g" \`
		`+ + egrep '/usr/lib[^/]/(platform-\|)python.\../.' \`
		`+ + \| sed -r -e "s@./usr/lib[^/]/(platform-\|)python(.\..)/.*@\1python(abi) = \2@g" \`
		`+ \| sort \| uniq`
		`+ ;;`
		`+ esac`
		`+ --`
		`+ 2.13.3`
		`+`

platform-python-bytecompile.patch

file added

+52

		`@@ -0,0 +1,52 @@`
		`+ From cc1a08b4a5222e74cae46ee17d4ef297ed63c2a9 Mon Sep 17 00:00:00 2001`
		`+ From: =?UTF-8?q?Miro=20Hron=C4=8Dok?= <miro@hroncok.cz>`
		`+ Date: Tue, 8 Aug 2017 12:27:48 +0200`
		`+ Subject: [PATCH] Bytecompile stuff in /usr/lib*/platform-python with`
		`+ platform-python`
		`+`
		`+ ---`
		`+ scripts/brp-python-bytecompile \| 14 ++++++++++----`
		`+ 1 file changed, 10 insertions(+), 4 deletions(-)`
		`+`
		`+ diff --git a/scripts/brp-python-bytecompile b/scripts/brp-python-bytecompile`
		`+ index 894fa3459..92460b982 100644`
		`+ --- a/scripts/brp-python-bytecompile`
		`+ +++ b/scripts/brp-python-bytecompile`
		`+ @@ -57,9 +57,15 @@ EOF`
		`+ # and below /usr/lib/python3.1/, we're targeting /usr/bin/python3.1`
		`+`
		`+ shopt -s nullglob`
		+ -for python_libdir in `find "$RPM_BUILD_ROOT" -type d\|grep -E "/usr/lib(64)?/python[0-9]\.[0-9]$"`;
		+ +for python_libdir in `find "$RPM_BUILD_ROOT" -type d\|grep -E "/usr/lib(64)?/(platform-)?python[0-9]\.[0-9]$"`;
		`+ do`
		`+ - python_binary=/usr/bin/$(basename $python_libdir)`
		`+ + python_basename=$(basename $python_libdir)`
		`+ + if [[ $python_basename == platform* ]];`
		`+ + then`
		`+ + python_binary=/usr/libexec/$python_basename`
		`+ + else`
		`+ + python_binary=/usr/bin/$python_basename`
		`+ + fi`
		`+ real_libdir=${python_libdir/$RPM_BUILD_ROOT/}`
		`+ echo "Bytecompiling .py files below $python_libdir using $python_binary"`
		`+`
		`+ @@ -88,14 +94,14 @@ if [ ! -x "$default_python" ]; then`
		`+ fi`
		`+`
		`+ # Generate normal (.pyc) byte-compiled files.`
		`+ -python_bytecompile "" $default_python "/bin/\|/sbin/\|/usr/lib(64)?/python[0-9]\.[0-9]\|/usr/share/doc" "$RPM_BUILD_ROOT" "$depth" "/"`
		`+ +python_bytecompile "" $default_python "/bin/\|/sbin/\|/usr/lib(64)?/(platform-)?python[0-9]\.[0-9]\|/usr/share/doc" "$RPM_BUILD_ROOT" "$depth" "/"`
		`+ if [ $? -ne 0 -a 0$errors_terminate -ne 0 ]; then`
		`+ # One or more of the files had a syntax error`
		`+ exit 1`
		`+ fi`
		`+`
		`+ # Generate optimized (.pyo) byte-compiled files.`
		`+ -python_bytecompile "-O" $default_python "/bin/\|/sbin/\|/usr/lib(64)?/python[0-9]\.[0-9]\|/usr/share/doc" "$RPM_BUILD_ROOT" "$depth" "/"`
		`+ +python_bytecompile "-O" $default_python "/bin/\|/sbin/\|/usr/lib(64)?/(platform-)?python[0-9]\.[0-9]\|/usr/share/doc" "$RPM_BUILD_ROOT" "$depth" "/"`
		`+ if [ $? -ne 0 -a 0$errors_terminate -ne 0 ]; then`
		`+ # One or more of the files had a syntax error`
		`+ exit 1`
		`+ --`
		`+ 2.13.3`
		`+`

rpm.spec

file modified

+83 -4

		`@@ -15,6 +15,16 @@`
		`# build with new db format`
		`%bcond_with ndb`

		`+ # which python subpackages to build (default: all)`
		`+ # so it can be built in different modules with different subpackages`
		`+ %bcond_without python2`
		`+ %bcond_without python3`
		`+ # Note that the rpm package with Patch908 applied has to be in`
		`+ # the buildroot before platform_python enabled build happens.`
		`+ # Hence platform_python is disbaled for this build, because this`
		`+ # is the first build with Patch908 (to be removed in the second build).`
		`+ %bcond_with platform_python`
		`+`
		`%define rpmhome /usr/lib/rpm`

		`%global rpmver 4.13.90`
		`@@ -29,7 +39,7 @@`
		`Summary: The RPM package management system`
		`Name: rpm`
		`Version: %{rpmver}`
		`- Release: %{?snapver:0.%{snapver}.}3%{?dist}`
		`+ Release: %{?snapver:0.%{snapver}.}4%{?dist}`
		`Group: System Environment/Base`
		`Url: http://www.rpm.org/`
		`Source0: http://ftp.rpm.org/releases/%{srcdir}/%{name}-%{srcver}.tar.bz2`
		`@@ -64,6 +74,13 @@`
		`# Probably to be upstreamed in slightly different form`
		`Patch907: rpm-4.13.90-ldflags.patch`

		`+ # Use platform-python for bytecompilation in /usr/lib*/platform-python`
		`+ Patch908: platform-python-bytecompile.patch`
		`+`
		`+ # This patch is currently not part of any built RPM, however, to be`
		`+ # consistent with python-rpm-generators, we are adding it anyway.`
		`+ Patch909: platform-python-abi.patch`
		`+`
		`# Partially GPL/LGPL dual-licensed and some bits with BSD`
		`# SourceLicense: (GPLv2+ and LGPLv2+ with exceptions) and BSD`
		`License: GPLv2+`
		`@@ -212,6 +229,7 @@`
		`%description sign`
		`This package contains support for digitally signing RPM packages.`

		`+ %if %{with python2}`
		`%package -n python2-%{name}`
		`Summary: Python 2 bindings for apps which will manipulate RPM packages`
		`Group: Development/Libraries`
		`@@ -228,13 +246,14 @@`

		`This package should be installed if you want to develop Python 2`
		`programs that will manipulate RPM packages and databases.`
		`+ %endif # with python2`

		`+ %if %{with python3}`
		`%package -n python3-%{name}`
		`Summary: Python 3 bindings for apps which will manipulate RPM packages`
		`Group: Development/Libraries`
		`BuildRequires: python3-devel`
		`%{?python_provide:%python_provide python3-%{name}}`
		`- %{?system_python_abi}`
		`Requires: %{name}-libs%{?_isa} = %{version}-%{release}`
		`Provides: %{name}-python3 = %{version}-%{release}`
		`Obsoletes: %{name}-python3 < %{version}-%{release}`
		`@@ -246,6 +265,20 @@`

		`This package should be installed if you want to develop Python 3`
		`programs that will manipulate RPM packages and databases.`
		`+ %endif # with python3`
		`+`
		`+ %if %{with platform_python}`
		`+ %package -n platform-python-%{name}`
		`+ Summary: Platform Python bindings for apps which will manipulate RPM packages`
		`+ Group: Development/Libraries`
		`+ BuildRequires: platform-python-devel`
		`+ Requires: %{name}-libs%{?_isa} = %{version}-%{release}`
		`+`
		`+ %description -n platform-python-%{name}`
		`+ The platform-python-rpm package contains a module that permits applications`
		`+ written in the Python programming language to use the interface`
		`+ supplied by RPM Package Manager libraries.`
		`+ %endif # with platform_python`

		`%package apidocs`
		`Summary: API documentation for RPM libraries`
		`@@ -358,13 +391,31 @@`
		`--enable-python \`
		`--with-crypto=openssl`

		`+`
		`+ %if %{with platform_python}`
		`+ # Copy the directory so python3 and platform_python do not share it,`
		`+ # as they may have the same version and the filenames could collide.`
		+ # Do it before `make` to avoid having anything built in there.
		`+ cp -a python platform_python`
		`+ %endif # with platform_python`
		`+`
		`make %{?_smp_mflags}`

		`pushd python`
		`+ %if %{with python2}`
		`%{__python2} setup.py build`
		`+ %endif # with python2`
		`+ %if %{with python3}`
		`%{__python3} setup.py build`
		`+ %endif # with python3`
		`popd`

		`+ %if %{with platform_python}`
		`+ pushd platform_python`
		`+ %{__platform_python} setup.py build`
		`+ popd`
		`+ %endif # with platform_python`
		`+`
		`%install`
		`rm -rf $RPM_BUILD_ROOT`

		`@@ -373,10 +424,20 @@`
		`# We need to build with --enable-python for the self-test suite, but we`
		`# actually package the bindings built with setup.py (#531543#c26)`
		`pushd python`
		`+ %if %{with python2}`
		`%{__python2} setup.py install --skip-build --root $RPM_BUILD_ROOT`
		`+ %endif # with python2`
		`+ %if %{with python3}`
		`%{__python3} setup.py install --skip-build --root $RPM_BUILD_ROOT`
		`+ %endif # with python3`
		`popd`

		`+ %if %{with platform_python}`
		`+ pushd platform_python`
		`+ %{__platform_python} setup.py install --skip-build --root $RPM_BUILD_ROOT`
		`+ popd`
		`+ %endif # with platform_python`
		`+`

		`# Save list of packages through cron`
		`mkdir -p ${RPM_BUILD_ROOT}%{_sysconfdir}/cron.daily`
		`@@ -547,13 +608,24 @@`
		`%{_bindir}/rpmsign`
		`%{_mandir}/man8/rpmsign.8*`

		`+ %if %{with python2}`
		`%files -n python2-%{name}`
		`- %{python_sitearch}/%{name}/`
		`- %{python_sitearch}/%{name}-%{version}*.egg-info`
		`+ %{python2_sitearch}/%{name}/`
		`+ %{python2_sitearch}/%{name}-%{version}*.egg-info`
		`+ %endif # with python2`
		`+`

		`+ %if %{with python3}`
		`%files -n python3-%{name}`
		`%{python3_sitearch}/%{name}/`
		`%{python3_sitearch}/%{name}-%{version}*.egg-info`
		`+ %endif # with python3`
		`+`
		`+ %if %{with platform_python}`
		`+ %files -n platform-python-%{name}`
		`+ %{platform_python_sitearch}/%{name}/`
		`+ %{platform_python_sitearch}/%{name}-%{version}*.egg-info`
		`+ %endif # with platform_python`

		`%files devel`
		`%{_mandir}/man8/rpmgraph.8*`
		`@@ -571,6 +643,13 @@`
		`%doc doc/librpm/html/*`

		`%changelog`
		`+ * Mon Aug 14 2017 Miro Hrončok <mhroncok@redhat.com> - 4.13.90-0.git14000.4`
		`+ - Add platform-python bytecompilation patch: platform-python-bytecompile.patch`
		`+ - Add platform python deps generator patch: platform-python-abi.patch`
		`+ - Add a platform-python subpackage and remove system python related declarations`
		`+ - Build rpm without platform_python for bytecompilation`
		`+ (https://fedoraproject.org/wiki/Changes/Platform_Python_Stack)`
		`+`
		`* Mon Aug 14 2017 Panu Matilainen <pmatilai@redhat.com> - 4.13.90-0.git14000.3`
		`- Disable macro argument quoting as a band-aid to #1481025`

ishcherb commented 6 years ago

Changes needed for Platform Python Stack change:
- Add platform-python bytecompilation patch: platform-python-bytecompile.patch
- Add platform python deps generator patch: platform-python-abi.patch
- Add a platform-python subpackage and remove system python related declarations
- Build rpm without platform_python for bytecompilation (already set in spec file)

When platform-python package is built, switch to %bcond_without platform_python
to build platform-python-rpm subpackage.

CC @churchyard

churchyard commented 6 years ago

These are changes needed for the Platform Python Stack Fedora change.

The Platform module on which everything else will depend will contain dnf, and that needs Python. To make everything work, we need to build an independent minimal Python stack containing all the libraries needed to build and run dnf, and that includes platform-python-rpm and also the patched scripts in this PR.

We're creating the pull request now, so you can review the changes.

The first boostraping step (this PR) does not depend on platform-python. The second will depend on it. The full build order is available in Bugzilla.

If you have any comments on the change, please let us know by this Thursday (2017-08-17). We will have to push and build everything by the end of the week to comply with the Fedora 27 schedule.
If the change looks good, you can merge and build it -- or let us know and we'll handle the merging and building for you.

churchyard commented 6 years ago

@mjw Could you please review this any time soon? It's the current blocker for all other packages. Thanks.

churchyard commented 6 years ago

Since this is at the beginning of the build order, we'll merge & build on Thursday morning if no feedback is given by then.

Pull-Request has been merged by churchyard

6 years ago

churchyard commented 6 years ago

Build running https://koji.fedoraproject.org/koji/taskinfo?taskID=21273370

churchyard commented 6 years ago

Also, f27 https://koji.fedoraproject.org/koji/taskinfo?taskID=21273823

churchyard commented 6 years ago

$ koji wait-repo f28-build --build=rpm-4.13.90-0.git14002.4.fc28
Warning: nvr rpm-4.13.90-0.git14002.4.fc28 is not current in tag f28-build
  latest build in f28-build is rpm-4.13.90-0.git14002.3.fc27

$ koji wait-repo f27-build --build=rpm-4.13.90-0.git14002.4.fc27
Warning: nvr rpm-4.13.90-0.git14002.4.fc27 is not current in tag f27-build
  latest build in f27-build is rpm-4.13.90-0.git14002.3.fc27

I guess we just need to wait?

churchyard commented 6 years ago

Bootstraped.