#104 fully automate repository changes
Merged a month ago by humaton. Opened 2 months ago by kparal.
rpms/ kparal/fedora-repos auto-update-repos  into  rawhide

file modified
+8 -4
@@ -11,18 +11,19 @@ 

  # testing and troubleshooting for development packages in conjunction

  # with new test releases.

  #

- # More information is available at http://fedoraproject.org/wiki/Testing 

+ # More information is available at http://fedoraproject.org/wiki/Testing

  #

  # Reproducible and reportable issues should be filed at

  # http://bugzilla.redhat.com/.

  #

  # Product: Fedora

  # Version: eln

+ 

  [eln-baseos]

  name=Fedora - ELN BaseOS - Developmental packages for the next Enterprise Linux release

  baseurl=https://odcs.fedoraproject.org/composes/production/latest-Fedora-ELN/compose/BaseOS/$basearch/os/

  #metalink=https://mirrors.fedoraproject.org/metalink?repo=eln&arch=$basearch

- enabled=1

+ enabled=AUTO_VALUE

  countme=1

  metadata_expire=6h

  repo_gpgcheck=0
@@ -36,6 +37,7 @@ 

  baseurl=https://odcs.fedoraproject.org/composes/production/latest-Fedora-ELN/compose/BaseOS/$basearch/debug/tree

  #metalink=https://mirrors.fedoraproject.org/metalink?repo=eln-debug&arch=$basearch

  enabled=0

+ metadata_expire=6h

  repo_gpgcheck=0

  type=rpm

  gpgcheck=1
@@ -47,6 +49,7 @@ 

  baseurl=https://odcs.fedoraproject.org/composes/production/latest-Fedora-ELN/compose/BaseOS/source/tree/

  #metalink=https://mirrors.fedoraproject.org/metalink?repo=eln-source&arch=$basearch

  enabled=0

+ metadata_expire=6h

  repo_gpgcheck=0

  type=rpm

  gpgcheck=1
@@ -59,7 +62,7 @@ 

  name=Fedora - ELN AppStream - Developmental packages for the next Enterprise Linux release

  baseurl=https://odcs.fedoraproject.org/composes/production/latest-Fedora-ELN/compose/AppStream/$basearch/os/

  #metalink=https://mirrors.fedoraproject.org/metalink?repo=eln&arch=$basearch

- enabled=1

+ enabled=AUTO_VALUE

  countme=1

  metadata_expire=6h

  repo_gpgcheck=0
@@ -73,6 +76,7 @@ 

  baseurl=https://odcs.fedoraproject.org/composes/production/latest-Fedora-ELN/compose/AppStream/$basearch/debug/tree

  #metalink=https://mirrors.fedoraproject.org/metalink?repo=eln-debug&arch=$basearch

  enabled=0

+ metadata_expire=6h

  repo_gpgcheck=0

  type=rpm

  gpgcheck=1
@@ -84,9 +88,9 @@ 

  baseurl=https://odcs.fedoraproject.org/composes/production/latest-Fedora-ELN/compose/AppStream/source/tree/

  #metalink=https://mirrors.fedoraproject.org/metalink?repo=eln-source&arch=$basearch

  enabled=0

+ metadata_expire=6h

  repo_gpgcheck=0

  type=rpm

  gpgcheck=1

  gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch

  skip_if_unavailable=False

- 

file modified
+4 -4
@@ -2,9 +2,9 @@ 

  name=Fedora Modular $releasever - $basearch

  #baseurl=http://download.example/pub/fedora/linux/releases/$releasever/Modular/$basearch/os/

  metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-$releasever&arch=$basearch

- enabled=0

+ enabled=AUTO_VALUE

  countme=1

- #metadata_expire=7d

+ metadata_expire=AUTO_VALUE

  repo_gpgcheck=0

  type=rpm

  gpgcheck=1
@@ -16,7 +16,7 @@ 

  #baseurl=http://download.example/pub/fedora/linux/releases/$releasever/Modular/$basearch/debug/tree/

  metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-debug-$releasever&arch=$basearch

  enabled=0

- metadata_expire=7d

+ metadata_expire=AUTO_VALUE

  repo_gpgcheck=0

  type=rpm

  gpgcheck=1
@@ -28,7 +28,7 @@ 

  #baseurl=http://download.example/pub/fedora/linux/releases/$releasever/Modular/source/tree/

  metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-source-$releasever&arch=$basearch

  enabled=0

- metadata_expire=7d

+ metadata_expire=AUTO_VALUE

  repo_gpgcheck=0

  type=rpm

  gpgcheck=1

file modified
+7 -6
@@ -11,7 +11,7 @@ 

  # testing and troubleshooting for development packages in conjunction

  # with new test releases.

  #

- # More information is available at http://fedoraproject.org/wiki/Testing 

+ # More information is available at http://fedoraproject.org/wiki/Testing

  #

  # Reproducible and reportable issues should be filed at

  # http://bugzilla.redhat.com/.
@@ -23,13 +23,13 @@ 

  name=Fedora - Modular Rawhide - Developmental packages for the next Fedora release

  #baseurl=http://download.example/pub/fedora/linux/development/rawhide/Modular/$basearch/os/

  metalink=https://mirrors.fedoraproject.org/metalink?repo=rawhide-modular&arch=$basearch

- enabled=1

+ enabled=AUTO_VALUE

  countme=1

  metadata_expire=6h

  repo_gpgcheck=0

  type=rpm

  gpgcheck=1

- gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch

+ gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch AUTO_VALUE

  skip_if_unavailable=False

  

  [rawhide-modular-debuginfo]
@@ -37,10 +37,11 @@ 

  #baseurl=http://download.example/pub/fedora/linux/development/rawhide/Modular/$basearch/debug/tree/

  metalink=https://mirrors.fedoraproject.org/metalink?repo=rawhide-modular-debug&arch=$basearch

  enabled=0

+ metadata_expire=6h

  repo_gpgcheck=0

  type=rpm

  gpgcheck=1

- gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch

+ gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch AUTO_VALUE

  skip_if_unavailable=False

  

  [rawhide-modular-source]
@@ -48,9 +49,9 @@ 

  #baseurl=http://download.example/pub/fedora/linux/development/rawhide/Modular/source/tree/

  metalink=https://mirrors.fedoraproject.org/metalink?repo=rawhide-modular-source&arch=$basearch

  enabled=0

+ metadata_expire=6h

  repo_gpgcheck=0

  type=rpm

  gpgcheck=1

- gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch

+ gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch AUTO_VALUE

  skip_if_unavailable=False

- 

file modified
+7 -6
@@ -11,7 +11,7 @@ 

  # testing and troubleshooting for development packages in conjunction

  # with new test releases.

  #

- # More information is available at http://fedoraproject.org/wiki/Testing 

+ # More information is available at http://fedoraproject.org/wiki/Testing

  #

  # Reproducible and reportable issues should be filed at

  # http://bugzilla.redhat.com/.
@@ -23,13 +23,13 @@ 

  name=Fedora - Rawhide - Developmental packages for the next Fedora release

  #baseurl=http://download.example/pub/fedora/linux/development/rawhide//Everything/$basearch/os/

  metalink=https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=$basearch

- enabled=1

+ enabled=AUTO_VALUE

  countme=1

  metadata_expire=6h

  repo_gpgcheck=0

  type=rpm

  gpgcheck=1

- gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch

+ gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch AUTO_VALUE

  skip_if_unavailable=False

  

  [rawhide-debuginfo]
@@ -37,10 +37,11 @@ 

  #baseurl=http://download.example/pub/fedora/linux/development/rawhide/Everything/$basearch/debug/tree/

  metalink=https://mirrors.fedoraproject.org/metalink?repo=rawhide-debug&arch=$basearch

  enabled=0

+ metadata_expire=6h

  repo_gpgcheck=0

  type=rpm

  gpgcheck=1

- gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch

+ gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch AUTO_VALUE

  skip_if_unavailable=False

  

  [rawhide-source]
@@ -48,9 +49,9 @@ 

  #baseurl=http://download.example/pub/fedora/linux/development/rawhide/Everything/source/tree/

  metalink=https://mirrors.fedoraproject.org/metalink?repo=rawhide-source&arch=$basearch

  enabled=0

+ metadata_expire=6h

  repo_gpgcheck=0

  type=rpm

  gpgcheck=1

- gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch

+ gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch AUTO_VALUE

  skip_if_unavailable=False

- 

file modified
+109 -18
@@ -1,4 +1,5 @@ 

  %global rawhide_release 35

+ %global updates_testing_enabled 0

  

  Summary:        Fedora package repositories

  Name:           fedora-repos
@@ -106,6 +107,9 @@ 

  Requires:       fedora-repos = %{version}-%{release}

  Obsoletes:      fedora-repos-rawhide < 33-0.7

  

+ %description rawhide

+ This package provides the rawhide repo definitions.

+ 

  %package archive

  Summary:        Fedora updates archive package repository

  Requires:       fedora-repos = %{version}-%{release}
@@ -115,9 +119,6 @@ 

  It is a package repository that contains any RPM that has made it to

  stable in Bodhi and been available in the Fedora updates repo in the past.

  

- %description rawhide

- This package provides the rawhide repo definitions.

- 

  %package rawhide-modular

  Summary:        Rawhide modular repo definitions

  Requires:       fedora-repos = %{version}-%{release}
@@ -187,20 +188,51 @@ 

  ln -s RPM-GPG-KEY-fedora-%{version}-primary RPM-GPG-KEY-%{version}-fedora

  popd

  

+ # Install repo files

+ install -d -m 755 $RPM_BUILD_ROOT/etc/yum.repos.d

+ for file in %{_sourcedir}/fedora*repo ; do

+   install -m 644 $file $RPM_BUILD_ROOT/etc/yum.repos.d

+ done

+ 

+ # Enable or disable repos based on current release cycle state.

+ %if %{rawhide_release} == %{version}

+ rawhide_enabled=1

+ stable_enabled=0

+ testing_enabled=0

+ %else

+ rawhide_enabled=0

+ stable_enabled=1

+ testing_enabled=%{updates_testing_enabled}

+ %endif

+ for repo in $RPM_BUILD_ROOT/etc/yum.repos.d/fedora-{rawhide,eln}*.repo; do

+     sed -i "s/^enabled=AUTO_VALUE$/enabled=${rawhide_enabled}/" $repo || exit 1

+ done

+ for repo in $RPM_BUILD_ROOT/etc/yum.repos.d/fedora{,-modular,-updates,-updates-modular}.repo; do

+     sed -i "s/^enabled=AUTO_VALUE$/enabled=${stable_enabled}/" $repo || exit 1

+ done

+ for repo in $RPM_BUILD_ROOT/etc/yum.repos.d/fedora-updates-testing{,-modular}.repo; do

+     sed -i "s/^enabled=AUTO_VALUE$/enabled=${testing_enabled}/" $repo || exit 1

+ done

+ 

  # Adjust Rawhide repo files to include Rawhide+1 GPG key.

  # This is necessary for the period when Rawhide gets bumped to N+1 and packages

  # start to be signed with a newer key. Without having the key specified in the

  # repo file, the system would consider the new packages as untrusted.

  rawhide_next=$((%{rawhide_release}+1))

- for repo in %{_sourcedir}/fedora-rawhide*.repo; do

-     sed -ir "s@^gpgkey=.*@& file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-${rawhide_next}-\$basearch@" \

+ for repo in $RPM_BUILD_ROOT/etc/yum.repos.d/fedora-rawhide*.repo; do

+     sed -i "/^gpgkey=/ s@AUTO_VALUE@file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-${rawhide_next}-\$basearch@" \

          $repo || exit 1

  done

  

- # Install repo files

- install -d -m 755 $RPM_BUILD_ROOT/etc/yum.repos.d

- for file in %{_sourcedir}/fedora*repo ; do

-   install -m 644 $file $RPM_BUILD_ROOT/etc/yum.repos.d

+ # Set appropriate metadata_expire in base repo files (6h before Final, 7d after)

+ %if "%{release}" < "1"

+ expire_value='6h'

+ %else

+ expire_value='7d'

+ %endif

+ for repo in $RPM_BUILD_ROOT/etc/yum.repos.d/fedora{,-modular}.repo; do

+     sed -i "/^metadata_expire=/ s/AUTO_VALUE/${expire_value}/" \

+         $repo || exit 1

  done

  

  # Install ostree remote config
@@ -210,23 +242,82 @@ 

  

  

  %check

- # assert all rawhide/eln repos are set to enabled only when this is rawhide

- for repo in $RPM_BUILD_ROOT/etc/yum.repos.d/fedora-{rawhide,eln}*.repo; do

-   %if %{rawhide_release} == %{version}

-     grep 'enabled=1' $repo

-   %else

-     grep 'enabled=1' $repo && exit 1 || :

-   %endif

+ # Make sure all repo variables were substituted

+ for repo in $RPM_BUILD_ROOT/etc/yum.repos.d/*.repo; do

+     if grep -q AUTO_VALUE $repo; then

+         echo "ERROR: Repo $repo contains an unsubstituted placeholder value"

+         exit 1

+     fi

+ done

+ 

+ # Make sure correct repos were enabled/disabled

+ enabled_repos=(fedora-cisco-openh264)

+ disabled_repos=(fedora-updates-archive)

+ %if %{rawhide_release} == %{version}

+ enabled_repos+=(fedora-rawhide fedora-rawhide-modular fedora-eln)

+ disabled_repos+=(fedora fedora-modular fedora-updates fedora-updates-modular \

+   fedora-updates-testing fedora-updates-testing-modular)

+ %else

+ enabled_repos+=(fedora fedora-modular fedora-updates fedora-updates-modular)

+ disabled_repos+=(fedora-rawhide fedora-rawhide-modular fedora-eln)

+ %if %{updates_testing_enabled}

+ enabled_repos+=(fedora-updates-testing fedora-updates-testing-modular)

+ %else

+ disabled_repos+=(fedora-updates-testing fedora-updates-testing-modular)

+ %endif

+ %endif

+ 

+ for repo in ${enabled_repos[@]}; do

+     if ! grep -q 'enabled=1' $RPM_BUILD_ROOT/etc/yum.repos.d/${repo}.repo; then

+         echo "ERROR: Repo $repo should have been enabled, but it isn't"

+         exit 1

+     fi

+ done

+ 

+ for repo in ${disabled_repos[@]}; do

+     if grep -q 'enabled=1' $RPM_BUILD_ROOT/etc/yum.repos.d/${repo}.repo; then

+         echo "ERROR: Repo $repo should have been disabled, but it isn't"

+         exit 1

+     fi

+ done

+ 

+ # Make sure updates-testing is not enabled in a Final (stable) release

+ %if "%{release}" >= "1"

+ for repo in $RPM_BUILD_ROOT/etc/yum.repos.d/fedora-updates-testing{,-modular}.repo; do

+     if grep -q 'enabled=1' $repo; then

+         echo "ERROR: Repo $repo should be disabled in a stable release, but it isn't"

+         exit 1

+     fi

+ done

+ %endif

+ 

+ # Make sure metadata_expire was correctly set

+ %if "%{release}" < "1"

+ expire_value='6h'

+ %else

+ expire_value='7d'

+ %endif

+ for repo in $RPM_BUILD_ROOT/etc/yum.repos.d/fedora{,-modular}.repo; do

+     lines=$(grep '^metadata_expire=' $repo | sort | uniq)

+     if [ "$(echo "$lines" | wc -l)" -ne 1 ]; then

+         echo "ERROR: Non-matching metadata_expire lines in $repo: $lines"

+         exit 1

+     fi

+     if test "$lines" != "metadata_expire=${expire_value}"; then

+         echo "ERROR: Wrong metadata_expire value in $repo: $lines"

+         exit 1

+     fi

  done

  

- # make sure the Rawhide+1 key wasn't forgotten to be created

+ # Make sure the Rawhide+1 key wasn't forgotten to be created

  rawhide_next=$((%{rawhide_release}+1))

+ test -n "$rawhide_next" || exit 1

  if ! test -f $RPM_BUILD_ROOT/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-${rawhide_next}-primary; then

      echo "ERROR: GPG key for Fedora ${rawhide_next} is not present"

      exit 1

  fi

  

- # make sure the Rawhide+1 key is present in Rawhide repo files

+ # Make sure the Rawhide+1 key is present in Rawhide repo files

  for repo in $RPM_BUILD_ROOT/etc/yum.repos.d/fedora-rawhide*.repo; do

      gpg_lines=$(grep '^gpgkey=' $repo)

      if test -z "$gpg_lines"; then

file modified
+1 -1
@@ -2,7 +2,7 @@ 

  name=Fedora Modular $releasever - $basearch - Updates

  #baseurl=http://download.example/pub/fedora/linux/updates/$releasever/Modular/$basearch/

  metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-released-modular-f$releasever&arch=$basearch

- enabled=0

+ enabled=AUTO_VALUE

  countme=1

  repo_gpgcheck=0

  type=rpm

@@ -2,7 +2,7 @@ 

  name=Fedora Modular $releasever - $basearch - Test Updates

  #baseurl=http://download.example/pub/fedora/linux/updates/testing/$releasever/Modular/$basearch/

  metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-testing-modular-f$releasever&arch=$basearch

- enabled=0

+ enabled=AUTO_VALUE

  countme=1

  repo_gpgcheck=0

  type=rpm

file modified
+1 -1
@@ -2,7 +2,7 @@ 

  name=Fedora $releasever - $basearch - Test Updates

  #baseurl=http://download.example/pub/fedora/linux/updates/testing/$releasever/Everything/$basearch/

  metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-testing-f$releasever&arch=$basearch

- enabled=0

+ enabled=AUTO_VALUE

  countme=1

  repo_gpgcheck=0

  type=rpm

file modified
+1 -1
@@ -2,7 +2,7 @@ 

  name=Fedora $releasever - $basearch - Updates

  #baseurl=http://download.example/pub/fedora/linux/updates/$releasever/Everything/$basearch/

  metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-released-f$releasever&arch=$basearch

- enabled=0

+ enabled=AUTO_VALUE

  countme=1

  repo_gpgcheck=0

  type=rpm

file modified
+4 -4
@@ -2,9 +2,9 @@ 

  name=Fedora $releasever - $basearch

  #baseurl=http://download.example/pub/fedora/linux/releases/$releasever/Everything/$basearch/os/

  metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=$basearch

- enabled=0

+ enabled=AUTO_VALUE

  countme=1

- #metadata_expire=7d

+ metadata_expire=AUTO_VALUE

  repo_gpgcheck=0

  type=rpm

  gpgcheck=1
@@ -16,7 +16,7 @@ 

  #baseurl=http://download.example/pub/fedora/linux/releases/$releasever/Everything/$basearch/debug/tree/

  metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-debug-$releasever&arch=$basearch

  enabled=0

- metadata_expire=7d

+ metadata_expire=AUTO_VALUE

  repo_gpgcheck=0

  type=rpm

  gpgcheck=1
@@ -28,7 +28,7 @@ 

  #baseurl=http://download.example/pub/fedora/linux/releases/$releasever/Everything/source/tree/

  metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-source-$releasever&arch=$basearch

  enabled=0

- metadata_expire=7d

+ metadata_expire=AUTO_VALUE

  repo_gpgcheck=0

  type=rpm

  gpgcheck=1

Instead of performing error-prone manual changes in .repo files several times
per cycle, this patchset brings fully automated handling of such edits. All the
releng person needs to do is to set rawhide_release and
updates_testing_enabled variables (and of course adjust Version and
Release), and all the repos will be auto-magically populated with the correct
values during build (which includes enabled=0/1 and metadata_expire=6h/7d).
The intention is to avoid human errors which inevitably happen (an example [1]).

This means:

  • Rawhide/ELN repo files will get enabled for Rawhide builds, disabled otherwise.
  • Standard repo files will get enabled for non-Rawhide builds, disabled otherwise.
  • Updates-testing repo will get enabled per specified configuration (ignored on
    Rawhide).
  • Base repo files will have short metadata expiration during development period
    ("Branched"), long expiration during stable period.

Everything is covered with tests, to prevent humans and automatons from doing
mistakes. That covers the automated changes to repo files, but also e.g. ensuring
that updates-testing is not enabled in a Final release by mistake.

Coupled with this changeset is an adjustment to certain repo files to make sure
all the sections defined in a single repo file have the same value
metadata_expire=, as it makes sense. And also in the spec file the rawhide
subpackage description was moved next to the subpackage definition, they were
split by mistake in the past, it seems.

If this gets merged, it will massively simplify Releng's Mass Branching SOP in
the future [2].

[1] https://src.fedoraproject.org/rpms/fedora-repos/c/08819dbf9428d57eedbe5cd978b516f995bb8b6a?branch=f34
[2] https://docs.pagure.org/releng/sop_mass_branching.html#fedora-repos

Build succeeded.

Pull-Request has been merged by humaton

a month ago