#2 Merge go-srpm-macros and add the automation described in https://fedoraproject.org/wiki/More_Go_packaging
Closed 6 years ago by jchaloup. Opened 6 years ago by nim.
rpms/ nim/go-compilers more-go-packaging  into  master

file modified
+113 -44
@@ -1,76 +1,145 @@ 

- Name:           go-compilers

- Version:        1

- Release:        17%{?dist}

- Summary:        Go language compilers for various architectures

- Group:          Development/Tools

- License:        GPLv3+

- Source0:        macros.golang-compiler

- Source1:        macros.gcc-go-compiler

+ Name:    go-compilers

+ Version: 3

+ Release: 0.16%{?dist}

+ Summary: rpm automation to create Go language (golang) packages

+ License: GPLv3+

+ 

+ Source0: macros.go-srpm

+ Source1: macros.go-rpm

+ 

+ # Read Go architecture definitions

+ %if 0%{?fedora} || 0%{?rhel} > 7

+ %{?load:%{SOURCE0}}

+ %else

+ # This file contains Go architecture definitions which are also used here

+ # Therefore, changing those requires a double rebuild

+ BuildRequires: go-srpm-macros

+ %endif

  

- ExclusiveArch:  %{go_arches}

+ # Autodeps

+ Source10: go.attr

+ Source11: go.prov

+ Source12: go.req

  

- # for install, cut and rm commands

- BuildRequires:  coreutils

- # for go specific macros

- BuildRequires:  go-srpm-macros

+ # Compiler definitions

+ Source20: macros.go-compilers-golang

+ Source21: macros.go-compilers-gcc

+ 

+ ExclusiveArch:  %{go_arches}

  

  %description

- The package provides correct golang language compiler

- base on an architectures.

+ This package provides rpm automation to simplify the creation of Go language

+ (golang) packages on various architectures:

+  — SRPM-stage macros

+  — Build-stage rpm macros

+  — Go rpm autodeps (provides and requires)

+  — correct Go compiler definitions for each supported architecture

+ 

+ %package -n go-srpm-macros

+ Summary:   SRPM-stage rpm automation for Go packages

+ BuildArch: noarch

+ Requires:  redhat-rpm-config >= 73

+ 

+ %description -n go-srpm-macros

+ This package provides SRPM-stage rpm automation to simplify the creation of Go

+ language (golang) packages on various architectures.

+ 

+ It limits itself to the automation subset required to create Go SRPM packages

+ and needs to be included in the default build root.

+ 

+ The rest of the automation is provided by the go-rpm-macros package, that

+ go-srpm-macros will pull in for Go packages only.

+ 

+ %package -n go-rpm-macros

+ Summary:   Build-stage rpm automation for Go packages

+ BuildArch: noarch

+ Requires:  lua-argparse

+ Requires:  compiler(go-compiler)

+ Requires:  go-srpm-macros = %{version}-%{release}

+ 

+ %description -n go-rpm-macros

+ This package provides build-stage rpm automation to simplify the creation of Go

+ language (golang) packages on various architectures.

+ 

+ It does not need to be included in the default build root: go-srpm-macros will

+ pull it in for Go packages only.

  

  %ifarch %{golang_arches}

- %package golang-compiler

- Summary:       compiler for golang

+ %package golang

+ Summary:   The reference implementation of the Go language compiler

+ Requires:  golang

+ Provides:  compiler(go-compiler) = 2

+ Provides:  compiler(golang)

+ Provides:  go-compilers-golang-compiler = %{version}-%{release}

+ Obsoletes: go-compilers-golang-compiler < %{version}-%{release}

  

- Requires:      golang

+ %description golang

+ The reference implementation of the Go language (golang) compiler.

  

- Provides:      compiler(go-compiler) = 2

- Provides:      compiler(golang)

- 

- %description golang-compiler

- Compiler for golang.

  %endif

  

  %ifarch %{gccgo_arches}

- %package gcc-go-compiler

- Summary:       compiler for gcc-go

- 

+ %package gcc

+ Summary:   The gcc implementation of the Go language compiler

  # GCC>=5 holds in Fedora now

- Requires:      gcc-go

- 

- Provides:      compiler(go-compiler) = 1

- Provides:      compiler(gcc-go)

- 

- %description gcc-go-compiler

- Compiler for gcc-go.

+ Requires:  gcc-go

+ Provides:  compiler(go-compiler) = 1

+ Provides:  compiler(gcc-go)

+ Provides:  go-compilers-gcc-go-compiler = %{version}-%{release}

+ Obsoletes: go-compilers-gcc-go-compiler < %{version}-%{release}

+ 

+ %description gcc

+ The gcc (gcc-go) implementation of the Go language (golang) compiler.

  %endif

  

- %prep

- # nothing to prep, just for hooks

+ %install

+ install -m 0755 -vd %{buildroot}%{_rpmconfigdir}/{macros.d,fileattrs}/

+ 

+ install -m 0644 -vp %{SOURCE0} %{SOURCE1} \

+                     %{buildroot}%{_rpmconfigdir}/macros.d/

  

- %build

- # nothing to build, just for hooks

+ install -m 0644 -vp %{SOURCE10} \

+                     %{buildroot}%{_rpmconfigdir}/fileattrs/

+ install -m 0755 -vp %{SOURCE11} %{SOURCE12} \

+                     %{buildroot}%{_rpmconfigdir}/

  

- %install

  %ifarch %{golang_arches}

- install -m 644 -D %{SOURCE0} %{buildroot}%{_rpmconfigdir}/macros.d/macros.golang-compiler

+ install -m 0644 -vp %{SOURCE20} \

+                     %{buildroot}%{_rpmconfigdir}/macros.d/

  %endif

  

  %ifarch %{gccgo_arches}

- install -m 644 -D %{SOURCE1} %{buildroot}%{_rpmconfigdir}/macros.d/macros.gcc-go-compiler

+ install -m 0644 -vp %{SOURCE21} \

+                     %{buildroot}%{_rpmconfigdir}/macros.d/

  %endif

  

+ %files -n go-srpm-macros

+ %{_rpmconfigdir}/macros.d/macros.go-srpm

+ 

+ %files -n go-rpm-macros

+ %{_rpmconfigdir}/macros.d/macros.go-rpm

+ %{_rpmconfigdir}/fileattrs/go.attr

+ %{_rpmconfigdir}/go.prov

+ %{_rpmconfigdir}/go.req

+ 

  %ifarch %{golang_arches}

- %files golang-compiler

- %{_rpmconfigdir}/macros.d/macros.golang-compiler

+ %files golang

+ %{_rpmconfigdir}/macros.d/macros.go-compilers-golang

  %endif

  

  %ifarch %{gccgo_arches}

- %files gcc-go-compiler

- %{_rpmconfigdir}/macros.d/macros.gcc-go-compiler

+ %files gcc

+ %{_rpmconfigdir}/macros.d/macros.go-compilers-gcc

  %endif

  

  %changelog

+ * Thu Feb 15 2018 Nicolas Mailhot <nim@fedoraproject.org> - 3-0.16

+ - merge go-srpm-macros and go-compilers to reduce the number of Go rpm macro

+   packages and make sure all the macros are kept in sync

+ - simplify subpackage naming

+ - add more Go packaging automation, including autodeps:

+   https://fedoraproject.org/wiki/More_Go_packaging

+ 

  * Wed Feb 07 2018 Fedora Release Engineering <releng@fedoraproject.org> - 1-17

  - Rebuilt for https://fedoraproject.org/wiki/Fedora_28_Mass_Rebuild

  

file added
+5
@@ -0,0 +1,5 @@ 

+ %__go_provides          %{_rpmconfigdir}/go.prov --goipath "%{?goipath}" --prefix "%{buildroot}" --gopath "%{gopath}" --version "%{?epoch:%{epoch}:}%{version}-%{release}" %{?commit:-a "(commit=%{commit})"} %{?branch:-a "(branch=%{branch})"} %{?tag:-a "(tag=%{tag})"}

+ %__go_requires          %{_rpmconfigdir}/go.req  --goipath "%{?goipath}" --prefix "%{buildroot}" --gopath "%{gopath}" --version "%{?epoch:%{epoch}:}%{version}-%{release}"

So basically both macros get invoked in the %files section of each subpackage? How the rpm knows it needs to call the %__go_provides? I read http://packaging.fedoraproject.narkive.com/qzTmfbYH/fedora-packaging-creating-an-autoprovides-files and http://rpm.org/user_doc/dependency_generators.html but still don't get it properly. What if the %{_fileattrsdir} contains go.attr, perl.attr and python.attr? How does the rpm knows the the go.attr is the right one? Is there any mechanism to say use go for the NAME?

EDIT: I see, the %__go_path must be set as well.

+ %__go_path              ^%{gopath}/src/.+/.*$

+ %__go_magic             ^((.*, )?directory|(broken )?symbolic link to .*)$

+ %__go_flags             magic_and_path

file added
+185
@@ -0,0 +1,185 @@ 

+ #!/bin/bash

+ 

+ usage() {

+ cat >&2 << EOF_USAGE

+ Usage: $0 -g <go import path> [-p <prefix>] [-G <go path>] [-v <version string>] [-a <attributes>]

+ <go import path>   the Go import path of the target package

+ <prefix>:          an eventual prefix path such as %{buildroot}

+ <go path>:         the root of the Go source tree

+ <version string>:  tag the provides with the <version string>

+ <attributes>:      attributes to add to the provides, for example

+                    (commit=XXXX)(branch=YYYY) or

+                    (vermod=alpha1)

+ 

+ Example:

+ echo filename | $0 -g golang.org/x/net -G /usr/share/gocode -v 2.5-0.1

+ echo filename | $0 -g golang.org/x/net -a '(commit=XXXX)(branch=YYYY)'

+ echo filename | $0 -g golang.org/x/net -a '(vermod=alpha1)'

+ EOF_USAGE

+ exit 1

+ }

+ 

+ if ! options=$(getopt -n $0 -o hg:p:G:v:a: -l help,goipath:,prefix:,gopath:,version:,attr:,attribute:,attributes: -- "$@")

+ then

+     # something went wrong, getopt will put out an error message for us

+     exit 2

+ fi

+ 

+ eval set -- "$options"

+ 

+ # Convert paths within gopath to version-constrained provides

+ # Never call it before checking the path belongs to the package import path

+ provides() {

+   local ppath="$1"

+   # Check for a vendor root, when not processing a symlink

+   if ! ( [[ -L "${ppath}" ]] || ( notvendored "${ppath}" ) ) ; then

+     local vroot="$(realpath -sm "${ppath}")/"

+     vroot="${vroot%/vendor/*}/vendor"

+     if [[ "${vroot}" != "${ppath}" ]] ; then

+       # Check if a parent with files exists within the vendor root

+       local tip="${ppath}"

+       local vipath=/dev/null

+       until [[ "${tip}" == "${vroot}" || "${ppath}" == "${vipath}/"* ]] ; do

+         [[ -n $(find "$tip" -maxdepth 1 -type f) ]] && vipath="$tip"

+         tip=$(dirname "$tip")

+       done

+       # If we didn't find a parent with files other than the path itself, it is

+       # the vendored import path root!

+       if [[ "$vipath" == "$ppath" ]] ; then

+         echo "bundled(golang(${ppath##*/vendor/}))"

+       fi

+     fi

+   else

+     local package="${ppath#${root}/src/}"

+     for index in "${!deco[@]}" ; do

+       echo "golang(${package})${deco[index]}${version}"

+     done

+   fi

+ }

+ 

+ # Resolve a symlink target in presence of a build root

+ resolvelink() {

+   local lt=$(realpath -m "$1")

+   echo "${prefix}${lt#${prefix}}"

+ }

+ 

+ # Resolve a symlink to its ultimate target in presence of a build root

+ ultimateresolvelink() {

+   local lt="$1"

+   until [[ ! -L ${lt} ]] ; do

+     lt=$(resolvelink "${lt}")

+   done

+   echo "${lt}"

+ }

+ 

+ # Test if a path is a directory within the package import path.

+ isgoipathdir() {

+   local lt="$1"

+   if [[ -d ${lt} ]] && [[ "${lt}"/ == "${root}/src/${goipath}"/* ]] ; then

+     true

+   else

+     false

+   fi

+ }

+ 

+ # Test if a path is vendored

+ notvendored() {

+   local p=$(realpath -sm "$1")

+   local vr="${p}/"

+   vr="${vr%/vendor/*}"

+   if [[ "${vr}" != "${p}/" ]] ; then

+     false

+   else

+     true

+   fi

+ }

+ 

+ # A symlink can point to a whole directory tree, but go.attr will only

+ # trigger on the root symlink.

+ # Therefore, check the symlink points within the processed import path, then

+ # walk all the target tree to generate symlink provides

+ #

+ # To process nested symlinks the function needs to be provided a working path

+ # to the symlink tip within the build root as second argument.

+ processlink() {

+   local link="$1"

+   local linktarget=$(ultimateresolvelink "$2")

+   if ( isgoipathdir "${linktarget}" ) && ( notvendored "${linktarget}" ) ; then

+     find "${linktarget}" -type d -print | while read subdir ; do

+       provides    "${link}${subdir#${linktarget}}"

+     done

+     find "${linktarget}" -type l -print | while read sublink ; do

+       processlink "${link}${sublink#${linktarget}}" "${sublink}"

+     done

+   fi

+ }

+ 

+ version=''

+ prefix=''

+ gopath=/usr/share/gocode

+ declare -A attributes

+ 

+ while [ $# -gt 0 ] ; do

+   case $1 in

+     -h|--help)    usage ;;

+     -g|--goipath) goipath="$2" ; shift;;

+     -p|--prefix)  prefix=$(realpath -sm "$2") ; shift;;

+     -G|--gopath)  gopath="$2" ; shift;;

+     -v|--version) version=" = $2" ; shift;;

+     -a|--attr|--attribute|--attributes)

+                   IFS=')' read -r -a newattrs <<< "$2"

+                   for index in "${!newattrs[@]}" ; do

+                     newattrs[index]=${newattrs[index]#\(}

+                     attributes[${newattrs[index]%%=*}]=${newattrs[index]#*=}

+                   done ; shift;;

+     (--)          shift; break;;

+     (-*)          echo "$0: error - unrecognized option $1" >&2; exit 3;;

+     (*)           break;;

+   esac

+   shift

+ done

+ 

+ # Detection of the package import path root is necessary to insure we do not

+ # generate useless provides such as golang(github).

+ #

+ # When go dep stabilizes and can be generalized import path root detection may

+ # be automated by searching for the first level of go dep lock files.

+ #

+ # However the requirement to set %{goipath} may remain to make sure we avoid

+ # processing directories created without applying the sanity checks documented

+ # in Fedora Packaging Guidelines.

+ [[ -z ${goipath} ]] && exit 0

+ 

+ root="${prefix}${gopath}"

+ 

+ deco=( "" )

+ # Some filtering may be needed in the future

+ for key in "${!attributes[@]}"; do

+   [ -n "${attributes[$key]}" ] && deco+=( "($key=${attributes[$key]})" )

+ done

+ 

+ # go.attr ensures that every time a package declares owning a directory or

+ # symlink under %{gopath}/src, the directory or symlink name will be piped to

+ # this script to compute the package Go provides.

+ #

+ # For legacy reason the script is supposed to be able to handle multiple

+ # inputs, even though modern rpm invokes it separately for each directory.

+ #

+ # As native Go tooling is focused on leaf developer needs (“what I need from

+ # others to build my code”, not “what I provide to help others build their

+ # code”) Go auto-provides do not invoke any Go binary and just record all the

+ # directories created under the import path root. This is actually faster and

+ # more reliable, since Go project seem to import pretty much any import path

+ # subdirectory, even if it contains assets without any .go file.

Maybe it is a silly question but does it make sense to import a go package that has no *.go file?

+ while read dir ; do

+   if [[ -L $dir ]] ; then

+     processlink "$dir" "$dir"

+   else

+     # If go.attr triggered on a normal directory just make sure it is under the

+     # package import path, then provide it

+     if [[ -d $dir ]] ; then

+       dir=$(realpath -sm "$dir")

+       [[ "${dir}"/ == "${root}/src/${goipath}"/* ]] && provides "$dir"

+     fi

+   fi

+ done | sort | uniq | grep -v '/\([._]\)\?\(internal\|vendor\)\([/)]\)'

file added
+160
@@ -0,0 +1,160 @@ 

+ #!/bin/bash

+ 

+ usage() {

+ cat >&2 << EOF_USAGE

+ Usage: $0 -g <go import path> [-p <prefix>] [-G <go path>] [-v <version string>]

+ <go import path>   the Go import path of the target package

+ <prefix>:          an eventual prefix path such as %{buildroot}

+ <go path>:         the root of the Go source tree

+ <version string>:  tag symlinked renamings with <version string>

+ 

+ Example:

+ echo filename | $0 -g golang.org/x/net -G /usr/share/gocode

+ EOF_USAGE

+ exit 1

+ }

+ 

+ if ! options=$(getopt -n $0 -o hg:p:G:v: -l help,goipath:,prefix:,gopath:,version: -- "$@")

+ then

+     # something went wrong, getopt will put out an error message for us

+     exit 2

+ fi

+ 

+ eval set -- "$options"

+ 

+ # Convert paths within gopath to version-constrained requires

+ # Never call it before checking the path belongs to the package import path

+ requires() {

+   local package="${1#${root}/src/}"

+   echo "golang(${package})${version}"

+ }

+ 

+ # Resolve a symlink target in presence of a build root

Wondering if it would be simpler to implement all the symlink resolving logic in Go. Why is the build root problematic wrt. symlinks?

I still wonder if it is the bash itself that makes the implementation hard to read or the complexity of the symlink processing itself.

+ resolvelink() {

+   local lt=$(realpath -m "$1")

+   echo "${prefix}${lt#${prefix}}"

+ }

+ 

+ # Resolve a symlink to its ultimate target in presence of a build root

+ ultimateresolvelink() {

+   local lt="$1"

+   until [[ ! -L ${lt} ]] ; do

+     lt=$(resolvelink "${lt}")

+   done

+   echo "${lt}"

+ }

+ 

+ # Test if a path is a directory within the package import path.

+ isgoipathdir() {

+   local lt="$1"

+   if [[ -d ${lt} ]] && [[ "${lt}"/ == "${root}/src/${goipath}"/* ]] ; then

+     true

+   else

+     false

+   fi

+ }

+ 

+ # Test if a path is vendored

+ notvendored() {

+   local p=$(realpath -sm "$1")

+   local vr="${p}/"

+   vr="${vr%/vendor/*}"

+   if [[ "${vr}" != "${p}/" ]] ; then

+     false

+   else

+     true

+   fi

+ }

+ 

+ # A symlink can point to a whole directory tree, but go.attr will only

+ # trigger on the root symlink.

Where do the symlinks come from? Are they created by the rpm itself?

+ # Therefore, check the symlink points within the processed import path, then

+ # walk all the target tree to generate symlink requires

+ #

+ # Requires is not recursive — it relies on providers requiring their own needs

+ processlink() {

+   local link="$1"

+   local nexttarget=$(resolvelink "$1")

+   local linktarget=$(ultimateresolvelink "${nexttarget}")

+   if ( isgoipathdir "${linktarget}" ) && ( notvendored "${linktarget}" ) ; then

+     find "${nexttarget}" -type d -print | while read subdir ; do

+       requires   "${subdir}"

+     done

+     find "${nexttarget}" -type l -print | while read sublink ; do

+       if isgoipathdir $(ultimateresolvelink "${sublink}") ; then

+         # The owner of the link will declare the correct requires

+         requires "${sublink}"

+       fi

+     done

+   fi

+ }

+ 

+ prefix=''

+ gopath=/usr/share/gocode

+ 

+ while [ $# -gt 0 ] ; do

+   case $1 in

+     -h|--help)    usage ;;

+     -g|--goipath) goipath="$2" ; shift;;

+     -p|--prefix)  prefix=$(realpath -sm "$2") ; shift;;

+     -G|--gopath)  gopath="$2" ; shift;;

+     -v|--version) version=" = $2" ; shift;;

+     (--)          shift; break;;

+     (-*)          echo "$0: error - unrecognized option $1" >&2; exit 3;;

+     (*)           break;;

+   esac

+   shift

+ done

+ 

+ # Detection of the package import path root is necessary when processing

+ # symlinks. We can only be sure the requires of a symlink target are correctly

+ # computed, if it is processed here as part of the package import path tree.

+ #

+ # When go dep stabilizes and can be generalized import path root detection may

+ # be automated by searching for the first level of go dep lock files.

+ #

+ # However the requirement to set %{goipath} may remain to make sure we avoid

+ # processing directories created without applying the sanity checks documented

+ # in Fedora Packaging Guidelines.

We should require at least some minimal metadata that does not need tarball processing. In order to allow simple analysis of a spec file. E.g. a bot analyzing all spec files in the distribution collecting project provider prefix, commit/revision and the import path prefix. E.g. for discovery reasons to be able to detect if a given import path is packaged in the distribution.

I have a tooling that does that and allows me to detect if a go project/dependency needs to be updated or not. For that I expecting %provider_prefix, %commit and %import_path to exist in each spec file. So at least for that reason I would require the goipath to be set manually,

At the end I don't mind using the goipath instead of the import_path. As long as it is clear it means import path prefix of the project.

+ [[ -z ${goipath} ]] && exit 0

+ 

+ root="${prefix}${gopath}"

+ 

+ # go.attr ensures that every time a package declares owning a directory or

+ # symlink under %{gopath}/src, the directory or symlink name will be piped to

+ # this script to compute the package Go requires.

+ #

+ # When go dep stabilizes and can be generalized it will be possible to simplify

+ # requires computation by running go dep on the import path root only. go dep

+ # will also make possible to compute requires version constrains. However,

+ # there will still be a need to trigger require computation on every symlink.

+ #

+ # For legacy reason the script is supposed to be able to handle multiple

+ # inputs, even though modern rpm invokes it separately for each directory.

+ while read dir ; do

+   if [[ -L $dir ]] ; then

+     processlink "$dir"

+   else

+     if [[ -d $dir ]] ; then

+       # When triggered on a directory, make sure it is within the import path,

+       # and not vendored, run go list to get the directory import, then filter

+       # out standard Go imports provided by golang itself.

+       # Invoking go is awfully slow. You *will* fear rpm has hung the first

+       # time that happens.

+       # Hopefully go dep will replace this with something faster soonish.

+       dir=$(realpath -sm "$dir")

+       if notvendored "$dir" ; then

+         package="${dir#${root}/src/}"

+         if $(ls ${dir}/*.go >/dev/null 2>&1) ; then

+           GOPATH=$root go list -e -f \

Let's build the macros on top of https://github.com/gofed/symbols-extractor/pull/137. Once implemented, I will prepare a patch so it can be built as part of the golang package itself as a separate rpm and then required in the go-srpm-macros. Once done, we can replace this entire file with a simple macro that just calls the golist binary.

+                 '{{ join .Imports "\n" }}' "$package" |\

+             while read import ; do

+               GOPATH=$root go list -e -f \

+                 '{{if not .Standard}}{{.ImportPath | printf "golang(%s)\n"}}{{end}}' "$import"

+             done

+         fi

+       fi

+     fi

+   fi

+ done | sort | uniq \

+      | grep -v '\(/\([._]\)\?\(internal\|testdata\|vendor\)\([/)]\)\|golang(C)\)'

macros.go-compilers-gcc macros.gcc-go-compiler
file renamed
file was moved with no change to the file
macros.go-compilers-golang macros.golang-compiler
file renamed
+9 -7
@@ -7,13 +7,15 @@ 

  

  # Define commands for building

  # BUILD_ID can be generated for golang build no matter of debuginfo

- %gobuild(o:) \

- %ifnarch ppc64 \

- go build -buildmode pie -compiler gc -tags=rpm_crashtraceback -ldflags "${LDFLAGS:-} -B 0x$(head -c20 /dev/urandom|od -An -tx1|tr -d ' \\n') -extldflags '%__global_ldflags %{?__golang_extldflags}'" -a -v -x %{?**};\

- %else \

- go build -compiler gc -tags=rpm_crashtraceback -ldflags "${LDFLAGS:-} -B 0x$(head -c20 /dev/urandom|od -An -tx1|tr -d ' \\n') -extldflags '%__global_ldflags %{?__golang_extldflags}'" -a -v -x %{?**};\

- %endif \

+ 

+ %gobuild(o:) %{expand:

+ # https://bugzilla.redhat.com/show_bug.cgi?id=995136#c12

+ %global _dwz_low_mem_die_limit 0

+ %ifnarch ppc64

+ go build -buildmode pie -compiler gc -tags=rpm_crashtraceback -ldflags "${LDFLAGS:-} -B 0x$(head -c20 /dev/urandom|od -An -tx1|tr -d ' \\n') -extldflags '%__global_ldflags %{?__golang_extldflags}'" -a -v -x %{?**};

+ %else

+ go build -compiler gc -tags=rpm_crashtraceback -ldflags "${LDFLAGS:-} -B 0x$(head -c20 /dev/urandom|od -An -tx1|tr -d ' \\n') -extldflags '%__global_ldflags %{?__golang_extldflags}'" -a -v -x %{?**};

+ %endif}

  

  # Define commands for testing

  %gotest() go test -buildmode pie -compiler gc -ldflags "${LDFLAGS:-} -extldflags '%__global_ldflags %{?__golang_extldflags}'" %{?**};

- 

file added
+165
@@ -0,0 +1,165 @@ 

+ # Copyright (c) 2018 Nicolas Mailhot <nim@fedoraproject.org>

+ #

+ # This file is distributed under the terms of GNU GPL license version 3, or

+ # any later version.

+ #

+ # This file contains macros needed at %%build %%install and %%check

+ # stage by Golang packages.

+ # The macros necessary at %%setup and srpm stage are in the sister file

+ # macros.go-srpm

+ 

+ # find directory filter to remove elements usually not needed to build other Go projects

+ %gofinddirfilter  -regextype egrep \! -iregex '.*/(.*[._-])?(example(s)?|test([._-])?data)/.*' \! -ipath '*/vendor/*' \! -iregex  '.*/[._].*'

+ 

+ # find filter to identify resources usually needed to build other Go projects

+ %gofindfilter     -regextype egrep -iregex '(.*\\.(go|c|h|s|tmpl|proto)|./Gopkg\\.(toml|lock))' \! -iname '*_test.go' \! -iregex  '.*/(.*[._-])?test(([._-])?case)?(s)?/.*' %{gofinddirfilter}

+ 

+ # find filter to identify unit tests

+ %gofindtestfilter                  -iname '*.go' %{gofinddirfilter}

+ 

+ # Collect md files spread in subdirectories

+ %gocollectmd %{expand:

+ for mdfile in $(find . -iname '*.md' %{gofinddirfilter}) ; do

+   suffix=$(dirname $mdfile | sed 's+^\./++g' | sed 's+/+·+g')

+   if [[ $suffix != '.' ]] ; then

+     cp -p "$mdfile" "$(echo $(basename $mdfile) | sed 's+\.md$++g')·${suffix}.md"

+   fi

+ done}

+ 

+ # Try to install Go package files in sensible locations, with strict directory

+ # ownership as required by Go autodeps

+ #

+ # %%goinstall takes a list of files as argument.

+ #

+ # %%goinstall will generate a file list that can be used in a %%files spec

+ # section. The default file list name is devel.file-list. It can be overriden

+ # by passing the -f argument to the macro with another filename.

+ #

+ # When invoked several times it will append to existing file lists not create

+ # a new one.

+ #

+ # When invoked several times with different file list names, it will attribute

+ # directories to the first file list that makes use of them only. This is

+ # intentional, to avoid triggering Go autodeps on the same Go directory in

+ # different subpackages. Therefore, splitting code in several subpackages

+ # requires careful though about %%goinstall invocation order.

+ %goinstall(f:) %{expand:

+ %global file_list %{?-f*}%{!?-f*:devel.file-list}

+ install -m 0755 -vd "%{buildroot}%{gopath}/src"

+ for file in %* ; do

+   file="${file#./}"

+   [[ -d "${file}" && ! -L "${file}" ]] && srcdir="${file}" || srcdir=$(dirname "${file}")

+   destdir="%{buildroot}%{gopath}/src/%{goipath}/${srcdir}"

+   destdir="${destdir%/.}"

+   dir="${destdir}"

+   dirs=()

+   while [[ ! -e "${dir}" ]] ; do

+     dirs=("$dir" "${dirs[@]}")

+     dir=$(dirname "${dir}")

+   done

+   for dir in "${dirs[@]}" ; do

+     install -m 0755 -vd "${dir}"

+     if $(echo "${dir}" | grep -q "^%{buildroot}%{gopath}/src/%{goipath}") ; then

+       touch -r             ".${dir#%{buildroot}%{gopath}/src/%{goipath}}" "${dir}"

+     fi

+     echo "%%dir \"${dir#%{buildroot}}\"" >> %{file_list}

+   done

+   if [[ -L "$file" ]] ; then

+     ln -s $(readlink "${file}") "${destdir}/$(basename ${file})"

+     touch -h      -r "${file}"  "${destdir}/$(basename ${file})"

+   fi

+   [[ -f "$file" && ! -L "$file" ]] && install -m 0644 -vp  "${file}" "${destdir}/"

+   [[ -f "$file" ||   -L "$file" ]] && echo "%%{gopath}/src/%%{goipath}/${file}" >> %{file_list} || :

+ done}

+ 

+ # Create a local Go build root

+ # Useful in %%build and %%check

+ %gobuildroot() %{expand:

+   GO_BUILD_PATH="$PWD/_build"

+   %global gobuildpath "$GO_BUILD_PATH"

+   install -m 0755 -vd "$(dirname %{gobuildpath}/src/%{goipath})"

+   ln -fs "$PWD" "%{gobuildpath}/src/%{goipath}"

+   cd "%{gobuildpath}/src/%{goipath}"

+   install -m 0755 -vd _bin

+   export GOPATH="%{gobuildpath}:%{gopath}"

+   export LDFLAGS="${LDFLAGS:-}%{?commit: -X %{goipath}/version.commit=%{commit}}%{?tag: -X %{goipath}/version.tag=%{tag}}%{?version: -X %{goipath}/version=%{version}}"

+ }

+ 

+ # Run %%{gotest} on all subdirectories except for those provided in parameters.

+ # THIS MACRO IS OPT-OUT.

+ # It only allows excluding specific subdirectories from the test run, with the

+ # following syntax :

+ #  — to exclude the tests in a specific subdirectory, not recursively:

+ #    pass it as parameter. Use . for the project root.

+ #    Example:

+ #      %gochecks . subdir1 sub2/dir

+ #  — to exclude the tests in a whole subtree:

+ #    use the -R or --root switch with the subtree root as argument.

+ #    Example:

+ #      %gochecks -R sub1/tree/root -root 'sub2/fixtures'

+ #  — to exclude using free-form egrep-style regexes:

+ #    use the -r or --regex switch with the regex as argument.

+ #    Example:

+ #      %gochecks --regex '.*/(.*[._-])?test(([._-])?case)?(s)?/.*'

+ #  — you can mix and match:

+ #      %gochecks . subdir1 -R sub/tree/root sub/dir2 -r './testcase(s)?/.*'

+ #  — arguments containing spaces or escaped quotes are not supported

+ %gochecks(rotegxR-) %{expand:%{lua:

+ local parameters = {}

+ -- Quite sufficient for our needs, otherwise look at rex.gmatch

+ for p in string.gmatch(rpm.expand("%**"), "%S+") do

+ --  p = string.gsub(p, '"(%S+)"', '%1')

+ --  p = string.gsub(p, "'(%S+)'", '%1')

+   table.insert(parameters, p)

+ end

+ local function notsubpath(path)

+   if not string.match(path, "^%./") then

+     path = "./" .. path

+   end

+   return "\! -regex '" .. path .. "'"

+ end

+ local function regexfilter(regex)

+   return "\! -regex '" .. regex .. "'"

+ end

+ local function rootfilter(root)

+   return notsubpath(root .. "/.*")

+ end

+ local function dirfilter(dir)

+   return notsubpath(dir .. "/[^/]*")

+ end

+ local argparse = require "argparse"

+ local parser = argparse()

+ parser:option   ("-r --regex")

+       :count    ("*")

+       :args     ("1")

+       :convert  (regexfilter)

+ parser:option   ("-R --root")

+       :count    ("*")

+       :args     ("1")

+       :convert  (rootfilter)

+ parser:argument ("dir")

+       :args     ("*")

+       :convert  (dirfilter)

+ local args = parser:parse(parameters)

+ local gofindtestfilter = rpm.expand("%{?gofindtestfilter}")

+ gofindtestfilter = gofindtestfilter .. " -regextype egrep " ..

+                    table.concat(args.regex, " ") .. " " ..

+                    table.concat(args.root, " ")  .. " " ..

+                    table.concat(args.dir, " ")

+ rpm.define("checkfilter " .. gofindtestfilter)

+ }

+ # The following relies on things like environment variable and other macros

+ # easier to use in shell

+ %gobuildroot

+ export PATH=$PATH:%{buildroot}%{_bindir}

+ find . %{checkfilter} -exec dirname '\{\}' \\;|sort|uniq|while read sub ; do

+   set +x

+   sub="${sub#./}"

+   echo "Testing ${sub}…"

+   pushd "%{gobuildpath}/src/%{goipath}/${sub}" >/dev/null

+   set -x

+   %{gotest}

+   set +x

+   popd >/dev/null

+ done

+ }

file added
+155
@@ -0,0 +1,155 @@ 

This clashes with go-srpm-macros package.

+ # Copyright (c) 2015-2018 Jakub Cajka <jcajka@redhat.com>,

+ #                         Jan Chaloupka <jchaloup@redhat.com>,

+ #                         Nicolas Mailhot <nim@fedoraproject.org>

+ # This file is distributed under the terms of GNU GPL license version 3, or

+ # any later version.

+ 

+ # This file contains macros for building projects in golang for packages

+ # with golang compiler or gcc-go compiler based on an architecture.

+ # Golang is primarly for primary architectures, gcc-go for secondary.

+ #

+ # This file provides only macros and must not use any other package except

+ # redhat-rpm-macros.

+ 

+ # Define arches for PA and SA

+ %golang_arches   %{ix86} x86_64 %{arm} aarch64 ppc64le s390x

+ %gccgo_arches    %{mips}

+ %go_arches       %{golang_arches} %{gccgo_arches}

+ 

+ # Where to set GOPATH for builds

+ %gopath          %{_datadir}/gocode

+ 

+ # Define go_compilers macro to signal go-compiler package is available

+ %go_compiler     1

+ 

+ # Sanitize a Go import path that can then serve as rpm package name

+ # Mandatory parameter: a Go import path

+ %gorpmname() %{lua:

+ local goname = rpm.expand("%1")

+ -- add a golang prefix and lowercase

+ goname       = string.lower("golang-" .. goname .. "/")

+ -- replace various separators rpm does not like with -

+ -- special-case x.y.z number-strings as that’s an exception in our naming

+ -- guidelines

+ goname       = string.gsub(goname, "^([^/]+)%.([^%./]+)/", "%1/")

+ goname       = string.gsub(goname, "(%d)%.(%d)",           "%1|%2")

+ goname       = string.gsub(goname, "[%._/%-]+",            "-")

+ -- Tokenize along - separators and remove duplicates to avoid

+ -- golang-foo-foo-bar-foo names

+ local result = ""

+ local tokens = {}

+ tokens["go"]     = true

+ tokens["git"]    = true

+ for token in string.gmatch(goname, "[^%-]+") do

+    if not tokens[token] then

+       result = result .. "-" .. token

+       tokens[token] = true

+    end

+ end

+ -- reassemble the string, restore x.y.z runs, convert the vx.y.z

+ -- Go convention to x.y.z as prefered in rpm naming

+ result = string.gsub(result, "^-", "")

+ result = string.gsub(result, "|", ".")

+ result = string.gsub(result, "%-v([%.%d])", "%1")

+ print(result)

+ }

+ 

+ # Map Go information to rpm metadata. This macro will compute default spec

+ # variable values.

+ #

+ # The following spec variable MUST be set before calling the macro:

+ #

+ #   goipath   the packaged Go project import path

+ #

+ # The following spec variables SHOULD be set before calling the macro:

+ #

+ #   forgeurl  the project url on the forge, strongly recommended, if it can not

+ #             be deduced from goipath; alternatively, use -u <url>

+ #   Version   if applicable, set it with Version: <version>

+ #   tag       if applicable

+ #   commit    if applicable

+ #

+ # The macro will attempt to compute and set the following variables if they are

+ # not already set by the packager:

+ #

+ #   goname         an rpm-compatible package name derived from goipath

+ #   gosource       an URL that can be used as SourceX: value

+ #   gourl          an URL that can be used as URL: value

+ #

+ # It will delegate processing to the forgemeta macro for:

+ #

+ #   forgesource    an URL that can be used as SourceX: value

+ #   forgesetupargs the correct arguments to pass to %setup for this source

+ #                  used by %forgesetup and %forgeautosetup

+ #   archivename    the source archive filename, without extentions

+ #   archiveext     the source archive filename extensions, without leading dot

+ #   archiveurl     the url that can be used to download the source archive,

+ #                  without renaming

+ #   scm            the scm type, when packaging code snapshots: commits or tags

+ #

+ # If the macro is unable to parse your forgeurl value set at least archivename

+ # and archiveurl before calling it.

+ #

+ # Most of the computed variables are both overridable and optional. However,

+ # the macro WILL REDEFINE %{dist} when packaging a snapshot (commit or tag).

+ # The previous %{dist} value will be lost. Don’t call the macro if you don’t

+ # wish %{dist} to be changed.

+ #

+ # Optional parameters:

+ #   -u <url>  Ignore forgeurl even if it exists and use <url> instead. Note

+ #             that the macro will still end up setting <url> as the forgeurl

+ #             spec variable if it manages to parse it.

+ #   -s  Silently ignore problems in forgeurl, use it if it can be parsed,

+ #       ignore it otherwise.

+ #   -p  Restore problem handling, override -s.

+ #   -v  Be verbose and print every spec variable the macro sets.

+ #   -i  Print some info about the state of spec variables the macro may use or

+ #       set at the end of the processing.

+ %gometa(u:spvi) %{expand:%{lua:

+ local forgeurl    = rpm.expand("%{?-u*}")

+ if (forgeurl == "") then

+   forgeurl        = rpm.expand("%{?forgeurl}")

+ end

+ -- Be explicit about the spec variables we’re setting

+ local function explicitset(rpmvariable,value)

+   rpm.define(rpmvariable .. " " .. value)

+   if (rpm.expand("%{?-v}") ~= "") then

+     rpm.expand("%{echo:Setting %%{" .. rpmvariable .. "} = " .. value .. "}")

+   end

+ end

+ -- Never ever stomp on a spec variable the packager already set

+ local function safeset(rpmvariable,value)

+   if (rpm.expand("%{?" .. rpmvariable .. "}") == "") then

+     explicitset(rpmvariable,value)

+   end

+ end

+ -- All the Go packaging automation relies on goipath being set

+ local goipath = rpm.expand("%{?goipath}")

+ if (goipath == "") then

+   rpm.expand("%{error:Please set the Go import path in the “goipath” variable before calling “gometa”!}")

+ end

+ -- Compute and set spec variables

+ if (forgeurl ~= "") then

+   rpm.expand("%forgemeta %{?-v} %{?-i} %{?-s} %{?-p} -u " .. forgeurl .. "\\n")

+   safeset("gourl", forgeurl)

+ else

+   safeset("gourl", "https://" .. goipath)

+   rpm.expand("%forgemeta %{?-v} %{?-i} -s     %{?-p} -u %{gourl}\\n")

+ end

+ if (rpm.expand("%{?forgesource}") ~= "") then

+   safeset("gosource", "%{forgesource}")

+ else

+   safeset("gosource", "%{gourl}/%{archivename}.%{archiveext}")

+ end

+ safeset("goname", "%gorpmname %{goipath}")

+ -- Final spec variable summary if the macro was called with -i

+ if (rpm.expand("%{?-i}") ~= "") then

+   rpm.expand("%{echo:Go-specific packaging variables}")

+   rpm.expand("%{echo:  goipath:         %{?goipath}}")

+   rpm.expand("%{echo:  goname:          %{?goname}}")

+   rpm.expand("%{echo:  gourl:           %{?gourl}}")

+   rpm.expand("%{echo:  gosource:        %{?gosource}}")

+ end}

+ BuildRequires: go-rpm-macros

+ ExclusiveArch: %{go_arches}

+ }

This PR adds the automation described in:
https://fedoraproject.org/wiki/More_Go_packaging

It was requested in:
https://bugzilla.redhat.com/show_bug.cgi?id=1526721

It addresses the comments made during the review of:
https://src.fedoraproject.org/rpms/go-srpm-macros/pull-request/1

  1. moves some files to go-compilers as requested.
  2. adds regex handling to %gochecks and cleans it up a lot
  3. adds more comments

It includes the following enhancements:

  1. nested symlink handling (lightly tested)
  2. some vendoring detection to generate bundled() provides (only for simple vendoring in a "vendor" directory, does not handle variants), as requested during public discussion
  3. merging of go-srpm-macros as a subpackage because the macros really want to be kept in a single repo and updated in sync
  4. find filter updates
  5. macro package descriptions cleanup

Conflicts: redhat-rpm-config < 73

1 new commit added

  • fix exclusion of the project root
6 years ago

2 new commits added

  • generalize special version registering in LDFLAGS
  • fix typo in lua conditionnal
6 years ago

Hi Nicolas,

what if [[ -f "$file" || -L "$file" ]] will exits with non-zero? Since it is used in the last command inside for statement, for exits with non-zero and hence %goinstall exits with non-zero, which probably stops package building (or maybe I just oversaw something).

I propose using if [[ -f "$file" || -L "$file" ]]; then ... fi here.

Jirka

Let's build the macros on top of https://github.com/gofed/symbols-extractor/pull/137. Once implemented, I will prepare a patch so it can be built as part of the golang package itself as a separate rpm and then required in the go-srpm-macros. Once done, we can replace this entire file with a simple macro that just calls the golist binary.

@jkucera
No one hit this so far probably because people just feed the macro file lists and never empty directories. However, it is quite trivial to protect against this failure (like you propose or via an empty ||), so I will change it as you propose.

@jchaloup
I'm all for using something else than go list as it's terribly slow. As long as it does not recurse and only processes the files in a specific subdirectory (since subdirectories can be ventilated among several subpackages) it should be trivial to drop it in the "real directory" branch.

The other branch is much more tricky to replace unless you already do symlink handling. You may have noticed in the script that handling nested symlinks in presence of a build root is quite nasty:). And, there are little gains to be had – the shell code is ugly but is is fast (compared to invoking go at least).

So how about merging this version as initial POC people can play with, and replace the engine later as soon as golist is available in a separate package? Unless there is something in the shell code that would prevent dropping in golist later, but I don't see what?

1 new commit added

  • prevent %goinstall from exiting with a problem exit code when the last processed item is a directory
6 years ago

We should put the entire goipatch -> pkgname transformation into a library with comments for each gsub and the for loop stating why is the change needed. Or at least describe the process in a multi-line comment. Then call a single (e.g. go) binary that will the trick.

The same way as here https://github.com/gofed/gofedlib/blob/master/gofedlib/distribution/packagenamegenerator.py. So the transformation can be called independently of the spec file.

We should require at least some minimal metadata that does not need tarball processing. In order to allow simple analysis of a spec file. E.g. a bot analyzing all spec files in the distribution collecting project provider prefix, commit/revision and the import path prefix. E.g. for discovery reasons to be able to detect if a given import path is packaged in the distribution.

I have a tooling that does that and allows me to detect if a go project/dependency needs to be updated or not. For that I expecting %provider_prefix, %commit and %import_path to exist in each spec file. So at least for that reason I would require the goipath to be set manually,

At the end I don't mind using the goipath instead of the import_path. As long as it is clear it means import path prefix of the project.

Maybe it is a silly question but does it make sense to import a go package that has no *.go file?

Wondering if it would be simpler to implement all the symlink resolving logic in Go. Why is the build root problematic wrt. symlinks?

I still wonder if it is the bash itself that makes the implementation hard to read or the complexity of the symlink processing itself.

Where do the symlinks come from? Are they created by the rpm itself?

I have actually encountered with a project that imports path that contains the example keyword. I guess the example was so good but did not get enough attention to be promoted to a proper Go package. Just saying.

It is faster to call go test over a list of packages rather than calling go test over each package separately

How does it get expanded? As:
goipath: github,com/some/path
?

In case the file is a symlink, it gets owned but not copied to the destdir? Where/when is it created then?

If I invoke %goinstall file1.go file2.go ... fileN.go what will be value of the file_list? The file1.gofile2.go...fileN.go? If so, how does the rpm know which file_list belongs to which subpackage? Based on the script the file_list is a file that contains a list of owned files/dirs.

So basically both macros get invoked in the %files section of each subpackage? How the rpm knows it needs to call the %__go_provides? I read http://packaging.fedoraproject.narkive.com/qzTmfbYH/fedora-packaging-creating-an-autoprovides-files and http://rpm.org/user_doc/dependency_generators.html but still don't get it properly. What if the %{_fileattrsdir} contains go.attr, perl.attr and python.attr? How does the rpm knows the the go.attr is the right one? Is there any mechanism to say use go for the NAME?

EDIT: I see, the %__go_path must be set as well.

1 new commit added

  • comment renaming
6 years ago

We should put the entire goipatch -> pkgname transformation into a library with comments for each gsub and the for loop stating why is the change needed. Or at least describe the process in a multi-line comment.

I've added a few comments, the code is pretty basic, it's not intended to be perfect just good enough for 99% of projects that do not use weird naming. The packager is expected to fix the result manually in its spec when hitting those (for example azure sdk for go, I didn't bother hardcoding "for go" stripping just for one project)

Then call a single (e.g. go) binary that will the trick. The same way as here https://github.com/gofed/gofedlib /blob/master/gofedlib/distribution/packagenamegenerator.py.

Remember that the naming rules are needed at srpm level and therefore in the default distro build root. The current macro only depends on the rpm built-in lua interpreter. Anything else will mean either shipping a new static binary in the default build root, or depending on the interpreters available in the build root (and their API breaks over time, and much fun for the people that need dealing with EL).

But, the logic is now commented, feel free to re-implement it in whatever tech you prefer, and call the resulting utility from the macro, if you feel being able to call it independently is worth the re-implementation pain.

We should require at least some minimal metadata
that does not need tarball processing.

You already have that in the repodata

In order to allow simple analysis of a spec file. E.g. a
bot analyzing all spec files in the distribution collecting
project provider prefix, commit/revision and the
import path prefix.

That's quite hard to do reliably without a full spec interpreter. You need to process specs with rpmspec, not rely on basic grepping

And even a full spec interpreter won't see the metadata additions generated by autodeps. As I already told you spec level is the wrong level to do this kind of checker, you need to work from the state of built packages in the repository (of course working from built packages won't catch packages that do not export their Go code at all)

E.g. for discovery reasons to be able to detect if a
given import path is packaged in the distribution.

dnf repoquery --disablerepo=* --enablerepo=koji-f28-builds --enablerepo=koji-rawhide-builds --whatprovides "golang($my_import_path)" --qf "#%{name}|%{source_name}|%{evr}|%{provides}"

and pipe the result to whatever checker you want. Change --enablerepo to point to the Fedora branches you want to check.

You can replace "golang($my_import_path)" with 'golang(*)' if you want a full repository dump.

Note that some packages won't have any commit id because they're built from full releases.

You can try on
https://copr.fedorainfracloud.org/coprs/nim/More_Go_Packaging/repo/epel-7/nim-More_Go_Packaging-epel-7.repo

while the repo is populated (I tend to scratch it quite often)

Maybe it is a silly question but does it make sense to
import a go package that has no *.go file?

Unfortunately, yes. Some go packages (directories) only contain resource files (assets) that can be imported from other projects, you have project roots with no .go files but a lot of subdirectories that contain the go files and so on.

I'm quite sure I've seen code that imported asset directories, not too sure about the second case but a packager can certainly shorten an import path in its BuildRequires and hit a root with no go files inside.

That's why I gave up on go list for Provides, it only works if the directory contains .go files.

Maybe it is a silly question but does it make sense to
import a go package that has no *.go file?

Unfortunately, yes. Some go packages (directories) only contain resource files (assets) that can be imported from other projects, you have project roots with no .go files but a lot of subdirectories that contain the go files and so on.
I'm quite sure I've seen code that imported asset directories, not too sure about the second case but a packager can certainly shorten an import path in its BuildRequires and hit a root with no go files inside.
That's why I gave up on go list for Provides, it only works if the directory contains .go files.

How imported? As there is some path somewhere where are some random files? Or as "import fmt", i.e. package as Go understands it(https://golang.org/ref/spec#Packages)?

It is faster to call go test over a list of packages rather
than calling go test over each package separately

Yes but that's not how rpm autodeps work. autodeps are triggered by the presence of something in a package file list, they do not tell you if the other elements in the build root are also in the same package file list. They may have been assigned to a different subpackage by the packager.

The script assumes that a directory is indivisible (because Go imports exist at the directory level, so splitting directories would break the language logic), but its subdirectories can belong to another subpackage so processing them won't work. For example, it is very tempting to split a project like Go kit that provides connectors to lots of other projects, along those other project lines (and go kit seems structured to allow splitting the subdirectories you don't want).

Being able to split is quite a mess from a Go dependency generator POW, my understanding is that go dep won't support it right now, but there are good reasons for splitting so not sure if that limitation will stand. IMHO it will end up with the possibility to ask go dep to emancipate a subtree and generate a separate lock file in this subtree

How imported? As there is some path somewhere
where are some random files? Or as "import fmt", i.e.
package as Go understands it(https://golang.or
/ref/spec#Packages)?

I'm pretty sure it was just an import in the Go source code. There's nothing in https://golang.org/ref/spec that states the result must contain go files, only that it's a way to state your code needs the result contents. Some code will definitely need assets to work.

tag symlinked renamings? What does it mean again? Can you provide an example?

That basically means that when encountering a "fake" Go package created via a symlink it will create a Requires for the symlink target with the evr of the package itself, so compat subpackages containing symlinks are version-locked with the subpackages containing the target code.

You have an example of the result there:
https://fedoraproject.org/wiki/More_Go_packaging#Example

Wondering if it would be simpler to implement all the symlink resolving logic in Go.

If it's simpler for you, by all means do it, I wasn't comfortable enough to do symlink manipulation in anything else than the shell

Why is the build root problematic wrt. symlinks?

At the time the autodep logic is called all files are installed in the buildroot which means all absolute symlinks that point to something that will be installed by the package are dangling.

And that's recursive, you have paths like

$GOPATH/src/something/SYMLINK1/somethingelse/SYMLINK2

that depend on several levels of dangling symlinks, to generate the
Provides: golang(something/SYMLINK1/somethingelse/SYMLINK2)

other packages expect you need to resolve SYMLINK1, inside the buildroot, check the target is a directory in the target import path, find that SYMLINK2 exists in the target directory, check it also resolves to a directory (it may point to a normal file, or outside the import path), and so on

I still wonder if it is the bash itself that makes the implementation hard to read or the complexity of the symlink processing itself.

That of course depends to your relative familiarity with bash or alternative scripting/coding languages, you can't have the answer without trying to reimplement it yourself.

I freely admit I'm not a great shell coder, I did it this because I needed it, the result seems to work in my usage, someone could probably rewrite it better.

Where do the symlinks come from? Are they created by the rpm itself?

Some may have been created by the Fedora packager, some may exist in the original project, I've encountered both cases

An example of nice to have packager symlink creation is here
https://fedoraproject.org/wiki/More_Go_packaging#Example

How does it get expanded? As: goipath: github,com/some/path ?

local goipath = rpm.expand("%{?goipath}")

means:
1. create a local lua variable named goipath (not seen outside the function)
2. assign it the value of whatever rpm thinks %{?goipath} means at the point the lua function is called in the spec (rpm.expand is just the way lua uses to read within rpm state)
3. in rpm spec logic %{?goipath} means the value of the goipath variable if it is set, empty tring otherwise

In case the file is a symlink, it gets owned but not copied to the destdir?

The symlink is recreated in line 68, as a security measure, to avoid copying complex fileattrs such as selinux contexts

Normal files are recreated line 72 with install doing the metadata scrubbing. IIRC using install on a symlink will copy the symlink target, not recreate a symlink in the new location.

The alternative would be to use cp -p but I'm quite uncomfortable blindly copying security attributes from an archive created by a third party. Especially since the security guys keep inventing new kinds of security metadata with non-obvious effects.

If I invoke %goinstall file1.go file2.go ... fileN.go what will be value of the file_list? The file1.gofile2.go...fileN.go? If so, how does the rpm know which file_list belongs to which subpackage? Based on the script the file_list is a file that contains a list of owned files/dirs.

%global file_list %{?-f*}%{!?-f*:devel.file-list}

Means assign the value of the -f parameter to the variable file_list, and use devel.file-list as default if -f was not set

So by default the macro will append to devel.file-list, and if you want to create a separate file list for another subpackage you just call if with -f someotherfilelistname

I haven't tested it for quite a long time BTW, that's not a pattern we're using. I did it originally for unit test subpackages before deciding they were not worth the dependency logic complexity

How imported? As there is some path somewhere
where are some random files? Or as "import fmt", i.e.
package as Go understands it(https://golang.or
/ref/spec#Packages)?

I'm pretty sure it was just an import in the Go source code. There's nothing in https://golang.org/ref/spec that states the result must contain go files, only that it's a way to state your code needs the result contents. Some code will definitely need assets to work.

Yes, but it is not an golang package per definition, so it shouldn't use the golang() provides that should be reserved only for golang packages(i.e. one discoverable by "go list", conforming to the language specification). Specifying path/file dependencies via import statement is undefined/implementation dependent(what I see in language specifications only mentions packages as defined there) and could be broken at any time as it is basically a hack.

So basically both macros get invoked in the %files section of each subpackage? How the rpm knows it needs to call the %__go_provides

There is not explicit macro call, autodeps are a different, declarative (not imperative) mechanism.

Once all other package construction steps are done and before creating an ar archive for every created binary rpm, rpmbuild will

  1. read all the .attr files installed in rpm

  2. for each rpm binary package it is supposed to create, and for each element in this target binary rpm filelist, try to match it with one or several attr definitions (path or file type magic). In our case we declare both path and flags
    `%__go_path ^%{gopath}/src/.+/.*$

%__go_magic ^((., )?directory|(broken )?symbolic link to .)$and tell rpm both must match%__go_flags magic_and_pathThe__goprefix tells rpm to flag matching elements with thego` label (completely arbitrary, I could have declared golang, go1, goB or even several parallel matchers for the same files)

  1. pipe the paths within the buildroot of matching elements to the corresponding requires and provides engines if they exist. In our case
    %__go_provides %{_rpmconfigdir}/go.prov --goipath "%{?goipath}" --prefix "%{buildroot}" --gopath "%{gopath}" --version "%{?epoch:%{epoch}:}%{version}-%{release}" %{?commit:-a "(commit=%{commit})"} %{?branch:-a "(branch=%{branch})"} %{?tag:-a "(tag=%{tag})"} + %__go_requires %{_rpmconfigdir}/go.req --goipath "%{?goipath}" --prefix "%{buildroot}" --gopath "%{gopath}" --version "%{?epoch:%{epoch}:}%{version}-%{release}"
    and take the engine output as Requires (respectively) Provides definitions, one line at a time.

The engines are not told what binary package will own the path they’re piped, or if other paths in the build root will be owned by the same package (or any package). That’s why you can only compute Provides/Requires one directory at a time, without taking subdirectories into account. (rpm of course keeps track of the binary package for which it called autodeps). Or why you need to declare symlink target as Requires, because the target may be part of another subpackage, so if you don't require it things may fail.

That's somewhat a feature, it means engine behaviour is completely invariant and can not morph depending on the package being processed (it's a PITA if you wanted to create separate unit test packages with specific autodep processing. That would probably require installing tests in a separate gopath root so autodep engines do not clash)

It is possible for one element to match several flags, so in theory, a go import path that contains .py files can match both python and go autodep engines (I've not checked it the python attr file restricts python autodeps to a specific root in the filesystem like the proposed go one). That's an actual feature, packagers do not have to choose a particular engine, all matching engines are executed in parallel.

Hardly it is not part of minimal build root. Or am I missing something?

Why version 3 and release 5?
If you think that this is significant change I'm not against version 2 release 1.

This should be part of the go-srpm-macros package.

Hardly it is not part of minimal build root. Or am I missing something?

That's the description of the source package that contains all files. The files needed at srpm step will end up in the go-srpm-macros subpackage which is part of the build root (that also means that the go-srpm-macros spec can be killed if this PR is accepted).

Having all files originating from the same .src.rpm means you can version-lock easily the rpm part to the srpm part

The drawback is the double rebuild since the spec uses the go arch definition in files it is itself responsible of. It may be possible to define the go arches within the spec, and inject the result in a macro file for other packages to avoid the package dep-looping on itself, but my rpm skills are not up to the task.

This clashes with go-srpm-macros package.

Why version 3 and release 5? If you think that this is significant change I'm not against version 2 release 1.

The PR updates and replaces both go-srpm-macros and go-compilers. go-srpm-macros is currently at version 2, so the most logical version for a major change was 3.

It started at version 3 and release 1 but comments and requests during review caused the revving up to 5. In hindsight (insight is always 100%) I should have sent a PR for version 3 release 0.1 and revved up from there.

If you feel strongly about it I can bump it back to 3 version 0.5, it'll be a pain for the currently running mass rebuild tests (internal and in copr) that already have 3-5 in their repositories, but that's doable

I would prefer original naming as it is more obvious from which srpm the subpackage has been generated.

This clashes with go-srpm-macros package.

Does not clash with it, replaces it

If it is not just net benefit I'm against merge of those two packages. IMHO both parts should be independent.

It's a net maintenance benefit, the more logic you add over time to improve the integration, the more you'll want to change both rpm and srpm parts in lockstep. Merging helps making sure the specs are consistent with one another (even for non-technical but human-critical parts such as descriptions). And that will avoid situations when downstream EL has one part but not the other.

However, if you do wish to continue jungling with two separate but closely tied source packages, just say so, I'll separate them again. The first effect will be two PRs to manage, going back and forth between them, trying to avoid commenting on the wrong one and or losing parts while switching from one to the other.

I would prefer original naming as it is more obvious from which srpm the subpackage has been generated.

That would be quite easy to restore, however as an end user, may I point out the original naming is quite verbose and human-unfriendly? I've lost the number of times I mistyped it.

Is it such a problem to userpm -qi, less rpmfile, or dnf repoquery […] --qf "%{source_name}"?

Nicolas, thanks for all the replies so far. I am worried about the upcoming maintenance of the macros. Given there are no tests verifying functionality of the scripts, the current solution will be very sensitive to regressions and bugs. I would rather re-implement what we can in Go, create new repository for it (e.g. github.com/gofed/go-srpm-macros) and run a set of unit-tests over each PR. I would be happy to re-implement it as long as you can provide a feedback and review (cause I will most likely forget something).

Hi Jan,

Reimplementing in solid code would be great, however that will take some time, and the better plans not always work out

May I suggest you merge this PR as reference implementation (baring bugs that need fixing of course), and then replace the parts you don't like one element at a time, once they are ready? That way this won't block cleaning up the specs in the repo, you'll have a reference implementation to test against, and we'll have the feedback and experience return of all the other Fedora Go packagers.

A lot of tools in Fedora have evolved like this, first prototype in shell or python, then progressive rewrite in C (or another compiled language) to speed up and help maintenance.

That avoids death star failures where initial implementation is exposed to too few people to catch design mistakes
https://kkovacs.eu/the-death-star-design-pattern

I have actually encountered with a project that imports path that contains the example keyword.

I have encountered this case too (in unit tests) . So far I've refused to introduce dependencies on non-production code, and disabled the corresponding tests, but that's a judgment call, not sure if it was the best solution or not. I quite hate when upstreams put us in such situations.

Yes, but it is not an golang package per definition, so it shouldn't use the golang() provides that should be reserved only for golang packages(i.e. one discoverable by "go list", conforming to the language specification). Specifying path/file dependencies via import statement is undefined/implementation dependent(what I see in language specifications only mentions packages as defined there) and could be broken at any time as it is basically a hack.

What can I say, I am not the one writing those imports, upstream projects are, so visibly this implementation detail works in current official go tools. And no upstream project is going to remove such an import just because Fedora decided to be more strict than go tools on other platforms

Making it work at rpm level is one less item the packager has to worry about, and making it work is just listing directories, so it's not like it is expensive for us to do

1 new commit added

  • make filtering of dot-dirs work – though maybe they should also be filtered by default in %gochecks, like they are in %goinstall?
6 years ago

1 new commit added

  • reset release to pre version
6 years ago

1 new commit added

  • minor cleanups
6 years ago

2 new commits added

  • it’s a lot more convenient when all the go tooling packages use the go- prefix, while go packages use golang-
  • why stop at commit and tag in ldflags when we can also do version and the heketi packager wants version too
6 years ago

2 new commits added

  • filter .dirs/.files in tests like in install
  • Make the compiler subpackages match the srpm name to make Jakub happy, adjust filenames accordingly. Going further would require renaming the srpm go ot go-macros, pass through the Fedora nename process, and I don’t know how to submit a PR for a not existing yet package.
6 years ago

2 new commits added

  • automate _dwz_low_mem_die_limit workaround; does not work on EL
  • read Go arch definition in the file that will be packaged
6 years ago

Most of the changes in this PR are already integrated.
The go-srpm-macros rpm now provides:
- %gorpmname (unchanged)
- %gocraftmeta (re-implementation of the %gometa as the %forge* macros are available only in Rawide, I plan to update f26+ branches as well)
- %gosetup (defined by the %gocraftmeta)
The go-compilers packages now provides:
- autogenerated Provides and Requires
- %goinstall
- %gochecks
- %gobuildroot
- + other few macros

Given, the new macros are[-about-to-be] available in Fedora only, it does not make much sense to keep all the with_* constructs anymore. However, sooner or later we will have push the go-srpm-macros and go-compilers packages into epel7/RHEL/CentOS as well. So all the spec files are as close to each other as possible.

Nicolas, feel free to open another PR if the current changes are not sufficient for you. Closing the current one. Thank you for all the ideas and effort making this happen.

Pull-Request has been closed by jchaloup

6 years ago

Hi Jan

Since you're now offline, another point I couldn't discuss with you: your go-compiler version removes the go-srpm-macros integration I had posted, so nothing will pull your macros and a go compiler in the build root in presence of a go package

Or am I missing something?

Hi Nicolas,

the go-srpm-macros is part of the minimal buildroot so it is always installed. When you run the %gocraftmeta, it will set the BuildRequires the way the go-compilers-golang-compiler rpm is installed with all the remaining macros.

I have already updated ~10 Go packages in the Rawhide.

Regards