#145 [WIP]: Use eu-elfclassify instead of file in brp-mangle-shebangs
Closed 2 years ago by churchyard. Opened 2 years ago by churchyard.
rpms/ churchyard/redhat-rpm-config eu-elfclassify  into  rawhide

file modified
+4 -8
@@ -71,17 +71,13 @@ 

  cd "$RPM_BUILD_ROOT"

  

  # Large packages such as kernel can have thousands of executable files.

- # We take care to not fork/exec thousands of "file"s and "grep"s,

+ # We take care to not fork/exec thousands of "find"s and "eu-elfclassify"s,

  # but run just two of them.

- # (Take care to exclude filenames which would mangle "file" output).

- find -executable -type f ! -path '*:*' ! -path $'*\n*' \

- | file -N --mime-type -f - \

- | grep -P ".+(?=: text/)" \

+ find -executable -type f -print0 \

+ | eu-elfclassify --stdin0 --not-elf --print0 \

  | {

  fail=0

- while IFS= read -r line; do

-   f=${line%%:*}

- 

+ while IFS= read -r -d $'\0' f; do

    # Remove the dot

    path="${f#.}"

  

file modified
+8 -2
@@ -6,7 +6,7 @@ 

  

  Summary: Red Hat specific rpm configuration files

  Name: redhat-rpm-config

- Version: 198

+ Version: 199

  Release: 1%{?dist}

  # No version specified.

  License: GPL+
@@ -116,10 +116,12 @@ 

  Requires: (annobin-plugin-clang if clang)

  

  # for brp-mangle-shebangs

+ Requires: %{_bindir}/eu-elfclassify

  Requires: %{_bindir}/find

- Requires: %{_bindir}/file

  Requires: %{_bindir}/grep

  Requires: %{_bindir}/sed

+ 

+ # for brp-strip-lto

  Requires: %{_bindir}/xargs

  

  # for brp-llvm-compile-lto-elf
@@ -215,6 +217,10 @@ 

  %{_rpmconfigdir}/macros.d/macros.kmp

  

  %changelog

+ * Fri Sep 17 2021 Miro Hrončok <mhroncok@redhat.com> - 199-1

+ - Use eu-elfclassify instead of file in brp-mangle-shebangs

+ - Fixes: rhbz#1998924

+ 

  * Mon Aug 30 2021 Florian Weimer <fweimer@redhat.com> - 198-1

  - ELN: Enable -march=x86-64-v2 for Clang as well

  

The brp-mangle-shebangs script should do the following thing:

1) Get all executable files in the buildroot
2) Get all "text" files from those
3a) Mangle shebangs that are "wrong" (e.g. #!/usr/bin/env node)
3b) Remove executable bits from "text" files without shebangs

Detection of "text" files is a tricky business.
The script previously grepped the MIME type detected by file for text/,
however, some obvious text files are reported as application/javascript or similar
and hence were never considered in (2) at all.
OTOH some "binary" files, such as zip archives,
may have shebangs that need mangling.

Now, we reliably detect non-ELF files by eu-elfclassify from the elfuitls package instead.
This is a new dependency, but rpm-build already depends on it,
so it does not pull in additional stuff to the default buildroot.

To my best knowledge, we only support two kinds of executable files:

  • files with shebangs
  • ELF files

Executable files with are neither of those are not good.
Technically, a Shell might execute them in itself,
but Shell scripts without shebangs
were already made not executable by the script before this change.

Fixes https://bugzilla.redhat.com/1998924

With the change described above,
we no longer need to grep the output of the file command
and hence we can process all files,
including files with weird characters in paths,
so we use null-terminated paths now.

Thanks to Neal Gompa for reporting this problem.
Thanks to Florian Weimer for suggesting eu-elfclassify.

This change needs some testing before we land it. The package where this was found is now mangled like this:

+ /usr/lib/rpm/redhat/brp-mangle-shebangs
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/emojis-list/README.md is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/emojis-list/LICENSE.md is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/micromatch/index.js is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/micromatch/LICENSE is executable but has no shebang, removing executable bit
mangling shebang in /usr/lib/node_modules/yarn/node_modules/ajv/scripts/prepare-tests from /usr/bin/env sh to #!/usr/bin/sh
mangling shebang in /usr/lib/node_modules/yarn/node_modules/ajv/scripts/travis-gh-pages from /usr/bin/env bash to #!/usr/bin/bash
mangling shebang in /usr/lib/node_modules/yarn/node_modules/ajv/scripts/info from /usr/bin/env node to #!/usr/bin/node
mangling shebang in /usr/lib/node_modules/yarn/node_modules/ajv/scripts/publish-built-version from /usr/bin/env bash to #!/usr/bin/bash
mangling shebang in /usr/lib/node_modules/yarn/node_modules/esprima/bin/esparse.js from /usr/bin/env node to #!/usr/bin/node
mangling shebang in /usr/lib/node_modules/yarn/node_modules/esprima/bin/esvalidate.js from /usr/bin/env node to #!/usr/bin/node
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/request/index.js is executable but has no shebang, removing executable bit
mangling shebang in /usr/lib/node_modules/yarn/node_modules/sshpk/bin/sshpk-verify from /usr/bin/env node to #!/usr/bin/node
mangling shebang in /usr/lib/node_modules/yarn/node_modules/sshpk/bin/sshpk-sign from /usr/bin/env node to #!/usr/bin/node
mangling shebang in /usr/lib/node_modules/yarn/node_modules/sshpk/bin/sshpk-conv from /usr/bin/env node to #!/usr/bin/node
mangling shebang in /usr/lib/node_modules/yarn/node_modules/loose-envify/cli.js from /usr/bin/env node to #!/usr/bin/node
mangling shebang in /usr/lib/node_modules/yarn/node_modules/js-yaml/bin/js-yaml.js from /usr/bin/env node to #!/usr/bin/node
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/dnscache/examples/sample.js is executable but has no shebang, removing executable bit
mangling shebang in /usr/lib/node_modules/yarn/node_modules/is-ci/bin.js from /usr/bin/env node to #!/usr/bin/node
mangling shebang in /usr/lib/node_modules/yarn/node_modules/uuid/bin/uuid from /usr/bin/env node to #!/usr/bin/node
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/array-unique/README.md is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/array-unique/index.js is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/array-unique/package.json is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/array-unique/LICENSE is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/uri-js/README.md is executable but has no shebang, removing executable bit
*** WARNING: Cannot read the first line from ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/esnext/uri.js.map, removing executable bit
*** WARNING: Cannot read the first line from ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/esnext/regexps-uri.js.map, removing executable bit
*** WARNING: Cannot read the first line from ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/esnext/util.js.map, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/esnext/util.d.ts is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/esnext/regexps-uri.js is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/esnext/index.js is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/esnext/uri.d.ts is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/esnext/schemes/ws.d.ts is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/esnext/schemes/http.js is executable but has no shebang, removing executable bit
*** WARNING: Cannot read the first line from ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/esnext/schemes/urn.js.map, removing executable bit
*** WARNING: Cannot read the first line from ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/esnext/schemes/mailto.js.map, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/esnext/schemes/https.js is executable but has no shebang, removing executable bit
*** WARNING: Cannot read the first line from ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/esnext/schemes/urn-uuid.js.map, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/esnext/schemes/urn.js is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/esnext/schemes/wss.d.ts is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/esnext/schemes/wss.js is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/esnext/schemes/urn-uuid.js is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/esnext/schemes/https.d.ts is executable but has no shebang, removing executable bit
*** WARNING: Cannot read the first line from ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/esnext/schemes/ws.js.map, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/esnext/schemes/ws.js is executable but has no shebang, removing executable bit
*** WARNING: Cannot read the first line from ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/esnext/schemes/wss.js.map, removing executable bit
*** WARNING: Cannot read the first line from ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/esnext/schemes/http.js.map, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/esnext/schemes/urn-uuid.d.ts is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/esnext/schemes/urn.d.ts is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/esnext/schemes/http.d.ts is executable but has no shebang, removing executable bit
*** WARNING: Cannot read the first line from ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/esnext/schemes/https.js.map, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/esnext/schemes/mailto.js is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/esnext/schemes/mailto.d.ts is executable but has no shebang, removing executable bit
*** WARNING: Cannot read the first line from ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/esnext/index.js.map, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/esnext/uri.js is executable but has no shebang, removing executable bit
*** WARNING: Cannot read the first line from ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/esnext/regexps-iri.js.map, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/esnext/regexps-iri.d.ts is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/esnext/regexps-iri.js is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/esnext/index.d.ts is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/esnext/util.js is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/esnext/regexps-uri.d.ts is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/es5/uri.all.js is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/es5/uri.all.min.d.ts is executable but has no shebang, removing executable bit
*** WARNING: Cannot read the first line from ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/es5/uri.all.js.map, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/es5/uri.all.d.ts is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/es5/uri.all.min.js is executable but has no shebang, removing executable bit
*** WARNING: Cannot read the first line from ./usr/lib/node_modules/yarn/node_modules/uri-js/dist/es5/uri.all.min.js.map, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/uri-js/package.json is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/uri-js/LICENSE is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/uri-js/yarn.lock is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/arr-diff/LICENSE is executable but has no shebang, removing executable bit
mangling shebang in /usr/lib/node_modules/yarn/node_modules/gunzip-maybe/bin.js from /usr/bin/env node to #!/usr/bin/node
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/ecc-jsbn/README.md is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/ecc-jsbn/index.js is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/ecc-jsbn/package.json is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/ecc-jsbn/test.js is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/ecc-jsbn/LICENSE is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/ecc-jsbn/lib/sec.js is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/ecc-jsbn/lib/ec.js is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/ecc-jsbn/lib/LICENSE-jsbn is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/object-path/.travis.yml is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/object-path/.npmignore is executable but has no shebang, removing executable bit
mangling shebang in /usr/lib/node_modules/yarn/node_modules/semver/bin/semver from /usr/bin/env node to #!/usr/bin/node
mangling shebang in /usr/lib/node_modules/yarn/node_modules/rimraf/bin.js from /usr/bin/env node to #!/usr/bin/node
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/mkdirp-promise/LICENSE is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/mkdirp-promise/lib/index.js is executable but has no shebang, removing executable bit
mangling shebang in /usr/lib/node_modules/yarn/node_modules/json5/lib/cli.js from /usr/bin/env node to #!/usr/bin/node
mangling shebang in /usr/lib/node_modules/yarn/node_modules/mkdirp/bin/cmd.js from /usr/bin/env node to #!/usr/bin/node
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/arr-flatten/README.md is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/node_modules/yarn/node_modules/arr-flatten/LICENSE is executable but has no shebang, removing executable bit
mangling shebang in /usr/lib/node_modules/yarn/bin/yarnpkg from /usr/bin/env node to #!/usr/bin/node
mangling shebang in /usr/lib/node_modules/yarn/bin/yarn from /bin/sh to #!/usr/bin/sh
mangling shebang in /usr/lib/node_modules/yarn/bin/yarn.js from /usr/bin/env node to #!/usr/bin/node

To my best knowledge, we only support two kinds of executable files:

  • files with shebangs
  • ELF files

Executable files with are neither of those are not good.
Technically, a Shell might execute them in itself,
but Shell scripts without shebangs
were already made not executable by the script before this change.

Is this assumption correct?

1 new commit added

  • Remove unused dependency on xargs, brp-mangle-shebangs does not use it
2 years ago

2 new commits added

  • Update comment near the xargs dependency, brp-mangle-shebangs does not use it
  • Use eu-elfclassify instead of file in brp-mangle-shebangs
2 years ago

Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci

Is this assumption correct?

Technically it's not, as you can register other binary handlers with the kernel. See https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html

In Fedora, this is at least used by wine to enable execution of windows binaries. Looking for things that drop files into /*/binfmt.d , I see there are at least a couple of other things that do this (qemu and ksh). But I have no idea whether we are shipping any binaries that depend on one of these things being set up. I thought wine might do it (for things like notepad or regedit) but those turn out to be shell wrappers.

It's probably a small enough concern that the best solution is documentation. Disabling the mangling is already sufficiently simple.

Thanks. Let's built some wine (and mingw) packages in Copr and see what happens.

Files like this are also executable:

$ file /usr/lib/mono-zeroconf/MZClient.exe
/usr/lib/mono-zeroconf/MZClient.exe: PE32 executable (console) Intel 80386 Mono/.Net assembly, for MS Windows

Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci

A mono package:

+ /usr/lib/rpm/redhat/brp-mangle-shebangs
mangling shebang in /usr/bin/mzclient from /bin/bash to #!/usr/bin/bash
*** WARNING: ./usr/lib/mono-zeroconf/MZClient.exe.mdb is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/mono-zeroconf/MZClient.exe is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/mono-zeroconf/Mono.Zeroconf.Providers.AvahiDBus.dll.mdb is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/mono-zeroconf/Mono.Zeroconf.Providers.AvahiDBus.dll is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/mono-zeroconf/Mono.Zeroconf.Providers.Bonjour.dll.config is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/mono-zeroconf/Mono.Zeroconf.Providers.Bonjour.dll.mdb is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/mono-zeroconf/Mono.Zeroconf.Providers.Bonjour.dll is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/mono/gac/policy.4.0.Mono.Zeroconf/0.0.0.0__e60c4f4a95e1099e/policy.4.0.Mono.Zeroconf.dll is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/mono/gac/policy.3.0.Mono.Zeroconf/0.0.0.0__e60c4f4a95e1099e/policy.3.0.Mono.Zeroconf.dll is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/mono/gac/policy.4.5.Mono.Zeroconf/0.0.0.0__e60c4f4a95e1099e/policy.4.5.Mono.Zeroconf.dll is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/mono/gac/policy.1.0.Mono.Zeroconf/0.0.0.0__e60c4f4a95e1099e/policy.1.0.Mono.Zeroconf.dll is executable but has no shebang, removing executable bit
*** WARNING: ./usr/lib/mono/gac/Mono.Zeroconf/4.0.0.90__e60c4f4a95e1099e/Mono.Zeroconf.dll is executable but has no shebang, removing executable bit

No idea if the files require executable bits to function, will ask some mono expert.

@tpokorra Do you know?

No idea if the files require executable bits to function, will ask some mono expert.

@tpokorra Do you know?

*.dll files should not be executable, so dropping the executable flag is fine.
The same for *.config and *.mdb

*.exe files are usually called with mono, eg. mono MZClient.exe
See also /usr/bin/mcs (from mono-core-6.12.0-4.fc34.x86_64) which is an executable bash script with this content:

#!/usr/bin/sh
exec /usr/bin/mono $MONO_OPTIONS /usr/lib/mono/4.5/mcs.exe "$@"

I just checked on my Fedora Desktop, if I run in my bash shell:
/usr/lib/mono/xbuild/12.0/bin/xbuild.exe
Then wine starts, and executes xbuild.exe
So perhaps the executable flag makes sense here?

But without considering wine, the executable flag on *.exe files could be dropped as well, in my opinion.
/usr/bin/mcs will still execute /usr/lib/mono/4.5/mcs.exe even if I drop the executable flag:

chmod a-x /usr/lib/mono/4.5/mcs.exe

I just checked on my Fedora Desktop, if I run in my bash shell:
/usr/lib/mono/xbuild/12.0/bin/xbuild.exe
Then wine starts, and executes xbuild.exe

That is because of /usr/lib/binfmt.d/wine.conf which sets up wine as an executable handler for windows PE executables.

So perhaps the executable flag makes sense here?

If you didn't have wine installed, you would have different behavior. Honestly I'm not even sure that executing that with wine is what you would want. So there is probably no one correct answer; the question is then which of the possible behaviors (that can actually be implemented reasonably) meets the needs of the greatest number of packages.

I'm thinking that just removing the executable bit from those PE executables makes the most sense. The very rare package that needs different behavior can easily disable the brp script.

Now that the thread has died out, what should we do?

I have this on my TODO, but unfortunately, I've been dealing with more pressing issues. I will eventually get back to this.

Pull-Request has been closed by churchyard

2 years ago