#6 Add wasm32-unknown-unknown builds of -std-static and -analysis
Opened a month ago by ivanmironov. Modified a month ago
https://github.com/im-0/fedora-rpm.rust add-std-static-wasm  into  master

Add wasm32-unknown-unknown builds of -std-static and -analysis
Ivan Mironov • a month ago  
0001-Use-lld-provided-by-system-for-wasm.patch
file added
+26

@@ -0,0 +1,26 @@

+ From ca5d5805792a83a96e7d77efa70d66ad0d37539f Mon Sep 17 00:00:00 2001

+ From: Ivan Mironov <mironov.ivan@gmail.com>

+ Date: Sun, 8 Dec 2019 17:23:08 +0500

+ Subject: [PATCH] Use lld provided by system for wasm

+ 

+ ---

+  src/librustc_target/spec/wasm32_base.rs | 3 +--

+  1 file changed, 1 insertion(+), 2 deletions(-)

+ 

+ diff --git a/src/librustc_target/spec/wasm32_base.rs b/src/librustc_target/spec/wasm32_base.rs

+ index 6f00245b0094..db41f27dc7ce 100644

+ --- a/src/librustc_target/spec/wasm32_base.rs

+ +++ b/src/librustc_target/spec/wasm32_base.rs

+ @@ -111,8 +111,7 @@ pub fn options() -> TargetOptions {

+          // arguments just yet

+          limit_rdylib_exports: false,

+  

+ -        // we use the LLD shipped with the Rust toolchain by default

+ -        linker: Some("rust-lld".to_owned()),

+ +        linker: Some("lld".to_owned()),

+          lld_flavor: LldFlavor::Wasm,

+  

+          // No need for indirection here, simd types can always be passed by

+ -- 

+ 2.23.0

+ 

rust.spec
file modified
+151 -16

@@ -47,9 +47,17 @@

  %bcond_with lldb

  %endif

  

+ # WebAssembly support requires wasm-aware lld, which is known to

+ # be good enough starting at LLVM 8.0.0.

+ %if 0%{?fedora} < 30 || %{with bundled_llvm}

+ %bcond_with wasm

+ %else

+ %bcond_without wasm

+ %endif

+ 

  Name:           rust

  Version:        1.39.0

- Release:        2%{?dist}

+ Release:        3%{?dist}

  Summary:        The Rust Programming Language

  License:        (ASL 2.0 or MIT) and (BSD and MIT)

  # ^ written as: (rust itself) and (bundled libraries)

@@ -62,6 +70,7 @@

  %global rustc_package rustc-%{channel}-src

  %endif

  Source0:        https://static.rust-lang.org/dist/%{rustc_package}.tar.xz

+ Source1:        strip-if-not-wasm

  

  # Revert https://github.com/rust-lang/rust/pull/57840

  # We do have the necessary fix in our LLVM 7.

@@ -79,6 +88,11 @@

  # https://github.com/rust-lang/rust/pull/66317

  Patch4:         rust-pr66317-bindir-relative.patch

  

+ # By default, rust tries to use "rust-lld" as a linker for WebAssembly.

+ # There is no "rust-lld" in Fedora packages, but we can use regular lld

+ # instead.

+ Patch5:         0001-Use-lld-provided-by-system-for-wasm.patch

+ 

  # Get the Rust triple for any arch.

  %{lua: function rust_triple(arch)

    local abi = "gnu"

@@ -95,6 +109,14 @@

  

  %global rust_triple %{lua: print(rust_triple(rpm.expand("%{_target_cpu}")))}

  

+ %if %with wasm

+ %global rust_triple_cross wasm32-unknown-unknown

+ %else

+ %global rust_triple_cross %{nil}

+ %endif

+ 

+ %global rust_triple_targets %{rust_triple},%{rust_triple_cross}

+ 

  %if %defined bootstrap_arches

  # For each bootstrap arch, add an additional binary Source.

  # Also define bootstrap_source just for the current target.

@@ -108,7 +130,7 @@

    local target_arch = rpm.expand("%{_target_cpu}")

    for i, arch in ipairs(bootstrap_arches) do

      print(string.format("Source%d: %s-%s.tar.xz\n",

-                         i, base, rust_triple(arch)))

+                         i + 1, base, rust_triple(arch)))

      if arch == target_arch then

        rpm.define("bootstrap_source "..i)

      end

@@ -171,9 +193,17 @@

  %global llvm llvm

  %global llvm_root %{_prefix}

  %endif

+ %if %with wasm

+ BuildRequires:  %{llvm}-devel >= 8.0

+ %else

  BuildRequires:  %{llvm}-devel >= 6.0

+ %endif

  %if %with llvm_static

- BuildRequires:  %{llvm}-static

+ %if %with wasm

+ BuildRequires:  %{llvm}-static >= 8.0

+ %else

+ BuildRequires:  %{llvm}-static >= 6.0

+ %endif

  BuildRequires:  libffi-devel

  %endif

  %endif

@@ -235,12 +265,48 @@

  This package includes the Rust compiler and documentation generator.

  

  

- %package std-static

+ %if %with wasm

+ # GNU strip removes symbol table from static library files, but this table

+ # is required for linking WebAssembly binaries using lld. lld produces errors

+ # like following when trying to link with stripped libraries:

+ #

+ #     lld: error: /.../libstd-....rlib: archive has no index; run ranlib to add one

+ #

+ # To circumvent this, we use wrapper script for strip utility, which filters out

+ # archives containing WebAssembly binaries and prevents them from being stripped.

+ %global __strip	"%{python} %{SOURCE1}"

+ %endif

+ 

+ 

+ %{lua: do

+   for triple in string.gmatch(rpm.expand("%{rust_triple_targets}"), "([^,]+)") do

+     local pkg_name_suffix, triple_description

+     if triple == rpm.expand("%{rust_triple}") then

+       pkg_name_suffix = ""

+       triple_description = triple .. " (native for host)"

+     else

+       pkg_name_suffix = "-" .. triple

+       triple_description = triple

+     end

+     local wasm_dep

+     if string.sub(triple, 1, 4) == "wasm" then

+ 		wasm_dep = "Requires: lld >= 8.0"

+     else

+ 		wasm_dep = ""

+     end

+     print(string.format([[

+ %%package std-static%s

  Summary:        Standard library for Rust

+ %s

  

- %description std-static

+ %%description std-static%s

  This package includes the standard libraries for building applications

- written in Rust.

+ written in Rust for target %s.

+ 

+ 

+ ]], pkg_name_suffix, wasm_dep, pkg_name_suffix, triple_description))

+   end

+ end}

  

  

  %package debugger-common

@@ -386,16 +452,30 @@

  useful as a reference for code completion tools in various editors.

  

  

- %package analysis

+ %{lua: do

+   for triple in string.gmatch(rpm.expand("%{rust_triple_targets}"), "([^,]+)") do

+     local pkg_name_suffix

+     if triple == rpm.expand("%{rust_triple}") then

+       pkg_name_suffix = ""

+     else

+       pkg_name_suffix = "-" .. triple

+     end

+     print(rpm.expand(string.format([[

+ %%package analysis%s

  Summary:        Compiler analysis data for the Rust standard library

- Requires:       rust-std-static%{?_isa} = %{version}-%{release}

+ Requires:       rust-std-static%s%%{?_isa} = %%{version}-%%{release}

  

- %description analysis

+ %%description analysis%s

  This package contains analysis data files produced with rustc's -Zsave-analysis

  feature for the Rust standard library. The RLS (Rust Language Server) uses this

  data to provide information about the Rust standard library.

  

  

+ ]], pkg_name_suffix, pkg_name_suffix, pkg_name_suffix)))

+   end

+ end}

+ 

+ 

  %prep

  

  %ifarch %{bootstrap_arches}

@@ -412,6 +492,10 @@

  %patch2 -p1

  %patch3 -p1

  %patch4 -p1

+ %if %with wasm

+ %patch5 -p1

+ %endif

+ 

  

  %if "%{python}" == "python3"

  sed -i.try-py3 -e '/try python2.7/i try python3 "$@"' ./configure

@@ -526,16 +610,36 @@

    %{?codegen_units_std} \

    --release-channel=%{channel}

  

+ # Build everything for host triple.

  %{python} ./x.py build

  %{python} ./x.py doc

  

+ # For other triples, build only "libstd" and "analysis" files.

+ %{lua: do

+   for triple in string.gmatch(rpm.expand("%{rust_triple_cross}"), "([^,]+)") do

+     print(rpm.expand(string.format([[

+ %%{python} ./x.py build --host=%%{rust_triple} --target=%s src/libstd

+ ]], triple)))

+   end

+ end}

  

  %install

  %{?cmake_path:export PATH=%{cmake_path}:$PATH}

  %{?rustflags:export RUSTFLAGS="%{rustflags}"}

  

+ # Install everything for host triple.

  DESTDIR=%{buildroot} %{python} ./x.py install

  

+ # For other triples, install only "libstd" and "analysis" files.

+ %{lua: do

+   for triple in string.gmatch(rpm.expand("%{rust_triple_cross}"), "([^,]+)") do

+     print(rpm.expand(string.format([[

+ DESTDIR=%%{buildroot} %%{python} ./x.py install --host=%%{rust_triple} --target=%s src/libstd

+ DESTDIR=%%{buildroot} %%{python} ./x.py install --host=%%{rust_triple} --target=%s analysis

+ ]], triple, triple)))

+   end

+ end}

+ 

  # Make sure the shared libraries are in the proper libdir

  %if "%{_libdir}" != "%{common_libdir}"

  mkdir -p %{buildroot}%{_libdir}

@@ -632,11 +736,25 @@

  %exclude %{_bindir}/*miri

  

  

- %files std-static

- %dir %{rustlibdir}

- %dir %{rustlibdir}/%{rust_triple}

- %dir %{rustlibdir}/%{rust_triple}/lib

- %{rustlibdir}/%{rust_triple}/lib/*.rlib

+ %{lua: do

+   for triple in string.gmatch(rpm.expand("%{rust_triple_targets}"), "([^,]+)") do

+     local pkg_name_suffix

+     if triple == rpm.expand("%{rust_triple}") then

+       pkg_name_suffix = ""

+     else

+       pkg_name_suffix = "-" .. triple

+     end

+     print(rpm.expand(string.format([[

+ %%files std-static%s

+ %%dir %%{rustlibdir}

+ %%dir %%{rustlibdir}/%s

+ %%dir %%{rustlibdir}/%s/lib

+ %%{rustlibdir}/%s/lib/*.rlib

+ 

+ 

+ ]], pkg_name_suffix, triple, triple, triple)))

+   end

+ end}

  

  

  %files debugger-common

@@ -716,11 +834,28 @@

  %{rustlibdir}/src

  

  

- %files analysis

- %{rustlibdir}/%{rust_triple}/analysis/

+ %{lua: do

+   for triple in string.gmatch(rpm.expand("%{rust_triple_targets}"), "([^,]+)") do

+     local pkg_name_suffix

+     if triple == rpm.expand("%{rust_triple}") then

+       pkg_name_suffix = ""

+     else

+       pkg_name_suffix = "-" .. triple

+     end

+     print(rpm.expand(string.format([[

+ %%files analysis%s

+ %%{rustlibdir}/%s/analysis/

+ 

+ 

+ ]], pkg_name_suffix, triple)))

+   end

+ end}

  

  

  %changelog

+ * Sun Dec 08 2019 Ivan Mironov <mironov.ivan@gmail.com> - 1.39.0-3

+ - Add wasm32-unknown-unknown builds of -std-static and -analysis

+ 

  * Tue Nov 12 2019 Josh Stone <jistone@redhat.com> - 1.39.0-2

  - Fix a couple build and test issues with rustdoc.

  

strip-if-not-wasm
file added
+63

@@ -0,0 +1,63 @@

+ #!/usr/bin/python3

+ 

+ from __future__ import print_function

+ 

+ import os

+ import subprocess

+ import sys

+ 

+ 

+ _STRIP = '/usr/bin/strip'

+ _AR = '/usr/bin/ar'

+ 

+ _MAGIC_AR = b'!<arch>'

+ _MAGIC_WASM = b'\0asm'

+ 

+ 

+ def _not_wasm(ar_fname):

+     with open(ar_fname, 'rb') as f:

+         magic = f.read(len(_MAGIC_AR))

+     if magic != _MAGIC_AR:

+         return True

+ 

+     fnames = subprocess.check_output((_AR, 't', ar_fname))

+     fnames = fnames.decode('ascii').strip().split()

+ 

+     for fname in fnames:

+         proc = subprocess.Popen(

+             (_AR, 'p', ar_fname, fname),

+             stdin=subprocess.PIPE,

+             stdout=subprocess.PIPE,

+             stderr=open(os.devnull, 'wb'))

+         proc.stdin.close()

+ 

+         magic = proc.stdout.read(len(_MAGIC_WASM))

+         proc.stdout.close()

+         proc.wait()

+ 

+         if magic == _MAGIC_WASM:

+             print(ar_fname, 'is WebAssembly static library, skipping', file=sys.stderr)

+             return False

+ 

+     return True

+ 

+ 

+ only_files_left = False

+ strip_opts = []

+ strip_files = []

+ for arg in sys.argv[1:]:

+     if only_files_left:

+         strip_files.append(arg)

+     elif arg == '--':

+         only_files_left = True

+     elif arg[:1] in ('-', '@'):

+         strip_opts.append(arg)

+     elif os.path.exists(arg):

+         strip_files.append(arg)

+     else:

+         strip_opts.append(arg)

+ 

+ strip_files = [fname for fname in strip_files if _not_wasm(fname)]

+ 

+ if strip_files:

+     os.execvp(_STRIP, [_STRIP] + strip_opts + ['--'] + strip_files)

no initial comment

This enables out of the box support of building code written in Rust
into WebAsssembly binaries. On the LLVM side everything is there
already.

I hope that I did not break anything by this patch.

To test that building to WebAssembly actually works, I successfully built and ran TodoMVC example[1] from yew[2] using cargo-web[3].

Also, the same code may be used to add -std-static and -analysis packages for more target triples already supported by rustc/LLVM (see rustc --print target-list).

[1] https://github.com/yewstack/yew/tree/e044f73fdb1c8eb3b25f3228f070af9b3351922a/examples/todomvc
[2] https://yew.rs/
[3] https://github.com/koute/cargo-web

Hey there. This looks mostly OK -- the lua loops may be overkill, but I guess we might want more such targets in the future. Fedora doesn't really bother with cross-compiling in general, but wasm32-wasi might be another appropriate "cross" target when that gets more traction. A mingw person was also asking about packaging on the rust list.

I actually did look at this a while ago myself, but I wasn't really happy that the target package couldn't be noarch. There seems to be some metadata in the result specific to the host compiler, which affects the hash of all the libraries, but in theory there's no reason for that.

I noticed the rust-doc package in the simple-koji-ci build here actually looks like the wasm docs, e.g. with std::os::raw::c_ulong = u32, whereas x86_64 should be u64. I think we just need to be more specific with install commands, like ./x.py install --target $HOST for all the native packages, then just ./x.py install --target $OTHER libstd rust-analysis.

Upstream, there's currently some discussion of dynamically building std in some cases, which would make this PR moot -- with our rust-src package, cargo should be able to dynamically build std for wasm32 or anything else.

There's also cargo-xbuild (source) which may allow this already. I haven't tried it with Fedora's toolchain, but if that works maybe we should just package that tool.

rebased onto d6dd25e

a month ago

Hi @jistone.

I actually did look at this a while ago myself, but I wasn't really happy that the target package couldn't be noarch. There seems to be some metadata in the result specific to the host compiler, which affects the hash of all the libraries, but in theory there's no reason for that.

Are there any upstream bug reports about this? Given that rustup uses only one tarball per host_ver * target_triple (see here for example), not one per host_ver * host_triple * target_triple, I think that the resulting files are actually compatible and should be bit to bit equal.

I noticed the rust-doc package in the simple-koji-ci build here actually looks like the wasm docs, e.g. with std::os::raw::c_ulong = u32, whereas x86_64 should be u64. I think we just need to be more specific with install commands, like ./x.py install --target $HOST for all the native packages, then just ./x.py install --target $OTHER libstd rust-analysis.

Thanks! I fixed this. By the way, specifying only --target without --host for x.py install leads to weird result: it silently skips all the libstd files and installs only analysis somehow.

Upstream, there's currently some discussion of dynamically building std in some cases, which would make this PR moot -- with our rust-src package, cargo should be able to dynamically build std for wasm32 or anything else.

Unless this hits stable rust really soon (which I doubt), providing prebuilt libstd packages is still useful.

There's also cargo-xbuild (source) which may allow this already. I haven't tried it with Fedora's toolchain, but if that works maybe we should just package that tool.

There is a problem: standard library uses unstable features internally, so there is no currently supported way to build it on stable rust. Source code of cargo-xbuild confirms that. There is a hack though: RUSTC_BOOTSTRAP environment variable. But, as I understand, the only supported use of this hack is bootstrapping the whole compiler using stable version of itself.

Fedora doesn't really bother with cross-compiling in general, but wasm32-wasi might be another appropriate "cross" target when that gets more traction.

But Fedora contains gcc cross toolchains for various architectures in standard repositories. Current rustc is able to compile for various targets already, but, given the situation with building libstd on stable, this is much less useful than cross gcc (which is able to compile libc from source).

Are there any upstream bug reports about this? Given that rustup uses only one tarball per host_ver * target_triple (see here for example), not one per host_ver * host_triple * target_triple, I think that the resulting files are actually compatible and should be bit to bit equal.

Yes, I think they are compatible, just different in insignificant ways. I haven't filed an upstream bug, but I will do so.

There is a problem: standard library uses unstable features internally, so there is no currently supported way to build it on stable rust. Source code of cargo-xbuild confirms that. There is a hack though: RUSTC_BOOTSTRAP environment variable. But, as I understand, the only supported use of this hack is bootstrapping the whole compiler using stable version of itself.

Ah, yes of course. And you're right, I don't generally want to support RUSTC_BOOTSTRAP in Fedora. Building std is a sort of bootstrapping though, so this special case of using rustc to build from rust-src of the same version could be OK.

But Fedora contains gcc cross toolchains for various architectures in standard repositories.

Support for those is limited -- take a look at their description:

Only building kernels is currently supported.  Support for cross-building
user space programs is not currently provided as that would massively multiply
the number of packages.

In that sense, current rustc is already similarly capable -- all of the normal target specs are included in the compiler itself, but we lack the "user space" libraries for cross targets. As you say, even the cross gcc requires you to build your own libc.