rust @1.66.0: trouble building universal arm64/x86_64

[kencu (Ken)]

the universal build of rust is needed for certain ports to build universal

hoping Marcus has some insights on how to proceed here

[MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)]

The error seems to be

:info:build <jemalloc>: Error allocating TSD

I have not encounter this problem.
On the same machine, does building with build_arch=x86_64 -universal work?
Does commenting out --set=rust.jemalloc in the Portfile allow a successful build?
At the very least, one of these two steps may help narrow down the problem.

[kencu (Ken)]

thank you for your attentions -- I will try both of those.

[kencu (Ken)]

building like this:

% sudo port -v destroot rust build_arch=x86_64 -universal

gives the same error it seems:

error: process didn't exit successfully: `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_rust/rust/work/rustc-1.66.0-src/build/bootstrap/debug/rustc -vV` (exit status: 254)
--- stdout

Did not run successfully: signal: 6 (SIGABRT)
"/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_rust/rust/work/rustc-1.66.0-src/build/x86_64-apple-darwin/stage1/bin/rustc" "-vV" "-Wrust_2018_idioms" "-Wunused_lifetimes" "-Wsemicolon_in_expressions_from_macros" "-Dwarnings" "-Clinker=/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_rust/rust/work/compwrap/ld/usr/bin/clang" "-Zunstable-options" "--check-cfg=values(bootstrap)" "-Z" "force-unstable-if-unmarked"
-------------

--- stderr
<jemalloc>: Error allocating TSD
rustc exited with signal: 6 (SIGABRT)

command did not execute successfully: "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_rust/rust/work/cargo-1.65.0-x86_64-apple-darwin/cargo/bin/cargo" "build" "--target" "x86_64-apple-darwin" "-Zcheck-cfg=names,values,output" "-Zbinary-dep-depinfo" "-j" "10" "-v" "-v" "--release" "--features" "panic-unwind backtrace" "--manifest-path" "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_rust/rust/work/rustc-1.66.0-src/library/test/Cargo.toml" "--message-format" "json-render-diagnostics"
expected success, got: exit status: 101

will try the other next.

[kencu (Ken)]

disabling jemalloc indeed allowed the build to finish.

There is a bit of a hiccup in the muniversal destrooting -- I'll see if I can fix that. New log attached.

Thanks, Marcus!

[MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)]

I built jemalloc on both an x86_64 machine and on an Apple Silicon machine with build_arch=x86_64.
The only difference I could find was #define CPU_SPINWAIT __asm__ volatile("pause") on the actual x86_64 machine.
That particular difference went away went I used version 1.1 of the muniveral PG.
I have no idea if that is the cause of the problem, but it might be worth a look.
Unfortunately, I cannot test it at the moment, but there is a ​pull request with what I tried.

[kencu (Ken)]

unfortunately same TSD error building universal rust with the new build of jemalloc. I'll go back to disabling it and see if I can fix that destroot issue for now.

I notice two things about jemalloc; first of all, there are these warnings during the build, involving interestingly the TSD area in question:

ccache /usr/bin/clang -std=gnu11 -Werror=unknown-warning-option -Wall -Wextra -Wshorten-64-to-32 -Wsign-compare -Wundef -Wno-format-zero-length -Wpointer-arith -Wno-missing-braces -Wno-missing-field-initializers -pipe -g3 -Wimplicit-fallthrough -O3 -funroll-loops -pipe -Os -isysroot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.sdk -arch arm64 -c -I/opt/local/include -isysroot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.sdk -D_REENTRANT -Iinclude -Iinclude -o src/bin_info.o src/bin_info.c
In file included from src/jemalloc_cpp.cpp:10:
In file included from include/jemalloc/internal/jemalloc_internal_includes.h:51:
In file included from include/jemalloc/internal/prof_structs.h:4:
In file included from include/jemalloc/internal/ckh.h:4:
In file included from include/jemalloc/internal/tsd.h:310:
include/jemalloc/internal/tsd_generic.h:91:24: warning: suggest braces around initialization of subobject [-Wmissing-braces]
                        tsd_t initializer = TSD_INITIALIZER;
                                            ^~~~~~~~~~~~~~~
include/jemalloc/internal/tsd.h:150:9: note: expanded from macro 'TSD_INITIALIZER'
                                TSD_DATA_SLOW_INITIALIZER               \
                                ^~~~~~~~~~~~~~~~~~~~~~~~~
include/jemalloc/internal/tsd.h:121:24: note: expanded from macro 'TSD_DATA_SLOW_INITIALIZER'
    /* tcache_slow */           TCACHE_SLOW_ZERO_INITIALIZER,           \
                                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
include/jemalloc/internal/tcache_types.h:22:39: note: expanded from macro 'TCACHE_SLOW_ZERO_INITIALIZER'
#define TCACHE_SLOW_ZERO_INITIALIZER {0}
                                      ^
In file included from src/jemalloc_cpp.cpp:10:
In file included from include/jemalloc/internal/jemalloc_internal_includes.h:51:
In file included from include/jemalloc/internal/prof_structs.h:4:
In file included from include/jemalloc/internal/ckh.h:4:
In file included from include/jemalloc/internal/tsd.h:310:
include/jemalloc/internal/tsd_generic.h:134:22: warning: suggest braces around initialization of subobject [-Wmissing-braces]
        tsd_t initializer = TSD_INITIALIZER;
                            ^~~~~~~~~~~~~~~
include/jemalloc/internal/tsd.h:150:9: note: expanded from macro 'TSD_INITIALIZER'
                                TSD_DATA_SLOW_INITIALIZER               \
                                ^~~~~~~~~~~~~~~~~~~~~~~~~
include/jemalloc/internal/tsd.h:121:24: note: expanded from macro 'TSD_DATA_SLOW_INITIALIZER'
    /* tcache_slow */           TCACHE_SLOW_ZERO_INITIALIZER,           \
                                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
include/jemalloc/internal/tcache_types.h:22:39: note: expanded from macro 'TCACHE_SLOW_ZERO_INITIALIZER'
#define TCACHE_SLOW_ZERO_INITIALIZER {0}
                                      ^

so perhaps doing something to stop those warnings might be of use to preventing the error in rust.

The other thing about jemalloc is that it forces itself to try to link against libstdc++:

DSO_LDFLAGS        : -shared -Wl,-install_name,$(LIBDIR)/$(@F)
LIBS               : -lstdc++ -pthread
RPATH_EXTRA        : 

I am not sure if it actually does link against it.

% otool -L /opt/local/lib/libjemalloc.dylib
/opt/local/lib/libjemalloc.dylib (architecture x86_64):
	/opt/local/lib/libjemalloc.2.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1300.36.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.0.0)
/opt/local/lib/libjemalloc.dylib (architecture arm64):
	/opt/local/lib/libjemalloc.2.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1300.36.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.0.0)

[kencu (Ken)]

success:

 % port -v installed rust
The following ports are currently installed:
  rust @1.66.0_0 requested_variants='' platform='darwin 22' archs='arm64' date='2022-12-29T12:00:46-0800'
  rust @1.66.0_0+universal (active) requested_variants='+universal' platform='darwin 22' archs='arm64 x86_64' date='2022-12-31T10:40:31-0800'

It's a little weird, in that the dylibs for each arch have different names:

% port contents rust | grep dylib | xargs file      
/opt/local/lib/librustc_driver-3b79dae3146d75e8.dylib:                          Mach-O 64-bit dynamically linked shared library x86_64
/opt/local/lib/librustc_driver-9cb14be8a64d2368.dylib:                          Mach-O 64-bit dynamically linked shared library arm64
/opt/local/lib/libstd-556e5c085d373926.dylib:                                   Mach-O 64-bit dynamically linked shared library x86_64
/opt/local/lib/libstd-c11d89f5f427923f.dylib:                                   Mach-O 64-bit dynamically linked shared library arm64
/opt/local/lib/libtest-127f34b82a1e8ad5.dylib:                                  Mach-O 64-bit dynamically linked shared library arm64
/opt/local/lib/libtest-61048579174a71f6.dylib:                                  Mach-O 64-bit dynamically linked shared library x86_64
/opt/local/lib/rustlib/aarch64-apple-darwin/lib/libstd-c11d89f5f427923f.dylib:  Mach-O 64-bit dynamically linked shared library arm64
/opt/local/lib/rustlib/aarch64-apple-darwin/lib/libtest-127f34b82a1e8ad5.dylib: Mach-O 64-bit dynamically linked shared library arm64
/opt/local/lib/rustlib/x86_64-apple-darwin/lib/libstd-556e5c085d373926.dylib:   Mach-O 64-bit dynamically linked shared library x86_64
/opt/local/lib/rustlib/x86_64-apple-darwin/lib/libtest-61048579174a71f6.dylib:  Mach-O 64-bit dynamically linked shared library x86_64

but perhaps that is how rust works. The binaries are fat:

% file /opt/local/bin/rustc
/opt/local/bin/rustc: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64:Mach-O 64-bit executable arm64]
/opt/local/bin/rustc (for architecture x86_64):	Mach-O 64-bit executable x86_64
/opt/local/bin/rustc (for architecture arm64):	Mach-O 64-bit executable arm64

we'll see how it works. I'll put up a PR for the minor rust changes.

[kencu (Ken)]

quick hack fix, full PR in a bit once universal rust is tested.

% cat ~/patch-rust-portfile-universal.diff
diff --git a/lang/rust/Portfile b/lang/rust/Portfile
index 41450176a4e..2e48cb7117e 100644
--- a/lang/rust/Portfile
+++ b/lang/rust/Portfile
@@ -132,9 +132,10 @@ if { ${os.platform} eq "darwin" && ${os.major} < 9 } {
     configure.args-append   --disable-rpath
 }
 if { "port:jemalloc" in ${depends_lib} } {
-    configure.args-append   --set=rust.jemalloc
+#    configure.args-append   --set=rust.jemalloc
 }
 
+muniversal.dont_diff        /opt/local/lib/rustlib/manifest-rustfmt-preview
 triplet.add_build           all
 triplet.add_host            all
 

[kencu (Ken)]

This approach doesn't work to build universal ports using rust so far.

With either version of the muniversal PortGroup, you just wind up with both of the built architectures being the host architecture.

According to this: ​https://rust-lang.github.io/rustup/cross-compilation.html what is needed instead is to get the one installed host version of rust to add the toolchain needed to support the arch you are cross-compiling to, and then tell it to build for that alternative toolchain using something like this:

cargo build --target=x86_64-apple-darwin22.2.0
cargo build --target=arm64-apple-darwin22.2.0

and then lipo those together with the muniversal PG.

SO -- step one is to get a cross-compiling version of rust/cargo working...

[MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)]

Plese forgive me if I am missing something, but aren't all of these steps already finished?
I have tested the universal Rust build much more thoroughly on i386 and x86_64 (as evidenced by the recently discovered bugs).
However, I believe the principles should be the same for arm64 and x86_64.

By installing the Rust for both architectures, you get the entire toolchains for both architectures.
cargo then builds the correct binary through ​CARGO_BUILD_TARGET.

Is there some reason to believe this strategy is not working?

I am in the process of trying to install arm64/x86_64 universal build of Rust to test things out, but I am not there yet.

[kencu (Ken)]

Looking forward to it! Yes, it's not working quite as planned as yet.

What you get when you try to install something universal arm64/x86_64 built with rust/cargo is the primary dylib of the finished build is just a text file saying that the attempted merge of the two dylibs differed and could not be merged :>

In the work folder, all the dylibs (x86_64 folder and arm64 folder) are arm64 dylibs, with nothing cross compiled.

Happy to work with you more when you can get there too -- it's hard to explain and do this remotely, and will be much easier when you see it for yourself.

% sudo port -v destroot rav1e +universal

% cd `port work rav1e`

% pwd                                              
/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_multimedia_rav1e/rav1e/work

% find . | grep librav1e.0.5.1.dylib | xargs file  
./destroot/opt/local/lib/librav1e.0.5.1.dylib:           ASCII text, with very long lines (411)
./destroot-arm64/opt/local/lib/librav1e.0.5.1.dylib:     Mach-O 64-bit dynamically linked shared library arm64
./destroot-ppc-intel/opt/local/lib/librav1e.0.5.1.dylib: Mach-O 64-bit dynamically linked shared library arm64
./destroot-intel/opt/local/lib/librav1e.0.5.1.dylib:     Mach-O 64-bit dynamically linked shared library arm64
./destroot-x86_64/opt/local/lib/librav1e.0.5.1.dylib:    Mach-O 64-bit dynamically linked shared library arm64

% cat ./destroot/opt/local/lib/librav1e.0.5.1.dylib
Binary files /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_multimedia_rav1e/rav1e/work/destroot-arm64//opt/local/lib/librav1e.0.5.1.dylib and /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_multimedia_rav1e/rav1e/work/destroot-ppc-intel//opt/local/lib/librav1e.0.5.1.dylib differ

[kencu (Ken)]

BTW to build universal rust on an M1, I believe I have either pushed or PR'd all the fixes needed to get through to universal rust, except for the small patch in this ticket for rust.

So if something fails to build, it's probably PR'd.

[MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)]

There is a ​pull request that hopefully fixes this problem.

[MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)]

In eb9275fb1b0b4b9cbdb76fca3624d01b7b74094f/macports-ports (master):

rust: fix arm64/x86_64 build & remove jemalloc dep

Rust builds its own copy of jemalloc
Removing it as a dependency as part of this commit minimizes rebuilds

Fixes #66591

[kencu (Ken)]

rust builds universal using the PR, thanks.

I’m still having trouble building universal ports with it.

Have you had success building, say, rav1e universal with it?

[MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)]

I have built rav1e +universal but only on i386/x86_64.

[kencu (Ken)]

I still get this ​https://trac.macports.org/ticket/66591#comment:11 , unfortunately, on an arm Mac.

want a full log?

[MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)]

That is very strange.
I will try to build arm64/x86_64 rav1e in a bit.

[kencu (Ken)]

It might be something specific to the rav1e build, I suppose. In the x86_64 target, all the dylibs are arm64, but the rav1e binary is x86_64:

% find ./rav1e-0.5.1-x86_64/target | grep dylib | xargs file
./rav1e-0.5.1-x86_64/target/aarch64-apple-darwin/release/librav1e.dylib:                       Mach-O 64-bit dynamically linked shared library arm64
./rav1e-0.5.1-x86_64/target/aarch64-apple-darwin/release/deps/librav1e.dylib:                  Mach-O 64-bit dynamically linked shared library arm64
./rav1e-0.5.1-x86_64/target/release/deps/libnoop_proc_macro-b6ab08a856a90633.dylib:            Mach-O 64-bit dynamically linked shared library arm64
./rav1e-0.5.1-x86_64/target/release/deps/libnum_derive-4fda59adadbd9587.dylib:                 Mach-O 64-bit dynamically linked shared library arm64
./rav1e-0.5.1-x86_64/target/release/deps/librust_hawktracer_proc_macro-3ebbe354c8c07b39.dylib: Mach-O 64-bit dynamically linked shared library arm64
./rav1e-0.5.1-x86_64/target/release/deps/libpaste-20e5d05a7d7e8cfe.dylib:                      Mach-O 64-bit dynamically linked shared library arm64
./rav1e-0.5.1-x86_64/target/release/deps/libsimd_helpers-f72d8d5113bb2161.dylib:               Mach-O 64-bit dynamically linked shared library arm64
./rav1e-0.5.1-x86_64/target/release/deps/libthiserror_impl-5a95860f9ea52449.dylib:             Mach-O 64-bit dynamically linked shared library arm64
./rav1e-0.5.1-x86_64/target/release/deps/libarg_enum_proc_macro-d93e60f432b7cc0d.dylib:        Mach-O 64-bit dynamically linked shared library arm64

 % find ./rav1e-0.5.1-x86_64/target | grep release/rav1e | xargs file
./rav1e-0.5.1-x86_64/target/x86_64-apple-darwin/release/rav1e.d:               ASCII text, with very long lines (25836)
./rav1e-0.5.1-x86_64/target/x86_64-apple-darwin/release/rav1e:                 Mach-O 64-bit executable x86_64
./rav1e-0.5.1-x86_64/target/aarch64-apple-darwin/release/rav1e-uninstalled.pc: ASCII text
./rav1e-0.5.1-x86_64/target/aarch64-apple-darwin/release/rav1e.pc:             ASCII text
./rav1e-0.5.1-x86_64/target/aarch64-apple-darwin/release/rav1e.h:              c program text, Unicode text, UTF-8 text

so how odd is that?

I will try another port and see what happens there.

[kencu (Ken)]

Good news! librsvg, which also builds with rust/cargo, is fine:

% port contents librsvg | grep dylib | xargs file
/opt/local/lib/librsvg-2.2.dylib: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit dynamically linked shared library x86_64] [arm64]
/opt/local/lib/librsvg-2.2.dylib (for architecture x86_64):	Mach-O 64-bit dynamically linked shared library x86_64
/opt/local/lib/librsvg-2.2.dylib (for architecture arm64):	Mach-O 64-bit dynamically linked shared library arm64

so increasingly something unique to rav1e it seems ...

[MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)]

Replying to kencu:

It might be something specific to the rav1e build, I suppose. In the x86_64 target, all the dylibs are arm64, but the rav1e binary is x86_64:

The problem seems to be that rav1e uses cargo-c in addition to cargo.
There is a ​pull request that attempts to fix the problem.

[MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)]

In 8d7c8273d406f18d7a2837852564bf7cbe253841/macports-ports (master):

rav1e: set target when calling cargo-c

See #66591#comment:19
No revbump since port either builds correctly or not at all