py-cryptography @3.4.7: ports failing to build for MacOS 10.12 and earlier

[mascguy (Christopher Nielsen)]

I'm seeing two potential linker-related issues:

     Running `rustc --crate-name cryptography_rust --edition=2018 src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type cdylib --emit=dep-info,link -C opt-level=3 -C lto=thin --crate-type cdylib -C link-arg=-undefined -C link-arg=dynamic_lookup -C metadata=f440199ca35a370a --out-dir /opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_python_py-cryptography/py39-cryptography/work/pyca-cryptography-b2536af/src/rust/target/x86_64-apple-darwin/release/deps --target x86_64-apple-darwin -L dependency=/opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_python_py-cryptography/py39-cryptography/work/pyca-cryptography-b2536af/src/rust/target/x86_64-apple-darwin/release/deps -L dependency=/opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_python_py-cryptography/py39-cryptography/work/pyca-cryptography-b2536af/src/rust/target/release/deps --extern pyo3=/opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_python_py-cryptography/py39-cryptography/work/pyca-cryptography-b2536af/src/rust/target/x86_64-apple-darwin/release/deps/libpyo3-1cc1820c704d7621.rlib -C linker=/opt/local/libexec/rust-compiler-wrap/bin/ld -C 'link-args=-Wl,-install_name,@rpath/_rust.abi3.so'`
error: linking with `/opt/local/libexec/rust-compiler-wrap/bin/ld` failed: exit code: 1

And:

  = note: ld: unknown option: -exported_symbols_list,/opt/local/var/macports/build

[mascguy (Christopher Nielsen)]

Here's one example from our buildbots:

​https://build.macports.org/builders/ports-10.12_x86_64-builder/builds/145844/steps/install-port/logs/stdio

[cjones051073 (Chris Jones)]

Yes, I saw that a while back. Its an upstream issue I think, their make file adds the above option.

[mascguy (Christopher Nielsen)]

Replying to cjones051073:

Yes, I saw that a while back. Its an upstream issue I think, their make file adds the above option.

Any ideas with regards to the first linker failure?

[cjones051073 (Chris Jones)]

probably the same issue I would guess.

[cjones051073 (Chris Jones)]

Not sure what's going on here. debugging a bit another port showing the same, fused, I patched the wrapper script to print the failing command (sorry, bit long....)

  = note: ld: unknown option: -exported_symbols_list,/opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/.tmp/rustcDGQ1So/list
          /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/.home/.cargo/macports/hyper-0.14.7
          /opt/local/libexec/llvm-11/bin/ld /opt/local/lib/libMacportsLegacySupport.a -L /opt/local/lib/rustlib/x86_64-apple-darwin/lib /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/hyper-a4b0e5e3a2a46c9f.hyper.diqsjegp-cgu.0.rcgu.o /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/hyper-a4b0e5e3a2a46c9f.hyper.diqsjegp-cgu.1.rcgu.o /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/hyper-a4b0e5e3a2a46c9f.hyper.diqsjegp-cgu.10.rcgu.o /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/hyper-a4b0e5e3a2a46c9f.hyper.diqsjegp-cgu.11.rcgu.o /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/hyper-a4b0e5e3a2a46c9f.hyper.diqsjegp-cgu.12.rcgu.o /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/hyper-a4b0e5e3a2a46c9f.hyper.diqsjegp-cgu.13.rcgu.o /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/hyper-a4b0e5e3a2a46c9f.hyper.diqsjegp-cgu.14.rcgu.o /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/hyper-a4b0e5e3a2a46c9f.hyper.diqsjegp-cgu.15.rcgu.o /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/hyper-a4b0e5e3a2a46c9f.hyper.diqsjegp-cgu.2.rcgu.o /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/hyper-a4b0e5e3a2a46c9f.hyper.diqsjegp-cgu.3.rcgu.o /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/hyper-a4b0e5e3a2a46c9f.hyper.diqsjegp-cgu.4.rcgu.o /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/hyper-a4b0e5e3a2a46c9f.hyper.diqsjegp-cgu.5.rcgu.o /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/hyper-a4b0e5e3a2a46c9f.hyper.diqsjegp-cgu.6.rcgu.o /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/hyper-a4b0e5e3a2a46c9f.hyper.diqsjegp-cgu.7.rcgu.o /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/hyper-a4b0e5e3a2a46c9f.hyper.diqsjegp-cgu.8.rcgu.o /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/hyper-a4b0e5e3a2a46c9f.hyper.diqsjegp-cgu.9.rcgu.o -o /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/libhyper-a4b0e5e3a2a46c9f.dylib -exported_symbols_list,/opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/.tmp/rustcDGQ1So/list /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/hyper-a4b0e5e3a2a46c9f.lkw390eqixpk2xz.rcgu.o -dead_strip -dynamiclib -dylib -L /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps -L /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/release/deps -L /opt/local/lib/rustlib/x86_64-apple-darwin/lib /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/libwant-a285df1206f555f6.rlib /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/libtry_lock-30a14a7d787f502e.rlib /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/libsocket2-0674e1316e6e4289.rlib /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/libh2-7ca1822a883635e0.rlib /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/libindexmap-01f155597b2be3c4.rlib /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/libhashbrown-0124d41b1d172c1d.rlib /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/libtokio_util-1afdc1c47f0bea0c.rlib /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/libfutures_sink-31fe2d91928d024b.rlib /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/libhttparse-8a874a201e159c78.rlib /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/libtower_service-3fee48f50ac4104e.rlib /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/libfutures_channel-e13571ccf8fa0e5a.rlib /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/libhttp_body-db223739896f0a40.rlib /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/libfutures_util-e8ee52e0558927f8.rlib /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/libfutures_io-c247ee0047fa4ba3.rlib /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/libslab-7c06fd9b66942bc5.rlib /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/libfutures_task-8750f938b14dbc70.rlib /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/libpin_utils-0b4dbfd71b220833.rlib /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/libfutures_core-87c62082a05b7f5a.rlib /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/libpin_project-8073a0f0c313dec2.rlib /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/libtokio-de99dacbb5ded683.rlib /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/libnum_cpus-111c717377c865f5.rlib /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/libmemchr-2af7b35b6d476da1.rlib /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/libmio-03b7383b1df40e4a.rlib /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/liblibc-927d181e4465fb44.rlib /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/libhttp-8263f0a6aeda2e12.rlib /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/libitoa-016bb70a773dd000.rlib /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/libbytes-57f3cbfbaa31d51f.rlib /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/libfnv-c0496f1bd67a4cd6.rlib /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/libtracing-cbc62f4e125d692b.rlib /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/libpin_project_lite-0daf4547ac550f74.rlib /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/libtracing_core-911dd28c0e1da407.rlib /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/liblazy_static-d1fc547ba207aea0.rlib /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/liblog-f9aefa55e03972e4.rlib /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/ffsend-0.2.72/target/x86_64-apple-darwin/release/deps/libcfg_if-ab51904cb344c487.rlib /opt/local/lib/rustlib/x86_64-apple-darwin/lib/libstd-85717d025629e453.rlib /opt/local/lib/rustlib/x86_64-apple-darwin/lib/libpanic_unwind-7b93fc750326a2be.rlib /opt/local/lib/rustlib/x86_64-apple-darwin/lib/libobject-e0b57a06c6fe8986.rlib /opt/local/lib/rustlib/x86_64-apple-darwin/lib/libaddr2line-1010a375b16ca14e.rlib /opt/local/lib/rustlib/x86_64-apple-darwin/lib/libgimli-53fd7cffb1a23848.rlib /opt/local/lib/rustlib/x86_64-apple-darwin/lib/librustc_demangle-7382aa49fc87fa87.rlib /opt/local/lib/rustlib/x86_64-apple-darwin/lib/libhashbrown-4713b0a77ad1af37.rlib /opt/local/lib/rustlib/x86_64-apple-darwin/lib/librustc_std_workspace_alloc-22bb2f853c557dc7.rlib /opt/local/lib/rustlib/x86_64-apple-darwin/lib/libunwind-664f179efd682082.rlib /opt/local/lib/rustlib/x86_64-apple-darwin/lib/libcfg_if-039cb2e28d943be0.rlib /opt/local/lib/rustlib/x86_64-apple-darwin/lib/liblibc-2e47240a720e4361.rlib /opt/local/lib/rustlib/x86_64-apple-darwin/lib/liballoc-a512143eec6cedff.rlib /opt/local/lib/rustlib/x86_64-apple-darwin/lib/librustc_std_workspace_core-b76b9c8edf8d9e8d.rlib /opt/local/lib/rustlib/x86_64-apple-darwin/lib/libcore-daf51c1efc4a00aa.rlib /opt/local/lib/rustlib/x86_64-apple-darwin/lib/libcompiler_builtins-64761f1fd53fac24.rlib -liconv -lSystem -lresolv -lc -lm -liconv

and indeed if I run that command by hand in the build area I get the same error

ld: unknown option: -exported_symbols_list,/opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/.tmp/rustcDGQ1So/list

thing is, the linker being used seems to understand it if I try with

 > /opt/local/libexec/llvm-11/bin/ld -exported_symbols_list
ld: missing file after -exported_symbols_list

so that suggests it does know about this flag...

[kencu (Ken)]

if you are calling LD directly, no comma:

-exported_symbols_list /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/.tmp/rustcDGQ1So/list

[kencu (Ken)]

if you are passing it to clang (or gcc) to send to the linker, as you usually (99.99999% of the time) are meant to:

clang -Wl,-exported_symbols_list,/opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_net_ffsend/ffsend/work/.tmp/rustcDGQ1So/list BLAHBLAHBLAH

[cjones051073 (Chris Jones)]

yep, just worked that out. I think the issue is the cargo PGs are explicitly telling the build to use ld as the linker, whereas it seems it actually wants to use the c compiler (clang) instead.

[kencu (Ken)]

Das is Verboten

Thou Shalt Never Call LD directly.

[kencu (Ken)]

seriously, the clang driver does 1000 things to make the link work properly that LD has no internal knowledge of. You should almost never call LD yourself.

[Chris Jones <jonesc@…>]

In 886d2e9acbada0ddbfd7cd3e417097b4226b48cb/macports-ports (master):

rust-compiler-wrap: Directly embed legacy-support flags, use clang to link

Closes: #62843

[mascguy (Christopher Nielsen)]

Rev-bumped py-cryptography, to rebuild against latest rust-compiler-wrap:

​https://github.com/macports/macports-ports/commit/97d07793e148398d587a56d746556f55e36b33de

[cjones051073 (Chris Jones)]

Damn, whilst everything worked fine for me locally here rust is now failing to build in the buildbots on 10.12 and older… I’ll take a local tomorrow..

[mascguy (Christopher Nielsen)]

Replying to cjones051073:

Damn, whilst everything worked fine for me locally here rust is now failing to build in the buildbots on 10.12 and older… I’ll take a local tomorrow..

If you need an additional set of eyes on anything, let me know. I'd need to add a lot of temporary ui_debug output to understand what's happening locally, but happy to help...

[mascguy (Christopher Nielsen)]

Looking at the failed builds, it appears that rustc is crashing with a SIGSEGV:

Building stage1 std artifacts (x86_64-apple-darwin -> x86_64-apple-darwin)
running: "/opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_lang_rust/rust/work/rustc-1.52.0-src/build/stage0-x86_64/bin/cargo" "build" "--target" "x86_64-apple-darwin" "-Zbinary-dep-depinfo" "-j" "8" "-v" "--release" "--frozen" "--features" "panic-unwind backtrace compiler-builtins-c" "--manifest-path" "/opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_lang_rust/rust/work/rustc-1.52.0-src/library/test/Cargo.toml" "--message-format" "json-render-diagnostics"
error: process didn't exit successfully: `/opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_lang_rust/rust/work/rustc-1.52.0-src/build/bootstrap/debug/rustc -vV` (exit code: 254)
--- stdout

Did not run successfully: signal: 11

The following suggests that it could be an unpatched issue in the LLVM toolchain:

​https://github.com/rust-lang/rust/issues/68731

Are we using the Xcode toolchain, or MacPorts-provided?

[mascguy (Christopher Nielsen)]

This is the bug report for the unpatched LLVM issue:

​https://bugzilla.redhat.com/show_bug.cgi?id=1797127

With it ultimately fixed via this LLVM commit:

​AtomicExpand Fix a crash bug when lowering unordered loads to cmpxchg