cargo-c fails to build on 10.13

[bK4gYuRo]

As far as I can check, the build fails on buildbots too.

[bK4gYuRo]

cargo-c build main.log

[mascguy (Christopher Nielsen)]

Bear in mind that the 10.13 buildbot has been offline for the past few days, and there is a backlog. (Including for this specific port, I believe.)

So the 10.13 build failure is likely outdated, and will be taken care of once the buildbot is brought back online.

[mascguy (Christopher Nielsen)]

As for your local build failure, I see a few different errors.

The most predominant is the following link error, but there's also an earlier link issue that may be the root cause:

:info:build           Undefined symbols for architecture x86_64:2143
:info:build             "___isPlatformVersionAtLeast", referenced from:2144
:info:build                 _sectransp_connect_common in libcurl_sys-9ad769ad2cc270c9.rlib(sectransp.o)2145
:info:build                 _sectransp_connect_step2 in libcurl_sys-9ad769ad2cc270c9.rlib(sectransp.o)2146
:info:build           ld: symbol(s) not found for architecture x86_642

[bit-hug]

I have the same local build failure here on my 10.13 system. It seems cargo-c-0.9.5_0.darwin_17.x86_64.tbz2 can't be found, which might be the root cause?

[mascguy (Christopher Nielsen)]

Replying to bit-hug:

I have the same local build failure here on my 10.13 system. It seems cargo-c-0.9.5_0.darwin_17.x86_64.tbz2 can't be found, which might be the root cause?

That's expected, as the 10.13 buildbot isn't running. So until it's brought back online, and catches up with its current backlog, binaries won't be available for 10.13.

Does that make sense?

[bit-hug]

Ah, I see, thank you! Is there any time estimate available for the buildbot to catch up?

[mascguy (Christopher Nielsen)]

Replying to bit-hug:

Ah, I see, thank you! Is there any time estimate available for the buildbot to catch up?

Asked for an ETA a little while ago, and waiting to hear back. Will let you know!

[mascguy (Christopher Nielsen)]

It may be a few more days before the 10.13 buildbot can be brought back online, so we should try to resolve your folks' local build failure.

Can you folks start by running sudo port -N rev-upgrade?

After that, can you try sudo port diagnose?

[bK4gYuRo]

Just an observation, the main.log from buildbot at ​https://build.macports.org/builders/ports-10.13_x86_64-builder/builds/128920/steps/install-port/logs/stdio has the same error that I see in my local build:

= note: Undefined symbols for architecture x86_64:
            "___isPlatformVersionAtLeast", referenced from:
                _sectransp_connect_common in libcurl_sys-9ad769ad2cc270c9.rlib(sectransp.o)
                _sectransp_connect_step2 in libcurl_sys-9ad769ad2cc270c9.rlib(sectransp.o)
          ld: symbol(s) not found for architecture x86_64
          clang: error: linker command failed with exit code 1 (use -v to see invocation)

I am afraid this is something caused by XCode version used in 10.13, and not just buildbot being down. I doubt that mere bringing up of the buildbot will fix the error.

[mascguy (Christopher Nielsen)]

This port built successfully for me locally, on my 10.13 systems. So I'm not (yet) convinced that it's an issue with the port, nor Xcode versions. (Nothing's 100% certain. But the fact that it's building for others, is a positive sign.)

Can you folks go through the steps mentioned in comment:12, and report back?

[bK4gYuRo]

sudo port rev-upgrade comes up clean, sudo port diagnose yields nothing. sudo port install cargo-c fails with the same errors in the main.log. To make sure we are comparing apples to apples, what is your version of XCode that succeeds to build cargo-c? Mine is 9.4.1. Also, if it is relevant, I have rustc 1.56.1.

[mascguy (Christopher Nielsen)]

Replying to bK4gYuRo:

sudo port rev-upgrade comes up clean, sudo port diagnose yields nothing. sudo port install cargo-c fails with the same errors in the main.log. To make sure we are comparing apples to apples, what is your version of XCode that succeeds to build cargo-c? Mine is 9.4.1. Also, if it is relevant, I have rustc 1.56.1.

Presently I'm building with Xcode 10.1. But I'll switch to 9.4.1, and rebuild with that. More to follow.

[mascguy (Christopher Nielsen)]

Thus far, I didn't consider that cargo-c may not be using the Xcode toolchain at all. And sure enough, it doesn't appear to, at least on 10.13:

$ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.13.6
BuildVersion:	17G14019

$ port info cargo-c
cargo-c @0.9.5 (devel)

Description:          cargo applet to build and install C-ABI compatibile dynamic and static libraries
Homepage:             https://github.com/lu-zero/cargo-c

Build Dependencies:   clang-13, cargo, pkgconfig
Library Dependencies: libgit2, libiconv, zlib, openssl3
Platforms:            darwin
License:              MIT
Maintainers:          Email: mcalhoun@macports.org, GitHub: MarcusCalhoun-Lopez
                      Policy: openmaintainer

And based on the build dependencies, it appears to use MacPorts' Clang.

I'll let Marcus confirm whether any Xcode-related components still come into play during the build, such that the Xcode version might have an impact. But if not, then we might need to look at an upstream dependency instead.

Marcus/Anyone, thoughts/ideas related to troubleshooting this ticket?

[kencu (Ken)]

isPlatformVersionAtLeast is a function in the compiler_rt library in newer versions of clang:

​https://github.com/llvm/llvm-project/blob/main/compiler-rt/lib/builtins/os_version_check.c

Normally it is found in the compiler_rt.a library that is automatically added to the link libraries by clang when clang drives the linker (ld64) on Darwin.

If you are driving the linker in some other way (ghc, rust perhaps, gcc), or if you are specifying nostdlibs then you have to add the full path to the static clang_rt library to get that function to be linked in.

Presumably one of those things is messing this up. I suspect that older xcode versions might not use that function but newer ones do, which could be why xcode9 works yet xcode10 fails.

Maybe.

[tomio-arisaka (Tomio Arisaka)]

This is my solution to the issue:

$ sudo port select --set clang mp-clang-13
$ sudo port install cargo-c
$ sudo port select --set clang none

Results:

$ port installed cargo-c
The following ports are currently installed:
  cargo-c @0.9.5_0 (active)
$ 
$ xcodebuild -version
Xcode 9.4.1
Build version 9F2000
$ 

I don't know why the issue happens. But, there are the following differences between the failed case and the succeeded case:

• The failed case log includes the next option: "-L /Library/Developer/CommandLineTools/usr/lib/clang/10.0.0/lib/darwin"

• The succeeded case log includes the next option: "-L /opt/local/libexec/llvm-13/lib/clang/13.0.0/lib/darwin"

I have installed the "Command Line Tools" before.

[mascguy (Christopher Nielsen)]

Replying to tomio-arisaka:

This is my solution to the issue:

$ sudo port select --set clang mp-clang-13
$ sudo port install cargo-c
$ sudo port select --set clang none

Results:

$ port installed cargo-c
The following ports are currently installed:
  cargo-c @0.9.5_0 (active)

$ xcodebuild -version
Xcode 9.4.1
Build version 9F2000

I don't know why the issue happens. But, there are the following differences between the failed case and the succeeded case:

• The failed case log includes the next option: "-L /Library/Developer/CommandLineTools/usr/lib/clang/10.0.0/lib/darwin"
• The succeeded case log includes the next option: "-L /opt/local/libexec/llvm-13/lib/clang/13.0.0/lib/darwin"

It's interesting that selecting Clang 13 as the default, fixes the issue.

On my 10.13 system, I was able to successfully build cargo-c with both Xcode 9.4.1, and 10.1, without an issue. (And without a default Clang set.)

@cjones, any thoughts/ideas?

[kencu (Ken)]

see what cctools and ld64 variants are installed

[tomio-arisaka (Tomio Arisaka)]

My understanding of the issue is as follows:

When building "cargo-c" with MacPorts on macOS High Sierra, the error occurs.

Extracts from the error log:

:info:build      Running `/opt/local/bin/rustc --crate-name cargo_cinstall --edition=2021 src/bin/cinstall.rs ...
  ...
  ... -C linker=/opt/local/bin/clang-mp-13 -L /Library/Developer/CommandLineTools/usr/lib/clang/10.0.0/lib/darwin ...
  ...
:info:build error: linking with `/opt/local/bin/clang-mp-13` failed: exit status: 1
:info:build   |
:info:build   = note: "/opt/local/bin/clang-mp-13" "-m64" "-arch" "x86_64" ...
  ...
  ...  "-L" "/Library/Developer/CommandLineTools/usr/lib/clang/10.0.0/lib/darwin" ...
  ...
  ...  "-lclang_rt.osx" ...
  ...
:info:build   = note: ...
  ...
:info:build           Undefined symbols for architecture x86_64:
:info:build             "___isPlatformVersionAtLeast", referenced from:
  ...
:info:build error: linking with `/opt/local/bin/clang-mp-13` failed: exit status: 1

It seems that the error occurs when the options of "/opt/local/bin/rustc" include "-C linker=/opt/local/bin/clang-mp-13 -L /Library/Developer/CommandLineTools/usr/lib/clang/10.0.0/lib/darwin".

I guess that

"clang-mp-13" searches the "/Library/Developer/CommandLineTools/usr/lib/clang/10.0.0/lib/darwin" directory for the "libclang_rt.osx" library,

and

"clang-mp-13" searches the "libclang_rt.osx" library for the "isPlatformVersionAtLeast" symbol.

But, cannot find it.

"Apple clang" includes the "libclang_rt.osx" library:

( It has been installed with the "xcode-select --install" command. Of course, it is not for "clang-mp-13" )

$ ls -lt /Library/Developer/CommandLineTools/usr/lib/clang/10.0.0/lib/darwin/ | grep 'libclang_rt.osx'
-rw-r--r--  1 root  admin   104360  9 26  2018 libclang_rt.osx.a
$ 
$ otool -L /Library/Developer/CommandLineTools/usr/lib/clang/10.0.0/lib/darwin/libclang_rt.osx.a | grep 'os_version_check'
/Library/Developer/CommandLineTools/usr/lib/clang/10.0.0/lib/darwin/libclang_rt.osx.a(os_version_check.c.o):
$ 
$ strings /Library/Developer/CommandLineTools/usr/lib/clang/10.0.0/lib/darwin/libclang_rt.osx.a | grep '___is'
___isOSVersionAtLeast
___isTargetPlatformNative
___isTargetVariantOSVersionAtLeast
$

So the "libclang_rt.osx" library does not include the "isPlatformVersionAtLeast" symbol. Then "clang-mp-13" cannot find it.

On the other hand, when the options of "/opt/local/bin/rustc" include "-C linker=/opt/local/bin/clang-mp-13 -L /opt/local/libexec/llvm-13/lib/clang/13.0.0/lib/darwin", the phase of building succeeds.

This case happens with the following commands:

$ sudo port select --set clang mp-clang-13
$ sudo port install cargo-c

"clang-13" includes the "libclang_rt.osx" library:

$ port contents clang-13 | grep 'libclang_rt.osx'
  /opt/local/libexec/llvm-13/lib/clang/13.0.0/lib/darwin/libclang_rt.osx.a
$ 
$ ls -lt /opt/local/libexec/llvm-13/lib/clang/13.0.0/lib/darwin/ | grep 'libclang_rt.osx'
-rw-r--r--  1 root  wheel   405024 10 25 06:17 libclang_rt.osx.a
$ 
$ otool -L /opt/local/libexec/llvm-13/lib/clang/13.0.0/lib/darwin/libclang_rt.osx.a | grep 'os_version_check'
/opt/local/libexec/llvm-13/lib/clang/13.0.0/lib/darwin/libclang_rt.osx.a(os_version_check.c.o):
$ 
$ strings /opt/local/libexec/llvm-13/lib/clang/13.0.0/lib/darwin/libclang_rt.osx.a | grep '___is'
___isOSVersionAtLeast
___isPlatformVersionAtLeast
$

So the "libclang_rt.osx" library includes the "isPlatformVersionAtLeast" symbol. Then "clang-mp-13" can find it.

[cjones051073 (Chris Jones)]

If running

sudo port select --set clang mp-clang-13

'fixes' things, then this indicates some part of the cargo-c build is not properly respecting macPorts choice of compiler, and is running clang instead of the configured compiler.

[Chris Jones <jonesc@…>]

In 3f0613b2c68ba42a0b1d062ab434d6116b6eeef4/macports-ports (master):

cargo PG: Create temporary bin dir to direct clang(++) to correct compiler
cargo/rust builds often do not properly respect MacPorts compiler selection,
so use the temporary bin directory trick to ensure 'clang' and 'clang++'
found via PATH point where we want.
See: #64088

[cjones051073 (Chris Jones)]

Please try with the above, which is the standard trick for mis-behaving build configurations like this.

[bit-hug]

Replying to tomio-arisaka:

On the other hand, when the options of "/opt/local/bin/rustc" include "-C linker=/opt/local/bin/clang-mp-13 -L /opt/local/libexec/llvm-13/lib/clang/13.0.0/lib/darwin", the phase of building succeeds.

This case happens with the following commands:

$ sudo port select --set clang mp-clang-13
$ sudo port install cargo-c

Just to mention, if it's helpful, in my case, selecting Clang 13 as the default does not fix the issue.

However, from my logs, it seems that the options of "/opt/local/bin/rustc" include "-C linker=/opt/local/bin/clang-mp-13 -L /Library/Developer/CommandLineTools/usr/lib/clang/10.0.0/lib/darwin" irrespective of the selected default clang.

Otherwise, I have the following

$ sudo port -N rev-upgrade
--->  Updating database of binaries
--->  Scanning binaries for linking errors
--->  No broken files found.                             
--->  No broken ports found.
$ sudo port diagnose
$ /usr/bin/xcodebuild -version
Xcode 9.2
Build version 9C40b

[kencu (Ken)]

yes, the clang being used to link should either not be forced to a specific clang library, or at least be forced to the matching libary if it is forced to something.

I wonder what witchcraft led rust to think it needed to set clang's runtime library specifically? (and wrongly).

[mascguy (Christopher Nielsen)]

Replying to cjones051073:

Please try with the above, which is the standard trick for mis-behaving build configurations like this.

Thanks @cjones!

I've also initiated rebuilds for cargo-c where necessary, on the buildbots.

[ryandesign (Ryan Carsten Schmidt)]

I've removed the cargo-c failcache entry for the 10.13 buildbot worker so that any other queued build that needs cargo-c will immediately reattempt to build cargo-c.

[ryandesign (Ryan Carsten Schmidt)]

Success: ​https://build.macports.org/builders/ports-10.13_x86_64-builder/builds/131733

[mascguy (Christopher Nielsen)]

Replying to ryandesign:

I've removed the cargo-c failcache entry for the 10.13 buildbot worker so that any other queued build that needs cargo-c will immediately reattempt to build cargo-c.

Ryan, can folks with buildbot access do this as well? If so, how is this done?

[cjones051073 (Chris Jones)]

Are you asking if someone can schedule a build of cargo-c for you, or *how* to schedule a build ?

[mascguy (Christopher Nielsen)]

Replying to cjones051073:

Are you asking if someone can schedule a build of cargo-c for you, or *how* to schedule a build ?

I know how to schedule a build, and can now do so.

Instead, what I'm specifically asking about, is whether folks with buildbot access can remove a prior failcache entry via the buildbot GUI. (And without having to make a dummy change to the portfile.) Make sense?

[cjones051073 (Chris Jones)]

I am not sure what you want, sorry. If the portfile has already been updated to remove a failcache setting for some OSes, then pushing that commit will trigger new build attempts (where previously it was skipped). Nothing needs to be done for this.

[cjones051073 (Chris Jones)]

Ok, I get what you are asking now....

I don't know ....

[MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)]

cargo-c seems to be successfully building on the ​buildbots, so I think this ticket can be closed.
Please feel free to reopen if this issue has not been resolved.