upgrade: newly installed dependencies inherit variants from the dependent port

[fhgwright (Fred Wright)]

The situation that caused an instance of #67320 was an upgrade of clang-14 +universal, that thought that it needed clang-11 +universal to build it. And the reason that clang-14 itself was +universal was most likely due to an earlier occurrence of the same bug.

I'm not sure if gcc needs to be +universal to build +universal targets (I've seen some evidence that it does), but this is certainly not true of clang. The only part of clang that needs to be built for the target architecture is the compiler-rt stuff, which is already built for multiple architectures regardless of the universal variant. And llvm normally builds for all supported target architectures, regardless of its universal variant.

I think the relevant code is in base, but I could be wrong about that.

[jmroot (Joshua Root)]

Please provide a log or debug output for an affected install. Base sets depends_skip_archcheck for compilers it adds as dependencies, provided they support the -arch flag. ​https://github.com/macports/macports-base/blob/v2.8.1/src/port1.0/portconfigure.tcl#L1695-L1699

Testing with a simple test port shows this is working as intended:

% port installed clang-11
The following ports are currently installed:
  clang-11 @11.1.0_7+defaultlibcxx+emulated_tls (active)
% port variants  
testport1 has the variants:
   universal: Build for multiple architectures
% port deps
Full Name: testport1 @1.1_0
Build Dependencies:   clang-11
Library Dependencies: libcxx
% sudo port -v build +universal
Password:
--->  Computing dependencies for testport1.
--->  Fetching distfiles for testport1
--->  Verifying checksums for testport1
--->  Extracting testport1
--->  Configuring testport1
Executing:  cd "/opt/local/var/macports/build/_Users_josh_dports-dev_local_test_testport1/testport1/work/testport1-1.1" && echo --prefix=/opt/local --disable-dependency-tracking 
--prefix=/opt/local --disable-dependency-tracking
--->  Building testport1

[fhgwright (Fred Wright)]

Getting back to the exact state where this happened might be nontrivial, but I believe it involved a case where the chosen compiler was either uninstalled or outdated at the time, and when it decided to install or upgrade it, it specified +universal, probably inherited from the original upgrade target. That's not the same as your test case above, where the chosen compiler is already active and up to date.

[kencu (Ken)]

This is not a bug, the universal build of clang and LLVM was allowed to mean clang itself was universal on purpose, not as an oversight of some kind.

For one, clang and LLVM provide several libraries that other ports link against, and they do need to be universal at times. So the universal clang is needed there.

For another, real universal on 10.5 and 10.6 was needed, as some 10.5 and 10.6 installations are unable to run the 64 bit versions.

our gcc builds are "fake universal" and by that we mean that only the relevant library parts are actually universal. So the gcc builds can build universal software, but are not themselves universal. This has caused a fair bit of headache at times over the years, but it can be lived with.

Varoius folks have stumbled across the fact that a non-universal clang/llvm can build universal software over the years, me included -- in the end, they have always decided that the best thing is to have clang/llvm actually be universal.

Of course, that could change, and could be worked around -- for no particular benefit. We are currently discussing doing the exact opposite move with gcc, and making it "real universal" instead of "fake universal" due to headaches we are having with, for example, gcc10-bootstrap and other gcc versions.

[jmroot (Joshua Root)]

Replying to fhgwright:

Getting back to the exact state where this happened might be nontrivial, but I believe it involved a case where the chosen compiler was either uninstalled or outdated at the time, and when it decided to install or upgrade it, it specified +universal, probably inherited from the original upgrade target. That's not the same as your test case above, where the chosen compiler is already active and up to date.

OK, so that's the behaviour of all variants explicitly requested on the command line: they are applied to dependencies as well as the requested port. This might be considered a duplicate of #48051?

[fhgwright (Fred Wright)]

Replying to kencu:

This is not a bug, the universal build of clang and LLVM was allowed to mean clang itself was universal on purpose, not as an oversight of some kind.

That's about the host architecture, not the target architecture.

For one, clang and LLVM provide several libraries that other ports link against, and they do need to be universal at times. So the universal clang is needed there.

Clang provides two completely different kinds of libraries - some involved in running clang itself (only related to the host architecture), and the compiler-rt libraries which only pertain to the target architecture. It would probably be cleaner if the compiler-rt stuff were a separate subport, and there even seems to be upstream support for doing that.

For another, real universal on 10.5 and 10.6 was needed, as some 10.5 and 10.6 installations are unable to run the 64 bit versions.

Again, host architecture, not target architecture.

our gcc builds are "fake universal" and by that we mean that only the relevant library parts are actually universal. So the gcc builds can build universal software, but are not themselves universal. This has caused a fair bit of headache at times over the years, but it can be lived with.

Gcc and clang are completely different. Gcc expects the target architecture to be a single-valued build-time option, and the only way to get a gcc that builds for multiple targets is by bundling multiple gcc builds with a front end that selects the appropriate one. Clang and llvm are fundamentally designed to make the target architecture a straightforward runtime option, fully decoupled from the host architecture.

Varoius folks have stumbled across the fact that a non-universal clang/llvm can build universal software over the years, me included -- in the end, they have always decided that the best thing is to have clang/llvm actually be universal.

There's no need to "stumble across" something that's part of the design. The only reason to confuse host and target architectures is that MacPorts does that itself.

Of course, that could change, and could be worked around -- for no particular benefit. We are currently discussing doing the exact opposite move with gcc, and making it "real universal" instead of "fake universal" due to headaches we are having with, for example, gcc10-bootstrap and other gcc versions.

The biggest problem with gcc is just that it's a horrible mess in general.

Replying to jmroot:

Replying to fhgwright:

Getting back to the exact state where this happened might be nontrivial, but I believe it involved a case where the chosen compiler was either uninstalled or outdated at the time, and when it decided to install or upgrade it, it specified +universal, probably inherited from the original upgrade target. That's not the same as your test case above, where the chosen compiler is already active and up to date.

OK, so that's the behaviour of all variants explicitly requested on the command line: they are applied to dependencies as well as the requested port. This might be considered a duplicate of #48051?

If you mean the immediate command that triggered it, then no, it was just upgrade outdated. If you mean some option on some command weeks or months in the past, subsequently propagating +universal across ports while confusing host and target architectures, then perhaps.

The real problem is that MacPorts fails to properly represent host and target architectures distinctly, and works around some of the ensuing bugs with ad hoc kludges, like depends_skip_archcheck. Emphasis on some. Not to mention the way "universal" became overloaded with the introduction of x86_64, and more so with the advent of arm64. I've seen builds fail due to confusion about whether "universal" meant "ppc+i386" or "i386+x86_64".

[jmroot (Joshua Root)]

Replying to fhgwright:

Getting back to the exact state where this happened might be nontrivial, but I believe it involved a case where the chosen compiler was either uninstalled or outdated at the time, and when it decided to install or upgrade it, it specified +universal, probably inherited from the original upgrade target. That's not the same as your test case above, where the chosen compiler is already active and up to date.

OK, so an outdated dependency does not exhibit this behaviour, but an uninstalled one does. It's not specific to universal; all variants from the dependent are inherited by the newly installed dependency.

[fhgwright (Fred Wright)]

Replying to jmroot:

Replying to fhgwright:

Getting back to the exact state where this happened might be nontrivial, but I believe it involved a case where the chosen compiler was either uninstalled or outdated at the time, and when it decided to install or upgrade it, it specified +universal, probably inherited from the original upgrade target. That's not the same as your test case above, where the chosen compiler is already active and up to date.

OK, so an outdated dependency does not exhibit this behaviour, but an uninstalled one does. It's not specific to universal; all variants from the dependent are inherited by the newly installed dependency.

Clearly a bug in many cases.

For example, after making a minor fix to dia, I made the mistake of trying a universal build as a test before submitting a PR. I'd already built dia -universal so all needed dependencies were already present for that case. 23.5 hours later, it finally finished populating dependencies for the +universal case. And this was on a 3.47 GHz 12-core, dual hyperthreaded machine with 128 GiB of RAM. The new universal builds included seven different versions of clang/llvm, plus cmake, cctools, ld64, git, curl, ... (you get the idea).

Universal builds are fairly often broken, largely because they're fairly often untested. And one reason they're fairly often untested is because testing them can be so excruciatingly painful, between the rampant gratuitous universality and the fact that universal builds are rarely available as precompiled binaries.

It's indeed not just an issue for universal; consider a GUI build tool with a quartz/x11 choice, being used to build a port with a quartz/x11 choice. Those choices should be independent in this circumstance, which is another host vs. target case.

[jmroot (Joshua Root)]

In 7be167c23956c184920354a66f57469bad70b659/macports-base (master):

_upgrade: set newly installed ports' variants better

Rather than inheriting all the variants of the depending port in this
case, apply those requested on the command line and in variants.conf,
adding +universal if required to get archs compatible with those of the
depending port. This also adds error reporting in this case if the
archs cannot be made compatible.

Closes: #67637