openmpi: fix builds for MacOS 10.8 and earlier; mark known_fail, for cases not easily fixable

[jmroot (Joshua Root)]

A lot of the subports of openmpi fail to build on some subset of OS versions. For example, openmpi-gcc10 fails on 10.7 and 10.8: ​https://ports.macports.org/port/openmpi-gcc10/builds

Ideally all the build failures would be fixed of course, but until and unless that happens, each subport should set known_fail yes on platforms where it is failing. Otherwise the buildbot wastes a lot of time running a new build of all the subports that previously failed each time the Portfile is modified in any way.

[mascguy (Christopher Nielsen)]

Agreed, we should block builds which fail, where there's no easy fix. Most of the failures, though, relate to the use of the Intel AVX instruction set. And that's easily disabled for those older MacOS releases.

That's something I'll take care of ASAP.

[Christopher Nielsen <mascguy@…>]

In 740316e26cae322a37a39cedd7918aacfdd5e14c/macports-ports (master):

openmpi: disable AVX instruction set for MacOS 10.8 and earlier

See: #62878

[mascguy (Christopher Nielsen)]

The other significant change that's coming over the next few weeks, is a simplification of both openmpi and mpich: We're going to significantly prune the target compiler list, which will greatly reduce the impact on the buildbots.

The compiler list will likely be Clang 9 through 11, and GCC 7 through 11.

See issue:62807

[mascguy (Christopher Nielsen)]

Based on local testing within VMs, disabling AVX instructions fixes most of the build failures for MacOS 10.7 and 10.8.

The buildbots currently have a sizeable backlog, so it's going to take a day or two before the changes work their way through the queue. But so far, things are looking much better.

[mascguy (Christopher Nielsen)]

I'm still waiting for the 10.7 buildbot to catch up, but things are looking quite good: Thus far, the AVX change has fixed virtually all subports for 10.8, along with some for 10.6 64-bit. (And even a few for 10.6 32-bit, where all dependencies are available.)

[mascguy (Christopher Nielsen)]

Sadly, it looks like the subports for gcc9 and later are failing, on MacOS 10.7 and earlier. The AVX instruction use issue is resolved, but now other Intel extensions are being balked at.

It might be possible to fix this - maybe - but it's probably easier to simply set those to known_fail. Users still have options like gcc7 and clang90, among others, which do work with those older MacOS releases.

[Christopher Nielsen <mascguy@…>]

In 94fbbd7401f75aa7aff64ab4875a4367ad5b3982/macports-ports (master):

openmpi: disable gcc9+ subports for MacOS 10.7, as builds fail

See: #62878

[Christopher Nielsen <mascguy@…>]

In d4ba8f5103edf9ff1593060bafab10572ebddfc5/macports-ports (master):

pg mpi: disable gcc9+ for MacOS 10.7; add support for gcc11

See: #62878

[kencu (Ken)]

success on 10.7 (with AVX enabled, as well):

$ port -v installed openmpi-gcc10
The following ports are currently installed:
  openmpi-gcc10 @4.1.1_0+fortran (active) platform='darwin 11' archs='x86_64' date='2021-05-14T19:55:14-0700'

SO -- how....

all the gcc versions on MacPorts (and all the macports-clang-N compilers) are built to use cctools assembler ${prefix}/bin/as. For all older systems this is the cctools-supplied version (for newer systems it might be a symlink to the one provided by Xcode).

${prefix}/bin/as has been set up to use a newer clang as the assembler if a newer clang is available, and it looks for macports-clang-9.0 ==> macports-clang-5.0 and then clang on 10.7+.

clang on 10.7 and 10.8 is very old now, and it doesn't understand the newer assembly instructions. macports-clang-9.0 is pretty recent, and easy to build for all systems all the way back to 10.5 Leopard Intel. So it's a pretty good choice for an assembler right now.

It understands the failing instruction, and also understands the previously-failed AVX instructions.

SO you can re-enable openmip-gcc10 on 10.7, and the AVX instructions on 10.7 and 10.8, if you make sure that clang-9.0 is installed when openmpi-gcc10 (etc) is built.

Something like this:

    # For MacOS 10.8 and earlier, system-provided clang assembler doesn't support newer instructions
    if {${os.major} <= 12} {
        ui_debug "installed clang-9.0 to use as assembler"
        depends_build-append port:clang-9.0
    }

I'm not at this second 100% sure if it is also needed at runtime as an assembler. It might be, and for caution, I would suggest perhaps it just be a depends_lib-append so it's always available.

[kencu (Ken)]

As an aside, almost 100% of users on systems < 10.9 have clang-9.0 installed,

So most anyone who actually tried to build openmpi-gcc10 before it was blocked from their systems actually succeeded.

[mascguy (Christopher Nielsen)]

Thanks for the info, this would be easy enough to implement.

But it sure would be nice if there was a lighter-weight way to install/use a more modern assembler, without having to add yet another compiler to the dependency list...

[mascguy (Christopher Nielsen)]

Is it possible to update the cctools assembler to support AVX, etc, on the older MacOS releases?

[kencu (Ken)]

Replying to mascguy:

Is it possible to update the cctools assembler to support AVX, etc, on the older MacOS releases?

no, that code is circa 2008 and will never be updated. All Apple efforts went into clang.

Don't worry about the clang-9.0 install. It's the defacto system compiler for MacPorts on all older systems, and all Intel users on almost all systems older than 10.12 or so will have it installed already.

Edit: if you mean could someone or some group take on the project of updating the code in the ancient "gas" component of cctools assembler to understand and use all the new Intel SIMD instructions, I guess sure someone could. Or they could port the current gnu assembler to Darwin. But no paid engineers will likely be doing this, and all such work is in llvm/clang now so I don't see it happening.

[mascguy (Christopher Nielsen)]

Replying to kencu:

Don't worry about the clang-9.0 install. It's the defacto system compiler for MacPorts on all older systems, and all Intel users on almost all systems older than 10.12 or so will have it installed already.

Sounds good Ken. Once the buildbots are caught up, I'll update openmpi to use clang-9.0 on MacOS 10.8 and earlier, and re-enable AVX instruction use.

[kencu (Ken)]

if you can also make sure it is an Intel-specific build dep, then I won't need to fix PowerPC later!

[mascguy (Christopher Nielsen)]

Replying to kencu:

if you can also make sure it is an Intel-specific build dep, then I won't need to fix PowerPC later!

Ah, good point. Thanks for the heads-up!

[Christopher Nielsen <mascguy@…>]

In 83662481aa05dd5ffa5dfbb9857b4d1d2eae0782/macports-ports (master):

openmpi/mpich: gcc builds: add clang 9 build dependency for macOS 10.8 and earlier

• clang 9 provides a modern assembler, allowing openmpi/mpich to build successfully

See: #62878
See: #62887

[Christopher Nielsen <mascguy@…>]

In 733aa5de852c084e87c7a01f595975efe6454bc7/macports-ports (master):

openmpi/mpich: disable Xcode clang builds for MacOS 10.6 and earlier

See: #62878
See: #62887

[Christopher Nielsen <mascguy@…>]

In 15d5dc2fe3fb22241ca35a1f93afcb25024b2795/macports-ports (master):

openmpi/mpich: disable default (gcc) build when necessary; cleanup

See: #62878
See: #62887

[Christopher Nielsen <mascguy@…>]

In 8ea6c676e0df04f9f90b2b9eb6190901e20a5e03/macports-ports (master):

openmpi/mpich: refactor note generation to pg mpiutil; cleanup

See: #62878
See: #62887

[mascguy (Christopher Nielsen)]

The vast majority of failures have either been fixed, or disabled. The few remaining are due to missing compiler dependencies - for example, clang 11 is not yet available for macOS 10.6 - but these fail quickly during the dependency install phase.

Marking as fixed.

[mascguy (Christopher Nielsen)]

Several of the disabled subports were still trying to build, due to unintended dependencies. (Both on the default subport, as well as legacy-support for earlier macOS releases.)

That's been fixed as of this writing, and that should further reduce any wasted buildbot time.

[mascguy (Christopher Nielsen)]

The aforementioned fixed are present in the following two commits:

​https://github.com/macports/macports-ports/commit/883e402ae5c48131366196332fc874ed0a26ecb2

​https://github.com/macports/macports-ports/commit/3c28142a5299c165c1a38a60a819bd82229d0c67

[Christopher Nielsen <mascguy@…>]

In a459e5eb7291e2e0e9d5cb3d9e5015bc195842dc/macports-ports (master):

openmpi/mpich: gcc subports: don't depend on clang 9 for PPC
See: #59239
See: #62878