openmpi/mpich: prune target compiler list; refactor duplicated logic, and simplify ports, via new portgroup

[mascguy (Christopher Nielsen)]

I envision this being a very iterative process, with the initial portgroup being something as simple as a few procs to eliminate copy-paste logic.

Also prune the target compiler list significantly, whilst still supporting MacOS releases back to 10.6.

[mascguy (Christopher Nielsen)]

The first two candidates are as follows. I'm not married to the proc names, but the point is to eliminate copy-paste logic:

proc mpi_add_compiler_depends_lib {cname} {
    set cport_name ""

    # As we are making wrappers, we depend on the compilers to exist.
    # Add them to depends_lib, not just depends_build.
    if {[regexp {clang[3-9]\d} ${cname}] == 1} {
        # Ports for Clang versions < 10 are named: clang-<major>.<minor>
        set cport_name          [regsub {(\d)(\d)} ${cname} {-\1.\2}]
    } elseif {[regexp {clang\d\d} ${cname}] == 1} {
        # Ports for Clang version >= 10 are named: clang-<major><minor>
        set cport_name          [regsub {(\d)(\d)} ${cname} {-\1\2}]
    } elseif {([regexp {gcc\d} ${cname}] == 1) || ([regexp {gcc\d\d} ${cname}] == 1)} {
        # Ports for GCC have names exactly matching our subports, so use as-is
        set cport_name          ${cname}
    }

    if {${cport_name} ne ""} {
        ui_debug "mpi_add_compiler_depends_lib: Adding compiler to depends_lib: ${cport_name}"
        depends_lib-append      port:${cport_name}
    } else {
        ui_debug "mpi_add_compiler_depends_lib: Not adding compiler to depends_lib; cname: ${cname}"
    }

    return 0
}

proc mpi_set_binary_eligibility {subport cname} {
    if {[lsearch -exact {mp llvm clang} ${cname}] != -1} {
        # Force local builds with Xcode-provided compilers (avoid issues with
        # different Xcode versions on buildbot and user machines)
        ui_debug "mpi_set_binary_eligibility: Disabling binary use for subport: ${subport}"
        archive_sites
    } else {
        ui_debug "mpi_set_binary_eligibility: Not disabling binary use for subport: ${subport}"
    }

    return 0
}

[mascguy (Christopher Nielsen)]

As for other shared logic, the two ports have so many differences - some subtle, and many not-so-subtle - that we might be limited in the refactoring effort.

Although the clist handling - including generation of subports, logic to exclude/fail based on compiler version and macOS release, etc - would be nice to refactor out.

If both ports supported exactly the same set of target compilers, it would be easier...

[eborisch (Eric A. Borisch)]

I'm all for them targeting the same set of compilers; what's the current delta?

Another thought: I've been toying with the idea of having all the non-default (-clang*, etc) flavors use the default libraries and headers rather than build their own, with a defaulted-on +wrapperonly variant.

In this mode, we would just make copies of mpicc, mpicxx, etc. with the appropriate changes to CC/CXX etc. in the script (and continue to depend upon the compilers to make sure they are installed.) Executables like mpiexec could just be a symlink to the -default mpiexec (likely easier to maintain than having a separate set of select files for the variants).

It's typically rare interested in seeing how the different compilers do at generating the MPI libraries themselves, and you're more interested in using the features of a particular compiler for your (or another port's) code. This would drastically cut down the overhead for having multiple flavors of the wrappers around, while still leaving the options of disabling the variant if you want a full build of MPI libraries by that compiler.

It looks like openmpi uses a binary mpicc (symlinked to a binary, actually), so I'm not sure how easy this would be to implement on openmpi, but it should be fairly easy on mpich.

[mascguy (Christopher Nielsen)]

This is the current list we support:

openmpi-clang                  @4.1.1          science/openmpi
openmpi-clang10                @4.1.1          science/openmpi
openmpi-clang11                @4.1.1          science/openmpi
openmpi-clang33                @4.1.1          science/openmpi
openmpi-clang34                @4.1.1          science/openmpi
openmpi-clang37                @4.1.1          science/openmpi
openmpi-clang50                @4.1.1          science/openmpi
openmpi-clang60                @4.1.1          science/openmpi
openmpi-clang70                @4.1.1          science/openmpi
openmpi-clang80                @4.1.1          science/openmpi
openmpi-clang90                @4.1.1          science/openmpi
openmpi-default                @4.1.1          science/openmpi
openmpi-gcc5                   @4.1.1          science/openmpi
openmpi-gcc6                   @4.1.1          science/openmpi
openmpi-gcc7                   @4.1.1          science/openmpi
openmpi-gcc8                   @4.1.1          science/openmpi
openmpi-gcc9                   @4.1.1          science/openmpi
openmpi-gcc10                  @4.1.1          science/openmpi
openmpi-gcc49                  @4.1.1          science/openmpi
mpich-clang                    @3.4.1          science/mpich
mpich-clang10                  @3.4.1          science/mpich
mpich-clang11                  @3.4.1          science/mpich
mpich-clang50                  @3.4.1          science/mpich
mpich-clang60                  @3.4.1          science/mpich
mpich-clang70                  @3.4.1          science/mpich
mpich-clang80                  @3.4.1          science/mpich
mpich-clang90                  @3.4.1          science/mpich
mpich-default                  @3.4.1          science/mpich
mpich-gcc5                     @3.4.1          science/mpich
mpich-gcc6                     @3.4.1          science/mpich
mpich-gcc7                     @3.4.1          science/mpich
mpich-gcc8                     @3.4.1          science/mpich
mpich-gcc9                     @3.4.1          science/mpich
mpich-gcc10                    @3.4.1          science/mpich
mpich-gcc49                    @3.4.1          science/mpich
mpich-gccdevel                 @3.4.1          science/mpich

[mascguy (Christopher Nielsen)]

Per the earlier discussion on the macports-users list, should we remove all GCC versions earlier than 7?

As for Clang, what's the oldest version we need to support?

[kencu (Ken)]

I am not aware of any reason to keep support for gcc older than gcc7 in the mpi family.

Likewise, anyone using a macports-clang probably wants at least macports-clang-9.0 by this point, but I don't know if there might be some niche user with an old code base that uses some older macports-clang for something.

maybe deprecate them in some way? Or just bin them?

[kencu (Ken)]

we do need to keep the older gcc versions for other purposes, and probably the older clangs too for similar reasons (bootstrapping or certain older code that needs it and can't/won't be updated).

[mascguy (Christopher Nielsen)]

Replying to kencu:

we do need to keep the older gcc versions for other purposes, and probably the older clangs too for similar reasons (bootstrapping or certain older code that needs it and can't/won't be updated).

Ken, are you referring to MacPorts as a whole, or just these two ports? I ask, as I'm speaking just for the latter...

[cjones051073 (Chris Jones)]

yes, we are not suggesting removing the primary clang/gcc ports here Ken, just the mpich/openmpi versions.

[kencu (Ken)]

exactly

[mascguy (Christopher Nielsen)]

So, would the following set of compilers be sufficient?

• Clang 9 - 11
• GCC 7 - 10

[mascguy (Christopher Nielsen)]

(Plus the host/Xcode versions, aka, variations "-clang" and "-default")

[mascguy (Christopher Nielsen)]

Replying to eborisch:

Another thought: I've been toying with the idea of having all the non-default (-clang*, etc) flavors use the default libraries and headers rather than build their own, with a defaulted-on +wrapperonly variant.

In this mode, we would just make copies of mpicc, mpicxx, etc. with the appropriate changes to CC/CXX etc. in the script (and continue to depend upon the compilers to make sure they are installed.) Executables like mpiexec could just be a symlink to the -default mpiexec (likely easier to maintain than having a separate set of select files for the variants).

It's typically rare interested in seeing how the different compilers do at generating the MPI libraries themselves, and you're more interested in using the features of a particular compiler for your (or another port's) code. This would drastically cut down the overhead for having multiple flavors of the wrappers around, while still leaving the options of disabling the variant if you want a full build of MPI libraries by that compiler.

It looks like openmpi uses a binary mpicc (symlinked to a binary, actually), so I'm not sure how easy this would be to implement on openmpi, but it should be fairly easy on mpich.

Interesting idea!

Since the default flavor doesn't utilize the buildbot binaries, does that mean an install would always cause a build from source (at least for the default libs)?