buildbot: add support for building ports with variants

[mojca (Mojca Miklavec)]

It would be very handy to have some limited support for building with variants.

Before trashing the subversion checkout on my buildbot, I wanted to attach partial patches.

See #51995.

[mojca (Mojca Miklavec)]

patches for master.cfg that add support for building with arbitrary variants (no sanity checking yet!)

[mojca (Mojca Miklavec)]

outdated patch for basic support of variants in mpbb

[mojca (Mojca Miklavec)]

The patches are pretty outdated, but they should give the first impression. I wanted to fix the second one, but I should probably write some code to parse options between install-port and <portname> first.

Some hints by Clemens about how to proceed:

tools/dependencies.tcl just opens all ports without any specified variants. See tools/dependencies.tcl line 66; the last parameter to mportopen, which is currently {}, is the list of variants. The format is easy to create:

array set variations {}
set variations(quartz) "+"
set variations(x11) "-"
mportopen ... [array get variations]

gives you an mport reference as if the port was installed with +quartz-x11. You should be able to find some suitable parsing code for variants given on the command line in src/port/port.tcl, line 1695 and following.

Looking at that file I have an impression that we forgot to consider global settings ([array get global_variations]):

if {!$index_only} {
    # Add any global_variations to the variations
    # specified for the port (so we get e.g. dependencies right)
    array unset merged_variations
    array set merged_variations [array get variations]
    foreach { variation value } [array get global_variations] {
        if { ![info exists merged_variations($variation)] } {
            set merged_variations($variation) $value
        }
    }

This means that even if someone would (theoretically) set up a buildbot with +quartz -x11 in configuration file, those setting would probably be ignored. Or do I misunderstand the concept?

Yes, I think the file doesn't handle global variants. I think it's also bad design that port/port.tcl has to do the merging of variant specs without at least some library support from macports1.0, but that's a story for a different refactoring.

An additional warning:

However, this output will only list positive variants, it will not list default variants that have been explicitly disabled. This might be a problem, because specifying -x11 +quartz will just give you a list of all dependencies with +quartz, but not without +x11, so if a port is written in a way that it does not automatically disable x11 if +quartz is given (for example if both are supported simultaneously and x11 is the default), this might lead to unexpected results. I haven't tested any of this, but we probably should.

[mojca (Mojca Miklavec)]

It looks like the core of the needed functionality was already added in r152489.

[jmroot (Joshua Root)]

Some of the logging probably needs to be enhanced with variant info, but that's about it AFAICT.

[mojca (Mojca Miklavec)]

I didn't check the logs, but among other things we also need:

• (straightforward/trivial) a revised version of my old attempt​ to add a field for variants to master.cfg
• (lower priority) some basic logic to automatically build +quartz for certain ports

[mojca (Mojca Miklavec)]

add a field to support variants

[mojca (Mojca Miklavec)]

The new (trivial) patch for master.cfg seems to work for me.

We need some basic safeguard (regexp?) though to make sure that only valid variant names are entered, otherwise people could end up writing port names or any other weird things. (I could imagine things equivalent to sql injections ending up here. Nothing to do with sql, but I hope you get the idea.)

This will be sufficient to get gkt3 +quartz and other binaries on the buildbots. The next (independent) step would be to write the logic to build that automatically for some ports.

[neverpanic (Clemens Lang)]

I've been working on getting this working, ideally even without modifications to the master.cfg of the buildbot, but I've hit some problems. [f6e46815a5033358212ce383956fd567eb51e53c/mpbb] changed the variant handling code in mpbb in a way that will cause failures when requesting specific variants to be installed. See the commit on GitHub for a description of the problem. We would have to revert this commit for this effort to proceed, but I'd like to wait for feedback from Joshua before doing that.

[neverpanic (Clemens Lang)]

See ​https://github.com/macports/mpbb/pull/5.

[jmroot (Joshua Root)]

I don't doubt the current code needs changes to work right when specifying arbitrary variants. Simply reverting that commit is not right either, though. The distinction between the canonical_active_variants and the variants that are being requested needs to be maintained.

[neverpanic (Clemens Lang)]

You're right. I thought about this yesterday and came to the conclusion that this is really a bug in base, since base propagates these variants down the dependency tree and attempts to install a combination that will not work.

Either the glib2 Portfile should ignore the user's request to change the variant and forcefully set a variant, or base should not request this invalid combination.

[neverpanic (Clemens Lang)]

In 45c59fe0a66e382927fbf6fe629d37898eb42b6b/macports-ports:

glib2: Default to +quartz when -x11 is given

When building pango -x11 in a clean prefix, glib2 will fail to install,
because the -x11 variant is being passed down to glib2, but glib2
requires either +quartz or +x11, and +quartz was not enabled by default
when x11 was disabled.

This caused problems when implementing variant support for the buildbot
in ​https://github.com/macports/mpbb/pull/5 and initially caused me to
think that I would have to revert

f6e46815a5033358212ce383956fd567eb51e53c/mpbb

which was added due to ​https://github.com/macports/mpbb/pull/4 and
​https://lists.macports.org/pipermail/macports-dev/2017-June/035978.html.

This solution should instead work without the revert and still allow the
buildbot to build both wine and +quartz-x11 ports.

See: #52742

[ryandesign (Ryan Carsten Schmidt)]

In 32fd667ad3d4cb8fe9be7bf2dc5058cc73d8e530/macports-ports:

glib2-devel: Default to +quartz when -x11 is given

When building pango -x11 in a clean prefix, glib2 will fail to install,
because the -x11 variant is being passed down to glib2, but glib2
requires either +quartz or +x11, and +quartz was not enabled by default
when x11 was disabled.

This caused problems when implementing variant support for the buildbot
in ​https://github.com/macports/mpbb/pull/5 and initially caused me to
think that I would have to revert

f6e46815a5033358212ce383956fd567eb51e53c/mpbb

which was added due to ​https://github.com/macports/mpbb/pull/4 and
​https://lists.macports.org/pipermail/macports-dev/2017-June/035978.html.

This solution should instead work without the revert and still allow the
buildbot to build both wine and +quartz-x11 ports.

See: #52742