xcode portgroup should set -derivedDataPath on Xcode versions that need it

[ryandesign (Ryan Carsten Schmidt)]

Ports that use the xcode 1.0 portgroup fail to build at least on Xcode 9.4 and 10.0 GM, because they cannot write to the derived data path. By default Xcode uses a derived data path inside the home directory, but fails to honor the HOME environment variable MacPorts specifies, so it tries to write into the macports user's home directory which is not writable.

To work around this, the poedit port manually specifies the derived data path, and I copied this usage to the MoltenVK port when I created it, but every other port that builds with xcodebuild (such a graphviz-gui or Readown) has failed to build for me in the same way, so I think we need to move that fix into the portgroup.

It looks like the -derivedDataPath flag is available in xcodebuild 5 and later (Mountain Lion and later). So we could add the flag for Xcode 5 and later, assuming it does no harm, though surely these ports haven't been broken since Mountain Lion. I suspect some new version of Xcode must have introduced a change. Do we know what version that might be?

[kencu (Ken)]

I noticed three things that appear to be general xcode PG issues while fixing aquaterm and looking over irrlicht...

1. why do we run a build phase, when the build always seems to occur again in the destroot phase anyway, and sometimes this causes errors?
2. the -derivedDataPath issue
3. at least on Xcode 10, the -derivedDataPath flag is only accepted if there is a scheme, which so far seems to be the same as ${xcode.target}

SO -- I am wondering if the xcode PG might just add the following as a default, as it seems to be working for me so far on several ports:

build                   {}
destroot.pre_args       -derivedDataPath ./DerivedData
xcode.scheme            ${xcode.target}

[ryandesign (Ryan Carsten Schmidt)]

Replying to kencu:

I noticed three things that appear to be general xcode PG issues while fixing aquaterm and looking over irrlicht...

1. why do we run a build phase, when the build always seems to occur again in the destroot phase anyway, and sometimes this causes errors?

I have noticed that some Xcode-using ports do that. I do not know why that happens. Some Xcode-using ports override the destroot phase entirely, perhaps in order to combat that. In MacPorts we should be building in the build phase and destrooting in the destroot phase. If we are not doing that for Xcode-using ports, we should figure out a way to fix that.

2. the -derivedDataPath issue
3. at least on Xcode 10, the -derivedDataPath flag is only accepted if there is a scheme, which so far seems to be the same as ${xcode.target}

SO -- I am wondering if the xcode PG might just add the following as a default, as it seems to be working for me so far on several ports:

build                   {}
destroot.pre_args       -derivedDataPath ./DerivedData
xcode.scheme            ${xcode.target}

I wasn't aware that Xcode required a scheme to use the derived data path.

A scheme is not the same thing as a target, that's why there are separate settings in the portgroup.

[kencu (Ken)]

I assume the xcode PG's "destroot" phase runs some "build-then-install" logic in Xcode that might be hard to dissect...

It could be you either choose destrooting in the build phase or building in the destroot phase :>

[ryandesign (Ryan Carsten Schmidt)]

It would be silly if we had to use either of those in general for all Xcode-using ports.

Perhaps there is something that some Xcode projects do that causes this to happen.

Maybe this used to work the way we wanted in older versions of Xcode, back when the portgroup was added, and something broke or changed in newer versions of Xcode.

[ddrum2000]

From building graphviz-gui

[ddrum2000]

Sorry for the double post I accidentally started above.

I suspect I have the same problem when trying to build graphviz-gui. I'm running Mojave with Xcode 10. I have attached a log file in the previous post.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ddrum2000:

So I suspect I have the same problem when trying to build graphviz-gui. I'm running Mojave with Xcode 10. I have attached a log file.

Right, that's the problem you'll see building most any port that uses xcodebuild, with new versions of Xcode.

We need to decide under what conditions to add the -derivedDataPath flag to the xcode portgroup. Should we add it for Xcode 5 and later, since the option appeared in that version? Or should we only add it on the versions of Xcode where it fails if we don't, and if so, what versions are those?

We also need clarity on the scheme issue Ken raised.

[jmroot (Joshua Root)]

Replying to kencu:

I assume the xcode PG's "destroot" phase runs some "build-then-install" logic in Xcode that might be hard to dissect...

Note that there is a confirmed bug with the new build system in Xcode 10 that causes problems with building and then destrooting separately, but not when doing both in one step. See [9cb3c083519503bb4b404fe57f19466a924523be/macports-ports] and ​IRC logs of 26th-27th September (starting around 22:40).

[ryandesign (Ryan Carsten Schmidt)]

Has duplicate #57223.

[jeremyhu (Jeremy Huddleston Sequoia)]

Has someone filed a radar about this issue? Please provide a reference to it.

[jmroot (Joshua Root)]

Replying to jeremyhu:

Has someone filed a radar about this issue? Please provide a reference to it.

Toby filed one about the problem I mentioned in comment:9. He didn't mention the number unfortunately; you'd have to ask him. I'm not clear on whether there are multiple separate issues here or if they're all related.

[ryandesign (Ryan Carsten Schmidt)]

Has duplicate #58276.

[ryandesign (Ryan Carsten Schmidt)]

Has duplicates #58126, #58297.

[kencu (Ken)]

So one issue is the way the xcode PortGroup works. If you look at the PortGroup file, it builds the software in the build{} phase, and then completely rebuilds the software all again in the destroot{} phase, and then destroots it. I presume this was so the build phase actually does something. I don't believe you can tell xcodebuild to separate the two. I haven't yet sorted out the exact extra bit the destroot{} command does to destroot it, but no doubt it's obvious enough with comparisons.

The old Xcode build system allowed this. The new Xcode build system will not allow redundant things in the build tree, so I presume that it errors out on those builds that do some extra stuff, like generating symlinks or building man or pdf files with a script. The new build system errors.

That is easy to fix. Just don't run the build{} phase. It does nothing useful to run the build{} phase anyway. I do not see a way to build in the build{} phase and just destroot in the destroot{} phase. AFAICT, xcodebuild cannot do that, although I'm not an expert on the inner workings of it. So this is a MacPorts' specific problem, as we do the double build. I do not see any Xcode fix coming for that.

The second problem is that as Ryan pointed out, Xcode dumps it's products in the home directory, which is locked out on MacPorts. This is another MacPorts' specific problem -- homebrew would not see it (as people run as the current user). You CAN set the -derivedDataPath ./DerivedData to force the derived data to go somewhere writable, but that only works for me if you also set a scheme. Perhaps every (recent) Xcode project file has a scheme we can explore for and use, I don't know. If so, if you set a scheme, then xcodebuild will accept the -derivedDataPath ./DerivedData. Each will have to be investigated for such a scheme, although often I've found that the target also works (not always).

For projects with no scheme, it appears you have to use the old build system, or accept the default -derivedDataPath.

[kencu (Ken)]

So what could we do?

Change the xcode PG defaults a bit for Xcode 10+.

1. Cancel the build phase for Xcode 10+, and just let it build in the destroot phase as it does already. That will get rid of all the symlink and similar errors. There is no other choice, AFAICT.

2. set the -derivedDataPath ./DerivedData in the pre-args for Xcode 10+. If there is no scheme set, default it to xcode target, which works most of the time.

3. make an override to use the old build system with a simple setting in the Portfile if the above doesn't work.

That will fix a lot of the broken builds. For the ones that still fail, the authour can try setting a proper scheme with a little investigation into what schemes there are in the project. And for the ones that fail THAT, you can set the old build system instead and give up.

The only other choice I can see would be to use the old build system all the time on Xcode 10+, which I actually kind of like as it's a one-line fix for every xcode 10 build failure I think -- but I know that idea would not fly.

[kencu (Ken)]

For those looking to get something built right now, this little block added to the Portfile will likely fix the xcode build for software that has not yet been updated to work with the new Xcode 10+ build system:

# this port has not yet been updated to build with the new build system Xcode 10+
if {${os.platform} eq "darwin" && ([vercmp $xcodeversion 10.0] > 0)} {
    build.pre_args-append    -UseModernBuildSystem=NO
    destroot.pre_args-append -UseModernBuildSystem=NO
}

[MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)]

In 377d0cc1a164f490be8f85ca3c7fb8020bcbe724/macports-ports (master):

MailtoMutt: allow build on newer systems

No revbump because port either builds correctly or not at all.
See #57137
Fixes #58455

[ryandesign (Ryan Carsten Schmidt)]

FYI it looks like Xcode 5 and up support -derivedDataPath. Still not sure why we're not setting that in the xcode portgroup.

[RJVB (René Bertin)]

Is there a safe os.version to check to determine if -UseModernBuildSystem is available or does one have to include the xcode PG?

[RJVB (René Bertin)]

Replying to RJVB:

Is there a safe os.version to check to determine if -UseModernBuildSystem is available or does one have to include the xcode PG?

Apparently xcodebuild doesn't bail on unknown options? On my OS X 10.9.5 where xcodebuild --help does NOT list the option:

:info:build mkdir -p ./tmp/modules/gui/macosx
:info:build cd "./modules/gui/macosx/" && cp *.h *.m /Volumes/VMs/MPbuild/_Volumes_Debian_MP9_site-ports_multimedia_VLC2/VLC2/work/vlc-2.2.8/tmp/modules/gui/macosx/
:info:build cd ./tmp/extras/package/macosx && \
:info:build         xcodebuild -target vlc -UseModernBuildSystem=NO SYMROOT=../../../build  DSTROOT=../../../build
:info:build User defaults from command line:
:info:build     UseModernBuildSystem = NO
:info:build Build settings from command line:
:info:build     DSTROOT = ../../../build
:info:build     SYMROOT = ../../../build

The build result is the same with and without the option, so it looks like it can be added unconditionally?!

[ryandesign (Ryan Carsten Schmidt)]

This ticket is about setting the derived data path, not about disabling the modern build system.

[mascguy (Christopher Nielsen)]

Add keyword portgroup, to pg-related tickets