Opened 2 years ago

Last modified 6 days ago

#57137 new enhancement

xcode portgroup should set -derivedDataPath on Xcode versions that need it

Reported by: ryandesign (Ryan Schmidt) Owned by:
Priority: Normal Milestone:
Component: ports Version:
Keywords: Cc: l2dy (Zero King), yan12125 (Chih-Hsuan Yen), ddrum2000, mndavidoff (Monte Davidoff), Schamschula (Marius Schamschula), eborisch (Eric A. Borisch), raimue (Rainer Müller), mtn88 (Max), inyeollee, wahspilihp (Philip Shaw), markemer (Mark Anderson), szhorvat (Szabolcs Horvát), AlD (Daniel Albers), jeremyhu (Jeremy Huddleston Sequoia), ShadSterling (Shad Sterling), TheLastLovemark, alexeyt820, majoc-at-astro (majoc-at-astro), telotortium (Robert Irelan), cooljeanius (Eric Gallager), mascguy (Chris N)
Port: xcode-1.0.tcl

Description

Ports that use the xcode 1.0 portgroup fail to build at least on Xcode 9.4 and 10.0 GM, because they cannot write to the derived data path. By default Xcode uses a derived data path inside the home directory, but fails to honor the HOME environment variable MacPorts specifies, so it tries to write into the macports user's home directory which is not writable.

To work around this, the poedit port manually specifies the derived data path, and I copied this usage to the MoltenVK port when I created it, but every other port that builds with xcodebuild (such a graphviz-gui or Readown) has failed to build for me in the same way, so I think we need to move that fix into the portgroup.

It looks like the -derivedDataPath flag is available in xcodebuild 5 and later (Mountain Lion and later). So we could add the flag for Xcode 5 and later, assuming it does no harm, though surely these ports haven't been broken since Mountain Lion. I suspect some new version of Xcode must have introduced a change. Do we know what version that might be?

Attachments (1)

main.log (352.8 KB) - added by ddrum2000 2 years ago.
From building graphviz-gui

Download all attachments as: .zip

Change History (35)

comment:1 Changed 2 years ago by yan12125 (Chih-Hsuan Yen)

Cc: yan12125 added

comment:2 Changed 2 years ago by kencu (Ken)

I noticed three things that appear to be general xcode PG issues while fixing aquaterm and looking over irrlicht...

  1. why do we run a build phase, when the build always seems to occur again in the destroot phase anyway, and sometimes this causes errors?
  2. the -derivedDataPath issue
  3. at least on Xcode 10, the -derivedDataPath flag was only accepted on aquaterm if there was also a scheme, which so far seems to be the same as ${xcode.target}

SO -- I am wondering if the xcode PG might just add the following as a default, as it seems to be working for me so far on several ports:

build                   {}
destroot.pre_args       -derivedDataPath ./DerivedData
xcode.scheme            ${xcode.target}
Last edited 2 years ago by kencu (Ken) (previous) (diff)

comment:3 in reply to:  2 Changed 2 years ago by ryandesign (Ryan Schmidt)

Replying to kencu:

I noticed three things that appear to be general xcode PG issues while fixing aquaterm and looking over irrlicht...

  1. why do we run a build phase, when the build always seems to occur again in the destroot phase anyway, and sometimes this causes errors?

I have noticed that some Xcode-using ports do that. I do not know why that happens. Some Xcode-using ports override the destroot phase entirely, perhaps in order to combat that. In MacPorts we should be building in the build phase and destrooting in the destroot phase. If we are not doing that for Xcode-using ports, we should figure out a way to fix that.

  1. the -derivedDataPath issue
  2. at least on Xcode 10, the -derivedDataPath flag is only accepted if there is a scheme, which so far seems to be the same as ${xcode.target}

SO -- I am wondering if the xcode PG might just add the following as a default, as it seems to be working for me so far on several ports:

build                   {}
destroot.pre_args       -derivedDataPath ./DerivedData
xcode.scheme            ${xcode.target}

I wasn't aware that Xcode required a scheme to use the derived data path.

A scheme is not the same thing as a target, that's why there are separate settings in the portgroup.

comment:4 Changed 2 years ago by kencu (Ken)

I assume the xcode PG's "destroot" phase runs some "build-then-install" logic in Xcode that might be hard to dissect...

It could be you either choose destrooting in the build phase or building in the destroot phase :>

comment:5 Changed 2 years ago by ryandesign (Ryan Schmidt)

It would be silly if we had to use either of those in general for all Xcode-using ports.

Perhaps there is something that some Xcode projects do that causes this to happen.

Maybe this used to work the way we wanted in older versions of Xcode, back when the portgroup was added, and something broke or changed in newer versions of Xcode.

Changed 2 years ago by ddrum2000

Attachment: main.log added

From building graphviz-gui

comment:6 Changed 2 years ago by ddrum2000

Sorry for the double post I accidentally started above.

I suspect I have the same problem when trying to build graphviz-gui. I'm running Mojave with Xcode 10. I have attached a log file in the previous post.

Last edited 2 years ago by ddrum2000 (previous) (diff)

comment:7 in reply to:  6 Changed 2 years ago by ryandesign (Ryan Schmidt)

Cc: ddrum2000 added

Replying to ddrum2000:

So I suspect I have the same problem when trying to build graphviz-gui. I'm running Mojave with Xcode 10. I have attached a log file.

Right, that's the problem you'll see building most any port that uses xcodebuild, with new versions of Xcode.

We need to decide under what conditions to add the -derivedDataPath flag to the xcode portgroup. Should we add it for Xcode 5 and later, since the option appeared in that version? Or should we only add it on the versions of Xcode where it fails if we don't, and if so, what versions are those?

We also need clarity on the scheme issue Ken raised.

comment:8 Changed 2 years ago by mndavidoff (Monte Davidoff)

Cc: mndavidoff added

comment:9 in reply to:  4 Changed 2 years ago by jmroot (Joshua Root)

Replying to kencu:

I assume the xcode PG's "destroot" phase runs some "build-then-install" logic in Xcode that might be hard to dissect...

Note that there is a confirmed bug with the new build system in Xcode 10 that causes problems with building and then destrooting separately, but not when doing both in one step. See [9cb3c083519503bb4b404fe57f19466a924523be/macports-ports] and IRC logs of 26th-27th September (starting around 22:40).

comment:10 Changed 2 years ago by Schamschula (Marius Schamschula)

Cc: Schamschula added

comment:11 Changed 2 years ago by eborisch (Eric A. Borisch)

Cc: eborisch added

comment:12 Changed 2 years ago by raimue (Rainer Müller)

Cc: raimue added

comment:13 Changed 2 years ago by ryandesign (Ryan Schmidt)

Cc: mtn88 added

Has duplicate #57223.

comment:14 Changed 2 years ago by inyeollee

Cc: inyeollee added

comment:15 Changed 2 years ago by wahspilihp (Philip Shaw)

Cc: wahspilihp added

comment:16 Changed 2 years ago by markemer (Mark Anderson)

Cc: markemer added

comment:17 Changed 2 years ago by szhorvat (Szabolcs Horvát)

Cc: szhorvat added

comment:18 Changed 2 years ago by AlD (Daniel Albers)

Cc: AlD added

comment:19 Changed 2 years ago by jeremyhu (Jeremy Huddleston Sequoia)

Has someone filed a radar about this issue? Please provide a reference to it.

comment:20 Changed 2 years ago by jeremyhu (Jeremy Huddleston Sequoia)

Cc: jeremyhu added

comment:21 in reply to:  19 Changed 23 months ago by jmroot (Joshua Root)

Replying to jeremyhu:

Has someone filed a radar about this issue? Please provide a reference to it.

Toby filed one about the problem I mentioned in comment:9. He didn't mention the number unfortunately; you'd have to ask him. I'm not clear on whether there are multiple separate issues here or if they're all related.

comment:22 Changed 20 months ago by ShadSterling (Shad Sterling)

Cc: ShadSterling added

comment:23 Changed 19 months ago by ryandesign (Ryan Schmidt)

Cc: TheLastLovemark added

Has duplicate #58276.

comment:24 Changed 19 months ago by ryandesign (Ryan Schmidt)

Cc: alexeyt820 added

Has duplicates #58126, #58297.

comment:25 Changed 19 months ago by kencu (Ken)

So one issue is the way the xcode PortGroup works. If you look at the PortGroup file, it builds the software in the build{} phase, and then completely rebuilds the software all again in the destroot{} phase, and then destroots it. I presume this was so the build phase actually does something. I don't believe you can tell xcodebuild to separate the two. I haven't yet sorted out the exact extra bit the destroot{} command does to destroot it, but no doubt it's obvious enough with comparisons.

The old Xcode build system allowed this. The new Xcode build system will not allow redundant things in the build tree, so I presume that it errors out on those builds that do some extra stuff, like generating symlinks or building man or pdf files with a script. The new build system errors.

That is easy to fix. Just don't run the build{} phase. It does nothing useful to run the build{} phase anyway. I do not see a way to build in the build{} phase and just destroot in the destroot{} phase. AFAICT, xcodebuild cannot do that, although I'm not an expert on the inner workings of it. So this is a MacPorts' specific problem, as we do the double build. I do not see any Xcode fix coming for that.

The second problem is that as Ryan pointed out, Xcode dumps it's products in the home directory, which is locked out on MacPorts. This is another MacPorts' specific problem -- homebrew would not see it (as people run as the current user). You CAN set the -derivedDataPath ./DerivedData to force the derived data to go somewhere writable, but that only works for me if you also set a scheme. Perhaps every (recent) Xcode project file has a scheme we can explore for and use, I don't know. If so, if you set a scheme, then xcodebuild will accept the -derivedDataPath ./DerivedData. Each will have to be investigated for such a scheme, although often I've found that the target also works (not always).

For projects with no scheme, it appears you have to use the old build system, or accept the default -derivedDataPath.

comment:26 Changed 19 months ago by kencu (Ken)

So what could we do?

Change the xcode PG defaults a bit for Xcode 10+.

  1. Cancel the build phase for Xcode 10+, and just let it build in the destroot phase as it does already. That will get rid of all the symlink and similar errors. There is no other choice, AFAICT.
  1. set the -derivedDataPath ./DerivedData in the pre-args for Xcode 10+. If there is no scheme set, default it to xcode target, which works most of the time.
  1. make an override to use the old build system with a simple setting in the Portfile if the above doesn't work.

That will fix a lot of the broken builds. For the ones that still fail, the authour can try setting a proper scheme with a little investigation into what schemes there are in the project. And for the ones that fail THAT, you can set the old build system instead and give up.

The only other choice I can see would be to use the old build system all the time on Xcode 10+, which I actually kind of like as it's a one-line fix for every xcode 10 build failure I think -- but I know that idea would not fly.

comment:27 Changed 19 months ago by majoc-at-astro (majoc-at-astro)

Cc: majoc-at-astro added

comment:28 Changed 19 months ago by kencu (Ken)

For those looking to get something built right now, this little block added to the Portfile will likely fix the xcode build for software that has not yet been updated to work with the new Xcode 10+ build system:

# this port has not yet been updated to build with the new build system Xcode 10+
if {${os.platform} eq "darwin" && ([vercmp $xcodeversion 10.0] > 0)} {
    build.pre_args-append    -UseModernBuildSystem=NO
    destroot.pre_args-append -UseModernBuildSystem=NO
}

comment:29 Changed 18 months ago by MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)

In 377d0cc1a164f490be8f85ca3c7fb8020bcbe724/macports-ports (master):

MailtoMutt: allow build on newer systems

No revbump because port either builds correctly or not at all.
See #57137
Fixes #58455

comment:30 Changed 12 months ago by ryandesign (Ryan Schmidt)

FYI it looks like Xcode 5 and up support -derivedDataPath. Still not sure why we're not setting that in the xcode portgroup.

comment:31 Changed 12 months ago by telotortium (Robert Irelan)

Cc: telotortium added

comment:32 Changed 3 months ago by cooljeanius (Eric Gallager)

Cc: cooljeanius added

comment:33 Changed 4 weeks ago by mascguy (Chris N)

Cc: mascguy added

comment:34 Changed 6 days ago by RJVB (René Bertin)

Is there a safe os.version to check to determine if -UseModernBuildSystem is available or does one have to include the xcode PG?

Note: See TracTickets for help on using tickets.