openjdk21 @21.0.3_1+release+server : build fails on Sonoma 14.7 with "xattr: [Errno 13] Permission" denied error

[lima-bravo (Lodewijk Bonebakker)]

While building openjdk21 with sudo port install openjdk21 , the build fails. Scanning the compilation log (attached) the error seems to be:

:info:build gmake[3]: Leaving directory '/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_java_openjdk21/openjdk21/work/jdk-21.0.4+7/make'
:info:build xattr: [Errno 13] Permission denied: '/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_java_openjdk21/openjdk21/work/jdk-21.0.4+7/build/release/jdk/conf/management/jmxremote.password.template'
:info:build gmake[3]: *** [ModuleWrapper.gmk:105: /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_java_openjdk21/openjdk21/work/jdk-21.0.4+7/build/release/jdk/conf/management/jmxremote.password.template] Error 1
:info:build gmake[3]: *** Deleting file '/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_java_openjdk21/openjdk21/work/jdk-21.0.4+7/build/release/jdk/conf/management/jmxremote.password.template'
:info:build gmake[3]: *** Waiting for unfinished jobs....
:info:build gmake[3]: Leaving directory '/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_java_openjdk21/openjdk21/work/jdk-21.0.4+7/make'
:info:build gmake[2]: *** [make/Main.gmk:157: jdk.management.agent-copy] Error 2
:info:build gmake[2]: *** Waiting for unfinished jobs....

System information:

bash-5.2# uname -a
Darwin iMac 23.6.0 Darwin Kernel Version 23.6.0: Wed Jul 31 20:48:44 PDT 2024; root:xnu-10063.141.1.700.5~1/RELEASE_X86_64 x86_64

bash-5.2# xcodebuild -version
Xcode 16.0
Build version 16A242d

bash-5.2# xcode-select --version
xcode-select version 2408.

bash-5.2# pkgutil --pkg-info=com.apple.pkg.CLTools_Executables
package-id: com.apple.pkg.CLTools_Executables
version: 16.0.0.0.1.1724870825
volume: /
location: /
install-time: 1726939429

bash-5.2# ls -al /usr/bin/llvm-gcc
-rwxr-xr-x  77 root  wheel  119008 Sep  5 11:17 /usr/bin/llvm-gcc

bash-5.2# /usr/bin/llvm-gcc -v
Apple clang version 16.0.0 (clang-1600.0.26.3)
Target: x86_64-apple-darwin23.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

[lima-bravo (Lodewijk Bonebakker)]

file: /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_java_openjdk21/openjdk21/main.log

[breun (Nils Breunese)]

The ticket title mentions 21.0.3_1, but the logs refer to 21.0.4. Can you confirm you actually tried installing 21.0.4_1?

I don't have access to macOS 14 myself anymore (already upgraded my machine to macOS 15), but both on ​GitHub and the buildbots (​Silicon, ​Intel) this build succeeded on macOS 14, so there may be something special in your environment.

The core error indeed seems to be this:

xattr: [Errno 13] Permission denied: '/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_java_openjdk21/openjdk21/work/jdk-21.0.4+7/build/release/jdk/conf/management/jmxremote.password.template'

I'm afraid I don't know why you would get a permission denied error when xattr is run on that file.

Can you reproduce this issue after cleaning this build?

[lima-bravo (Lodewijk Bonebakker)]

I retried install today, and it completed successfully, without any issues. Not materially doing anything different than yesterday.

sudo port uninstall openjdk21
sudo port clean openjdk21
sudo port install openjdk21

Today's output:

bash-5.2# port install openjdk21
--->  Computing dependencies for openjdk21
--->  Fetching archive for openjdk21
--->  Attempting to fetch openjdk21-21.0.4_1+release+server.darwin_23.x86_64.tbz2 from https://packages.macports.org/openjdk21
--->  Attempting to fetch openjdk21-21.0.4_1+release+server.darwin_23.x86_64.tbz2.rmd160 from https://packages.macports.org/openjdk21
--->  Installing openjdk21 @21.0.4_1+release+server
--->  Activating openjdk21 @21.0.4_1+release+server
--->  Cleaning openjdk21
--->  Updating database of binaries
--->  Scanning binaries for linking errors
--->  No broken files found.
--->  No broken ports found.
--->  Some of the ports you installed have notes:
  openjdk21 has the following notes:
    If you want to make openjdk21 the default JDK, add this to shell profile:
    export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-21-macports.jdk/Contents/Home

Yesterday's output:

bash-5.2# port install openjdk21
--->  Computing dependencies for openjdk21
--->  Fetching archive for openjdk21
--->  Attempting to fetch openjdk21-21.0.4_1+release+server.darwin_23.x86_64.tbz2 from https://packages.macports.org/openjdk21
--->  Attempting to fetch openjdk21-21.0.4_1+release+server.darwin_23.x86_64.tbz2 from https://fra.de.packages.macports.org/openjdk21
--->  Attempting to fetch openjdk21-21.0.4_1+release+server.darwin_23.x86_64.tbz2 from https://mse.uk.packages.macports.org/openjdk21
--->  Fetching distfiles for openjdk21
--->  Verifying checksums for openjdk21
--->  Extracting openjdk21
--->  Applying patches to openjdk21
--->  Configuring openjdk21
--->  Building openjdk21
Error: Failed to build openjdk21: command execution failed
Error: See /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_java_openjdk21/openjdk21/main.log for details.
Error: Follow https://guide.macports.org/#project.tickets if you believe there is a bug.
Error: Processing of port openjdk21 failed
bash-5.2# cd /opt/local/

Machine was shutdown yesterday evening and restarted this morning. Maybe the image at mse.uk.packages.macports.org has an issue?

[jmroot (Joshua Root)]

The difference is simply that there wasn't a binary archive available yesterday and there was today, so today it didn't have to attempt to build from source at all. You would probably reproduce the issue if you ran sudo port build openjdk21.

[lima-bravo (Lodewijk Bonebakker)]

I can confirm that I can indeed reproduce the issue with sudo port build openjdk21. Comparing the offending file with my laptop, I see the following:

bash-5.2# ls -ltr
total 40
-rw-r--r--@ 1 macports  wheel   3997 Sep 25 20:20 jmxremote.access
-rw-r--r--@ 1 macports  wheel  14988 Sep 25 20:20 management.properties
bash-5.2# xattr jmxremote.access
com.apple.provenance

The only difference I can see between the two systems is that the failing system has com.apple.provenance set as xattr, while the other doesn't. Hunting down the directory tree, I discovered that in /opt/local/var/macports directories where set with xattr:

bash-5.2# ls -ltr@
total 24
drwxr-xr-x    3 macports  macports     96 Apr 17 14:05 home
drwxrwxrwt    3 root      wheel        96 Apr 18 19:34 sip-workaround
drwxr-xr-x    9 root      wheel       288 Apr 18 19:34 registry
drwxr-xr-x    3 root      wheel        96 Jun  2 20:50 reclaim
-rw-r--r--    1 root      wheel        11 Sep 21 18:08 last_reclaim
-rw-r--r--    1 root      wheel      5184 Sep 21 18:11 pingtimes
drwxr-xr-x@   4 root      wheel       128 Sep 23 14:38 sources
	com.apple.provenance	   11
drwxr-xr-x    6 root      wheel       192 Sep 23 14:48 cache
drwxr-xr-x@   3 macports  wheel        96 Sep 25 17:09 incoming
	com.apple.provenance	   11
drwxr-xr-x  354 root      wheel     11328 Sep 25 19:57 software
drwxr-xr-x   21 root      wheel       672 Sep 25 19:59 distfiles
drwxr-xr-x    4 root      wheel       128 Sep 25 20:19 build
drwxr-xr-x   10 root      wheel       320 Sep 25 20:19 logs\

I tried to remove the xattr with

bash-5.2# xattr -d -r com.apple.provenance sources/
bash-5.2# xattr -d -r com.apple.provenance incoming/

only to discover that they were restored again quickly.

So something is triggering com.apple.provenance to be set again, which leads to the compilation error.

[breun (Nils Breunese)]

Replying to lima-bravo:

I tried to remove the xattr with

bash-5.2# xattr -d -r com.apple.provenance sources/
bash-5.2# xattr -d -r com.apple.provenance incoming/

only to discover that they were restored again quickly.

So something is triggering com.apple.provenance to be set again, which leads to the compilation error.

Those directories indeed don't have that com.apple.provenance attribute on my machine. I don't know why you have them. As far as I know that extended attribute is used by macOS to indicate a quarantine status. Did you ever transfer these directories from another machine?

Can you confirm that the com.apple.provenance attribute is actually removed and that it is added again, or could it be that it's just not getting removed?

While looking for information about this I find posts from people that say you need to make sure that your terminal application (Terminal.app, iTerm2.app, etc.) has Full Disk Access. You can configure this via System Preferences > Security & Privacy > Privacy > Full Disk Access.

[lima-bravo (Lodewijk Bonebakker)]

After more testing, the attributes are not getting removed - I don't know how to remove them or prevent them from being set, without disabling SIP

[lima-bravo (Lodewijk Bonebakker)]

Terminal has full disk access

[jjstickel (Jonathan Stickel)]

I am experiencing the same problem, with even more folders in /opt/local/var/macports having that xattr set:

$ /bin/ls -ltr@ /opt/local/var/macports
total 8
drwxr-xr-x@    3 macports  macports     96 Aug 27  2023 home
	com.apple.provenance	   11
drwxrwxrwt     3 root      wheel        96 Jan 19  2024 sip-workaround
drwxr-xr-x     9 root      wheel       288 Jan 19  2024 registry
drwxr-xr-x@    3 root      wheel        96 Jan 19  2024 reclaim
	com.apple.provenance	   11
-rw-r--r--@    1 root      wheel         9 Jan 19  2024 last_reclaim
	com.apple.provenance	   11
drwxr-xr-x@    6 root      wheel       192 Aug 16 08:42 cache
	com.apple.provenance	   11
drwxr-xr-x@    4 root      wheel       128 Oct 31 20:51 sources
	com.apple.provenance	   11
drwxr-xr-x@ 1221 root      wheel     39072 Nov 11 19:24 software
	com.apple.provenance	   11
drwxr-xr-x     5 root      wheel       160 Nov 11 19:24 build
drwxr-xr-x@    4 root      wheel       128 Nov 11 19:24 logs
	com.apple.provenance	   11
drwxr-xr-x@    3 macports  wheel        96 Nov 11 19:24 incoming
	com.apple.provenance	   11
drwxr-xr-x   417 root      wheel     13344 Nov 11 19:24 distfiles

This is with arm64 and macOS 15.1. It seems that builds that rely on using xattr are going to fail randomly for some users. Are there no workarounds other than disabling SIP or waiting for the binary of the port?

[ryandesign (Ryan Carsten Schmidt)]

The workaround would be to remove the xattrs from those directories and to figure out why they were added in the first place and prevent that from happening to other users.

[jjstickel (Jonathan Stickel)]

Replying to ryandesign:

The workaround would be to remove the xattrs from those directories and to figure out why they were added in the first place and prevent that from happening to other users.

Yes, I agree. From the brief reading I've done so far, (this ticket and a couple minutes of googling), it seems like it arises from some black-box Apple background process as part of the so-called "systems integrity protection". A bit beyond my capability and interest to investigate right now, though. It seems like you can turn it on or off, but there are no user options to control its behavior, which might allow us to manage this xattr business.

[ryandesign (Ryan Carsten Schmidt)]

We can't recommend that users turn off all or part of SIP; it is a beneficial security feature. We need to work within its constraints.

[breun (Nils Breunese)]

I don't see the com.apple.provenance extended attribute for these folders on my macOS 15.1 machine:

❯ /bin/ls -ltr@ /opt/local/var/macports
total 40
-r--r--r--    1 root      wheel  11203 17 nov  2016 port-help.tcl
drwxr-xr-x    4 macports  staff    128 19 jul  2018 home
-rw-r--r--    1 root      wheel     37 16 mrt  2019 stats-uuid
drwxr-xr-x    3 root      admin     96 20 okt  2022 reclaim
drwxr-xr-x    6 root      admin    192 17 aug 11:25 cache
drwxrwxrwt    5 root      wheel    160 17 sep 02:39 sip-workaround
drwxr-xr-x    9 root      wheel    288 17 sep 02:39 registry
drwxr-xr-x    4 root      wheel    128  1 nov 01:14 sources
drwxr-xr-x    3 macports  admin     96 16 nov 04:09 incoming
drwxr-xr-x    3 root      wheel     96 16 nov 04:10 build
drwxr-xr-x  367 root      wheel  11744 16 nov 04:10 software
drwxr-xr-x   24 root      wheel    768 16 nov 04:10 logs
drwxr-xr-x  284 root      wheel   9088 16 nov 04:10 distfiles
-rw-r--r--    1 root      wheel     11 16 nov 04:10 last_reclaim

I don't know why your folders do have that attribute and mine don't, and I also don't know why they would be set again after you remove them. This article explains a bit more about when com.apple.provenance gets set. Did you ever move these MacPorts folders over the network in an archive or something?

[jjstickel (Jonathan Stickel)]

Replying to breun:

I don't see the com.apple.provenance extended attribute for these folders on my macOS 15.1 machine: <snip> I don't know why your folders do have that attribute and mine don't, and I also don't know why they would be set again after you remove them. This article explains a bit more about when com.apple.provenance gets set. Did you ever move these MacPorts folders over the network in an archive or something?

I have no idea either. No, these folders were never moved in from elsewhere; /opt/local and all its contents were created via a typical Macports installation. As per ​Lodewijk, I'm not the only one this is happening to. I've also tried running xattr commands, with sudo, to remove the attributes, and nothing changes.

[breun (Nils Breunese)]

I have a lot of questions:

• Why is this not a problem on the MacPorts CI infrastructure and my machine?
• Is the presence of the com.apple.provenance attribute the cause of your build failing? If so, why is the com.apple.provenance attribute present on your machines? Is this expected and reproducible for new MacPorts installations?
• Which xattr command is getting executed exactly and why does it result in a '[Errno 13] Permission denied' error?
• Why can't you remove the attribute? (I believe this should be possible when  > System Preferences > Security & Privacy > Privacy > Full Disk Access > Terminal is on.)

Sadly I don't have the answers to these questions and I also don't know how to find them.

Maybe you could post on the MacPorts users mailinglist to find out whether the presence of this attribute on those MacPorts folders is common or not, and if so, if more users are experiencing issues due to this attribute.

[jjstickel (Jonathan Stickel)]

Replying to breun:

• Is the presence of the com.apple.provenance attribute the cause of your build failing?

Yes.

• Which xattr command is getting executed exactly and why does it result in a '[Errno 13] Permission denied' error?

See ​https://trac.macports.org/ticket/70890#comment:2

• Why can't you remove the attribute? (I believe this should be possible when  > System Preferences > Security & Privacy > Privacy > Full Disk Access > Terminal is on.)

See ​https://trac.macports.org/ticket/70890?replyto=16#comment:5 and the next few comments. Terminal (for reporter) and iTerm2 (for me) have full disk access.

Maybe you could post on the MacPorts users mailinglist to find out whether the presence of this attribute on those MacPorts folders is common or not, and if so, if more users are experiencing issues due to this attribute.

Sure, that could be a path forward. I've moved on at the moment and am not using ports that require java.

I don't know the answers to your other questions about why this doesn't happen on CI and why com.apple.provenance appears. I'll ask a question of my own: is SIP enabled on the CI VM?

[breun (Nils Breunese)]

Replying to jjstickel:

Replying to breun:

• Is the presence of the com.apple.provenance attribute the cause of your build failing?

Yes.

I’m not saying it isn’t, but what exactly is the proof for this?

• Which xattr command is getting executed exactly and why does it result in a '[Errno 13] Permission denied' error?

See ​https://trac.macports.org/ticket/70890#comment:2

That is my own comment, but I don’t see what xattr command exactly is getting executed there, only the resulting error message.

• Why can't you remove the attribute? (I believe this should be possible when  > System Preferences > Security & Privacy > Privacy > Full Disk Access > Terminal is on.)

See ​https://trac.macports.org/ticket/70890?replyto=16#comment:5 and the next few comments. Terminal (for reporter) and iTerm2 (for me) have full disk access.

If removing this extended attribute doesn’t work for you, then I’m afraid only Apple can possibly fix that, so you may have to report that issue to them.

I don't know the answers to your other questions about why this doesn't happen on CI and why com.apple.provenance appears. I'll ask a question of my own: is SIP enabled on the CI VM?

I don’t know, but I would expect it to be. Ryan can probably confirm this. I do know it’s enabled on my personal macOS 15.1 machine, which doesn’t have this issue.

[jjstickel (Jonathan Stickel)]

Alright, I had a few minutes to check more into xattr and found ​this forum post. Apparently terminal is the only shell that can remove the xattr com.apple.provenance (it fails silently with iTerm2). This I have now confirmed. However, the build of openjdk21 still fails with the same line, even after extracting and removing the generated xattrs from the build folder.

So, yes, the main question remains: what is the actual xattr execution line being run during the build? Also, why is it being run? I'm afraid someone else will need to dig into the jdk build process to sort it out.

[josemgarcia (José María García)]

FWIW, I was having this particular issue when trying to install the port on Sequoia 15.2. As mentioned in ​https://trac.macports.org/ticket/70890#comment:6 I gave full disk access to the Terminal app, and after trying again it worked.