openjdk11-graalvm, openjdk17-graalvm, openjdk19-graalvm: buildbot activation fails: file LICENSE_NATIVEIMAGE.txt already exists and does not belong to a registered port

[mascguy (Christopher Nielsen)]

This occurs across-the-board, with the following error:

Error: Failed to activate openjdk17-graalvm:
Image error: /Library/Java/JavaVirtualMachines/openjdk17-graalvm/Contents/Home/LICENSE_NATIVEIMAGE.txt
already exists and does not belong to a registered port.

However, this issue does not occur with GitHub CI; refer to successful builds via recent PR:

​PR 19623 - openjdk17-graalvm: update to 17.0.8

Nor does the issue occur locally, based on tests by multiple users. (Including the reporter of this ticket.)

Relevant info: The item in question is actually a symlink, extracted from the upstream tarball:

$ ls -l /Library/Java/JavaVirtualMachines/openjdk17-graalvm/Contents/Home/LICENSE_NATIVEIMAGE.txt
lrwxr-xr-x  1 root  admin  31 25 jul 21:33 /Library/Java/JavaVirtualMachines/openjdk17-graalvm/Contents/Home/LICENSE_NATIVEIMAGE.txt -> lib/svm/LICENSE_NATIVEIMAGE.txt

See previous discussion via the following PR, for port jet, which requires openjdk17-graalvm to build:

​PR 19615 - jet: new submission (version 0.6.26)

[ryandesign (Ryan Carsten Schmidt)]

Replying to mascguy:

This occurs across-the-board, with the following error:

Error: Failed to activate openjdk17-graalvm:
Image error: /Library/Java/JavaVirtualMachines/openjdk17-graalvm/Contents/Home/LICENSE_NATIVEIMAGE.txt
already exists and does not belong to a registered port.

It's not just this file, and it's not just this port. On the buildbot workers, /Library/Java/JavaVirtualMachines contain directories openjdk11-graalvm, openjdk17-graalvm, and openjdk19-graalvm, each 80-90MB. Some previous version of the ports must have installed their files in a way that bypassed the destroot. That problem has now evidently been fixed, based on the fact that the activation failure message now occurs. Code should be added to the ports to delete these unregistered files at pre-activate time, and that code should be kept in the ports for one year.

However, this issue does not occur with GitHub CI; refer to successful builds via recent PR:

​PR 19623 - openjdk17-graalvm: update to 17.0.8

Not surprising since GitHub CI always starts from a fresh OS image on which MacPorts has never been installed. Buildbot workers and user systems, however, do not wipe themselves clean so they retain any problems caused by ports, which is why ports should clean up those problems.

Nor does the issue occur locally, based on tests by multiple users. (Including the reporter of this ticket.)

That only means that these users did not install whichever problematic versions of these ports installed these files outside of the destroot. I checked the 10.11 and 10.13 buildbot workers and the file had been created on May 10, 2023, so I suspect if you backdate your ports collection to that date and then install one of these ports, it would install its files outside the destroot, and you would then be able to reproduce the issue after updating your ports collection back to the current version and trying to install the current versions of the ports.

[mascguy (Christopher Nielsen)]

Ah, makes sense, thanks for detail Ryan!

Going forward, I'd strongly recommend that all of our JDK-related ports be installed into the MacPorts tree (perhaps under ${prefix}/libexec/java/openjdkXXXXX, or wherever it makes the most sense).

And then create a symlink in /Library/Java/JavaVirtualMachines/openjdkXXXX, to the target.

This is an approach I've been using for years, across numerous Java versions. And it works beautifully.

Better still, that symlink could be optional, perhaps only installed via a subport [selected via a variant].

With the caveat that our Java PG should search the agreed-upon standard area within MacPorts, to avoid requiring that a symlink exist.

[ryandesign (Ryan Carsten Schmidt)]

Installing in the MacPorts prefix and symlinking into the required system location in /Library is also the approach MacPorts base uses for launchd plists.

My understanding was that Java on macOS must be installed (or symlinked) into /Library/Java/JavaVirtualMachines in order to work.

[jmroot (Joshua Root)]

Replying to ryandesign:

That only means that these users did not install whichever problematic versions of these ports installed these files outside of the destroot. I checked the 10.11 and 10.13 buildbot workers and the file had been created on May 10, 2023, so I suspect if you backdate your ports collection to that date and then install one of these ports, it would install its files outside the destroot, and you would then be able to reproduce the issue after updating your ports collection back to the current version and trying to install the current versions of the ports.

It specifically appears to have been the former versions of the *-native-image subports, e.g. openjdk17-graalvm-native-image: [30d11901cc9927c643c8bc2f29557d67d7fae98c/macports-ports] (which incidentally looks like it should probably be setting replaced_by but isn't.)

[jmroot (Joshua Root)]

Replying to mascguy:

Going forward, I'd strongly recommend that all of our JDK-related ports be installed into the MacPorts tree (perhaps under ${prefix}/libexec/java/openjdkXXXXX, or wherever it makes the most sense).

And then create a symlink in /Library/Java/JavaVirtualMachines/openjdkXXXX, to the target.

That may be a good idea for other reasons, but it doesn't make it harder to bypass the destroot.

[breun (Nils Breunese)]

I don’t mind changing the Java ports to install their files under the MacPorts prefix and only creating a symlink under /Library/Java/JavaVirtualMachines, but I don’t see how that would fix this issue.

As I understand it, the builbots apparently had the conflicting file still on them from a previous installation where uninstalling the GraalVM port didn’t remove them? Do you have any idea how that could have happened? When I uninstall this port on my machine, the file is gone.

[ryandesign (Ryan Carsten Schmidt)]

Replying to breun:

I don’t mind changing the Java ports to install their files under the MacPorts prefix and only creating a symlink under /Library/Java/JavaVirtualMachines, but I don’t see how that would fix this issue.

As Josh said, it would not fix this issue.

As I understand it, the builbots apparently had the conflicting file still on them from a previous installation where uninstalling the GraalVM port didn’t remove them? Do you have any idea how that could have happened? When I uninstall this port on my machine, the file is gone.

As Josh said, it was "the *-native-image subports, e.g. openjdk17-graalvm-native-image". Those subports used to contain:

        if {[regexp {(?i)native-image} [exec ${java_home}/bin/gu list] match]} {
            system "sudo ${java_home}/bin/gu remove native-image"
        }

Presumably at some point either the regexp did not match the gu list output or else the gu remove native-image command failed.

[mascguy (Christopher Nielsen)]

Replying to ryandesign:

My understanding was that Java on macOS must be installed (or symlinked) into /Library/Java/JavaVirtualMachines in order to work.

Well, it's certainly beneficial to add symlinks there, since that's the standard macOS install location. (Which you folks already know, but just mentioning it for others who may be less familiar with all of these details.) And tools like /usr/libexec/java_home look there, when searching for installed JDKs. Ditto for Java-based apps and tools, if they're macOS-aware.

So to provide the best experience to our users - and minimize the chance of issues - it makes sense to create a symlink in that location.

Installing in the MacPorts prefix and symlinking into the required system location in /Library is also the approach MacPorts base uses for launchd plists.

Yep, exactly, which is why I'm advocating for installing in the MacPorts tree.

Pondering this a bit more, it might be good to install JDKs under ${prefix}/Library/Java/JavaVirtualMachines/openjdkXXXX, similar to how we handle Python and other components. But whichever area we choose, it would be preferable to installing directly to /Library/Java/JavaVirtualMachines.

Thoughts all?

[catap (Kirill A. Korinsky)]

Replying to mascguy:

Pondering this a bit more, it might be good to install JDKs under ${prefix}/Library/Java/JavaVirtualMachines/openjdkXXXX, similar to how we handle Python and other components. But whichever area we choose, it would be preferable to installing directly to /Library/Java/JavaVirtualMachines.

Thoughts all?

Java works a bit different than python on macOS.

When you call java it checks JAVA_HOME environment variable to determine which version it should use.

Unfortetnly, at least one, we had quite a pain here. Do you remebere BigSur java issue? :)

[breun (Nils Breunese)]

I've created a first PR for one of my openjdk* ports to install all files under ${prefix} and only create a symlink under /Library/Java/JavaVirtualMachines: ​https://github.com/macports/macports-ports/pull/19857

I don't believe there is anything I can do to clean up the dangling symlink on the buildbots, or is there?

[catap (Kirill A. Korinsky)]

berun, the issue is not onlu buildbots, but also users machines.

Maybe you should add pre-install which cleanup it? Let keep it for a while like a year to be on safe side.

[breun (Nils Breunese)]

openjdk11-graalvm still has a openjdk11-graalvm-native-image subport which runs ${java_home}/bin/gu remove native-image on deactivation, so I guess that should be fine? Or does the probleem seem to be that this command doesn't remove everything that the install counterpart installed?

openjdk19-graalvm has already been obsoleted and replaced by openjdk20-graalvm, which no longer has a separate subport for native image support, but includes it.

The ​update to openjdk17-graalvm 17.0.7 indeed removed the openjdk17-graalvm-native-image subport, because this feature got included in the main openjdk17-graalvm update. If the deactivation ran, then the ${java_home}/bin/gu remove native-image should have been run on deactivation.

Is this problem only this 1 file getting left behind, or are there more that need to be cleaned up?

[mascguy (Christopher Nielsen)]

Replying to breun:

The ​update to openjdk17-graalvm 17.0.7 indeed removed the openjdk17-graalvm-native-image subport, because this feature got included in the main openjdk17-graalvm update. If the deactivation ran, then the ${java_home}/bin/gu remove native-image should have been run on deactivation.

Is this problem only this 1 file getting left behind, or are there more that need to be cleaned up?

We should cleanup everything, to avoid multiple rounds of fixes.

[mascguy (Christopher Nielsen)]

Replying to mascguy:

We should cleanup everything, to avoid multiple rounds of fixes.

Nils, don't hesitate to CC me on all of these PRs. That way I'll immediately be notified, rather than having to look for them.

[breun (Nils Breunese)]

We should cleanup everything (...)

Do you mean running rm -rf /Library/Java/JavaVirtualMachines/${name} in pre-install?

[mascguy (Christopher Nielsen)]

Replying to breun:

We should cleanup everything (...)

Do you mean running rm -rf /Library/Java/JavaVirtualMachines/${name} in pre-install?

Yes, albeit via the MacPorts macro delete.

[mascguy (Christopher Nielsen)]

Replying to mascguy:

Replying to breun:

We should cleanup everything (...)

Do you mean running rm -rf /Library/Java/JavaVirtualMachines/${name} in pre-install?

Yes, albeit via the MacPorts macro delete.

But only for these openjdkXX-graalvm ports, of course.

[breun (Nils Breunese)]

Can you review ​https://github.com/macports/macports-ports/pull/20034 for openjdk17-graalvm? When merged, I'll apply the fix to the other openjdk*-graalvm ports too.

[breun (Nils Breunese)]

PR for openjdk11-graalvm: ​https://github.com/macports/macports-ports/pull/20036

[breun (Nils Breunese)]

PRs for cleaning up existing files has been merged for openjdk11-graalvm and openjdk17-graalvm.

openjdk20-graalvm never had a subport for native image support, its first release already included it.

openjdk19-graalvm is an obsoleted port, does that need any fixing? If so, how could that be done? If not, I think this ticket is done.

[mascguy (Christopher Nielsen)]

Replying to breun:

openjdk19-graalvm is an obsoleted port, does that need any fixing? If so, how could that be done?

You can follow the pattern which I used for libfmt, along with your pre-install cleanup:

# -*- coding: utf-8; mode: tcl; tab-width: 4; indent-tabs-mode: nil; c-basic-offset: 4 -*- vim:fenc=utf-8:ft=tcl:et:sw=4:ts=4:sts=4

PortSystem          1.0
PortGroup           stub 1.0

name                libfmt
version             9.0.0
revision            2

categories          devel
description         Obsolete stub port, replaced by libfmt8/libfmt9
long_description    {*}${description}

# This port has purposely been made a stub, to eliminate any potential use.
# Otherwise, it will clash with the segregated versions - libfmt8 and libfmt9 - breaking dependents.

# Obsolete Date: 2022-08-10

[mascguy (Christopher Nielsen)]

Replying to mascguy:

You can follow the pattern which I used for libfmt, along with your pre-install cleanup

Eliminate the lines for supported_archs and platforms though, as we want it to actually build/install everywhere.

[breun (Nils Breunese)]

openjdk19-graalvm is an obsoleted port since July 4th with openjdk20-graalvm marked as its replacement, so I expect openjdk19-graalvm will already have been uninstalled for most if not all users.

​https://ports.macports.org/port/openjdk19-graalvm/ also reports 0 installations, so I don't expect making changes to openjdk19-graalvm will have any effect?

Or is the goal to clean up the buildbots this way?

[mascguy (Christopher Nielsen)]

Replying to breun:

Or is the goal to clean up the buildbots this way?

Yep, the goal is to cleanup the buildbots, if needed.

Ryan, have you already cleaned up the remnants of these? Or would it be helpful to forcibly do so with a stub port?

[catap (Kirill A. Korinsky)]

Christopher, Nils I guess you must revbum *all* ports which has all affected JVMs (all javas?) as lib or run dependencies.

Without it the desister has been made because port may have *hardcoded* path to JVM and it might be /Library/Java/JavaVirtualMachines/openjdk11/Contents/Home/bin/java.

[catap (Kirill A. Korinsky)]

Ok, this isn't relevant to your update. It is different kind of issue.

[breun (Nils Breunese)]

Replying to catap:

port may have *hardcoded* path to JVM and it might be /Library/Java/JavaVirtualMachines/openjdk11/Contents/Home/bin/java.

The /Library/Java/JavaVirtualMachines/${name} symlink should make sure that paths that start with with /Library/Java/JavaVirtualMachines/ should just keep working.

[catap (Kirill A. Korinsky)]

BTW shall we move bootstrap JVMs also to prefix? I guess this one doesn't required symlink at all.

[breun (Nils Breunese)]

Yeah, I guess there is no need for bootstrap JVMs to have a presence under /Library/Java/JavaVirtualMachines indeed.

[mascguy (Christopher Nielsen)]

Replying to mascguy:

Replying to breun:

Or is the goal to clean up the buildbots this way?

Yes, the goal is to cleanup the buildbots, if needed.

Ryan, have you already cleaned up the remnants of these? Or would it be helpful to forcibly do so with a stub port?

Ryan...?

[catap (Kirill A. Korinsky)]

Meanwhile ABCL on macOS-11 picks:

/opt/local/bin/abcl: line 2: /Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home/bin/java: No such file or directory

because of:

DEBUG: Running callback java::java_set_env
DEBUG: java-portgroup: Trying to find JVM version: 11
DEBUG: java-portgroup: Found multiple installations for JVM 11
DEBUG: java-portgroup: Detected JVMs: 11 /Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home
DEBUG: java-portgroup: Discovered matching JAVA_HOME: /Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home
DEBUG: Adding dependency on JDK fallback openjdk11
DEBUG: Finished running callback java::java_set_env

See: ​https://build.macports.org/builders/ports-11_x86_64-builder/builds/124444/steps/install-port/logs/stdio

[breun (Nils Breunese)]

@catap Is that good/bad/expected/unexpected? And exactly how is that related to GraalVM ports leaving files behind after uninstallation?

[catap (Kirill A. Korinsky)]

This is another jvm which left behind. It doesn't related to graalvm ports, but you have cleaned a lot of JVMs left behind by this ticket and it seems logical to point that we have one more :)

[breun (Nils Breunese)]

Are you sure /Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk is from a MacPorts port? I think I only recently started using the .jdk suffix in paths of MacPorts JDK ports. AdoptOpenJDK changed into Eclipse Temurin two years ago (​https://blog.adoptopenjdk.net/2021/08/goodbye-adoptopenjdk-hello-adoptium/).

[catap (Kirill A. Korinsky)]

I see, when it is another manually installed JVM :(

[mascguy (Christopher Nielsen)]

Let's close this as fixed.

Great work and collaboration folks!