Symlinks in prefix make destroot layout unpredictable

[Dave-Allured (Dave Allured)]

port install meson fails to install a working executable for meson command, causing build failures in other ports. I am building from sources into a custom prefix, on Monterey 12.7.2.

rackmac1:~/port/291/bin 45> where ninja
/Users/dallured/port/291/bin/ninja
rackmac1:~/port/291/bin 46> where meson
rackmac1:~/port/291/bin 47> ls -go | grep -e ninja -e meson
lrwxr-xr-x    1         39 Feb 12 06:52 meson -> /Users/dallured/port/291/bin/meson-3.12
-rwxr-xr-x    1     410168 Feb 11 18:56 ninja
rackmac1:~/port/291/bin 48> ./meson
./meson: Command not found.

I am not sure whether this problem is in port meson, or port py312-meson. It could be either. The build log for py312-meson shows many suspicious warnings for py312-meson. "Operation not permitted", etc. See attached build log.

[Dave-Allured (Dave Allured)]

Build log for py312-meson and meson

[Dave-Allured (Dave Allured)]

On closer examination:

• Most, if not all, of "Operation not permitted" warnings are normal for installing without root privileges. Therefore those warnings are not relevant, and can be ignored.

• This looks like the real problem. Port py312-meson is not correctly installing into the custom prefix. How to fix?

  Warning: violation by /Volumes
  Warning: py312-meson violates the layout of the ports-filesystems!
  

[kencu (Ken)]

just to be clear here.

you have this file:

/Users/dallured/port/291/bin/meson

it is a symlink to this:

/Users/dallured/port/291/bin/meson -> /Users/dallured/port/291/bin/meson-3.12

yet when you do this:

cd /Users/dallured/port/291/bin
./meson

you get this:

./meson: Command not found

is that all correct?

If it is, does this exist?

/Users/dallured/port/291/bin/meson-3.12

[Dave-Allured (Dave Allured)]

Yes, that is all correct. That ls -go followed by grep was supposed to show that, but that was a bit obscure, sorry.

The meson sym link was put in by the meson port. It was expecting that py312-meson already existed in the same directory, but it is not there.

Also to clarify further, my custom prefix is /Users/dallured/port/291. Everything is supposed to be installing under here. This has worked out quite well for hundreds of ports, but it hit this meson thing today.

[ryandesign (Ryan Carsten Schmidt)]

The meson port installs the $prefix/bin/meson symlink that points to $prefix/bin/meson-3.12

The py312-meson port installs $prefix/bin/meson-3.12

So if you're missing meson-3.12, then the py312-meson port's build/installation log is the one we need to be examining.

[kencu (Ken)]

you might also do a

port contents py312-meson

to see where bin/meson-3.12 actually is

that should be fun ;>

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign:

the py312-meson port's build/installation log is the one we need to be examining.

And I see it is included in the file you attached, but you appear to have provided the output of running port with the -v flag, so it's missing all the additional detail that we would have gotten from a main.log file or running port with the -d flag.

py-meson, like all py-* ports, uses the python portgroup, which is where the code that creates the bin symlinks is (in a post-destroot block). If you're missing this symlink for the py312-meson port I would expect you to be missing the symlinks for all other py-* ports too.

[Dave-Allured (Dave Allured)]

Output from port contents py312-meson

[Dave-Allured (Dave Allured)]

List of all my installed ports

[kencu (Ken)]

looks like your home folder is symlink to something else?

[Dave-Allured (Dave Allured)]

symlink to something else?

Yes. I had never noticed that. It is a shared mac within a server room. in /Users, there are a bunch of sym links to physical user home directories.

[Dave-Allured (Dave Allured)]

I uninstalled and re-ran port -d install py312-meson only, to get the requested extra debug info. Here is the console output. There was no main.log because, I guess, the build thought it was successful.

[ryandesign (Ryan Carsten Schmidt)]

Yes, main.log files are deleted after successful builds unless you set keeplogs yes in macports.conf or use the -k flag on the command line.

[ryandesign (Ryan Carsten Schmidt)]

From your re-run log​, this line is where the portgroup would create the symlink that you're missing:

DEBUG: Executing proc-post-org.macports.destroot-destroot-0

Unfortunately the portgroup doesn't log anything when it does or does not create the symlink. You could try adding some logging code there to see what's happening.

Replying to Dave-Allured:

symlink to something else?

Yes. I had never noticed that. It is a shared mac within a server room. in /Users, there are a bunch of sym links to physical user home directories.

From your port contents​ we can see that some parts of MacPorts are evidently resolving that symlink:

Port py312-meson contains:
  /Users/dallured/port/291/share/py312-meson/bash-completion/completions/meson

while other parts aren't:

  /Volumes/RM1_Homes/dallured/port/291/Library/Frameworks/Python.framework/Versions/3.12/bin/meson

I don't know if that's the source of the problem but you could try reinstalling MacPorts base with --prefix=/Volumes/RM1_Homes/dallured/port/291 instead of --prefix=/Users/dallured/port/291, and similarly changing applications_dir and frameworks_dir if you've customized them. If reinstalling py312-meson after doing that fixes the problem, then you could use port contents on your installed ports to locate those that mention paths in in /Users/dallured/port/291 and then rebuild those ports.

[reneeotten (Renee Otten)]

the intention of the meson port is to install the pyXY-meson port, where XY is MacPorts’ current default Python version. Additionally, it will create the symlink(s) so that you and other ports (including the meson PortGroup) should/will JustWork. That does work where I have tested it and there have not been any other reports of issues, so it appears this is something unusual on your specific setup. Not sure if I (or anyone else here) can fix this for you..

[Dave-Allured (Dave Allured)]

it appears this is something unusual on your specific setup

Apparently so. Ken and Ryan have correctly identified that user home directories on this machine go through sym links. My guess is that this really is the source of this particular problem. For myself, I can look at a workaround.

The lingering question is, should support for this rare special case (sym linked home directory) be added to Macports for general application?

[jmroot (Joshua Root)]

Cf. #21082, #39850. In short, things get really confusing when the prefix is not a fully resolved and normalised path. Whether the normalised or un-normalised version is created in the destroot, anything not checking for both will potentially get it wrong. I would guess the glob in the python portgroup is just not matching anything.

You probably don't need to reinstall base since you're not actually moving any files, just update macports.conf to use the resolved path for everything.

[reneeotten (Renee Otten)]

okay, so this has nothing to do with the py-meson or meson ports.

[kencu (Ken)]

Replying to Dave-Allured:

The lingering question is, should support for this rare special case (sym linked home directory) be added to Macports for general application?

Ideally, sure, but in real life -- probably too many pitfalls to reliably do it.

If configuring macports to install into --prefix=/Volumes/RM1_Homes/dallured/port/291 fixes things up for you (it should) then someone could add that to the documentation as a workaround.

[jmroot (Joshua Root)]

It would likely require patching a lot of build systems to handle such paths consistently in the destroot phase. It might be python that is normalising the path in this case; you could check with python3.12 -c "import sysconfig; print(sysconfig.get_config_var('base'))".

[ryandesign (Ryan Carsten Schmidt)]

Let's leave it open for a minute to confirm whether fully resolving the MacPorts prefix fixes the problem. If it does, then the next question is if we can do anything about it. It may not be a bug in a specific python port or in the python portgroup but we might consider it to be a bug in base.

If it is, mentioning it in the documentation is a good start. Another good step could be for port diagnose to print an error if the prefix is not fully resolved. And finally, could the configure script do the hard work of determining if the path is fully resolved, and either error out if it is not or even better just fix it internally and use the fully resolved path instead?

[Dave-Allured (Dave Allured)]

Yes, leave this open for a bit, for the conversation. Thank you. I am in a crunch this week, but will try the prefix test in a few days. I think Renee has a point, that this may not have much to do specifically with the meson ports.

[Dave-Allured (Dave Allured)]

As suggested, I ran port contents to locate ports trying to use the fully resolved prefix that includes /Volumes. Only the py-* ports did this. Note that base ports python310, python311, python312 respected the original prefix and did not try to use the fully resolved prefix. It seems that this problem may be isolated to py-* ports only, so a portgroup solution might be feasible.

rackmac1:~/port 56> foreach p ( `cat logfiles/py312-meson.0212e.installed.list ` )
foreach? echo `port contents $p |  grep -c /Volumes` "  $p" |& tee -a counts.volumes-str.0212e
foreach? end
0 	aom
0 	autoconf
0 	automake
0 	bash
0 	bison
0 	bison-runtime
0 	brotli
0 	bzip2
0 	cctools
0 	clang-16
0 	clang_select
0 	cmake
0 	cmake-bootstrap
0 	curl
0 	curl-ca-bundle
0 	db48
0 	djvulibre
0 	expat
0 	fftw-3
0 	flex
0 	fontconfig
0 	freetype
0 	gawk
0 	gcc13
0 	gcc13-libcxx
0 	gcc_select
0 	gdbm
0 	gettext
0 	gettext-runtime
0 	gettext-tools-libs
0 	giflib
0 	git
0 	gmake
0 	gmp
0 	gperf
0 	grep
0 	gsed
0 	hatchling_select
0 	help2man
0 	icu
0 	imath
0 	isl
0 	jasper
0 	jbig2dec
0 	jbigkit
0 	kerberos5
0 	lapack
0 	lcms2
0 	ld64
0 	ld64-xcode
0 	lerc
0 	libarchive
0 	libb2
0 	libcomerr
0 	libcxx
0 	libde265
0 	libdeflate
0 	libedit
0 	libffi
0 	libgcc
0 	libgcc13
0 	libiconv
0 	libidn
0 	libidn2
0 	libjpeg-turbo
0 	libmpc
0 	libomp
0 	libpaper
0 	libpng
0 	libpsl
0 	libraw
0 	libtextstyle
0 	libtool
0 	libunistring
0 	libuv
0 	libxml2
0 	libyaml
0 	llvm-16
0 	llvm_select
0 	lmdb
0 	lz4
0 	lzip
0 	lzo2
0 	m4
0 	mpfr
0 	nasm
0 	ncurses
0 	nghttp2
0 	ninja
0 	openexr
0 	openjpeg
0 	openssl
0 	openssl3
0 	ossp-uuid
0 	p5.34-authen-sasl
0 	p5.34-b-cow
0 	p5.34-canary-stability
0 	p5.34-capture-tiny
0 	p5.34-cgi
0 	p5.34-clone
0 	p5.34-common-sense
0 	p5.34-compress-raw-bzip2
0 	p5.34-compress-raw-zlib
0 	p5.34-config-autoconf
0 	p5.34-cpan-meta
0 	p5.34-cpan-meta-check
0 	p5.34-cpan-meta-requirements
0 	p5.34-cpan-meta-yaml
0 	p5.34-data-optlist
0 	p5.34-data-section
0 	p5.34-devel-checkbin
0 	p5.34-devel-symdump
0 	p5.34-digest-hmac
0 	p5.34-digest-sha1
0 	p5.34-encode
0 	p5.34-encode-locale
0 	p5.34-error
0 	p5.34-extutils-cbuilder
0 	p5.34-extutils-makemaker
0 	p5.34-extutils-manifest
0 	p5.34-file-slurper
0 	p5.34-getopt-long
0 	p5.34-gssapi
0 	p5.34-html-parser
0 	p5.34-html-tagset
0 	p5.34-http-date
0 	p5.34-http-message
0 	p5.34-inc-latest
0 	p5.34-io-compress
0 	p5.34-io-compress-brotli
0 	p5.34-io-html
0 	p5.34-io-socket-ip
0 	p5.34-io-socket-ssl
0 	p5.34-ipc-cmd
0 	p5.34-json
0 	p5.34-json-xs
0 	p5.34-locale-gettext
0 	p5.34-locale-maketext-simple
0 	p5.34-lwp-mediatypes
0 	p5.34-mixin-linewise
0 	p5.34-module-build
0 	p5.34-module-load
0 	p5.34-module-load-conditional
0 	p5.34-module-metadata
0 	p5.34-mozilla-ca
0 	p5.34-mro-compat
0 	p5.34-net-libidn
0 	p5.34-net-smtp-ssl
0 	p5.34-net-ssleay
0 	p5.34-params-check
0 	p5.34-params-util
0 	p5.34-perl-ostype
0 	p5.34-perlio-utf8_strict
0 	p5.34-pod-coverage
0 	p5.34-pod-coverage-trustpod
0 	p5.34-pod-escapes
0 	p5.34-pod-eventual
0 	p5.34-pod-parser
0 	p5.34-pod-simple
0 	p5.34-podlators
0 	p5.34-regexp-ipv6
0 	p5.34-scalar-list-utils
0 	p5.34-socket
0 	p5.34-software-license
0 	p5.34-sub-exporter
0 	p5.34-sub-install
0 	p5.34-sub-name
0 	p5.34-sub-uplevel
0 	p5.34-term-readkey
0 	p5.34-test-cpan-meta
0 	p5.34-test-cpan-meta-json
0 	p5.34-test-deep
0 	p5.34-test-exception
0 	p5.34-test-failwarnings
0 	p5.34-test-fatal
0 	p5.34-test-leaktrace
0 	p5.34-test-more-utf8
0 	p5.34-test-needs
0 	p5.34-test-nowarnings
0 	p5.34-test-pod
0 	p5.34-test-pod-coverage
0 	p5.34-test-simple
0 	p5.34-test-warn
0 	p5.34-test-warnings
0 	p5.34-text-parsewords
0 	p5.34-text-template
0 	p5.34-time-hires
0 	p5.34-time-local
0 	p5.34-timedate
0 	p5.34-try-tiny
0 	p5.34-types-serialiser
0 	p5.34-uri
0 	p5.34-version
0 	p5.34-xsloader
0 	pcre2
0 	perl5
0 	perl5.34
0 	pkgconfig
0 	popt
0 	py-bootstrap-modules
28 	py311-build
12 	py311-calver
0 	py311-cython-compat
11 	py311-editables
175 	py311-flit_core
202 	py311-hatchling
43 	py311-installer
49 	py311-packaging
29 	py311-pathspec
27 	py311-pluggy
970 	py311-pygments
19 	py311-pyproject_hooks
687 	py311-setuptools
91 	py311-setuptools_scm
12 	py311-trove-classifiers
7 	py311-typing_extensions
88 	py311-wheel
60 	py311-yaml
28 	py312-build
175 	py312-flit_core
43 	py312-installer
692 	py312-meson
49 	py312-packaging
19 	py312-pyproject_hooks
687 	py312-setuptools
88 	py312-wheel
0 	pygments_select
0 	python3_select
0 	python3_select-311
0 	python3_select-312
0 	python310
0 	python311
0 	python312
0 	python_select
0 	python_select-311
0 	python_select-312
0 	re2c
0 	readline
0 	rsync
0 	sqlite3
0 	svt-av1
0 	texinfo
0 	tiff
0 	urw-fonts
0 	webp
0 	x265
0 	xar
0 	xxhashlib
0 	xz
0 	yasm
0 	zlib
0 	zstd

[Dave-Allured (Dave Allured)]

Here is a fix for the python portgroup that will create the missing sym links. I prefer this over changing my prefix or config file. In short, it provides the missing path normalization at the right place. Thanks @jmroot for the tip about glob not finding the correct binaries, that was the exact problem.

PR submitted: ​https://github.com/macports/macports-ports/pull/22735

Edit 2024 Feb 27: This PR was cancelled, superseded by the following fixes to base, by @jmroot. Looks good.

[jmroot (Joshua Root)]

In b69f0cf9b8f8ce4b87cd565d2e7575ffb0ea414e/macports-base (master):

Resolve filesystem paths read from macports.conf

Closes: #69327

[Dave-Allured (Dave Allured)]

@jmroot et al, nice fix in that commit to Macports base. Thank you!

I have an imminent need for that fix, for testing something else that was not cured by my portgroup fix. What do you recommend? Copy the single file src/macports1.0/macports.tcl into my local Macports install? Copy only the single patch? Clone the full development snapshot? Something else? When is the next base release expected?

[jmroot (Joshua Root)]

If you don't mind being on the bleeding edge, you can install from master. Otherwise you can install from the release-2.9 branch, or apply just the desired patches directly to your installation. If you do the latter, make sure to get the followup commit [981ae2bed378c5cb513fcdb3fea788d3f61972e9/macports-base] as well.