py-scipy/py-numpy: opportunistic linking to py-mkl breaks both of these ports

[essandess (Steve Smith)]

$ python3 -c 'from scipy import special'

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/scipy/special/__init__.py", line 633, in <module>
    from . import _ufuncs
ImportError: dlopen(/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/scipy/special/_ufuncs.cpython-39-darwin.so, 2): Library not loaded: @rpath/libmkl_rt.dylib
  Referenced from: /opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/scipy/special/_ufuncs.cpython-39-darwin.so
  Reason: image not found

$ find /opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/scipy -type f -name '*.so' -exec fgrep -l '@rpath' {} ';' | xargs otool -L | fgrep '@rpath'
	@rpath/libmkl_rt.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libmkl_rt.dylib (compatibility version 0.0.0, current version 0.0.0)
	…
	@rpath/libmkl_rt.dylib (compatibility version 0.0.0, current version 0.0.0)

Is there a Portfile one-liner to fix this, or is it necessary to search the build exhaustively for all the .so's and install_name_tool-correct them?

[essandess (Steve Smith)]

I tried that attached patch, to no avail.

I just did this brute-force. Please see ​https://github.com/macports/macports-ports/pull/10453

[mf2k (Frank Schima)]

I cannot reproduce this issue. Maybe a simple revision increase is needed somewhere?

[kencu (Ken)]

I don't see this either (BigSur):

% find /opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/scipy -type f -name '*.so' | xargs otool -L | grep '@rpath'       

[essandess (Steve Smith)]

My build environment:

macOS 10.15.7 19H524
Xcode 12.4 12D4e

printenv SDKROOT
<not set>

xcode-select -p
/Library/Developer/CommandLineTools

xcrun --show-sdk-path
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk

xcrun --show-sdk-version
10.15.6

I’ll try with sudo xcode-select -r.

[essandess (Steve Smith)]

I see the same issue with sudo xcode-select -r. Log file attached for sudo port destroot py39-scipy.

find "$(port work py39-scipy)/scipy-1.6.2/build" -type f -name '*.so' -exec fgrep -l '@rpath' {} ';'

/opt/local/var/macports/build/_opt_local_ports_python_py-scipy/py39-scipy/work/scipy-1.6.2/build/lib.macosx-10.15-x86_64-3.9/scipy/odr/__odrpack.cpython-39-darwin.so
…
/opt/local/var/macports/build/_opt_local_ports_python_py-scipy/py39-scipy/work/scipy-1.6.2/build/lib.macosx-10.15-x86_64-3.9/scipy/spatial/qhull.cpython-39-darwin.so

[cjones051073 (Chris Jones)]

Just to add, I do see this on macOS11

Oberon ~/Projects/MacPorts/ports > sudo port select python3 python38
Selecting 'python38' for 'python3' succeeded. 'python38' is now active.
Oberon ~/Projects/MacPorts/ports > python3 -c 'from scipy import special'
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/__init__.py", line 22, in <module>
    from . import multiarray
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/multiarray.py", line 12, in <module>
    from . import overrides
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/overrides.py", line 7, in <module>
    from numpy.core._multiarray_umath import (
ImportError: dlopen(/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-darwin.so, 2): Library not loaded: @rpath/libmkl_rt.dylib
  Referenced from: /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-darwin.so
  Reason: image not found

Oberon ~/Projects/MacPorts/ports > xcode-select -p
/Applications/Xcode.app/Contents/Developer
Oberon ~/Projects/MacPorts/ports > xcrun --show-sdk-path
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk
Oberon ~/Projects/MacPorts/ports > xcrun --show-sdk-version
11.1

[Chris Jones <jonesc@…>]

In 3079b51048e18e185d8e7135a1a67b9d6cf919d8/macports-ports (master):

py-numpy: rev-bump to force rebuild
Closes: #62578

[cjones051073 (Chris Jones)]

Underlying issues appears to have been with py-numpy. Rev-bumping to force a rebuild of both numpy and scipy seems to have fixed things.

[mascguy (Christopher Nielsen)]

I seemingly encounter this issue with py-numpy, each time grpc is updated: The latter causes breakage to py-pyarrow, which MacPorts tries to rebuild. But during the configure phase, py-pyarrow attempts to import py-numpy - which fails due to rpath issue - subsequently causing the build for py-pyarrow to fail.

So, I'm just curious: Given that py-numpy is clearly linked to py-mkl, shouldn't the latter be a formal lib dependency of py-numpy? Or is there some Python witchcraft I'm missing - either related to Cython, or whatever - such that we wouldn't want said dependency?

[mascguy (Christopher Nielsen)]

Reopening, as it's clear that this is a recurring issue. And whatever solutions we've used thus far - whether rev-bumping or whatever - aren't cutting the mustard!

[mascguy (Christopher Nielsen)]

SO... just uninstalled py39-numpy, and forced a source install. And yet, the rpath dylib issue is still there, causing f2py3.9-3.9, etc, to fail.

What am I missing with all of these interdependencies...?

[mascguy (Christopher Nielsen)]

p.s. Do we need to muck with the py-numpy portfile, and forcibly fix this via install_name_tool in the post-destroot phase? This is nuts...

[mascguy (Christopher Nielsen)]

FYI, this is what I just had to do, to fix my installation of py39-numpy:

sudo bash -c 'port contents py39-numpy | ggrep -E "darwin.so" | gxargs -r --max-args=1 install_name_tool -add_rpath "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib"'

So the $100,000 question is, why is this necessary in the first place, particularly when I just rebuilt py39-numpy from source...?

[mascguy (Christopher Nielsen)]

Okay, so another piece to the puzzle: When py-mkl isn't installed, everything works properly.

So this looks like a bug: py-numpy is opportunistically linking with py-mkl, when it's installed.

[mascguy (Christopher Nielsen)]

Of note, the opportunistic linking against py-mkl is occurring for at least two ports, and perhaps others: py-numpy and py-scipy.

Thankfully our very own Michael Dickens maintains those two, so reassigning to him to fix.

p.s. Donations kindly appreciated. Maybe some gourmet chocolate chip cookies? ;-)

[mascguy (Christopher Nielsen)]

Building py39-numpy via trace mode - notably, when py39-mkl was also active - yielded the following warnings. No surprise at all, but just to validate the prior conclusion of opportunistic linking:

Warning: The following existing files were hidden from the build system by trace mode:
  /opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/libmkl_rt.dylib
  /opt/local/bin/clang-mp-10
  /opt/local/bin/clang-mp-9.0
  /opt/local/bin/svnversion
  /opt/local/lib/libunwind.dylib
Warning: The following file inside the MacPorts prefix not installed by a port was accessed:
  /opt/local/lib/lib
--->  Staging py39-numpy into destroot
Warning: The following existing files were hidden from the build system by trace mode:
  /opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/libmkl_rt.dylib
  /opt/local/bin/clang-mp-10
  /opt/local/bin/clang-mp-9.0
  /opt/local/bin/svnversion
  /opt/local/lib/libunwind.dylib
  /var/root/.CFUserTextEncoding
Warning: The following file inside the MacPorts prefix not installed by a port was accessed:
  /opt/local/lib/lib

Full log attached; filename: py39-numpy-trace-install-10.13.log.xz

[mascguy (Christopher Nielsen)]

Renee, can you offer any insight, as to what the proper fix is for this? My knowledge of the Python space is limited, but just curious: Is there an accepted approach to avoid this, via our Python port framework? Or is this more project-specific, necessitating appropriate configure/build arguments to disable use of py-mkl? (Or perhaps tweaks to site.cfg?)

Of course, all of this assumes that py-numpy doesn't need/want py-mkl in the first place. If that's not the case, then we'll need to rethink the approach. (And perhaps consider fixing the binaries via install_name_tool, if necessary.)

Thoughts?

[reneeotten (Renee Otten)]

I don't see this issue myself; nothing links to py-mkl for scipy/numpy. If needed, it ​seems one can supply NPY_LAPACK_ORDER='!MKL' as an environment variable to avoid using MKL. So that could be added to a variant and/or switched off by default. I'll leave it to Michael to decide what to do with it.

[mascguy (Christopher Nielsen)]

Replying to reneeotten:

I don't see this issue myself; nothing links to py-mkl for scipy/numpy. If needed, it ​seems one can supply NPY_LAPACK_ORDER='!MKL' as an environment variable to avoid using MKL. So that could be added to a variant and/or switched off by default. I'll leave it to Michael to decide what to do with it.

When I tried building with that env var set, it failed with the following:

ValueError: Environment variable 'NPY_LAPACK_ORDER' may not mix negated an non-negated items: '!mkl'

As for the linking to py-mkl, this is what I see in the NumPy build log, when the former is active. (And without using trace mode.)

:info:build Processing numpy/random/mtrand.pyx
:info:build Cythonizing sources
:info:build blas_opt_info:
:info:build blas_mkl_info:
:info:build customize UnixCCompiler
:info:build   FOUND:
:info:build     libraries = ['mkl_rt', 'pthread']
:info:build     library_dirs = ['/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib']
:info:build     define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
:info:build     include_dirs = ['/usr/local/include', '/opt/local/include', '/opt/local/Library/Frameworks/Python.framework/Versions/3.9/include']
:info:build   FOUND:
:info:build     libraries = ['mkl_rt', 'pthread']
:info:build     library_dirs = ['/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib']
:info:build     define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
:info:build     include_dirs = ['/usr/local/include', '/opt/local/include', '/opt/local/Library/Frameworks/Python.framework/Versions/3.9/include']
:info:build non-existing path in 'numpy/distutils': 'site.cfg'
:info:build lapack_opt_info:
:info:build lapack_mkl_info:
:info:build   FOUND:
:info:build     libraries = ['mkl_rt', 'pthread']
:info:build     library_dirs = ['/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib']
:info:build     define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
:info:build     include_dirs = ['/usr/local/include', '/opt/local/include', '/opt/local/Library/Frameworks/Python.framework/Versions/3.9/include']
:info:build   FOUND:
:info:build     libraries = ['mkl_rt', 'pthread']
:info:build     library_dirs = ['/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib']
:info:build     define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
:info:build     include_dirs = ['/usr/local/include', '/opt/local/include', '/opt/local/Library/Frameworks/Python.framework/Versions/3.9/include']
:info:build Warning: attempted relative import with no known parent package
:info:build /opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py:274: UserWarning: Unknown distribution option: 'define_macros'
:info:build   warnings.warn(msg)
:info:build running build

[mascguy (Christopher Nielsen)]

@michaeldd/@cjones, while this is easy enough to work around - either by deactivating py-mkl beforehand, or building py-numpy/py-scipy in trace mode - neither is a great long-term solution for our users.

Are you folks able to provide a formal fix for this?

[reneeotten (Renee Otten)]

Replying to mascguy:

@michaeldd/@cjones, while this is easy enough to work around - either by deactivating py-mkl beforehand, or building py-numpy/py-scipy in trace mode - neither is a great long-term solution for our users.

Are you folks able to provide a formal fix for this?

I started to look into this and the environment variable I suggested in an earlier comment (or some variant thereof) should work; I'll try to come up with a solution to exclude MKL unless specifically requested. Having said that, this is a rather old issue with very few comments so I guess not many people run into this... so I don't think there is this much of a time rush as you're making it sound ;) It will. not be an issue for most probably because they don't build from source and the builds from the buildbot will not link to MKL as it isn't installed there.

[mascguy (Christopher Nielsen)]

Replying to reneeotten:

I started to look into this and the environment variable I suggested in an earlier comment (or some variant thereof) should work; I'll try to come up with a solution to exclude MKL unless specifically requested. Having said that, this is a rather old issue with very few comments so I guess not many people run into this... so I don't think there is this much of a time rush as you're making it sound ;) It will. not be an issue for most probably because they don't build from source and the builds from the buildbot will not link to MKL as it isn't installed there.

No worries Renee, this definitely isn't an emergency by any means. Particularly now that we have two easy workarounds.

Just want to make sure we don't forget about it. Nagging works! ;-)

And thanks for your help, as always. Greatly appreciated!

[reneeotten (Renee Otten)]

Christopher, I cannot get py-numpy linked with py-mkl even if I try ;) I just install py39-mkl followed by py39-mkl and see nothing:

% otool -L `port contents py39-numpy | grep "\.so"` | grep mkl

can you please confirm that you *only* see this because you mucked around with install_name to remove the rpath business in py39-mkl? I do agree that this should be fixed (and I was always assuming that I actually used the MKL libraries.....} and I'm on track of trying to do so, but would like to understand why you're having this problem in the first place.

[mascguy (Christopher Nielsen)]

Just attached logs covering the opportunistic linking case; filename: py39-numpy-non-trace-install-10.13.log.xz​

[mascguy (Christopher Nielsen)]

Replying to reneeotten:

Christopher, I cannot get py-numpy linked with py-mkl even if I try ;) I just install py39-mkl followed by py39-mkl and see nothing:

% otool -L `port contents py39-numpy | grep "\.so"` | grep mkl

can you please confirm that you *only* see this because you mucked around with install_name to remove the rpath business in py39-mkl? I do agree that this should be fixed (and I was always assuming that I actually used the MKL libraries.....} and I'm on track of trying to do so, but would like to understand why you're having this problem in the first place.

Note that I didn't touch the py39-mkl components; I only modified those for py39-numpy. Also, the log I just attached, relates to a subsequent clean py39-numpy source installation. So I'm reasonably confident that I wiped any remnants of my install_name_tool adventures.

Does that make sense?

[reneeotten (Renee Otten)]

one more thing to consider... your log shows you're using macOS 10.13.6 which presumably means that you have py-mkl version 2019.0, correct? If so, that's likely the difference as I am on macOS 10.15.7 and therefore get 2021.2.0.

[mascguy (Christopher Nielsen)]

Replying to reneeotten:

one more thing to consider... your log shows you're using macOS 10.13.6 which presumably means that you have py-mkl version 2019.0, correct? If so, that's likely the difference as I am on macOS 10.15.7 and therefore get 2021.2.0.

I'm fairly certain I experienced the same issue on 10.15 as well. I'll try to find time to retest a source build there too, with py-mkl installed, later today.

[reneeotten (Renee Otten)]

Replying to mascguy:

Replying to reneeotten:

one more thing to consider... your log shows you're using macOS 10.13.6 which presumably means that you have py-mkl version 2019.0, correct? If so, that's likely the difference as I am on macOS 10.15.7 and therefore get 2021.2.0.

I'm fairly certain I experienced the same issue on 10.15 as well. I'll try to find time to retest a source build there too, with py-mkl installed, later today.

Great! Can you also please double-check then that you haven't installed the MKL libraries any other way and that is the reason that in numpy/distutils/system_info.py it decides that the libraries are available (and then still links to the one in the MacPorts prefix?

Also, please make sure to install py39-numpy @1.21.3_1, your log shows revision 2 which is not present in the main tree. You're also using the development version of MacPorts, which I don't do... but I doubt that will make a difference here.

[mascguy (Christopher Nielsen)]

Replying to reneeotten:

Great! Can you also please double-check then that you haven't installed the MKL libraries any other way and that is the reason that in numpy/distutils/system_info.py it decides that the libraries are available (and then still links to the one in the MacPorts prefix?

To run a fairer apples-to-apples test, be sure to install py39-pytorch first. That's why py39-mkl is installed on my systems, and I'm guessing it's the same situation for Steve. (Since he also does quite a bit of work relative to machine learning and such.)

[reneeotten (Renee Otten)]

Replying to mascguy:

Replying to reneeotten:

Great! Can you also please double-check then that you haven't installed the MKL libraries any other way and that is the reason that in numpy/distutils/system_info.py it decides that the libraries are available (and then still links to the one in the MacPorts prefix?

To run a fairer apples-to-apples test, be sure to install py39-pytorch first. That's why py39-mkl is installed on my systems, and I'm guessing it's the same situation for Steve. (Since he also does quite a bit of work relative to machine learning and such.)

That shouldn't matter; I installed py39-mkl and py39-mkl-include these are the only ports that should be relevant here. Sorry, I don't really feel like building all the other ports for which I have no interest.

[mascguy (Christopher Nielsen)]

Replying to reneeotten:

That shouldn't matter; I installed py39-mkl and py39-mkl-include these are the only ports that should be relevant here. Sorry, I don't really feel like building all the other ports for which I have no interest.

No worries, I was thinking that py39-pytorch (and most dependencies) would install reasonably quickly, via the buildbot binaries. But you're probably right, hopefully it shouldn't matter.

[mascguy (Christopher Nielsen)]

Just ran a quick test on my 10.15.7 VM, and couldn't reproduce the issue. I'll retry on my physical 10.15.6 install, when I can find time to reboot, as that's where I hit it last time.

[mascguy (Christopher Nielsen)]

It looks like Steve hit this on 10.15 though, per his original attached log (main.log.xz). So that would also be worth reviewing, if you haven't yet.

[mascguy (Christopher Nielsen)]

Sadly, I just tried to reproduce the issue on my physical macOS 10.15.6 installation, and now I can't.

Haven't given up yet, but need to step back from this for at least a day or two. Thoughts/suggestions welcome, though!

[reneeotten (Renee Otten)]

Replying to mascguy:

Sadly, I just tried to reproduce the issue on my physical macOS 10.15.6 installation, and now I can't.

Haven't given up yet, but need to step back from this for at least a day or two. Thoughts/suggestions welcome, though!

For some reason the current NumPy version doesn't find MKL even if it’s installed. I’ll investigate that and try to fix it and then put it in a variant. Additionally, when using “openblas” or “atlas” we can make sure it will not look for MKL, that part is easy. So don’t waste time on trying to reproduce it; We should fix this and I know what to do, just not yet how 😉

[reneeotten (Renee Otten)]

okay, so I've update the py-numpy Portfile and added a mkl variant, making sure the MKL library isn't found when this variant is not selected - this will avoid the opportunistic linking.

So, the library name is libmkl_rt.1.dylib so I do a reinplace to change the library name from mkl_rt to mkl_rt.1 in numpy/distutils/system_info.py. I *think* that works as intended as the MKL library is found and linked to when installing py-numpy with the mkl variant.

However when importing the package it fails with:

Original error was: dlopen(/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/numpy/core/_multiarray_umath.cpython-310-darwin.so, 2): Library not loaded: @rpath/libmkl_rt.1.dylib
  Referenced from: /opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/numpy/core/_multiarray_umath.cpython-310-darwin.so
  Reason: image not found

and looking with otool gives:

% otool -L /opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/libmkl_rt.1.dylib
/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/libmkl_rt.1.dylib:
	@rpath/libmkl_rt.1.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)

% otool -l /opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/libmkl_rt.1.dylib | grep path
         name @rpath/libmkl_rt.1.dylib (offset 24)
         path /opt/local/lib (offset 12)
         path /opt/local/lib/libomp (offset 12)

Now there is some install_name_tool -add_rpath stuff happening in the py-mkl port, but I cannot figure out whether that is the issue or not. I've tried a variety of (random) things with install_name_tool, but cannot get it to work. Can anyone educate me here on what the problem is and whether this should be fixed in the py-mkl port or if it's a NumPy issue?

[reneeotten (Renee Otten)]

In 00fc836eb9ee2ced138aaa068983e2386bbb1402/macports-ports (master):

py-numpy: use specified BLAS/LAPACK libraries

• add mkl variant; patch MKL library name; use install_name_tool to remove @path references to MKL library

Closes: #62578