scotch @7.0.2: can no longer be dlopen'ed

[FreddieWitherden (Freddie Witherden)]

Historically, the scotch port has carried a patch (in line with Ubuntu and several other Linux distributions) to enable it to be dlopen'ed. However, this appears to no longer be present whence:

$ python3
Python 3.9.6 (default, Oct 18 2022, 12:41:40)
[Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes import *
>>> CDLL('/opt/local/lib/libscotch.dylib')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/ctypes/__init__.py", line 366, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(/opt/local/lib/libscotch.dylib, 0x0006): symbol not found in flat namespace '_SCOTCH_errorPrint'

[ryandesign (Ryan Carsten Schmidt)]

According to ​https://gitlab.inria.fr/scotch/scotch/-/issues/16#note_794947 this was fixed in 7.0.3; the port could be updated to that version.

[catap (Kirill A. Korinsky)]

Well.. Scotch is a bit different on macOS :)

Long story short: it is split into the library (libscotch.dylib) and two possible error handlers: libscotcherr.dylib which prints errors, and libscotcherrexit.dylib which exits on error. And scotch needs function _SCOTCH_errorPrint which is exported from an error handler.

Unfortunately python's CDLL can't load two library into one namespace. But you make a wrapper. A dummy .dylib which links against libscotch.dylib and desired error handler.

For example:

√ /tmp % clang -shared -L/opt/local/lib -lscotch -lscotcherr -o wscotch.dylib    
√ /tmp % python3
Python 3.9.16 (main, Dec  7 2022, 02:41:07) 
[Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes import *
>>> CDLL('/tmp/wscotch.dylib')
<CDLL '/tmp/wscotch.dylib', handle 213934560 at 0x10a0837f0>
>>> 

[FreddieWitherden (Freddie Witherden)]

So the above code fragment *used* to work on macOS as I believe previous port files applied the same fix as most Linux distributions to enable the library to be dlopen'ed. Was anything explicitly broken by the old approach that resulted in its removal?

[catap (Kirill A. Korinsky)]

Replying to FreddieWitherden:

So the above code fragment *used* to work on macOS as I believe previous port files applied the same fix as most Linux distributions to enable the library to be dlopen'ed. Was anything explicitly broken by the old approach that resulted in its removal?

The new version of scotch had reworked the way how it is build and special reworked a build of shared libraries :(

Before MacPorts builds the scotch with one of error handler, now it is used an official way to build it. If you would like I may include dylib like I've shown in example to the port.

[FreddieWitherden (Freddie Witherden)]

Is there any reason not to make the dylib you proposed the default libscotch.dylib? This would be consistent with what several Linux distributions do (including Ubuntu) and restore cross-platform compatibility (since all one would need to do for macOS is append .dylib rather than .so which is what has worked historically).

[catap (Kirill A. Korinsky)]

Replying to FreddieWitherden:

Is there any reason not to make the dylib you proposed the default libscotch.dylib? This would be consistent with what several Linux distributions do (including Ubuntu) and restore cross-platform compatibility (since all one would need to do for macOS is append .dylib rather than .so which is what has worked historically).

For some reason, upstream authors decided to use dynamic linking for users: ​https://gitlab.inria.fr/scotch/scotch/-/issues/26

I've opened an upstream issue about this case: ​https://gitlab.inria.fr/scotch/scotch/-/issues/27

For the time being, I suggest using a wrapper that covers an issue.

[catap (Kirill A. Korinsky)]

Freddie,

I have some time to think about this issue, and you may use following example as the right way to load scotch into python:

Python 3.9.16 (main, Dec  7 2022, 02:41:07) 
[Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
>>> ctypes.CDLL('/opt/local/lib/libscotcherr.dylib', mode=ctypes.RTLD_GLOBAL)
<CDLL '/opt/local/lib/libscotcherr.dylib', handle 219368560 at 0x10cd9cf70>
>>> ctypes.CDLL('/opt/local/lib/libscotch.dylib')
<CDLL '/opt/local/lib/libscotch.dylib', handle 21936a700 at 0x10cd84f70>
>>>