Opened 3 years ago

Closed 3 years ago

#62023 closed defect (fixed)

python39 @3.9.1 does not build universal on BigSur

Reported by: kencu (Ken) Owned by: jmroot (Joshua Root)
Priority: Normal Milestone:
Component: ports Version:
Keywords: BigSur universal Cc:
Port: python39

Description

The lack of a universal python is blocking the installation of a number of universal ports on BigSur at present.

Building python39 as universal arm64/x86_64 fails on BigSur (both Intel and arm machines) with this error:

if test "x-extract i386" != "x" ; then \
		rm -f /opt/local/var/macports/build/_opt_macports-ports_lang_python39/python39/work/destroot/opt/local/Library/Frameworks/Python.framework/Versions/3.9/binpython3.9-32; \
		lipo -extract i386 \
			-output /opt/local/var/macports/build/_opt_macports-ports_lang_python39/python39/work/destroot/opt/local/Library/Frameworks/Python.framework/Versions/3.9/bin/python3.9-32 \
			/opt/local/var/macports/build/_opt_macports-ports_lang_python39/python39/work/destroot/opt/local/Library/Frameworks/Python.framework/Versions/3.9/bin/python3.9; \
	fi
fatal error: /Library/Developer/CommandLineTools/usr/bin/lipo: -extract i386 specified but fat file: /opt/local/var/macports/build/_opt_macports-ports_lang_python39/python39/work/destroot/opt/local/Library/Frameworks/Python.framework/Versions/3.9/bin/python3.9 does not contain that architecture

that can be overcome quite easily by overriding the default setting for LIPO_32BIT_FLAGS in the Makefile, and commenting it out (for a quick fix):

#LIPO_32BIT_FLAGS=-extract i386

and then python39 indeed finishes installing as +universal on BigSur.

However, there is an error in the installed files:

--->  Cleaning python39
--->  Removing work directory for python39
--->  Updating database of binaries
--->  Scanning binaries for linking errors
Could not open /opt/local/var/macports/build/_opt_macports-ports_lang_python39/python39/work/destroot/opt/local/Library/Frameworks/Python.framework/Versions/3.9/Python: Error opening or reading file (referenced from /opt/local/Library/Frameworks/Python.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python)
--->  Found 1 broken file, matching files to ports
--->  Found 1 broken port, determining rebuild order
You can always run 'port rev-upgrade' again to fix errors.
The following ports will be rebuilt: python39 @3.9.1+universal

and it looks like at least one of them has the wrong library reference embedded, for some reason:

$ otool -L /opt/local/Library/Frameworks/Python.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python
/opt/local/Library/Frameworks/Python.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python:
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1770.255.0)
	/opt/local/var/macports/build/_opt_macports-ports_lang_python39/python39/work/destroot/opt/local/Library/Frameworks/Python.framework/Versions/3.9/Python (compatibility version 3.9.0, current version 3.9.0)
	/opt/local/lib/libintl.8.dylib (compatibility version 10.0.0, current version 10.5.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.60.1)

not quite sure why that is wrong just now. I will try to fix it manually first, and then if that works, perhaps we can sort out why this is happening.

Attachments (3)

python39-changes-from-patch.txt (13.9 KB) - added by kencu (Ken) 3 years ago.
python39-fixed-changes-from-patch.diff (14.1 KB) - added by kencu (Ken) 3 years ago.
patch-portfile-fix-lipo-archs.diff (1.2 KB) - added by kencu (Ken) 3 years ago.

Download all attachments as: .zip

Change History (15)

comment:1 Changed 3 years ago by kencu (Ken)

This manual fix:

$ sudo install_name_tool -change /opt/local/var/macports/build/_opt_macports-ports_lang_python39/python39/work/destroot/opt/local/Library/Frameworks/Python.framework/Versions/3.9/Python /opt/local/Library/Frameworks/Python.framework/Versions/3.9/Python /opt/local/Library/Frameworks/Python.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python

repairs the port:

$ port -v installed | grep active
  bzip2 @1.0.8_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T09:41:38-0800'
  expat @2.2.10_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T09:41:39-0800'
  gettext @0.19.8.1_2+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T09:41:47-0800'
  libedit @20191231-3.1_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T09:41:51-0800'
  libffi @3.3_1+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T09:41:51-0800'
  libiconv @1.16_1+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T09:41:40-0800'
  ncurses @6.2_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T09:41:41-0800'
  openssl @1.1.1i_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T09:51:51-0800'
  pcre @8.44_1 (active) platform='darwin 20' archs='x86_64' date='2020-12-29T11:44:01-0800'
  pkgconfig @0.29.2_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T09:52:11-0800'
  python3_select @0.0_2 (active) platform='darwin 20' archs='noarch' date='2020-12-29T11:43:37-0800'
  python39 @3.9.1_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T10:07:17-0800'
  python_select @0.3_9 (active) platform='darwin 20' archs='noarch' date='2020-12-29T11:43:37-0800'
  sqlite3 @3.34.0_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T09:52:16-0800'
  the_silver_searcher @2.2.0_0 (active) platform='darwin 20' archs='x86_64' date='2021-01-10T09:58:02-0800'
  xz @5.2.5_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T09:52:20-0800'
  zlib @1.2.11_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T09:41:52-0800'

$ sudo port -v rev-upgrade
--->  Scanning binaries for linking errors
--->  No broken files found.
--->  No broken ports found.

comment:2 Changed 3 years ago by kencu (Ken)

Here are the changes induced into python39 by the patch phase and post-patch reinplaces:

generated by this:

   35  sudo port clean
   36  sudo port -v extract subport=python39 +universal
   37  cd work
   38  sudo chmod -R a+rw ./*
   39  cd P*
   40  git init
   41  git add .
   42  git commit -m init > /dev/null
   43  cd ../..
   44  sudo port -v patch subport=python39 +universal
   45  cd work
   46  cd P*
   47  git status
   48  git diff
   49  git diff --no-prefix > ~/python39-changes-from-patch.txt

I'll attach the whole diff, but it looks like this part in configure is what needs fixing:

             intel)
-               UNIVERSAL_ARCH_FLAGS="-arch i386 -arch x86_64"
+               UNIVERSAL_ARCH_FLAGS="-arch arm64 -arch x86_64"
                LIPO_32BIT_FLAGS="-extract i386"
                ARCH_RUN_32BIT="/usr/bin/arch -i386"
                ;;

Changed 3 years ago by kencu (Ken)

comment:3 Changed 3 years ago by kencu (Ken)

(sidebar) during work on this, I see this issue:

/usr/bin/clang -bundle -undefined dynamic_lookup -arch arm64 -arch x86_64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX11.1.sdk -L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX11.1.sdk -arch arm64 -arch x86_64 build/temp.macosx-11.1-x86_64-3.9/opt/local/var/macports/build/_opt_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/_ctypes.o build/temp.macosx-11.1-x86_64-3.9/opt/local/var/macports/build/_opt_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/callbacks.o build/temp.macosx-11.1-x86_64-3.9/opt/local/var/macports/build/_opt_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/callproc.o build/temp.macosx-11.1-x86_64-3.9/opt/local/var/macports/build/_opt_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/cfield.o build/temp.macosx-11.1-x86_64-3.9/opt/local/var/macports/build/_opt_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/malloc_closure.o build/temp.macosx-11.1-x86_64-3.9/opt/local/var/macports/build/_opt_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/stgdict.o -L/opt/local/lib -lffi -ldl -o build/lib.macosx-11.1-x86_64-3.9/_ctypes.cpython-39-darwin.so
ld: warning: ignoring file build/temp.macosx-11.1-x86_64-3.9/opt/local/var/macports/build/_opt_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/callbacks.o, building for macOS-arm64 but attempting to link with file built for unknown-x86_64
ld: warning: ignoring file build/temp.macosx-11.1-x86_64-3.9/opt/local/var/macports/build/_opt_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/_ctypes.o, building for macOS-arm64 but attempting to link with file built for unknown-x86_64
ld: warning: ignoring file build/temp.macosx-11.1-x86_64-3.9/opt/local/var/macports/build/_opt_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/callproc.o, building for macOS-arm64 but attempting to link with file built for unknown-x86_64
ld: warning: ignoring file build/temp.macosx-11.1-x86_64-3.9/opt/local/var/macports/build/_opt_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/cfield.o, building for macOS-arm64 but attempting to link with file built for unknown-x86_64
ld: warning: ignoring file build/temp.macosx-11.1-x86_64-3.9/opt/local/var/macports/build/_opt_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/malloc_closure.o, building for macOS-arm64 but attempting to link with file built for unknown-x86_64
ld: warning: ignoring file build/temp.macosx-11.1-x86_64-3.9/opt/local/var/macports/build/_opt_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/stgdict.o, building for macOS-arm64 but attempting to link with file built for unknown-x86_64
Version 0, edited 3 years ago by kencu (Ken) (next)

comment:4 Changed 3 years ago by kencu (Ken)

This patch to the portfile allows a universal arm/x86_64 build to finish. I tried to handle the 32bit situation, but I have not as yet tested it:

% diff -u `port file python39` Portfile
--- /opt/local/var/macports/sources/rsync.macports.org/macports/release/tarballs/ports/lang/python39/Portfile	2020-12-08 03:30:58.000000000 -0800
+++ Portfile	2021-01-10 11:52:59.000000000 -0800
@@ -174,12 +174,20 @@
 
 variant universal {
     post-patch {
+
+        # first null out any existing lipo extract archs
+        reinplace \
+          "s|LIPO_32BIT_FLAGS=\".*\"|LIPO_32BIT_FLAGS=\"\"|" \
+          ${worksrcpath}/configure
+
         set universal_arch_flags {}
         set arch_run_32bit {}
+        set lipo_32bit_flags {}
         foreach arch ${universal_archs} {
             lappend universal_arch_flags -arch ${arch}
             if {${arch} eq "i386" || ${arch} eq "ppc"} {
                 lappend arch_run_32bit -${arch}
+                lappend lipo_32bit_flags -extract ${arch}
             }
         }
         reinplace \
@@ -189,6 +197,9 @@
             reinplace \
               "s|ARCH_RUN_32BIT=\".*\"|ARCH_RUN_32BIT=\"arch ${arch_run_32bit}\"|" \
               ${worksrcpath}/configure
+            reinplace \
+              "s|LIPO_32BIT_FLAGS=\".*\"|LIPO_32BIT_FLAGS=\"${lipo_32bit_flags}\"|" \
+              ${worksrcpath}/configure
         }
     }
     if {${configure.sdkroot} ne ""} {

I will attach the diff for the pre-patch and post-patch source tree.

Changed 3 years ago by kencu (Ken)

Changed 3 years ago by kencu (Ken)

comment:5 Changed 3 years ago by kencu (Ken)

And for some reason, I no longer see that issue that requires install_name_tool to fix, so that Portfile patch appears to be all that is needed. I will have to test the universal build on older systems to see if it actually works properly, however.

And I'm not sure what this means <https://trac.macports.org/ticket/62023#comment:3> if anything.

comment:6 Changed 3 years ago by kencu (Ken)

That same simple patch unfortunately is not enough to fix python38, which is still getting backport fixes for arm64 upstream I understand:

DYLD_FRAMEWORK_PATH=/opt/local/var/macports/build/_opt_myports_lang_python38/python38/work/Python-3.8.7 ./python.exe -E -S -m sysconfig --generate-posix-vars ;\
	if test $? -ne 0 ; then \
		echo "generate-posix-vars failed" ; \
		rm -f ./pybuilddir.txt ; \
		exit 1 ; \
	fi
Traceback (most recent call last):
  File "/opt/local/var/macports/build/_opt_myports_lang_python38/python38/work/Python-3.8.7/Lib/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/local/var/macports/build/_opt_myports_lang_python38/python38/work/Python-3.8.7/Lib/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/local/var/macports/build/_opt_myports_lang_python38/python38/work/Python-3.8.7/Lib/sysconfig.py", line 712, in <module>
    _main()
  File "/opt/local/var/macports/build/_opt_myports_lang_python38/python38/work/Python-3.8.7/Lib/sysconfig.py", line 700, in _main
    _generate_posix_vars()
  File "/opt/local/var/macports/build/_opt_myports_lang_python38/python38/work/Python-3.8.7/Lib/sysconfig.py", line 402, in _generate_posix_vars
    pybuilddir = 'build/lib.%s-%s' % (get_platform(), _PY_VERSION_SHORT)
  File "/opt/local/var/macports/build/_opt_myports_lang_python38/python38/work/Python-3.8.7/Lib/sysconfig.py", line 679, in get_platform
    osname, release, machine = _osx_support.get_platform_osx(
  File "/opt/local/var/macports/build/_opt_myports_lang_python38/python38/work/Python-3.8.7/Lib/_osx_support.py", line 495, in get_platform_osx
    raise ValueError(
ValueError: Don't know machine value for archs=('arm64', 'x86_64')
generate-posix-vars failed
make: *** [pybuilddir.txt] Error 1
make: Leaving directory `/opt/local/var/macports/build/_opt_myports_lang_python38/python38/work/Python-3.8.7'
Command failed:  cd "/opt/local/var/macports/build/_opt_myports_lang_python38/python38/work/Python-3.8.7" && /usr/bin/make -j8 -w all 

comment:7 Changed 3 years ago by kencu (Ken)

There is perhaps another approach to be explored, here:

It is possible to build a number of flavours of the universal binary build,
the default is a 32-bit only binary (i386 and ppc). The flavour can be
specified using the option ``--with-universal-archs=VALUE``. The following
values are available:

  * ``32-bit``:   ``ppc``, ``i386``

  * ``64-bit``:   ``ppc64``, ``x86_64``

  * ``all``:      ``ppc``, ``ppc64``, ``i386``, ``x86_64``

  * ``3-way``:	  ``ppc``, ``i386`` and ``x86_64``

  * ``intel``:	  ``i386``, ``x86_64``

To build a universal binary that includes a 64-bit architecture, you must build
on a system running OSX 10.5 or later. The ``all`` flavour can only be built on
OSX 10.5.

The makefile for a framework build will install ``python32`` and ``pythonw32`` 
binaries when the universal architecures includes at least one 32-bit architecture
(that is, for all flavours but ``64-bit``).

comment:8 Changed 3 years ago by kencu (Ken)

building it universal on 10.6 i386/x86_64 seems to be fine. Patching:

Executing:  cd "/opt/local/var/macports/build/_opt_myports_lang_python39/python39/work/Python-3.9.1" && /usr/bin/patch -p0 < '/opt/myports/lang/python39/files/implicit.patch'
patching file configure
--->  Patching cgi.py: s|@@PREFIX@@|/opt/local|g
--->  Patching dyld.py: s|@@PREFIX@@|/opt/local|g
--->  Patching Makefile.pre.in: s|/setup.py|/setup.py --no-user-cfg|
--->  Patching configure: s|@@APPLICATIONS_DIR@@|/Applications/MacPorts|
--->  Patching configure: s|LIPO_32BIT_FLAGS=".*"|LIPO_32BIT_FLAGS=""|
--->  Patching configure: s|UNIVERSAL_ARCH_FLAGS=".*"|UNIVERSAL_ARCH_FLAGS="-arch x86_64 -arch i386"|
--->  Patching configure: s|ARCH_RUN_32BIT=".*"|ARCH_RUN_32BIT="arch -i386"|
--->  Patching configure: s|LIPO_32BIT_FLAGS=".*"|LIPO_32BIT_FLAGS="-extract i386"|
--->  Configuring python39

Archs and running:

 file "/Applications/MacPorts/Python 3.9/IDLE.app/Contents/MacOS/Python"
/Applications/MacPorts/Python 3.9/IDLE.app/Contents/MacOS/Python: Mach-O universal binary with 2 architectures
/Applications/MacPorts/Python 3.9/IDLE.app/Contents/MacOS/Python (for architecture x86_64):	Mach-O 64-bit executable x86_64
/Applications/MacPorts/Python 3.9/IDLE.app/Contents/MacOS/Python (for architecture i386):	Mach-O executable i386
macbookpro21:python39 cunningh$ python3.9
Python 3.9.1 (default, Jan 10 2021, 18:21:38) 
[Clang 9.0.1 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> ^D

So it appears that this Lipo_32bit_flags business works there.

comment:9 Changed 3 years ago by kencu (Ken)

JMR - would you like a PR, or would you rather not fix / fix differently?

comment:10 Changed 3 years ago by jmroot (Joshua Root)

TBH, I lost track of what you were actually proposing a few comments ago, so I just went with [4cfc3d6e2fa4d507771d17c8c347dfe6734b191a/macports-ports]. I don't have easy access to a system that can do universal builds, so please test.

comment:11 Changed 3 years ago by kencu (Ken)

This was what I proposed <https://trac.macports.org/attachment/ticket/62023/patch-portfile-fix-lipo-archs.diff> but I'll try out what you did. Thanks.

comment:12 Changed 3 years ago by kencu (Ken)

Resolution: fixed
Status: assignedclosed
Note: See TracTickets for help on using tickets.