python27: importing ctypes fails on Monterey / Arm64

[jonwilts]

Upgraded my M1 Mini to Monterey, and following the Migration instructions for macports. No idea if this is Arm64 only.

During Migration, py27-setuptools fails to build:

:info:build Executing:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_python_py-setuptools/py27-setuptools/work/setuptools-44.1.1" && /opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 setup.py --no-user-cfg build
:debug:build system:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_python_py-setuptools/py27-setuptools/work/setuptools-44.1.1" && /opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 setup.py --no-user-cfg build
:info:build Traceback (most recent call last):
:info:build   File "setup.py", line 9, in <module>
:info:build     import setuptools
:info:build   File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_python_py-setuptools/py27-setuptools/work/setuptools-44.1.1/setuptools/__init__.py", line 20, in <module>
:info:build     from setuptools.dist import Distribution, Feature
:info:build   File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_python_py-setuptools/py27-setuptools/work/setuptools-44.1.1/setuptools/dist.py", line 36, in <module>
:info:build     from setuptools import windows_support
:info:build   File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_python_py-setuptools/py27-setuptools/work/setuptools-44.1.1/setuptools/windows_support.py", line 2, in <module>
:info:build     import ctypes
:info:build   File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ctypes/__init__.py", line 7, in <module>
:info:build     from _ctypes import Union, Structure, Array
:info:build ImportError: No module named _ctypes
:info:build Command failed:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_python_py-setuptools/py27-setuptools/work/setuptools-44.1.1" && /opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 setup.py --no-user-cfg build
:info:build Exit code: 1
:error:build Failed to build py27-setuptools: command execution failed
:debug:build Error code: CHILDSTATUS 33447 1
:debug:build Backtrace: command execution failed
:debug:build     while executing
:debug:build "system {*}$notty {*}$callback {*}$nice $fullcmdstring"
:debug:build     invoked from within
:debug:build "command_exec -callback portprogress::target_progress_callback build"
:debug:build     (procedure "portbuild::build_main" line 8)
:debug:build     invoked from within
:debug:build "$procedure $targetname"
:error:build See /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_python_py-setuptools/py27-setuptools/main.log for details.

py27-setuptools is a dependency for a number of other ports including py27-libxml2 which seems to be used a lot.

[jmroot (Joshua Root)]

Looks like _ctypes may not have been built. The build log for python27 may have information as to why.

[jonwilts]

Ok - removed and reinstalled python27 with -v. You were right:

building '_ctypes' extension
creating build/temp.macosx-12.0-arm64-2.7/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.18/Modules/_ctypes/darwin
creating build/temp.macosx-12.0-arm64-2.7/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.18/Modules/_ctypes/libffi_osx
creating build/temp.macosx-12.0-arm64-2.7/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.18/Modules/_ctypes/libffi_osx/x86
creating build/temp.macosx-12.0-arm64-2.7/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.18/Modules/_ctypes/libffi_osx/powerpc
/usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -arch arm64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I_ctypes/darwin -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.18/Mac/Include -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.18/Modules/_ctypes/libffi_osx/include -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.18/Modules/_ctypes/libffi_osx/powerpc -I. -IInclude -I./Include -I/opt/local/include -I/opt/local/include/db48 -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.18/Include -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.18 -c /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.18/Modules/_ctypes/_ctypes.c -o build/temp.macosx-12.0-arm64-2.7/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.18/Modules/_ctypes/_ctypes.o -DUSING_MALLOC_CLOSURE_DOT_C=1 -DMACOSX
In file included from /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.18/Modules/_ctypes/_ctypes.c:113:
/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.18/Modules/_ctypes/libffi_osx/include/ffi.h:65:3: error: "Unsupported MacOS X CPU type"
#       error "Unsupported MacOS X CPU type"
        ^
In file included from /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.18/Modules/_ctypes/_ctypes.c:113:
In file included from /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.18/Modules/_ctypes/libffi_osx/include/ffi.h:73:
/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.18/Modules/_ctypes/libffi_osx/include/ffitarget.h:12:2: error: "Unsupported CPU type"
#error "Unsupported CPU type"
 ^
In file included from /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.18/Modules/_ctypes/_ctypes.c:113:
In file included from /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.18/Modules/_ctypes/libffi_osx/include/ffi.h:74:
/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.18/Modules/_ctypes/libffi_osx/include/fficonfig.h:51:2: error: "Unknown CPU type"
#error "Unknown CPU type"
 ^
In file included from /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.18/Modules/_ctypes/_ctypes.c:113:
/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.18/Modules/_ctypes/libffi_osx/include/ffi.h:171:5: error: unknown type name 'ffi_abi'
                                ffi_abi         abi;
                                ^
/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.18/Modules/_ctypes/libffi_osx/include/ffi.h:193:2: error: unknown type name 'ffi_sarg'
        ffi_sarg        sint;
        ^
/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.18/Modules/_ctypes/libffi_osx/include/ffi.h:194:2: error: unknown type name 'ffi_arg'
        ffi_arg         uint;
        ^
/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.18/Modules/_ctypes/libffi_osx/include/ffi.h:307:11: error: unknown type name 'ffi_abi'
                                                                                ffi_abi                 abi,
                                                                                ^
In file included from /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.18/Modules/_ctypes/_ctypes.c:129:
/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.18/Modules/_ctypes/ctypes.h:99:5: error: unknown type name 'ffi_closure'
    ffi_closure *pcl_write; /* the C callable, writeable */
    ^
8 errors generated.

Python build finished, but the necessary bits to build these modules were not found:
bsddb185           dl                 imageop         
linuxaudiodev      ossaudiodev        spwd            
sunaudiodev                                           
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


Failed to build these modules:
_ctypes            _Qt                                

[jmroot (Joshua Root)]

It shouldn't be using the included copy of libffi in the first place. Please just attach the whole log.

[ryandesign (Ryan Carsten Schmidt)]

FYI there was no problem building python27 and all the py27 modules needed to run the buildbot worker on our M1 Mac mini.

[jonwilts]

python27 build log

[jonwilts]

/opt/local/include/arm64-ffi.h

[jonwilts]

/opt/local/include/arm64-ffi.h

[jonwilts]

I noticed at the top of the log this message

Header file /opt/local/include/ffi.h does not define LIBFFI_H or ffi_wrapper_h 

/opt/local/include/ffi.h contains this:

#ifdef __arm64__
#include "arm64-ffi.h"
#else
#include "ppcintel-ffi.h"
#endif

I've added my /opt/local/include/arm64-ffi.h in case that's useful.

[jmroot (Joshua Root)]

Looks like some third-party version of libffi is installed in the MacPorts prefix. The libffi port doesn't install a file called arm64-ffi.h.

[jmroot (Joshua Root)]

In 9719fb3cf851e6c7888a1a855079a76b3f3a0364/macports-ports (master):

python27: libffi detection fixes

Fix getting libffi include dir from pkgconfig. Fix ffi.h validation to
recognise muniversal builds.

See: #63730

[jmroot (Joshua Root)]

It seems that that is just what the headers installed by libffi +universal look like, due to tricks employed by the muniversal portgroup. Please try out the committed fix.

[jmroot (Joshua Root)]

Assuming fixed in the absence of any report to the contrary. Please reopen if not.