Opened 2 years ago

Closed 2 years ago

Last modified 22 months ago

#63782 closed defect (fixed)

python3{8,9,10} +universal: ImportError: dynamic module does not define module export function (PyInit__posixsubprocess)

Reported by: casr (Chris Rawnsley) Owned by: MarcusCalhoun-Lopez <MarcusCalhoun-Lopez@…>
Priority: Normal Milestone:
Component: ports Version:
Keywords: monterey arm64 Cc: jmroot (Joshua Root), mascguy (Christopher Nielsen), ShadSterling (Shad Sterling), tehfink, MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)
Port: python38 python39 python310

Description (last modified by ryandesign (Ryan Carsten Schmidt))

I have built python38 with the +universal variant, however, when I use the arm64 version it fails to create a virtualenv directory and complains with:

Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 64, in <module>
    import msvcrt
ModuleNotFoundError: No module named 'msvcrt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 185, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 144, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/venv/__init__.py", line 10, in <module>
    import subprocess
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 69, in <module>
    import _posixsubprocess
ImportError: dynamic module does not define module export function (PyInit__posixsubprocess)

I am running macOS 12.0.1 21A559 arm64, Xcode 13.1 13A1030d.

To reproduce:

# works
( cd $(mktemp -d) && arch -x86_64 python3.8 -m venv venv )

# fails
( cd $(mktemp -d) && arch -arm64 python3.8 -m venv venv )

# fails
( cd $(mktemp -d) && python3.8 -m venv venv )

Seems similar to the problem described in #61282

Attachments (2)

main.log (2.9 MB) - added by ryankanno 2 years ago.
main.log output of sudo port destroot python39 +universal
dep_util.py.patch (1.1 KB) - added by jmroot (Joshua Root) 2 years ago.

Change History (42)

comment:1 Changed 2 years ago by ryandesign (Ryan Carsten Schmidt)

Cc: jmroot added
Description: modified (diff)
Keywords: monterey arm64 added
Port: python38 added

comment:2 Changed 2 years ago by jmroot (Joshua Root)

Summary: python38 @3.8.12 has a problem when using venv: ModuleNotFoundError: No module named 'msvcrt'python38 @3.8.12 has a problem when using venv: ImportError: dynamic module does not define module export function (PyInit__posixsubprocess)

comment:3 Changed 2 years ago by casr (Chris Rawnsley)

I noticed something peculiar with my MacPorts installation in that port was using a tclsh that was running an Intel binary. I've since reinstalled MacPorts making sure my terminal wasn't running in Rosetta-mode, retried the command and the virtual environment is created as expected.

I think this can be moved to fixed if you're happy to do so. Happy to provide any follow up in case it is useful too.

comment:4 Changed 2 years ago by casr (Chris Rawnsley)

Actually, I just tried the opposite way around and the same error occurred. In other words, this time around MacPorts is installed under arm64 and the x86_64 version is busted.

$ arch -x86_64 python3.8 -m venv venv
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 64, in <module>
    import msvcrt
ModuleNotFoundError: No module named 'msvcrt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 185, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 144, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/venv/__init__.py", line 10, in <module>
    import subprocess
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 69, in <module>
    import _posixsubprocess
ImportError: dynamic module does not define module export function (PyInit__posixsubprocess)

comment:5 Changed 2 years ago by jmroot (Joshua Root)

What kind of a binary is _posixsubprocess.cpython-38-darwin.so?

comment:6 Changed 2 years ago by casr (Chris Rawnsley)

It seems to be in order...

$ port contents python38 | grep -i posixsub
  /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload/_posixsubprocess.cpython-38-darwin.so

$ file /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload/_posixsubprocess.cpython-38-darwin.so
/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload/_posixsubprocess.cpython-38-darwin.so: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit bundle x86_64] [arm64:Mach-O 64-bit bundle arm64]
/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload/_posixsubprocess.cpython-38-darwin.so (for architecture x86_64):	Mach-O 64-bit bundle x86_64
/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload/_posixsubprocess.cpython-38-darwin.so (for architecture arm64):	Mach-O 64-bit bundle arm64

comment:7 Changed 2 years ago by jmroot (Joshua Root)

Port: python39 python310 added

Duplicate #64234 reports the same issue with python39 and python310. At present, I have no idea what is causing this, and unfortunately I am unable to test universal builds.

comment:8 Changed 2 years ago by jmroot (Joshua Root)

Summary: python38 @3.8.12 has a problem when using venv: ImportError: dynamic module does not define module export function (PyInit__posixsubprocess)python3{8,9,10} +universal: ImportError: dynamic module does not define module export function (PyInit__posixsubprocess)

comment:9 Changed 2 years ago by mascguy (Christopher Nielsen)

Cc: mascguy added

comment:10 Changed 2 years ago by ryankanno

If you think it's worth trying (and of any help), I can go through and check out some of the older 3.9 versions to see if I see the same error.

comment:11 Changed 2 years ago by jmroot (Joshua Root)

The first thing you could rule out is the other variants. See if building with -lto, -optimizations, or both has any effect on this (still with +universal of course; we already know -universal is OK). If not, and if there's an older version that was known not to have this issue, bisecting would certainly be helpful. If there's no known good version, going back to older versions may be futile, but I guess it's still data.

Last edited 2 years ago by jmroot (Joshua Root) (previous) (diff)

comment:12 in reply to:  11 Changed 2 years ago by ryankanno

Replying to jmroot:

The first thing you could rule out is the other variants. See if building with -lto, -optimizations, or both has any effect on this (still with +universal of course; we already know -universal is OK). If not, and if there's an older version that was known not to have this issue, bisecting would certainly be helpful. If there's no known good version, going back to older versions may be futile, but I guess it's still data.

Great idea. Following your advice, here's a matrix of what I came up with - it looks like the arm64 archs are failing. As a note, this is the opposite of what I saw in https://trac.macports.org/ticket/64234.

Installation command Output of port -vq installed python39 Arch Command Result
$ port install python39 +universal -lto python39 @3.9.9_0+optimizations+universal (active) requested_variants='+universal-lto' platform='darwin 21' archs='arm64 x86_64' date='2021-12-18T07:52:13+0900' x86_64 arch -x86_64 /opt/local/bin/python3.9 -c 'import subprocess;' success
$ port install python39 +universal -lto python39 @3.9.9_0+optimizations+universal (active) requested_variants='+universal-lto' platform='darwin 21' archs='arm64 x86_64' date='2021-12-18T07:52:13+0900' arm64 arch -arm64 /opt/local/bin/python3.9 -c 'import subprocess;' ModuleNotFoundError: No module named 'msvcrt'
$ port install python39 +universal -optimizations python39 @3.9.9_0+lto+universal (active) requested_variants='+universal-optimizations' platform='darwin 21' archs='arm64 x86_64' date='2021-12-18T08:46:14+0900' x86_64 arch -x86_64 /opt/local/bin/python3.9 -c 'import subprocess;' success
$ port install python39 +universal -optimizations python39 @3.9.9_0+lto+universal (active) requested_variants='+universal-optimizations' platform='darwin 21' archs='arm64 x86_64' date='2021-12-18T08:46:14+0900' arm64 arch -arm64 /opt/local/bin/python3.9 -c 'import subprocess;' ModuleNotFoundError: No module named 'msvcrt'
$ port install python39 +universal -lto -optimizations python39 @3.9.9_0+universal (active) requested_variants='+universal-lto-optimizations' platform='darwin 21' archs='arm64 x86_64' date='2021-12-18T08:56:59+0900' x86_64 arch -x86_64 /opt/local/bin/python3.9 -c 'import subprocess;' success
$ port install python39 +universal -lto -optimizations python39 @3.9.9_0+universal (active) requested_variants='+universal-lto-optimizations' platform='darwin 21' archs='arm64 x86_64' date='2021-12-18T08:56:59+0900' arm64 arch -arm64 /opt/local/bin/python3.9 -c 'import subprocess;' ModuleNotFoundError: No module named 'msvcrt'

Will continue to add more data points from older portfiles as I try them out.

Last edited 2 years ago by ryankanno (previous) (diff)

comment:13 Changed 2 years ago by jmroot (Joshua Root)

Thanks. Note that the ModuleNotFoundError is not relevant; msvcrt only exists on Windows, so the subprocess module catches that error and imports _posixsubprocess instead, which is where the actual failure happens.

comment:14 Changed 2 years ago by jmroot (Joshua Root)

Just to check, what does your sys.path look like?

comment:15 in reply to:  13 Changed 2 years ago by ryankanno

Replying to jmroot:

Thanks. Note that the ModuleNotFoundError is not relevant; msvcrt only exists on Windows, so the subprocess module catches that error and imports _posixsubprocess instead, which is where the actual failure happens.

Gotcha - makes sense. As a note, these results above are the opposite of what I saw in my original ticket #64234. To verify the original ticket and for my own sanity, I ran the following:

Installation command Output of port -vq installed python39 Arch Command Result
$ port install python39 +universal python39 @3.9.9_0+lto+optimizations+universal (active) requested_variants='+universal' platform='darwin 21' archs='arm64 x86_64' date='2021-12-18T09:38:51+0900' x86_64 arch -x86_64 /opt/local/bin/python3.9 -c 'import subprocess;' ImportError: dynamic module does not define module export function (PyInit__posixsubprocess)
$ port install python39 +universal python39 @3.9.9_0+lto+optimizations+universal (active) requested_variants='+universal' platform='darwin 21' archs='arm64 x86_64' date='2021-12-18T09:38:51+0900' arm64 arch -arm64 /opt/local/bin/python3.9 -c 'import subprocess;' success

For the sys.path, here is the path(s) from the python above:

arch -x86_64 /opt/local/bin/python3.9 -c 'import sys; print(sys.path)'

['', '/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python39.zip', '/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9', '/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload', '/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages']

arch -arm64 /opt/local/bin/python3.9 -c 'import sys; print(sys.path)'

['', '/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python39.zip', '/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9', '/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload', '/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages']
Last edited 2 years ago by ryandesign (Ryan Carsten Schmidt) (previous) (diff)

comment:16 Changed 2 years ago by jmroot (Joshua Root)

OK, so nothing weird in sys.path that could be causing issues. As per comment:4, which arch is broken does seem to depend on the hardware or at least the build_arch, even though a universal binary should be functionally the same in either case. If you install from a binary archive, it may well have been built on a system different to yours.

comment:17 Changed 2 years ago by ryankanno

I wrote a small script to run in the background that would check out a previous portfile, install w/ +universal, output port -vq installed python39, then execute the two commands importing subprocess. Unfortunately, the previous 3.9 versions exhibited the same behavior w/ x86_64 success and failing with arm64. I skipped 3.9.1. Apologies for the messy / verbose tables, but here's the results. Let me know if there's anything else that y'all would like me to do. Thanks again.

Version Installation command Output of port -vq installed python39 Arch Command Result
3.9.2 $ port install +universal python39 @3.9.2_0+universal (active) requested_variants='+universal' platform='darwin 21' archs='arm64 x86_64' date='2021-12-18T10:46:45+0900' x86_64 arch -x86_64 /opt/local/bin/python3.9 -c 'import subprocess;' success
3.9.2 $ port install +universal python39 @3.9.2_0+universal (active) requested_variants='+universal' platform='darwin 21' archs='arm64 x86_64' date='2021-12-18T10:46:45+0900' arm64 arch -arm64 /opt/local/bin/python3.9 -c 'import subprocess;' ImportError: dynamic module does not define module export function (PyInit__posixsubprocess)
3.9.3 $ port install +universal python39 @3.9.3_0+universal (active) requested_variants='+universal' platform='darwin 21' archs='arm64 x86_64' date='2021-12-18T11:03:33+0900' x86_64 arch -x86_64 /opt/local/bin/python3.9 -c 'import subprocess;' success
3.9.3 $ port install +universal python39 @3.9.3_0+universal (active) requested_variants='+universal' platform='darwin 21' archs='arm64 x86_64' date='2021-12-18T11:03:33+0900' arm64 arch -arm64 /opt/local/bin/python3.9 -c 'import subprocess;' ImportError: dynamic module does not define module export function (PyInit__posixsubprocess)
3.9.4 $ port install +universal python39 @3.9.4_0+universal (active) requested_variants='+universal' platform='darwin 21' archs='arm64 x86_64' date='2021-12-18T11:12:53+0900' x86_64 arch -x86_64 /opt/local/bin/python3.9 -c 'import subprocess;' success
3.9.4 $ port install +universal python39 @3.9.4_0+universal (active) requested_variants='+universal' platform='darwin 21' archs='arm64 x86_64' date='2021-12-18T11:12:53+0900' arm64 arch -arm64 /opt/local/bin/python3.9 -c 'import subprocess;' ImportError: dynamic module does not define module export function (PyInit__posixsubprocess)
3.9.5 $ port install +universal python39 @3.9.5_0+universal (active) requested_variants='+universal' platform='darwin 21' archs='arm64 x86_64' date='2021-12-18T11:23:41+0900' x86_64 arch -x86_64 /opt/local/bin/python3.9 -c 'import subprocess;' success
3.9.5 $ port install +universal python39 @3.9.5_0+universal (active) requested_variants='+universal' platform='darwin 21' archs='arm64 x86_64' date='2021-12-18T11:23:41+0900' arm64 arch -arm64 /opt/local/bin/python3.9 -c 'import subprocess;' ImportError: dynamic module does not define module export function (PyInit__posixsubprocess)
3.9.6 $ port install +universal python39 @3.9.6_0+universal (active) requested_variants='+universal' platform='darwin 21' archs='arm64 x86_64' date='2021-12-18T11:33:02+0900' x86_64 arch -x86_64 /opt/local/bin/python3.9 -c 'import subprocess;' success
3.9.6 $ port install +universal python39 @3.9.6_0+universal (active) requested_variants='+universal' platform='darwin 21' archs='arm64 x86_64' date='2021-12-18T11:33:02+0900' arm64 arch -arm64 /opt/local/bin/python3.9 -c 'import subprocess;' ImportError: dynamic module does not define module export function (PyInit__posixsubprocess)
3.9.7 $ port install +universal python39 @3.9.7_0+universal (active) requested_variants='+universal' platform='darwin 21' archs='arm64 x86_64' date='2021-12-18T12:05:48+0900' x86_64 arch -x86_64 /opt/local/bin/python3.9 -c 'import subprocess;' success
3.9.7 $ port install +universal python39 @3.9.7_0+universal (active) requested_variants='+universal' platform='darwin 21' archs='arm64 x86_64' date='2021-12-18T12:05:48+0900' arm64 arch -arm64 /opt/local/bin/python3.9 -c 'import subprocess;' ImportError: dynamic module does not define module export function (PyInit__posixsubprocess)
3.9.8 $ port install +universal python39 @3.9.8_0+lto+optimizations+universal (active) requested_variants='+universal' platform='darwin 21' archs='arm64 x86_64' date='2021-12-18T12:21:15+0900' x86_64 arch -x86_64 /opt/local/bin/python3.9 -c 'import subprocess;' success
3.9.8 $ port install +universal python39 @3.9.8_0+lto+optimizations+universal (active) requested_variants='+universal' platform='darwin 21' archs='arm64 x86_64' date='2021-12-18T12:21:15+0900' arm64 arch -arm64 /opt/local/bin/python3.9 -c 'import subprocess;' ImportError: dynamic module does not define module export function (PyInit__posixsubprocess)
Last edited 2 years ago by ryandesign (Ryan Carsten Schmidt) (previous) (diff)

comment:18 in reply to:  17 Changed 2 years ago by jmroot (Joshua Root)

Replying to ryankanno:

Let me know if there's anything else that y'all would like me to do.

Well, I haven't actually seen a universal build log yet, so if you could run sudo port destroot python39 +universal and add the resulting main.log as an attachment, there might be some clue in it.

Changed 2 years ago by ryankanno

Attachment: main.log added

main.log output of sudo port destroot python39 +universal

comment:19 Changed 2 years ago by jmroot (Joshua Root)

Some more findings: Looking at the binary, it's not just _posixsubprocess that is affected, every single extension module in lib-dynload appears to have no symbol table in the x86_64 slice. For example:

% xcrun llvm-objdump --macho --arch=arm64 -t python39-3.9.9_0+lto+optimizations+universal.darwin_21.arm64-x86_64/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload/_uuid.cpython-39-darwin.so 
python39-3.9.9_0+lto+optimizations+universal.darwin_21.arm64-x86_64/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload/_uuid.cpython-39-darwin.so:
SYMBOL TABLE:
0000000000003e94 l     F __TEXT,__text _uuid_exec
0000000000003eb8 l     F __TEXT,__text _py_uuid_generate_time_safe
0000000000008000 l     O __DATA,__data _uuidmodule
0000000000008068 l     O __DATA,__data _uuid_methods
00000000000080a8 l     O __DATA,__data _uuid_slots
0000000000000000      d  *UND* /opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_lang_python39/python39/work/Python-3.9.9/Modules/
0000000000000000      d  *UND* _uuidmodule.c
0000000000000000      d  *UND* /tmp/lto.o
0000000000003e88      d  *UND* 
0000000000003e88      d  *UND* _PyInit__uuid
000000000000000c      d  *UND* 
000000000000000c      d  *UND* 
0000000000003e94      d  *UND* 
0000000000003e94      d  *UND* _uuid_exec
0000000000000024      d  *UND* 
0000000000000024      d  *UND* 
0000000000003eb8      d  *UND* 
0000000000003eb8      d  *UND* _py_uuid_generate_time_safe
0000000000000078      d  *UND* 
0000000000000078      d  *UND* 
0000000000008000      d  *UND* _uuidmodule
0000000000008068      d  *UND* _uuid_methods
00000000000080a8      d  *UND* _uuid_slots
0000000000000000      d  *UND* 
0000000000003e88 g     F __TEXT,__text _PyInit__uuid
0000000000000000         *UND* _PyModuleDef_Init
0000000000000000         *UND* _PyModule_AddIntConstant
0000000000000000         *UND* __Py_BuildValue_SizeT
0000000000000000         *UND* __Py_NoneStruct
0000000000000000         *UND* ___stack_chk_fail
0000000000000000         *UND* ___stack_chk_guard
0000000000000000         *UND* _uuid_generate_time
% xcrun llvm-objdump --macho --arch=x86_64 -t python39-3.9.9_0+lto+optimizations+universal.darwin_21.arm64-x86_64/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload/_uuid.cpython-39-darwin.so
/python39-3.9.9_0+lto+optimizations+universal.darwin_21.arm64-x86_64/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload/_uuid.cpython-39-darwin.so:
SYMBOL TABLE:

The python interpreter and framework appear to be fine in both slices though.

comment:20 Changed 2 years ago by casr (Chris Rawnsley)

I'm hunting around documentation and wondering if the configure flag --with-universal-archs=universal2 might help.

https://github.com/python/cpython/blob/f4c03484da59049eb62a9bf7777b963e2267d187/Mac/README.rst#21-flavors-of-universal-binaries

comment:21 Changed 2 years ago by jmroot (Joshua Root)

Someone with an M1 Mac could certainly try that and see if it helps. I don't think it does anything that the port doesn't already do in a more general way, though.

comment:22 in reply to:  20 Changed 2 years ago by ryankanno

Replying to casr:

I'm hunting around documentation and wondering if the configure flag --with-universal-archs=universal2 might help.

https://github.com/python/cpython/blob/f4c03484da59049eb62a9bf7777b963e2267d187/Mac/README.rst#21-flavors-of-universal-binaries

I can definitely try this, but just so it's clear on my end, what would the full command be that you'd like me to run?

comment:23 Changed 2 years ago by jmroot (Joshua Root)

You would edit the Portfile and add that flag to configure.args, and then install the port with sudo port -s install python39 +universal (or whichever python version). Or if it's already installed, sudo port -ns upgrade --force python39 +universal will force a rebuild.

comment:24 in reply to:  23 Changed 2 years ago by ryankanno

Replying to jmroot:

You would edit the Portfile and add that flag to configure.args, and then install the port with sudo port -s install python39 +universal (or whichever python version). Or if it's already installed, sudo port -ns upgrade --force python39 +universal will force a rebuild.

Here is what I did, let me know if there's anything else you'd like me to try.

Edited the Portfile as follows:

diff --git i/lang/python39/Portfile w/lang/python39/Portfile
index fc4f95fb92e..85387f02849 100644
--- i/lang/python39/Portfile
+++ w/lang/python39/Portfile
@@ -66,7 +66,8 @@ configure.args      --enable-framework=${frameworks_dir} \
                     --with-computed-gotos \
                     --with-ensurepip=no \
                     --with-system-expat \
-                    --with-system-ffi
+                    --with-system-ffi \
+                    --with-universal-archs=universal2

Ran sudo port uninstall python39 to be safe, cd into the folder with the modified Portfile above and ran sudo port -s install +universal, but unfortunately still see failures w/ arm64

$ arch -x86_64 /opt/local/bin/python3.9
Python 3.9.9 (main, Dec 24 2021, 16:53:51)
[Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> ^D

$ arch -arm64 /opt/local/bin/python3.9
Python 3.9.9 (main, Dec 24 2021, 16:53:51)
[Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 73, in <module>
    import msvcrt
ModuleNotFoundError: No module named 'msvcrt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 78, in <module>
    import _posixsubprocess
ImportError: dynamic module does not define module export function (PyInit__posixsubprocess)
>>>

comment:25 Changed 2 years ago by casr (Chris Rawnsley)

I just got around to trying my suggestion too and I had the same issue as ryankanno.

comment:26 Changed 2 years ago by casr (Chris Rawnsley)

This seems to work for me on python39 which I was testing on:

diff --git a/lang/python39/Portfile b/lang/python39/Portfile
index 1f2689f3ab2..4bb3d914d02 100644
--- a/lang/python39/Portfile
+++ b/lang/python39/Portfile
@@ -124,10 +124,6 @@ platform darwin {
         reinplace -E s|${lfs_pattern}|${lfs_replacement}| \
             ${buildlibdir}/_sysconfigdata__darwin_darwin.py
 
-        # remove -arch flags from the config
-        reinplace -E {s|-arch [a-z0-9_]+||g} \
-            ${buildlibdir}/_sysconfigdata__darwin_darwin.py
-
         # also remove gettext overlinking
         reinplace "s|-lintl||" \
             ${buildlibdir}/_sysconfigdata__darwin_darwin.py

comment:27 Changed 2 years ago by jmroot (Joshua Root)

So actually the problem is that things are being rebuilt in the destroot phase.

comment:28 Changed 2 years ago by casr (Chris Rawnsley)

Is this the right fix or is it just creating another problem? I'd be happy to raise this as a PR unless you think we should consider other options.

comment:29 Changed 2 years ago by casr (Chris Rawnsley)

So actually the problem is that things are being rebuilt in the destroot phase.

By the way, the lines I changed were in the post-build phase. I realise my terse diff was probably confusing as this reinplace can appear in both phases depending on the Python version.

comment:30 Changed 2 years ago by jmroot (Joshua Root)

Not removing the -arch flags means all modules will be always be built for exactly the same archs as python (can't be built non-universal), so yes, it's a problem. The correct fix is to not rebuild the things in the destroot phase (without -arch flags) that have already been built in the build phase (with -arch flags).

comment:31 Changed 2 years ago by ShadSterling (Shad Sterling)

Cc: ShadSterling added

comment:32 Changed 2 years ago by tehfink

Cc: tehfink added

comment:33 Changed 2 years ago by jmroot (Joshua Root)

In 9c2928159239d0a8c5026c87c5d3f2b1f0be5d0a/macports-ports (master):

python*: preserve mtime of pyconfig.h

See: #63782

comment:34 Changed 2 years ago by jmroot (Joshua Root)

Please try with that change. If it doesn't help, please try adding the attached patch to the port to add some extra debug output, try another build, and attach the resulting log to the ticket.

Changed 2 years ago by jmroot (Joshua Root)

Attachment: dep_util.py.patch added

comment:35 Changed 2 years ago by MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)

Cc: MarcusCalhoun-Lopez added

comment:36 Changed 2 years ago by MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)

There is a pull request that attempts to solve a similar problem by moving the patch of _sysconfigdata__darwin_darwin.py from post-build to post-destroot.

comment:37 Changed 2 years ago by MarcusCalhoun-Lopez <MarcusCalhoun-Lopez@…>

Owner: set to MarcusCalhoun-Lopez <MarcusCalhoun-Lopez@…>
Resolution: fixed
Status: newclosed

In df937b0e8d2efacf47bbf4e7d5c96baa383a86e5/macports-ports (master):

python310: fix universal build (https://github.com/macports/macports-ports/pull/14786)

Patching the file _sysconfigdatadarwin_darwin.py in post-build
seems to cause rebuilds (without the -arch flags) during the
destroot phase.

Revbump is needed since the universal build can be incorrect.

Fixes: #63782

Co-authored-by: Joshua Root <jmr@…>

comment:38 Changed 2 years ago by jmroot (Joshua Root)

I'll roll out the fix to the other python versions soon. Thanks, Marcus!

comment:39 Changed 2 years ago by jmroot (Joshua Root)

In dc0835bc3a9f2eb3e718575e0b72388080b5e4f6/macports-ports (master):

python3{7,8,9}: backport universal fix from python310

See: #63782

comment:40 Changed 22 months ago by casr (Chris Rawnsley)

I've just gotten around to testing this out and everything appears to be working as expected. Thanks to everyone working on this!

Note: See TracTickets for help on using tickets.