Opened 2 years ago

Closed 19 months ago

#61282 closed defect (fixed)

python39 @3.9.0, python38 @3.8.7, python37 @3.7.10, python36 @3.6.13: +universal does not work with longer than normal CFLAGS

Reported by: ryandesign (Ryan Schmidt) Owned by: jmroot (Joshua Root)
Priority: Normal Milestone:
Component: ports Version: 2.6.99
Keywords: Cc:
Port: python39, python38, python37, python36

Description

python39 installs with or without the universal variant, but when installed with the universal variant, it doesn't work. This is a regression from python38 and earlier versions which work fine universal or not.

For example when python39 is installed universal, py39-setuptools fails to build:

Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 73, in <module>
    import msvcrt
ModuleNotFoundError: No module named 'msvcrt'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/local/var/macports/build/_Users_rschmidt_macports_macports-ports-ryandesign-fork_python_py-setuptools/py39-setuptools/work/setuptools-50.3.0/setup.py", line 10, in <module>
    import setuptools
  File "/opt/local/var/macports/build/_Users_rschmidt_macports_macports-ports-ryandesign-fork_python_py-setuptools/py39-setuptools/work/setuptools-50.3.0/setuptools/__init__.py", line 10, in <module>
    import distutils.core
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/core.py", line 16, in <module>
    from distutils.dist import Distribution
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 19, in <module>
    from distutils.util import check_environ, strtobool, rfc822_escape
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/util.py", line 14, in <module>
    from distutils.spawn import spawn
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/spawn.py", line 11, in <module>
    import subprocess
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 78, in <module>
    import _posixsubprocess
ModuleNotFoundError: No module named '_posixsubprocess'

This can be reproduced easily at the command line:

$ python3.9
Python 3.9.0 (default, Oct  6 2020, 19:05:10)
[Clang 9.1.0 (clang-902.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 73, in <module>
    import msvcrt
ModuleNotFoundError: No module named 'msvcrt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 78, in <module>
    import _posixsubprocess
ModuleNotFoundError: No module named '_posixsubprocess'
>>> ^D

Examining the contents of python39, I see that when it is installed non-universal, it contains the file:

/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload/_posixsubprocess.cpython-39-darwin.so

But when it is installed universal, that file name contains "failed":

/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload/_posixsubprocess.cpython-39-darwin_failed.so

There are many other "failed" files in the universal install.

Attachments (7)

main.log (7.3 MB) - added by ryandesign (Ryan Schmidt) 19 months ago.
main.log - vanilla
config.log (844.1 KB) - added by ryandesign (Ryan Schmidt) 19 months ago.
config.log - vanilla
main.2.log (7.5 MB) - added by ryandesign (Ryan Schmidt) 19 months ago.
main.log - -Werror=implicit-function-declaration
config.2.log (861.9 KB) - added by ryandesign (Ryan Schmidt) 19 months ago.
config.log - -Werror=implicit-function-declaration
Portfile.diff (1.6 KB) - added by jmroot (Joshua Root) 19 months ago.
sysconfig.py.diff (507 bytes) - added by jmroot (Joshua Root) 19 months ago.
main.success.log.bz2 (166.7 KB) - added by ryandesign (Ryan Schmidt) 19 months ago.
successful build of python39 @3.9.2 with jmr's change

Change History (30)

comment:1 Changed 2 years ago by ryandesign (Ryan Schmidt)

This was on macOS 10.13.6.

comment:2 Changed 22 months ago by ryandesign (Ryan Schmidt)

The problem remains with python 3.9.1.

$ port -vq installed python39
  python39 @3.9.1_0+universal (active) platform='darwin 17' archs='i386 x86_64' date='2020-12-18T07:33:31-0600'
$ python3.9
Python 3.9.1 (default, Dec 18 2020, 07:27:59)
[Clang 9.1.0 (clang-902.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 73, in <module>
    import msvcrt
ModuleNotFoundError: No module named 'msvcrt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 78, in <module>
    import _posixsubprocess
ModuleNotFoundError: No module named '_posixsubprocess'
>>> ^D

comment:3 Changed 21 months ago by ryandesign (Ryan Schmidt)

Port: python38 added
Summary: python39 @3.9.0+universal does not workpython39 @3.9.0, python38 @3.8.7: +universal does not work

The problem now also occurs with python38 @3.8.7. See duplicate #61915.

comment:4 Changed 21 months ago by kencu (Ken)

A universal build of python39 with arm/x86_64, installed as per #62023, seems to work (on BigSur Intel at present):

% port -vq installed python39
  python39 @3.9.1_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T10:07:17-0800'

% python3.9                  
Python 3.9.1 (default, Jan 10 2021, 10:06:23) 
[Clang 12.0.0 (clang-1200.0.32.28)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> ^D
Last edited 21 months ago by kencu (Ken) (previous) (diff)

comment:5 Changed 21 months ago by kencu (Ken)

Universal on i386/86_64 is OK on 10.6.8:

$ port -vq installed python39
  python39 @3.9.1_0+universal (active) platform='darwin 10' archs='i386 x86_64' date='2021-01-10T18:29:47-0800'

$ file "/Applications/MacPorts/Python 3.9/IDLE.app/Contents/MacOS/Python"
/Applications/MacPorts/Python 3.9/IDLE.app/Contents/MacOS/Python: Mach-O universal binary with 2 architectures
/Applications/MacPorts/Python 3.9/IDLE.app/Contents/MacOS/Python (for architecture x86_64):	Mach-O 64-bit executable x86_64
/Applications/MacPorts/Python 3.9/IDLE.app/Contents/MacOS/Python (for architecture i386):	Mach-O executable i386

$ python3.9
Python 3.9.1 (default, Jan 10 2021, 18:21:38) 
[Clang 9.0.1 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> ^D

comment:6 Changed 21 months ago by kencu (Ken)

Ah -- but the i386 build is broken:

$ arch -i386 python3.9
Python 3.9.1 (default, Jan 10 2021, 18:21:38) 
[Clang 9.0.1 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 73, in <module>
    import msvcrt
ModuleNotFoundError: No module named 'msvcrt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 78, in <module>
    import _posixsubprocess
ImportError: dynamic module does not define module export function (PyInit__posixsubprocess)
>>> ^D

comment:7 Changed 19 months ago by hehongbo (HeHongbo)

Same on macOS 11.2.1 when I try installing python39.

comment:8 Changed 19 months ago by ryandesign (Ryan Schmidt)

Port: python37 added
Summary: python39 @3.9.0, python38 @3.8.7: +universal does not workpython39 @3.9.0, python38 @3.8.7, python37 @3.7.10: +universal does not work

python37 is affected now too after having been updated to 3.7.10.

We need to identify whatever change they've put in and backported to all these versions and get rid of it.

comment:9 Changed 19 months ago by ryandesign (Ryan Schmidt)

Port: python36 added
Summary: python39 @3.9.0, python38 @3.8.7, python37 @3.7.10: +universal does not workpython39 @3.9.0, python38 @3.8.7, python37 @3.7.10, python36 @3.6.13: +universal does not work

python36 is also affected now after being updated to 3.6.13.

comment:10 Changed 19 months ago by jmroot (Joshua Root)

Not that much changed between 3.7.9 and 3.7.10: https://github.com/python/cpython/compare/v3.7.9..v3.7.10
or between 3.6.12 and 3.6.13: https://github.com/python/cpython/compare/v3.6.12..v3.6.13

comment:11 Changed 19 months ago by ryandesign (Ryan Schmidt)

Aha! Thank you for pointing that out. Then I think it is something I changed on my machine some months ago: I build with this change to base:

--- src/port1.0/portconfigure.tcl	(revision 155969)
+++ src/port1.0/portconfigure.tcl	(working copy)
@@ -235,8 +235,8 @@
         configure.classpath
 # compiler flags section
 default configure.optflags      -Os
-default configure.cflags        {${configure.optflags}}
-default configure.objcflags     {${configure.optflags}}
+default configure.cflags        {${configure.optflags} -Werror=implicit-function-declaration}
+default configure.objcflags     {${configure.optflags} -Werror=implicit-function-declaration}
 default configure.cppflags      {[portconfigure::configure_get_cppflags]}
 proc portconfigure::configure_get_cppflags {} {
     global prefix

If I rebuild python39 @3.9.2 +universal without that, then it works.

Let me get clean build logs both with and without that and let's see how they differ.

Last edited 19 months ago by ryandesign (Ryan Schmidt) (previous) (diff)

comment:12 Changed 19 months ago by ryandesign (Ryan Schmidt)

It looks like python's build system must do some kind of word wrapping when it creates the file _sysconfigdata__darwin_darwin.py (perhaps to comply with PEP-8). The Portfile uses reinplace to remove -arch flags from _sysconfigdata__darwin_darwin.py (see #39669 where that was first added) but because my CFLAGS are longer, some of the -arch flags happen to end up broken across multiple lines due to the word wrapping, so reinplace doesn't remove all of them. (Even in the default build, it doesn't remove all of them—look at CONFIGURE_CFLAGS, PY_CORE_CFLAGS, and PY_STDMODULE_CFLAGS.) Here's the diff between how that file got generated in a standard "vanilla" build and how it got generated with -Werror=implicit-function-declaration added:

  • python39-3.9.2_0+universal.darwin_17.i386-x86_64/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/_sysconfigdata__darwin_darwin.py

    old new  
    2525 'CCSHARED': '',
    2626 'CFLAGS': '-Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common '
    2727           '-dynamic -DNDEBUG -g -fwrapv -O3 -Wall   '
    28            '-pipe -Os  ',
     28           '-pipe -Os -Werror=implicit-function-declaration  -arch '
     29           'i386',
    2930 'CFLAGSFORSHARED': '',
    3031 'CFLAGS_ALIASING': '-fno-strict-aliasing',
    3132 'CFLAGS_NODIST': '',
    3233 'CONFIGFILES': 'configure configure.ac acconfig.h pyconfig.h.in '
    3334                'Makefile.pre.in',
    34  'CONFIGURE_CFLAGS': '  -pipe -Os  -arch '
     35 'CONFIGURE_CFLAGS': '  -pipe -Os '
     36                     '-Werror=implicit-function-declaration  -arch '
    3537                     'i386',
    3638 'CONFIGURE_CFLAGS_NODIST': '-std=c99 -Wextra -Wno-unused-result '
    3739                            '-Wno-unused-parameter '
     
    5052                "'--with-computed-gotos' '--with-ensurepip=no' "
    5153                "'--with-system-expat' '--with-system-ffi' "
    5254                "'--enable-universalsdk=/' 'CC=/usr/bin/clang' 'CFLAGS=-pipe "
    53                 "-Os  ' 'LDFLAGS=-L/opt/local/lib "
     55                '-Os -Werror=implicit-function-declaration  -arch '
     56                "i386' 'LDFLAGS=-L/opt/local/lib "
    5457                "-Wl,-headerpad_max_install_names  ' "
    5558                "'CPPFLAGS=-I/opt/local/include'",
    5659 'CONFINCLUDEDIR': '/opt/local/Library/Frameworks/Python.framework/Versions/3.9/include',
     
    649652 'PY_BUILTIN_MODULE_CFLAGS': '-Wno-unused-result -Wsign-compare '
    650653                             '-Wunreachable-code -fno-common -dynamic -DNDEBUG '
    651654                             '-g -fwrapv -O3 -Wall   '
    652                              '-pipe -Os   -std=c99 '
    653                              '-Wextra -Wno-unused-result -Wno-unused-parameter '
     655                             '-pipe -Os -Werror=implicit-function-declaration '
     656                             '  -std=c99 -Wextra '
     657                             '-Wno-unused-result -Wno-unused-parameter '
    654658                             '-Wno-missing-field-initializers '
    655659                             '-Wstrict-prototypes '
    656660                             '-Werror=implicit-function-declaration '
     
    659663                             '-DPy_BUILD_CORE_BUILTIN',
    660664 'PY_CFLAGS': '-Wno-unused-result -Wsign-compare -Wunreachable-code '
    661665              '-fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall  '
    662               ' -pipe -Os  ',
     666              ' -pipe -Os -Werror=implicit-function-declaration '
     667              ' ',
    663668 'PY_CFLAGS_NODIST': '-std=c99 -Wextra -Wno-unused-result '
    664669                     '-Wno-unused-parameter -Wno-missing-field-initializers '
    665670                     '-Wstrict-prototypes '
     
    668673 'PY_COERCE_C_LOCALE': 1,
    669674 'PY_CORE_CFLAGS': '-Wno-unused-result -Wsign-compare -Wunreachable-code '
    670675                   '-fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch '
    671                    'x86_64  -pipe -Os   '
    672                    '-std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter '
    673                    '-Wno-missing-field-initializers -Wstrict-prototypes '
    674                    '-Werror=implicit-function-declaration -fvisibility=hidden  '
    675                    '-I./Include/internal -I. -I./Include -I/opt/local/include '
    676                    '-DPy_BUILD_CORE',
     676                   'x86_64  -pipe -Os '
     677                   '-Werror=implicit-function-declaration  -arch '
     678                   'i386 -std=c99 -Wextra -Wno-unused-result '
     679                   '-Wno-unused-parameter -Wno-missing-field-initializers '
     680                   '-Wstrict-prototypes -Werror=implicit-function-declaration '
     681                   '-fvisibility=hidden  -I./Include/internal -I. -I./Include '
     682                   '-I/opt/local/include -DPy_BUILD_CORE',
    677683 'PY_CORE_LDFLAGS': '  -L/opt/local/lib '
    678684                    '-Wl,-headerpad_max_install_names  ',
    679685 'PY_CPPFLAGS': '-I. -I./Include -I/opt/local/include',
     
    685691 'PY_SSL_DEFAULT_CIPHER_STRING': 0,
    686692 'PY_STDMODULE_CFLAGS': '-Wno-unused-result -Wsign-compare -Wunreachable-code '
    687693                        '-fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall '
    688                         '  -pipe -Os  -arch '
    689                         'i386 -std=c99 -Wextra -Wno-unused-result '
     694                        '  -pipe -Os '
     695                        '-Werror=implicit-function-declaration  '
     696                        ' -std=c99 -Wextra -Wno-unused-result '
    690697                        '-Wno-unused-parameter -Wno-missing-field-initializers '
    691698                        '-Wstrict-prototypes '
    692699                        '-Werror=implicit-function-declaration '

I suspect that that's what's causing the problem. Or if not, it should be fixed anyway, perhaps by disabling the word wrapping or by doing the -arch flag removal before that file gets generated (perhaps in whatever script builds it—that might then be suitable to send upstream).

Last edited 19 months ago by ryandesign (Ryan Schmidt) (previous) (diff)

Changed 19 months ago by ryandesign (Ryan Schmidt)

Attachment: main.log added

main.log - vanilla

Changed 19 months ago by ryandesign (Ryan Schmidt)

Attachment: config.log added

config.log - vanilla

Changed 19 months ago by ryandesign (Ryan Schmidt)

Attachment: main.2.log added

main.log - -Werror=implicit-function-declaration

Changed 19 months ago by ryandesign (Ryan Schmidt)

Attachment: config.2.log added

config.log - -Werror=implicit-function-declaration

comment:13 Changed 19 months ago by ryandesign (Ryan Schmidt)

Note that in both main.logs some stuff gets compiled in the destroot phase, with that modified sysconfig. In main.2.log, you can see some things being compiled with only -arch i386. The post-build block tries to save off the sysconfig mtime before changing it and restoring it afterward so that things don't get rebuilt, and yet...

Changed 19 months ago by jmroot (Joshua Root)

Attachment: Portfile.diff added

Changed 19 months ago by jmroot (Joshua Root)

Attachment: sysconfig.py.diff added

comment:14 Changed 19 months ago by jmroot (Joshua Root)

Summary: python39 @3.9.0, python38 @3.8.7, python37 @3.7.10, python36 @3.6.13: +universal does not workpython39 @3.9.0, python38 @3.8.7, python37 @3.7.10, python36 @3.6.13: +universal does not work with longer than normal CFLAGS

Try these.

comment:15 Changed 19 months ago by ryandesign (Ryan Schmidt)

Ooh, thanks. I was just thinking about making this my project for the evening, but you beat me to it.

It looks good to me. Attaching my log with this change (plus a revbump to 1).

Changing the awk to a reinplace... is that just stylistic or is there a functional change?

Any idea why some stuff still gets built in the destroot phase?

I wonder if this will also fix #59466. (Edit: no, this issue only affects python34 and later but that issue was with python27.)

Last edited 19 months ago by ryandesign (Ryan Schmidt) (previous) (diff)

Changed 19 months ago by ryandesign (Ryan Schmidt)

Attachment: main.success.log.bz2 added

successful build of python39 @3.9.2 with jmr's change

comment:16 Changed 19 months ago by ryandesign (Ryan Schmidt)

Looks like this sysconf thing was new in python 2.7 so the fix wouldn't have to be brought farther back than python27.

comment:17 in reply to:  15 ; Changed 19 months ago by jmroot (Joshua Root)

Replying to ryandesign:

I was just thinking about making this my project for the evening, but you beat me to it.

More time to spend on getting the 10.15 builder back up? :)

It looks good to me. Attaching my log with this change (plus a revbump to 1).

I'm not going to rev bump unless there's evidence that anyone not making changes to base was affected.

Changing the awk to a reinplace... is that just stylistic or is there a functional change?

Awk was only used in the first place so that the second line could be removed. Reinplace is simpler for a single line.

Any idea why some stuff still gets built in the destroot phase?

PythonLauncher gets built in the destroot phase normally (not sure why), but I don't know why you're seeing modules getting rebuilt. I don't see that with a non-universal build on 10.15.

comment:18 in reply to:  17 ; Changed 19 months ago by ryandesign (Ryan Schmidt)

Replying to jmroot:

More time to spend on getting the 10.15 builder back up? :)

Just did :)

I'm not going to rev bump unless there's evidence that anyone not making changes to base was affected.

Well all users are affected, not just me. See comment:12:

Even in the default build, it doesn't remove all of them—look at CONFIGURE_CFLAGS, PY_CORE_CFLAGS, and PY_STDMODULE_CFLAGS.

I'm not sure what the significance of having wrong single -arch flags in those sysconfig values is, but why not revbump it to fix it? It's consistent with our guidance to increase the revision when installed files change.

Awk was only used in the first place so that the second line could be removed. Reinplace is simpler for a single line.

Ahh right.

comment:19 in reply to:  18 Changed 19 months ago by jmroot (Joshua Root)

Replying to ryandesign:

Even in the default build, it doesn't remove all of them—look at CONFIGURE_CFLAGS, PY_CORE_CFLAGS, and PY_STDMODULE_CFLAGS.

I'm not sure what the significance of having wrong single -arch flags in those sysconfig values is, but why not revbump it to fix it? It's consistent with our guidance to increase the revision when installed files change.

Ah, I see what you mean now. But python27 at least appears to be unaffected.

comment:20 Changed 19 months ago by ryandesign (Ryan Schmidt)

Yeah looks like python 3.4 is the first version in which _sysconfigdata.py gets word wrapped.

comment:22 Changed 19 months ago by ryandesign (Ryan Schmidt)

comment:23 Changed 19 months ago by jmroot (Joshua Root)

Resolution: fixed
Status: assignedclosed

In 9fd97292e6a751db08d9fe5e9e59bec71b868494/macports-ports (master):

python3*: protect against issues with long CFLAGS

Prevent line wrapping in _sysconfigdata.py so that '-arch foo' can't be
split across multiple lines, preventing the reinplace from removing it
correctly. In practice this only seems to have happened with universal
builds.

All variables being on a single line also means that awk is no longer
needed to edit LINKFORSHARED.

Also fix implicit declaration errors in python35's configure script.

Closes: #61282

Note: See TracTickets for help on using tickets.