Opened 7 months ago

Last modified 7 months ago

#56458 new defect

python*: Compiler path is baked into _sysconfigdata.py

Reported by: ryandesign (Ryan Schmidt) Owned by:
Priority: Normal Milestone:
Component: ports Version:
Keywords: snowleopard, haspatch Cc: fvaccari
Port: python27 python31 python32 python33 python34 python35 python36 python37-devel

Description

The paths to the compilers used to build python27 are baked into the installed file /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_sysconfigdata.py:

$ grep /usr/bin/clang /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_sysconfigdata.py
 'BLDSHARED': '/usr/bin/clang -bundle -undefined dynamic_lookup   -L/opt/local/lib -Wl,-headerpad_max_install_names -L/opt/local/lib/db48  ',
 'CC': '/usr/bin/clang',
 'CONFIG_ARGS': "'--prefix=/opt/local' '--disable-dependency-tracking' '--enable-framework=/opt/local/Library/Frameworks' '--enable-ipv6' '--with-system-expat' '--with-system-ffi' '--enable-universalsdk=/' 'CC=/usr/bin/clang' 'CFLAGS=-pipe -Os  ' 'LDFLAGS=-L/opt/local/lib -Wl,-headerpad_max_install_names -L/opt/local/lib/db48  ' 'CPPFLAGS=-I/opt/local/include -I/opt/local/include/db48'",
 'CXX': '/usr/bin/clang++',
 'LDCXXSHARED': '/usr/bin/clang++ -bundle -undefined dynamic_lookup',
 'LDSHARED': '/usr/bin/clang -bundle -undefined dynamic_lookup   -L/opt/local/lib -Wl,-headerpad_max_install_names -L/opt/local/lib/db48  ',
 'LINKCC': '/usr/bin/clang',
 'MAINCC': '/usr/bin/clang',

Some ports that build with python use these compilers, rather than using the compiler MacPorts told them to use. This may be due to a bug in the python portgroup that should be fixed. But even if the portgroup is fixed, it doesn't help ports not using that portgroup, and it doesn't help software built outside of MacPorts but still using MacPorts python.

The problem occurs when the paths baked into that file at build time don't exist at runtime. For example, python built on our Snow Leopard buildbot build worker will use gcc-4.2, because the build worker has our recommended version of Xcode, 3.2.6, installed. But if a user has Xcode 4.2 installed, then gcc-4.2 will not exist, and the user will encounter build failures because of that. See for example #55831 and #55833.

The problem only affects Snow Leopard, because only on Snow Leopard do we default to a compiler (gcc-4.2) with one version of Xcode (3.2.6) which does not exist in later versions of Xcode (4.2). On all other macOS versions, we use the same default compiler regardless of Xcode version.

The problem affects python27 and later. python26 and earlier don't have a _sysconfigdata.py file so I don't think they are affected. In python36, the filename changed to _sysconfigdata_m_darwin_darwin.py.

Some possible solutions occur to me:

  1. Force python* to build with llvm-gcc-4.2 on 10.6, since llvm-gcc-4.2 is available in all versions of Xcode on 10.6. At first this seemed workable, since llvm-gcc-4.2 is the default compiler for Xcode 4.0.x–4.2.x, until I realized that llvm-gcc-4.2 is blacklisted for python33 and later; see #45667. I confirmed just now that this blacklisting is still required for python33 and later; they fail to build with llvm-gcc-4.2.
  2. Force python* to build with clang on 10.6, since clang is available in all versions of Xcode on 10.6. This may not be a good idea because it would make other python modules build with that very early version of clang, and that seems likely to result in build failures.
  3. In post-destroot, replace the real compiler paths in _sysconfigdata.py with placeholders, and in post-activate, replace the placeholders with the current values from ${configure.cc} and ${configure.cxx}. I can't think of a reason why this wouldn't work. For Xcode 4.2, that would still end up being clang, which might be undesirable as in (2) above. We could special-case 10.6 and use llvm-gcc-4.2 there instead of clang. I am assuming that the build failure of python33 and later with llvm-gcc-4.2 is specific to the source code of python itself, and would not manifest itself when building most modules.

Change History (2)

comment:1 in reply to:  description Changed 7 months ago by ryandesign (Ryan Schmidt)

Keywords: haspatch added

Replying to ryandesign:

  1. In post-destroot, replace the real compiler paths in _sysconfigdata.py with placeholders, and in post-activate, replace the placeholders with the current values from ${configure.cc} and ${configure.cxx}. I can't think of a reason why this wouldn't work. For Xcode 4.2, that would still end up being clang, which might be undesirable as in (2) above. We could special-case 10.6 and use llvm-gcc-4.2 there instead of clang. I am assuming that the build failure of python33 and later with llvm-gcc-4.2 is specific to the source code of python itself, and would not manifest itself when building most modules.

I've implemented this here: https://github.com/macports/macports-ports/pull/1773

comment:2 Changed 7 months ago by ryandesign (Ryan Schmidt)

You closed the PR, so I'll respond here:

Sysconfig is supposed to record the compiler that was used to build python. This would render it incorrect.

I understand that.

You'll notice it records a lot of other things like the configure args that were used.

Right. This PR doesn't change the CONFIG_ARGS line. It's possible there are other values the PR should not change, such as CC and CXX, and I'm happy to change the PR for whatever decision is reached in that regard. I do not know the distinction between the various variables.

Distutils does not blindly use that compiler to build things, it has code to choose a default compiler that exists (if one is not specified).

In some cases, yes, it definitely does use these compilers to build other things, resulting in the problems reported in the above-mentioned tickets.

Note: See TracTickets for help on using tickets.