python extension module builds missing -I/opt/local/include

[ned-deily (Ned Deily)]

When building third-party Python packages (i.e. those not in the Python standard library), the MacPorts configuration is causing C extension modules to be compiled without -I/opt/local/include but linked with -L/opt/local/lib. This definitely causes problems for packages that contain C modules that make use of OS X libraries that are shadowed in MacPorts. A good example are the openssl libraries, libssl and libcrypto, which are shipped with OS X and with newer versions provided in MacPorts.

The problem is caused by the port files only including -I/opt/local/include in CPPFLAGS but not in CFLAGS. For Python that is not enough because the value of CPPFLAGS does not influence extension modules built by Python's Distutils. There are a few long-open issues about this behavior, for example ​http://bugs.python.org/issue4010, but that's the way things have worked for a long time and continue to.

If you examine the logs of various py* ports which include C extension modules, you can see the difference between the compile and link compiler calls. This could affect ports in subtle ways if there is a library shadowing situation. The py*-openssl port seems to be vulnerable though I have not verified that it causes problems.

This issue came up while helping debug a segfault a user was seeing with a third-party Python package when built with a MacPorts Python. (​https://bitbucket.org/tiran/backports.pbkdf2/issue/1/running-benchmarkpy-on-os-x-1085-using). It was determined that the root cause was this issue in that the package was being compiled with system-provided openssl include files but linked with MacPorts openssl libs. Here is a simplified demo of the problem:

$ /opt/local/bin/python2.7 -c 'import sysconfig;print(sysconfig.get_config_var("CFLAGS"))'
-fno-strict-aliasing -fno-common -dynamic   -pipe -Os -fwrapv   -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes
$ /opt/local/bin/python2.7 -c 'import sysconfig;print(sysconfig.get_config_var("CPPFLAGS"))'
-I. -IInclude -I./Include -I/opt/local/include -I/opt/local/include/db46
$ /opt/local/bin/python2.7 -c 'import sysconfig;print(sysconfig.get_config_var("LDFLAGS"))'
  -isysroot / -L/opt/local/lib -Wl,-headerpad_max_install_names -L/opt/local/lib/db46

$ /opt/local/bin/pip-2.7 install https://bitbucket.org/tiran/backports.pbkdf2/get/a6791ee674c7.zip
[...]
    building 'backports.pbkdf2._pbkdf2' extension
    /usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -pipe -Os -fwrapv -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c backports/pbkdf2/_pbkdf2.c -o build/temp.macosx-10.8-x86_64-2.7/backports/pbkdf2/_pbkdf2.o
    backports/pbkdf2/_pbkdf2.c:22:15: warning: 'ERR_peek_last_error' is deprecated [-Wdeprecated-declarations]
        errcode = ERR_peek_last_error();
                  ^
    /usr/include/openssl/err.h:274:15: note: 'ERR_peek_last_error' declared here
    unsigned long ERR_peek_last_error(void) DEPRECATED_IN_MAC_OS_X_VERSION_10_7_AND_LATER;
                  ^
    backports/pbkdf2/_pbkdf2.c:27:5: warning: 'ERR_clear_error' is deprecated [-Wdeprecated-declarations]
        ERR_clear_error();
        ^
    /usr/include/openssl/err.h:278:6: note: 'ERR_clear_error' declared here
    void ERR_clear_error(void ) DEPRECATED_IN_MAC_OS_X_VERSION_10_7_AND_LATER;
         ^
[...]
    31 warnings generated.
    /usr/bin/clang -bundle -undefined dynamic_lookup -isysroot / -L/opt/local/lib -Wl,-headerpad_max_install_names -L/opt/local/lib/db46 build/temp.macosx-10.8-x86_64-2.7/backports/pbkdf2/_pbkdf2.o -lcrypto -o build/lib.macosx-10.8-x86_64-2.7/backports/pbkdf2/_pbkdf2.so

[...]
$ /opt/local/bin/pip-2.7 uninstall backports.pbkdf2

$ export CFLAGS="$(/opt/local/bin/python2.7 -c 'import sysconfig;print(sysconfig.get_config_var("CFLAGS"))') -I/opt/local/include"
$ echo $CFLAGS
-fno-strict-aliasing -fno-common -dynamic -pipe -Os -fwrapv -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/local/include
$ /opt/local/bin/pip-2.7 install --user https://bitbucket.org/tiran/backports.pbkdf2/get/a6791ee674c7.zip
[...]
    building 'backports.pbkdf2._pbkdf2' extension
    /usr/bin/clang -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fno-strict-aliasing -fno-common -dynamic -pipe -Os -fwrapv -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/local/include -I/opt/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c backports/pbkdf2/_pbkdf2.c -o build/temp.macosx-10.8-x86_64-2.7/backports/pbkdf2/_pbkdf2.o
    /usr/bin/clang -bundle -undefined dynamic_lookup -isysroot / -L/opt/local/lib -Wl,-headerpad_max_install_names -L/opt/local/lib/db46 -fno-strict-aliasing -fno-common -dynamic -pipe -Os -fwrapv -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/local/include build/temp.macosx-10.8-x86_64-2.7/backports/pbkdf2/_pbkdf2.o -lcrypto -o build/lib.macosx-10.8-x86_64-2.7/backports/pbkdf2/_pbkdf2.so

I have not looked at the older, non-current python ports to see if they are configured and behave the same but chances are that they do. It is also quite possible that other py*-* ports have unknowingly run into this problem in the past and have worked around it with patches to the package port rather than fixing the root problem in the python* port.

[ci42]

Cc Me!

[vergus@…]

I was just about to submit a bug against pyyaml for not finding yaml.h, but this bug explains the issue.

[r.fagan@…]

can somebody please fix this already.

[mf2k (Frank Schima)]

Can someone please provide a patch?

[chris.jerdonek@…]

I also ran into this issue when installing PyYAML using pip. It was not finding libyaml's /opt/local/include/yaml.h installed by MacPorts.

[larryv (Lawrence Velázquez)]

Cc Me!

[chris.jerdonek@…]

Cc Me!

[petrrr]

Cc Me!