py27-numpy @1.7.1_0: import fails due to missing symbol

[jdgleeson]

I've used py27-numpy @1.7.1_0+atlas+gcc47 since mid April with no problems. After I forced an upgrade a couple days ago I get this traceback when I import numpy:

$ python
Python 2.7.5 (default, May 20 2013, 21:30:50) 
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt47/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/__init__.py", line 137, in <module>
    import add_newdocs
  File "/opt47/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/add_newdocs.py", line 9, in <module>
    from numpy.lib import add_newdoc
  File "/opt47/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/lib/__init__.py", line 4, in <module>
    from type_check import *
  File "/opt47/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/lib/type_check.py", line 8, in <module>
    import numpy.core.numeric as _nx
  File "/opt47/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/__init__.py", line 5, in <module>
    import multiarray
ImportError: dlopen(/opt47/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/multiarray.so, 2): Symbol not found: restGPRx
  Referenced from: /opt47/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/multiarray.so
  Expected in: dynamic lookup

This library has the missing symbol:

$ nm /opt47/local/lib/gcc47/libgcc_s.1.dylib | grep restGPRx
00005f98 t restGPRx

but multiarray.so does not link with it:

jdgleeson@Archimedes:~$ otool -L /opt47/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/multiarray.so
/opt47/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/multiarray.so:
	/usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 111.1.7)

I noted in the log file that gcc-4.0 is used to create the extension libraries (but the source is compiled with gcc-mp-4.7). I have the port -d output from the April build of numpy, and it confirms that the extension libraries were built with gcc-mp-4.7, not gcc-4.0, at that time.

I've attached the log file from the recent build of numpy.

[jdgleeson]

Main log of py27-numpy 1.7.1_0+atlas+gcc47 (sudo port -kdn upgrade --force) May 21

[jdgleeson]

Since numpy has not changed since it worked for me in April, I suspected a change in another port was to blame. Here is a list of numpy dependencies I've identified that I've upgraded since April 12 (when I first built 1.7.1_0):

atlas           3.10.1_3+gcc47  3.10.1_5+gcc47
db46            4.6.21_7        4.6.21_8
gcc47           4.7.2_2         4.7.3_0
glpk            4.47_0          4.48_0
libstdcxx       4.7.2_2         4.8.0_0
perl5.12        5.12.4_1        5.12.4_2
ppl             1.0_0           1.0_1
py27-distribute 0.6.36_0        0.6.39_0
python27        2.7.3_1         2.7.5_0
sqlite3         3.7.16.1_0      3.7.17_0
zlib            1.2.7_0         1.2.8_0

I thought changes in atlas and distribute were most likely to affect the way numpy is build, so I force-deactivated them and activated the versions that were active on April 12. When I rebuilt numpy the import still failed with the same error. But there is always the possibility that I did not do this correctly, since I've never tried it before.

[cooljeanius (Eric Gallager)]

Cc Me!

[jdgleeson]

I'm getting closer to figuring this out. When python 2.7.5 is active gcc-4.0 is used to created extension libraries for numpy. When python 2.7.3 is active gcc-mp-4.7 is used to created extension libraries for numpy.

$ sudo port activate python27 @2.7.5_0
$ sudo port clean py27-numpy
$ sudo port -kd build py27-numpy > termout16.txt 2>&1
$ grep -m 2 bundle termout16.txt
/usr/bin/gcc-4.0 -bundle -undefined dynamic_lookup -L/opt47/local/lib -L/opt47/local/lib/db46 -m32 -m32 -m32 -m32 build/temp.macosx-10.5-ppc-2.7/numpy/core/src/dummymodule.o -Lbuild/temp.macosx-10.5-ppc-2.7 -o build/lib.macosx-10.5-ppc-2.7/numpy/core/_dummy.so
/usr/bin/gcc-4.0 -bundle -undefined dynamic_lookup -L/opt47/local/lib -L/opt47/local/lib/db46 -m32 -m32 -m32 -m32 build/temp.macosx-10.5-ppc-2.7/numpy/core/src/multiarray/multiarraymodule_onefile.o -Lbuild/temp.macosx-10.5-ppc-2.7 -lnpymath -lnpysort -o build/lib.macosx-10.5-ppc-2.7/numpy/core/multiarray.so

$ sudo port activate python27 @2.7.3_1       
$ sudo port clean py27-numpy
$ sudo port -kd build py27-numpy > termout17.txt 2>&1
$ grep -m 2 bundle termout17.txt
/opt47/local/bin/gcc-mp-4.7 -bundle -undefined dynamic_lookup -L/opt47/local/lib -m32 -m32 -m32 build/temp.macosx-10.5-ppc-2.7/numpy/core/src/dummymodule.o -Lbuild/temp.macosx-10.5-ppc-2.7 -o build/lib.macosx-10.5-ppc-2.7/numpy/core/_dummy.so
/opt47/local/bin/gcc-mp-4.7 -bundle -undefined dynamic_lookup -L/opt47/local/lib -m32 -m32 -m32 build/temp.macosx-10.5-ppc-2.7/numpy/core/src/multiarray/multiarraymodule_onefile.o -Lbuild/temp.macosx-10.5-ppc-2.7 -lnpymath -lnpysort -o build/lib.macosx-10.5-ppc-2.7/numpy/core/multiarray.so

[jdgleeson]

The error first occurs with this CPython revision: ​http://hg.python.org/cpython/rev/1d061f83a6bf (I am grateful for hg bisect). Now I might have better luck creating a unit test.

[ned-deily (Ned Deily)]

Looking at the log, it appears that, for each C extension module, /opt47/local/bin/gcc-mp-4.7 is being used by Distutils for the compile step but /usr/bin/gcc-4.0 is used for the link step (-bundle). Normally, the same compiler should be used for both. By default, Distutils tries to use the same compiler and options as was used to build Python itself, unless overridden. In particular, the CC env variable is used to override the compiler in the CC config var and the LDSHARED env var is used to supply the compiler and link options from the LDSHARED config var. (You can inspect Python's configuration variables with sysconfig.get_config_var('CC') or ('LDSHARED').) With Python 2.7.3, setting a CC environment variable to override the compiler caused Distutils to update the compiler string in LDSHARED as well. Unfortunately, in Python 2.7.4 (and 2.7.5), the update of LDSHARED when CC is specified no longer happens; if you use CC env variable to supply a different compiler, you also need to inspect the LDSHARED config var and supply a complete modified string in an LDSHARED env variable. I've opened an upstream issue to restore the inadvertently changed behavior (​http://bugs.python.org/issue18080) and attached a patch that should fix the problem for 2.7.5. I have not looked into the Numpy build process nor the MacPorts changes to see if this is the same problem but reading the log here rang a bell. You could try to apply the patch to the MacPorts Python 2.7.5 and see if it fixes the problem. There were other rather extensive changes in Distutils to better support Xcode 4 for Python 2.7.4 so it is possible that something else in them may have affected Numpy builds.

[jdgleeson]

Thanks, Ned! Your patch does indeed fix the numpy build problem.

I've attached Ned's patch (---/+++ paths edited to match Macports patch convention) and the associated patch for the python27 Portfile.

Can this ticket be transferred from py27-numpy to the python27 port, or do I need to open a new ticket and link to this one?

[ned-deily (Ned Deily)]

Good news and I'm sorry for the trouble it caused you. FWIW, I'm in the process of applying the patch upstream. It should be official in a couple of hours.

[reid@…]

Cc Me!

[petrrr]

Cc Me!

[jyrkiwahlstedt]

Currently seems to be ok (with 2.7.6, 1.8.0)