Opened 4 months ago

Closed 3 months ago

Last modified 3 months ago

#57829 closed defect (fixed)

py27-scipy @1.2.0+gfortran: ImportError: dlopen: Symbol not found: _cbbcsd_

Reported by: renzresearch Owned by: seanfarley (Sean Farley)
Priority: Normal Milestone:
Component: ports Version: 2.5.4
Keywords: Cc: michaelld (Michael Dickens), dershow, jsalort (Julien Salort), EJFielding (Eric Fielding), ATL-Flaneur (Andreas Yankopolus), DanyPPage, majoc-at-astro (majoc-at-astro)
Port: py-scipy

Description

"from scipy import optimize" causes an ImportError. "import scipy.optimize" results in the same ImportError. "import scipy" is fine. But it does not import the sub-module optimize.

% python
Python 2.7.15 (default, Oct  1 2018, 15:59:56) 
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.11.45.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from scipy import optimize
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/optimize/__init__.py", line 387, in <module>
    from ._minimize import *
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/optimize/_minimize.py", line 26, in <module>
    from ._trustregion_dogleg import _minimize_dogleg
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/optimize/_trustregion_dogleg.py", line 5, in <module>
    import scipy.linalg
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/linalg/__init__.py", line 207, in <module>
    from ._decomp_update import *
  File "_decomp_update.pyx", line 1, in init scipy.linalg._decomp_update
ImportError: dlopen(/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/linalg/cython_lapack.so, 2): Symbol not found: _cbbcsd_
  Referenced from: /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/linalg/cython_lapack.so
  Expected in: flat namespace
 in /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/linalg/cython_lapack.so

Change History (31)

comment:1 Changed 4 months ago by ryandesign (Ryan Schmidt)

Cc: michaelld added
Owner: set to seanfarley
Status: newassigned
Summary: py27-scipy 1.2.0 ImportError: dlopen: Symbol not found: _cbbcsd_py27-scipy @1.2.0+gfortran: ImportError: dlopen: Symbol not found: _cbbcsd_

I can confirm this problem on High Sierra with py27-scipy @1.2.0_0+gfortran. The problem was not present with py27-scipy @1.1.0_1+gfortran.

comment:2 Changed 4 months ago by lockhart (Thomas Lockhart)

Can also confirm this issue on Sierra, affecting the scipy.linalg submodule. I've uninstalled and reinstalled every package that seems to be related but this did not help:

sudo port install py36-cython py36-numpy py36-matplotlib py36-bokeh py36-bottleneck py36-tables py36-numba py36-astropy py36-numexpr py36-scipy py36-pandas py36-scikit-learn

sudo port install opencv +python36

Last edited 4 months ago by lockhart (Thomas Lockhart) (previous) (diff)

comment:3 Changed 4 months ago by kencu (Ken)

See this thread <https://mail.python.org/pipermail/scipy-dev/2018-June/022856.html> when this issue turned up after a commit to master that has now been incorporated into 1.2.0.

In that thread, it looked like the issue is picking up the wrong libraries.

> The build logs show that it's picking up /usr/lib/libblas.* and
> /usr/lib/liblapack.* and using those. Apparently, they are missing
> symbols.
>

comment:4 Changed 4 months ago by dershow

This bug also breaks at least one other port, but probably others. I just upgraded to 10.14, and followed the migration instructions. Then, in python, I tried to do: import seaborn as sns and I got the same error as above. It seems that seaborn (py27-seaborn etc) uses scipy trying to use it shows the same issue.

comment:5 Changed 4 months ago by dershow

Cc: dershow added

comment:6 Changed 4 months ago by duncanmmacleod (Duncan Macleod)

You can resolve things be rebuilding with the +openblas variant, I think. Should that now be the default operation?

comment:7 Changed 4 months ago by NeilBowles

Yes, this does the trick and solves all the other problems with the updated scipy install - I was having problems with py37, spyder etc. Thanks for pointing this out, installing scipy with the +openblas variant solves the problem.

comment:8 Changed 4 months ago by mf2k (Frank Schima)

Port: py-scipy added; py27-scipy removed

comment:9 Changed 3 months ago by jsalort (Julien Salort)

Cc: jsalort added

comment:10 Changed 3 months ago by EJFielding (Eric Fielding)

Cc: EJFielding added

comment:11 Changed 3 months ago by EJFielding (Eric Fielding)

I ran into this same bug when I upgraded my py36-scipy to @1.2.0_0+gfortran. It used the prebuilt binary "py36-scipy-1.2.0_0+gfortran.darwin_17.x86_64.tbz2" for the install. I got the missing symbol error when I tried to run a Python3 scipy script. The same script worked when I activated the older @1.1.0_1+gfortran port. Trying the workaround to install with +openblas variant and, of course, it has to rebuild the whole py36-scipy and takes quite a while. I can confirm that py36-scipy @1.2.0_0+gfortran+openblas works.

comment:12 Changed 3 months ago by mf2k (Frank Schima)

Cc: ATL-Flaneur added

Has duplicate #57880.

comment:13 Changed 3 months ago by seanfarley (Sean Farley)

I just read the thread on the mailing list and it's unfortunate they dropped support for Apple's Accelerate framework (a big reason to use macs in academia). Well, if support has been dropped then it's dropped. But this is a much larger problem in general. So many things depend on scipy that we should decide if we want to revamp the port tree to use openblas by default. Having Apple's Accelerate mixed with openblas will lead to linking issues. And we can't depend on variants so I don't think +openblas should remain a variant for scipy.

I absolutely don't have the time to work on revamping this. Does anyone else have any ideas?

comment:14 Changed 3 months ago by mf2k (Frank Schima)

Cc: DanyPPage added

Has duplicate #57907.

comment:15 Changed 3 months ago by DanyPPage

How do I rebuild scipy with +openblast ?

comment:16 in reply to:  15 Changed 3 months ago by EJFielding (Eric Fielding)

Replying to DanyPPage:

How do I rebuild scipy with +openblast ?

install scipy port again with the extra variant (not it is +openblas, not +openblast), of course you might have a different Python version, so adjust that:

port install py36-scipy +gfortran +openblas

comment:17 Changed 3 months ago by kencu (Ken)

I don't use this but suggest we don't just leave it broken, so IMHO

  1. revert

or

  1. use openblas by default

Are there any other choices to be considered? If no, go with 2 and take it from there.

comment:18 Changed 3 months ago by DanyPPage

It worked and solved my problem: scipy seems to work properly. THANX !

comment:19 Changed 3 months ago by ATL-Flaneur (Andreas Yankopolus)

The present default config that fails to install seems less than optimal. Perhaps make openblas the default (option 2 above), with Apple's Accelerate being a configurable option.

comment:20 in reply to:  17 Changed 3 months ago by reneeotten (Renee Otten)

Replying to kencu:

I don't use this but suggest we don't just leave it broken, so IMHO

  1. revert

or

  1. use openblas by default

Are there any other choices to be considered? If no, go with 2 and take it from there.

there are similiar +gfortran issues with py-numpy (see ticket-56954, which are not really figured out yet as far as I can tell. I think changing the default (compiler) variants, should be done in an consistent manner for at least these two ports (and perhaps others in the Python scientific stack).

comment:21 Changed 3 months ago by michaelld (Michael Dickens)

I'll make (2) the default. Looking at the Portfile right now ...

comment:22 Changed 3 months ago by michaelld (Michael Dickens)

Resolution: fixed
Status: assignedclosed

In b10ed01c11fe815974c7d3acd766de3fcae93b8b/macports-ports (master):

py-scipy: Make +openblas a default variant

... at least temporarily, to overcome issues with Apple's /usr/lib/libblas.* and /usr/lib/liblapack.* missing a symbol.

Closes: #57829

comment:23 Changed 3 months ago by michaelld (Michael Dickens)

I made +openblas a default variant. I don't know if this change will take when doing an update from a prior version without this variant being enabled. But it will take care of new/fresh installs, which is progress.

comment:24 Changed 3 months ago by seanfarley (Sean Farley)

@michaelld it's a bit more difficult than that, though. I remember there was a package that added missing symbols to accelerate (veclibFort) that might be the way to quickly patch over this. Otherwise, as I mentioned above, they're be a lot of mismatched symbols. Perhaps Apple should just stop shipping /usr/lib/lib{blas,lapack}* if they're broken.

comment:25 Changed 3 months ago by michaelld (Michael Dickens)

hmmm ... OK I'll check out "veclibFort" & see if it helps. I'd prefer to not use +openblas as a default, but it does work (I've verified; it's actually my default, which is why I committed this in the first place, since it "just worked" for me LOL).

comment:26 Changed 3 months ago by michaelld (Michael Dickens)

I will note that when doing an "update", the new default variant is not selected. One has to do "install" to get it. Sigh.

comment:27 Changed 3 months ago by seanfarley (Sean Farley)

I agree with your change by the way; just making sure you're aware of the potential problems users might have.

comment:28 in reply to:  26 Changed 3 months ago by mf2k (Frank Schima)

Replying to michaelld:

I will note that when doing an "update", the new default variant is not selected. One has to do "install" to get it. Sigh.

Not entirely true. One can also do this:

sudo port upgrade --enforce-variants py37-scipy

comment:29 Changed 3 months ago by nwautomator (That Guy --^)

This does not work on a clean Macports (i.e. no installed ports) on MacOS Mojave 10.14.2. I'm getting an error that the clang37 port is not supported on Mojave.

comment:30 Changed 3 months ago by michaelld (Michael Dickens)

Well the clang37 port isn't supported on 10.14. No idea why it's trying to do that. port rdeps py27-scipy | grep -i clang returns empty, so SciPy doesn't by default depend on any Clang on 10.14 (which is what I'm running the query on). Maybe you have some default variants set in a config file?

comment:31 Changed 3 months ago by majoc-at-astro (majoc-at-astro)

Cc: majoc-at-astro added
Note: See TracTickets for help on using tickets.