python27 @2.7.3_0 Includes system paths in sys.path

[jameskyle@…]

Macports installation of python 2.7 includes system paths in sys.path. This causes mucho problemo with module lookups.

Example:

[, '/Library/Python/2.7/site-packages/pip-1.1-py2.7.egg', '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip', '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7', '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin', '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac', '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages', '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk', '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old', '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/readline', '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload', '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages', '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/aeosa', '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg-info', '/Library/Python/2.7/site-packages']

The behavior was observed on a fresh install of macports with only a single package installed (a python package) and duplicated across multiple others.

[jameskyle@…]

Ah, one commonality on all systems that I just realized.

They all had pip installed on the system level.

I'll poke around and add to the ticket as needed.

[ned-deily (Ned Deily)]

Adding /Library/Python/2.7/site-packages is an upstream "feature" as of Python 2.7 for all OS X framework builds; see ​http://bugs.python.org/issue4865. I must admit that I'm not convinced it is a good idea. Feel free to open an issue about on the Python bug tracker.

[eborisch (Eric A. Borisch)]

FWIW, this isn't the case in my install:

MacPro:~$ /opt/local/bin/python2.7
Python 2.7.3 (default, Apr 13 2012, 10:00:30) 
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.9)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> for n in sys.path:print(n)
... 

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/readline
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PIL
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyObjC
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/gtk-2.0
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg-info
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/wx-2.8-mac-unicode

10.6.8 / 4.0.2

[mf2k (Frank Schima)]

I get the same result as eborisch. Isn't this just an issue with the py-pip port?

[ned-deily (Ned Deily)]

It's only an issue on 10.7 and beyond since Apple first supplied a system Python 2.7 with 10.7. /Library/Python/2.7 won't exist on earlier systems.

[eborisch (Eric A. Borisch)]

Cc Me!

[jameskyle@…]

I'm not so sure this is just a pip thing. I did the following this morning:

• Uninstalled all python modules
• Uninstalled all packages depending on python
• Uninstalled python27 package
• Verified the /opt/local/Frameworks/Python directory was removed (indicating no left over cruft)

Then I

• Reinstalled python27 and nothing else

I then dropped into the python shell via:

% /opt/local/bin/python2.7

And printed sys.path, it's the exact same path as above.

I also did a sudo -u root -i for a clean $HOME/SHELL and inspected the path there, same path.

[ned-deily (Ned Deily)]

Right, it has nothing to do with pip. As I explained above, it's a change in Python 2.7. If you want the behavior changed, you should open an issue on the Python bug tracker.

[jameskyle@…]

Replying to nad@…:

Right, it has nothing to do with pip. As I explained above, it's a change in Python 2.7. If you want the behavior changed, you should open an issue on the Python bug tracker.

I'm not so sure this is an upstream issue.

When I compile my own python 2.7 on the same system with

./configure --prefix=/Users/jkyle/Downloads/Python && make -j8 && make install

And none of the non-install paths are present.

sys.path

[, '/Users/jkyle/Downloads/Python/lib/python27.zip', '/Users/jkyle/Downloads/Python/lib/python2.7', '/Users/jkyle/Downloads/Python/lib/python2.7/plat-darwin', '/Users/jkyle/Downloads/Python/lib/python2.7/plat-mac', '/Users/jkyle/Downloads/Python/lib/python2.7/plat-mac/lib-scriptpackages', '/Users/jkyle/Downloads/Python/lib/python2.7/lib-tk', '/Users/jkyle/Downloads/Python/lib/python2.7/lib-old', '/Users/jkyle/Downloads/Python/lib/python2.7/lib-dynload', '/Users/jkyle/Downloads/Python/lib/python2.7/site-packages']

I suspect the issue may lie in the macports python build itself. I'll look into this and update as needed.

[ned-deily (Ned Deily)]

Trust me, it's an upstream issue (I am part of upstream). You'll see the same behavior using a python.org OS X installer for 2.7. The reason you didn't see it in your test is that it only applies to OS X framework builds, which you did not select. And only on 10.7 and beyond. The code that does this is in Lib/site.py in getsitepackages, currently around line 313 or so (​http://hg.python.org/cpython/file/2.7/Lib/site.py).

[jameskyle@…]

Thanks for the explanation.

I'll file an upstream bug on this. Pulling in system paths for separate installs would seem to be a bad idea.

[kyounger@…]

Any legitimate hacks around this?

(Sorry if this might seem obvious, I'm generally pretty new to python and just ran into this issue tonight)

[kyounger@…]

Cc Me!

[ned-deily (Ned Deily)]

An upstream bug was filed as ​http://bugs.python.org/issue15048 and was rejected for a 2.7.x or 3.2.x maintenance release as breaking backward compatibility. If you have specific cases where this behavior causes problems and you can help to make a case that the current behavior should be changed, you could add them to the issue and ask that it be reopened. Otherwise, you could: try to avoid having duplicate distributions/packages installed for both the system Python 2.7 and MacPorts; or you could manually manipulate sys.path within your Python program; or Python's behavior could be patched in site.py by you or by the maintainer of the MacPorts Python ports.

[jmroot (Joshua Root)]

I vote we patch this behaviour out of the port.

[petrrr]

+1 for patching this! This is really kind of unexpected behavior. Especially because I always thought, this is why Python packages handling with MacPort would be clean and therefore preferable. But if I cannot rely on being independent from System's package installation this breaks.

I preferable, this could be configurable with a variant, but probably should default to activate the patch. As otherwise we are back here ...

[petrrr]

Cc Me!

[larryv (Lawrence Velázquez)]

Cc Me!

[smithsp (Sterling Smith)]

+1 for a patch, which is the default.

[petrrr]

By history this ticket is probably a duplicate of #31891. However, this ticket evidences the problem in a much clearer way.

So if we agree that the problems described here and in #31891 are basically equivalent or have the same cause, we may want to close #31891, instead of this one and keep tracking the problem here.

[petrrr]

Cc Me!

[petrrr]

Adding CC addresses from #31891 here.

[petrrr]

Adding reporters address of #31891 to CC as well

[larryv (Lawrence Velázquez)]

Fixed for python27 and python31 through python34 in r126840.