Opened 3 years ago

Last modified 4 months ago

#50058 new enhancement

Executables installed by pip are not linked into macports path (and are not “select”able in any way).

Reported by: gallafent Owned by: stromnov (Andrew Stromnov)
Priority: Normal Milestone:
Component: ports Version:
Keywords: Cc: hmijail (Horacio Mijail Antón Quiles), SickTeddyBear, wladston (Wladston Ferreira Filho), kierans (Kieran Simpson), franzinc (franzinc)
Port: py-pip

Description

There's some prior discussion of this problem at https://github.com/pypa/pip/issues/113

In short: when I install a python package using pip, its executables are installed into the appropriately versioned python tree, but those executables are not symlinked to somewhere which is on my PATH.

In my example, I have installed macports' pip, and have port selected python35 and pip35. I then used pip to install the aws cli tools, sudo pip install aws. The aws binary is installed by pip at /opt/local/Library/Frameworks/Python.framework/Versions/3.5/bin/aws. So far so good. The problem is that that path is not on my PATH (of course … port select can't alter my path, only the individual symlinks in /opt/local/bin to the objects which are owned by that particular port for which port select was run).

I would naively expect at first glance that issuing a sudo port select pip pip35 would cause any binaries subsequently installed by that version of pip to be linked into my macports path /opt/local/bin. Of course that doean't happen with a standard pip. There follows the question of how to achieve this, since standard pip doesn't know it's supposed to do something about any binaries installed /after/ the port select operation, other than to install them in the appropriate place in the python heirarchy for the selected version of pip. This puts the question firmly in MacPorts' world, as per the latest comments on the github issue.

Beporter's comment (third from last in that github issue at the time of writing) echos my feeling here … I would expect that something installed with the currently selected version of pip would end up on my path, though I completely understand why it doesn't at present.

There's clearly some precedent for linking binaries to do with a specific python version directly in this way, since for example idle is linked thus on my installation, so the desired end result (I end up with a link /opt/local/bin/aws -> /opt/local/Library/Frameworks/Python.framework/Versions/3.5/bin/aws) would be consistent with this:

idle -> /opt/local/bin/idle3.5
idle2.7 -> /opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/idle2.7
idle3 -> /opt/local/bin/idle3.5
idle3.4 -> /opt/local/Library/Frameworks/Python.framework/Versions/3.4/bin/idle3.4
idle3.5 -> /opt/local/Library/Frameworks/Python.framework/Versions/3.5/bin/idle3.5

One way to achieve this (the “best” that I can think of so far) would be to modify MacPorts' pip so that when a given version of pip installs a binary it also links that into /opt/local/bin. This shouldn't be too hard to achieve but it's pretty invasive w.r.t. pip's behaviour. It would, though, solve the oddness seen by me and others leading to pip-installed executables not appearing on my path. The port select pip pipXX command would also need on each invocation either to invoke pip or examine pip's registry in order to unlink the executables which were installed by the version of pip being deselected, and link those owned by the version of pip being selected. Not rocket science, but a bit complicated.

The alternative would seem to be to provide stub ports of some sort which use pip as the installation method under the hood but which handle the symlinking by providing port select machinery for each such port. That feels like a maintenance drag though (keeping up with pip's repositories …)

(Footnote: I can see that there's an existing port py-awscli which should provide what I need for now specifically in my example aws case, though it's pretty outdated, but that doesn't negate the general point about pip and paths in this bug …)

Change History (13)

comment:1 Changed 3 years ago by gallafent

Apologies, submitted too soon, forgot to CC maintainer stromnov@…, and can't see a way to do that after the event! Likewise I can't change the port field from the nonexistent pip to something which does exist, i.e. py35-pip!

comment:2 Changed 3 years ago by mf2k (Frank Schima)

Owner: changed from macports-tickets@… to stromnov@…
Port: py-pip added; pip removed
Type: defectenhancement
Version: 2.3.4

comment:3 Changed 3 years ago by stromnov (Andrew Stromnov)

It's a rather bad practice to mix up several different package managers in one place (port and pip). If you, for example, manually install tornado using macports pip-3.5, then you can entirely break your macports installation due to dependency chain update (setuptools, certify and so on).

So, it's preferably to install custom packages into separate dirs, or install custom packages using macports build system.

comment:4 in reply to:  3 Changed 3 years ago by gallafent

Replying to stromnov@…:

So, it's preferably to install custom packages into separate dirs, or install custom packages using macports build system.

I guess the “correct” response to the absence of a given package in macports is to request that it be added then (e.g. I might want there to exist a package py35-awscli (purely theoretically ;), in which case I should request that new port. It seems a pity not to be able to use the pip infrastructure in a more synergistic way with macports, but I guess we are where we are!

This does ask the rather more serious question though: if it's bad practice to install packages in macports python using pip, why does macports provide a port of pip? That's a discussion for another day though!

comment:5 Changed 3 years ago by stromnov (Andrew Stromnov)

The same situation exists in any other *nix-like OS: http://askubuntu.com/questions/431780/apt-get-install-vs-pip-install

You can use pip to install custom packages in any other location than system-wide (using --user or --target options).

comment:6 Changed 3 years ago by hmijail (Horacio Mijail Antón Quiles)

The "you can entirely break your macports installation" comment was scary, so I'd like to make sure that I got it right:

One should NOT use pip to install packages on the macports directory tree (which is what a macports-installed pip would do by default!). The recommended procedure would be:

  • Install with macports your python distribution
  • if that python doesn't have its own standard virtualenv and pip (like Python 3.4 and later include venv and ensurepip), then install with macports the appropriate pip and virtualenv for that python
  • Prepare a "new place" to install pip packages (like a virtualenv)
  • Once into the "new place", use pip to install whatever

A possible, but risky exception, would be: macports not carrying a pip-available package that has to be installed globally. That package could then be installed through pip into the macports directory tree, without fear of causing a conflict - until the day that macports *does* carry that package, and then the risk of conflict appears. Right?

Is this documented somewhere? I'd say it should!

Last edited 3 years ago by hmijail (Horacio Mijail Antón Quiles) (previous) (diff)

comment:7 Changed 3 years ago by hmijail (Horacio Mijail Antón Quiles)

Cc: hmijail@… added

Cc Me!

comment:8 Changed 3 years ago by SickTeddyBear

Cc: amcgee@… added

Cc Me!

comment:9 Changed 2 years ago by wladston (Wladston Ferreira Filho)

Cc: wladston@… added

Cc Me!

comment:10 Changed 2 years ago by wladston (Wladston Ferreira Filho)

Cc: wlad@… added

Cc Me!

comment:11 Changed 14 months ago by kierans (Kieran Simpson)

Cc: kierans added

comment:12 Changed 13 months ago by franzinc (franzinc)

Cc: franzinc added

comment:13 Changed 4 months ago by cogentParadigm (Ali Gangji)

I am dynamically referencing the correct bin directory based on the currently active version of python using this line in my .bash_profile:

export PATH="$PATH:/opt/local/Library/Frameworks/Python.framework/Versions/$(stat -f %Y /opt/local/bin/python | sed 's/[^0-9\.]*//g')/bin"

This won't automatically update your current shell when changing the active version of python. You'll have to source .bash_profile or start a new shell.

Note: See TracTickets for help on using tickets.