Opened 7 years ago

Closed 23 months ago

#52448 closed defect (wontfix)

python35 @3.5.2 missing ensurepip?

Reported by: hmijail (Horacio Mijail Antón Quiles) Owned by: jmroot (Joshua Root)
Priority: Normal Milestone:
Component: ports Version: 2.3.4
Keywords: Cc: ned-deily (Ned Deily), cooljeanius (Eric Gallager)
Port: python35

Description

$ python3.5 -m venv myvenv  
Error: Command '['/Users/mija/myvenv/bin/python3.5', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1

This worked some months ago, and keeps working currently in Homebrew's python3, which is also python 3.5.2.

MacOS Sierra 10.12 (16A323), XCode Version 8.0 (8A218a)

Change History (37)

comment:1 Changed 7 years ago by mf2k (Frank Schima)

Cc: jwa@… removed
Owner: changed from macports-tickets@… to jwa@…

comment:2 in reply to:  description Changed 7 years ago by larryv (Lawrence Velázquez)

Replying to hmijail@…:

$ python3.5 -m venv myvenv  
Error: Command '['/Users/mija/myvenv/bin/python3.5', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1

The command path suggests you’re using the interpreter from a venv. Is that what you intended? Your command works for me verbatim, using MacPorts’ Python explicitly.

% port installed python35
The following ports are currently installed:
  python35 @3.5.2_0+universal (active)
% cd /private/tmp
% /opt/local/bin/python3.5 -m venv myvenv
% ls myvenv
bin/		include/	lib/		pyvenv.cfg
%

comment:3 Changed 7 years ago by hmijail (Horacio Mijail Antón Quiles)

I think that this is not what is happening.

1:~$ mkdir test
2:~$ cd test
3:~/test$ python3.5 -m venv venv
Error: Command '['/Users/mija/test/venv/bin/python3.5', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1

My understanding is that venv creates the virtual environment and immediately goes into it and installs ensurepip. And I guess that in this last step is where the ensurepip installation is failing.

In fact, adding --without-pip it works.

$ python3.5 -m venv --without-pip venv2
$

comment:4 Changed 7 years ago by hmijail (Horacio Mijail Antón Quiles)

Just to make sure, I just uninstalled the python35 port and installed it again (I already tried that a couple of days ago, but who knows).

The problem is still here.

comment:5 in reply to:  4 Changed 7 years ago by larryv (Lawrence Velázquez)

I suspect there is an issue with your configuration. I am having no trouble with this at all.

Are you sure python3.5 is resolving to MacPorts’ Python? What does this return?

$ command -v python3.5

What happens when you create a venv the recommended way?

$ cd /private/tmp && /opt/local/bin/pyvenv-3.5 testenv

comment:6 Changed 7 years ago by hmijail (Horacio Mijail Antón Quiles)

1:~$ command -v python3.5
/opt/local/bin/python3.5
2:~$ cd /private/tmp && /opt/local/bin/pyvenv-3.5 testenv
Error: Command '['/private/tmp/testenv/bin/python3.5', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1
3:/private/tmp$

I actually don't discard some local configuration problem, since I'm seeing very strange breakage with pyinvoke installed with pip3. However, I don't see where the breakage can come from, and I still didn't have the time to dig into that.

I don't usually work with Python, by the way.

Last edited 7 years ago by hmijail (Horacio Mijail Antón Quiles) (previous) (diff)

comment:7 in reply to:  6 Changed 7 years ago by larryv (Lawrence Velázquez)

Replying to hmijail@…:

I don't usually work with Python, by the way.

Neither do I :)

Let’s try something. Do you see any output if you run that command manually?

$ cd /private/tmp && /private/tmp/testenv/bin/python3.5 -Im ensurepip --upgrade --default-pip; echo $?
Version 0, edited 7 years ago by larryv (Lawrence Velázquez) (next)

comment:8 Changed 7 years ago by hmijail (Horacio Mijail Antón Quiles)

(I just saw your request when I was about to post this; will do it in a second)

Just to make sure that I wasn't imagining things, I just uninstalled the python35 port, installed Homebrew's python3 (Python 3.5.2, formula version3.5.2_2), and it works OK.

10:/private/tmp$ /usr/local/bin/pyvenv-3.5 testenv2
11:/private/tmp$

In case it matters: I upgraded to macOS Sierra a few days ago, and I followed the MacPorts migration instructions (delete everything, install new MacPorts). I didn't do anything special with brew, since I didn't see any instructions; and in fact it all mostly seems to work, apart from some permissions problems.

Last edited 7 years ago by hmijail (Horacio Mijail Antón Quiles) (previous) (diff)

comment:9 Changed 7 years ago by hmijail (Horacio Mijail Antón Quiles)

(Regarding the migration, in fact I remember now that after deleting everything following the migration instructions, there were still a lot of symlinks under /opt. I didn't delete them manually, assuming they would get overwritten as needed and shouldn't be a problem...)

3:/private/tmp$ command -v python3.5
/opt/local/bin/python3.5
4:/private/tmp$ cd /private/tmp/testenv && /private/tmp/testenv/bin/python3.5 -Im ensurepip --upgrade --default-pip; echo $?
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ensurepip/__main__.py", line 4, in <module>
    ensurepip._main()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ensurepip/__init__.py", line 209, in _main
    default_pip=args.default_pip,
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ensurepip/__init__.py", line 116, in bootstrap
    _run_pip(args + [p[0] for p in _PROJECTS], additional_paths)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ensurepip/__init__.py", line 41, in _run_pip
    pip.main(args)
  File "/var/folders/2d/1kly9t3s23bclpxgs6l342vc0000gs/T/tmplhya0ke0/pip-8.1.1-py2.py3-none-any.whl/pip/__init__.py", line 215, in main
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/locale.py", line 594, in setlocale
    return _setlocale(category, locale)
locale.Error: unsupported locale setting
1

OMFG, locales?

I just recently moved into iTerm2, which has its own locale settings that I didn't explore yet. And it's known that Python has problems with non-GNU locales https://bugs.python.org/issue18378 , so maybe that's it.

(though that doesn't explain the problems with pyinvoke...)

comment:10 Changed 7 years ago by hmijail (Horacio Mijail Antón Quiles)

Well, I'm running Homebrew in the same iTerm2, so that can't be it all.

comment:11 Changed 7 years ago by larryv (Lawrence Velázquez)

Cc: jmr@… added

comment:12 Changed 7 years ago by hmijail (Horacio Mijail Antón Quiles)

For reference: the locale settings I have in iTerm are these:

$ locale
LANG=
LC_COLLATE="C"
LC_CTYPE="UTF-8"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=

The ones I have in Terminal are:

$ locale
LANG=
LC_COLLATE="C"
LC_CTYPE="C"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=

Anyway, in Terminal the failures are the same as in iTerm.

comment:13 Changed 7 years ago by jmroot (Joshua Root)

Not really sure why I'm on Cc here, but…

LC_CTYPE="UTF-8"

Pretty sure that's invalid. You can only use one of the values listed by locale -a. I vaguely recall some other bug where unvalidated environment variables being passed straight to setlocale caused problems (meld?).

Also, it looks like this may have been fixed in pip 8.1.2: https://github.com/pypa/pip/issues/3575

comment:14 Changed 7 years ago by hmijail (Horacio Mijail Antón Quiles)

I didn't set the locales like that. Terminal.app sets them like that when the checkbox [Preferences/Profiles/Advanced/Set locale environment variables on startup] is unchecked. iTerm sets them like that when the checkbox [Preferences/Profiles/Terminal/Set locale variables automatically] is checked. Meaning, that it is some kind of "standard", not some configuration I made myself. According to the python bug report I posted, this seems rather to be a question of BSD vs GNU.

Anyway, in both configurations, Homebrew's Python works and MacPorts' doesn't.

Last edited 7 years ago by hmijail (Horacio Mijail Antón Quiles) (previous) (diff)

comment:15 in reply to:  13 Changed 7 years ago by larryv (Lawrence Velázquez)

Replying to jmr@…:

Not really sure why I'm on Cc here, but…

Domain expertise? :)

comment:16 in reply to:  14 ; Changed 7 years ago by larryv (Lawrence Velázquez)

Replying to hmijail@…:

I didn't set the locales like that. Terminal.app sets them like that when the checkbox [Preferences/Profiles/Advanced/Set locale environment variables on startup] is unchecked. iTerm sets them like that when the checkbox [Preferences/Profiles/Terminal/Set locale variables automatically] is checked. Meaning, that it is some kind of "standard", not some configuration I made myself. According to the python bug report I posted, this seems rather to be a question of BSD vs GNU.

Have you fiddled with your Language & Region settings?

Anyway, in both configurations, Homebrew's Python works and MacPorts' doesn't.

Looks like they grab pip 8.1.2 separately.

comment:17 Changed 7 years ago by jmroot (Joshua Root)

Yeah, this is all laid out in the upstream bug linked in comment:9. Apple does something different to everyone else (but still conforms to the spec), python doesn't handle it well. The bundled version of pip in python35 doesn't have a workaround for the problem.

The workaround is easy, set your locale environment variables to something that python likes.

comment:18 Changed 7 years ago by hmijail (Horacio Mijail Antón Quiles)

As already noted, venvs worked well with MacPorts's Python 3.5 from a couple of months ago, just as it does now with Homebrew's current Python.

comment:19 in reply to:  18 Changed 7 years ago by larryv (Lawrence Velázquez)

And you’re sure you haven’t changed any Language & Region settings since then?

comment:20 Changed 7 years ago by hmijail (Horacio Mijail Antón Quiles)

As far as I can remember, I didn't change that, exactly because of the python problems: I found that disabling Terminal's location setting fixed some problem and left it at that.

On the other hand, as stated, I updated to macOS Sierra a few days ago, so I don't know if Apple changed some Terminal default.

The only reason iTerm is still using its enabled-by-default "set locale" pref is because I'm still getting into it and didn't want to change too many preferences at once.

comment:21 in reply to:  16 Changed 7 years ago by hmijail (Horacio Mijail Antón Quiles)

Looks like they grab pip 8.1.2 separately.

But this problem is about ensurepip, not about pip, which AFAICT is in its own MacPorts package, no? (py35-pip)

comment:22 Changed 7 years ago by hmijail (Horacio Mijail Antón Quiles)

Homebrew's python3's ensurepip is 8.1.1:

6:~/test$ python3 -m venv venv3
7:~/test$ . venv3/bin/activate
(venv3) 8:~/test$ pip --version
pip 8.1.1 from /Users/mija/test/venv3/lib/python3.5/site-packages (python 3.5)
(venv3) 9:~/test$
Last edited 7 years ago by hmijail (Horacio Mijail Antón Quiles) (previous) (diff)

comment:23 Changed 7 years ago by hmijail (Horacio Mijail Antón Quiles)

I mentioned strange breakage in pyinvoke. That ended up being a pykerberos problem in Python 3 (general, happened the same on Homebrew), being worked on. So the only strange thing remaining is in MacPorts' Python3's ensurepip. (Just confirmed in python34 too)

Last edited 7 years ago by hmijail (Horacio Mijail Antón Quiles) (previous) (diff)

comment:24 Changed 7 years ago by hmijail (Horacio Mijail Antón Quiles)

In case it helps anyone in the future: a way to fix MacPorts' Python3's ensurepip is

$ export LC_ALL=en_US.UTF-8
$ python -m venv mpvenv
$

...or instead of en_US.UTF-8, use C, or maybe any other option shown by locale -a.

...or just go with Homebrew, which seems to just work with the default configurations of... everything?

Last edited 7 years ago by hmijail (Horacio Mijail Antón Quiles) (previous) (diff)

comment:25 in reply to:  22 ; Changed 7 years ago by larryv (Lawrence Velázquez)

Replying to hmijail@…:

Homebrew's python3's ensurepip is 8.1.1:

That’s pip, not “ensurepip”. And I guess I don’t understand what that bit about 8.1.2 in the formula was about then.

This bit is curious. Does this work with Homebrew’s Python 3?

$ python3.5 -c 'import ensurepip'

comment:26 in reply to:  25 ; Changed 7 years ago by hmijail (Horacio Mijail Antón Quiles)

That’s pip, not “ensurepip”. And I guess I don’t understand what that bit about 8.1.2 in the formula was about then.

Why do you say it's not ensurepip, to begin with?

comment:27 in reply to:  26 ; Changed 7 years ago by larryv (Lawrence Velázquez)

ensurepip is a module that installs pip. It’s part of CPython; it doesn’t have a version. pip --version invokes pip.

comment:28 in reply to:  27 ; Changed 7 years ago by hmijail (Horacio Mijail Antón Quiles)

ensurepip is a module that installs pip. It’s part of CPython; it doesn’t have a version. pip --version invokes pip.

PEP 453 seems to disagree. -m ensurepip is an interface for cases where pip was not installed or uninstalled. "In most cases, end users won't need to use this CLI directly, as pip should have been installed automatically when installing Python or when creating a virtual environment. However, it is formally documented as a public interface to support at least these known use cases: ..."

"ensurepip itself (including the private copy of pip and its dependencies) will always be installed normally (as it is a regular part of the standard library), but an option will be provided to skip the invocation of ensurepip"

Now, PEP 453 also seems to say that ensurepip does nothing if an updated pip is installed.

https://www.python.org/dev/peps/pep-0453/

comment:29 Changed 7 years ago by hmijail (Horacio Mijail Antón Quiles)

Homebrew's python3:

$ python3 -c 'import ensurepip'
$ echo $?
0

comment:30 in reply to:  28 Changed 7 years ago by larryv (Lawrence Velázquez)

Cc: nad@… added

I don’t know what you’re trying to prove. I misspoke about the installing, but ensurepip still doesn’t have a “version”. It contains a private copy of pip, which itself has a version, which for Python 3.5.2 is 8.1.1.

Anyway I’m at a loss. The bug is in Python, not MacPorts. Homebrew’s packaging must be avoiding the problem somehow, but I cannot figure out what the formula is doing through all the Ruby syntax. And since it’s ultimately an unresolved upstream bug, I’m not really convinced that we should be trying to fix it.

Cc-ing an upstream dev to see if he has any ideas.

comment:31 Changed 7 years ago by hmijail (Horacio Mijail Antón Quiles)

I also don't know what you are trying to prove. Even MacPorts' python35's ensurepip (a) answers as pip, (b) does have a version number.

$ export LC_ALL=C
$ which python3.5
/opt/local/bin/python3.5
$ python3.5 -m venv venv10
$ . venv10/bin/activate
(venv10) $ pip --version
pip 8.1.1 from /Users/mija/test/venv10/lib/python3.5/site-packages (python 3.5)
(venv10) $ port installed py35-pip
The following ports are currently installed:
  py35-pip @8.1.2_0
$

comment:32 in reply to:  31 Changed 7 years ago by larryv (Lawrence Velázquez)

I am trying to get us on the same page with the terminology. That program you are running inside the venv is the private copy of pip provided by ensurepip. It is not ensurepip itself. Look inside, ensurepip is not to be found.

#!/private/tmp/myvenv/bin/python3.5

# -*- coding: utf-8 -*-
import re
import sys

from pip import main

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())
Last edited 7 years ago by larryv (Lawrence Velázquez) (previous) (diff)

comment:33 Changed 7 years ago by hmijail (Horacio Mijail Antón Quiles)

Oh, OK, I see now what you meant. Sorry, my fault.

comment:34 Changed 7 years ago by ned-deily (Ned Deily)

The only thing I can add to the discussion at the moment is that I can reproduce the OP's ensurepip failure when LC_CTYPE=UTF-8 and using the venv from the current MacPorts python35 (3.5.2). However, I don't see a similar failure when using the python3.5 from the python.org 3.5.2 installer for OS X which has the same bundled versions of pip (8.1.1) and setuptools (20.10.1). I took a very quick look at the current MacPort python35 patches and didn't see anything obvious. I'm afraid I don't have time right now to dig in any deeper. But, obviously as noted above, the easy workaround is to ensure locale variables are set to more standard values. I just throw in an export LANG=en_US.UTF-8 in my .bashrc.

comment:35 Changed 7 years ago by jmroot (Joshua Root)

Cc: jmroot removed
Owner: changed from jyrkiwahlstedt to jmroot
Status: newassigned

comment:36 Changed 3 years ago by cooljeanius (Eric Gallager)

Cc: cooljeanius added

comment:37 Changed 23 months ago by jmroot (Joshua Root)

Resolution: wontfix
Status: assignedclosed

If this wasn't fixed in python 3.5.x at some point, I guess it never will be.

Note: See TracTickets for help on using tickets.