Opened 10 years ago

Closed 8 years ago

#12369 closed enhancement (fixed)

Split python25 into python25-core and modules

Reported by: nirs@… Owned by: mww@…
Priority: Normal Milestone:
Component: ports Version:
Keywords: Cc: raimue (Rainer Müller), nirs@…, mww@…, nox@…, nirsof@…, mf2k (Frank Schima), bryan@…, akitada@…, blb@…, skymoo (Adam Mercer), free.as.in.speech@…, anthony.theocharis@…, arthurk@…, macports@…
Port: python25

Description

$ python2.5
Python 2.5.1 (r251:54863, Jul 29 2007, 14:28:00) 
[GCC 4.0.1 (Apple Computer, Inc. build 5367)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sha
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/local/lib/python2.5/sha.py", line 6, in <module>
    from hashlib import sha1 as sha
  File "/opt/local/lib/python2.5/hashlib.py", line 133, in <module>
    md5 = __get_builtin_constructor('md5')
  File "/opt/local/lib/python2.5/hashlib.py", line 60, in __get_builtin_constructor
    import _md5
ImportError: No module named _md5
>>> 

sha is important core module, many apps will break without it.

There was no problem with Python 2.5.0.

Environment: Mac OS X 10.4.8

Attachments (12)

python25-reenable-libs.diff (1.3 KB) - added by jezdez@… 8 years ago.
Patch to make the python25 port to behave as expected, e.g. like python24, python26, Apple's Python and python.org's Python.
python25-reenable-libs.2.diff (1.3 KB) - added by jezdez@… 8 years ago.
New patch that refers to http://lists.macosforge.org/pipermail/macports-dev/2009-April/008236.html and only removes the tk/tcl dependency.
python25-reenable-libs.3.diff (1.3 KB) - added by jezdez@… 8 years ago.
Updated patch for r49645
py25_coremodules.diff (28.2 KB) - added by arthurk@… 8 years ago.
python25-reenable-libs.4.diff (38.2 KB) - added by arthurk@… 8 years ago.
python25-reenable-libs.5.diff (38.2 KB) - added by arthurk@… 8 years ago.
enabled and added dep on tk
python25-reenable-libs.6.diff (38.2 KB) - added by arthurk@… 8 years ago.
fixed ncurses dep on python25 port
python25-reenable-libs.7.diff (28.3 KB) - added by arthurk@… 8 years ago.
Remove rev-bumping
python25-reenable-libs.8.diff (43.3 KB) - added by arthurk@… 8 years ago.
python25-reenable-libs.9.diff (43.3 KB) - added by arthurk@… 8 years ago.
Fixed 'port:port:' typos
python25-reenable-libs.10.diff (43.4 KB) - added by arthurk@… 8 years ago.
Update patch to r51758
python25-reenable-libs.11.diff (43.4 KB) - added by arthurk@… 8 years ago.

Download all attachments as: .zip

Change History (61)

comment:1 Changed 10 years ago by nirs@…

Seems that it is caused by setup.py patch

 # This global variable is used to hold the list of modules to be disabled.
7	-disabled_module_list = []
8	+disabled_module_list = ["zlib","_hashlib","_ssl","_bsddb","_sqlite3","_tkinter","bz2","gdbm","readline","_curses","_curses_panel"]

The comment is quite useless - the variable name is self explaining. The comment should describe WHY these modules are disabled.

Why these modules are disabled?

comment:2 Changed 10 years ago by nox@…

  • Milestone set to Port Bugs
  • Priority changed from Blocker to High
  • Summary changed from ImportError when importing sha on Python 2.5.1 to BUG: ImportError when importing sha on Python 2.5.1
  • Version 1.5.0 deleted

comment:3 Changed 10 years ago by nox@…

  • Cc nox@… added

Python _hashlib core module has been disabled in python itself because they are available through specific ports. As they should be core modules, I think their ports should be removed of the tree and reenabled in python25 portfiles.

The module ports are:

  • py25-zlib
  • py25-hashlib
  • py25-bsddb
  • py25-sqlite3
  • py25-tkinter
  • py25-bz2
  • py25-gdbm
  • py25-readline
  • py25-curses

comment:4 Changed 10 years ago by nox@…

  • Cc nirs@… added

I've added the reporter in the Cc list, at least, he will now he have to install py25-hashlib

comment:5 Changed 10 years ago by raimue (Rainer Müller)

  • Cc raimue@… added

Tickets #11931, #12586, #12369 are duplicates describing the same issue. The resolution is to install py25-hashlib at the moment. As nox suggested we could also move this back into the main python25 port. How should we handle it?

comment:6 Changed 10 years ago by nirs@…

Why these modules were made available as separate modules?

comment:7 Changed 10 years ago by mww@…

  • Resolution set to wontfix
  • Status changed from new to closed

If all modules of python were to be activated, python would have dependencies on:

tcl, tk, openssl, db44, bzip2, ncursesw, zlib, gdbm, readline & sqlite3

Which is quite heavy if you just want a python interpreter (perhaps even unknowingly as a dependency). If someone is keen on having all these in a bundle, please create a 'python25-full' meta port which drags in all the py25-modules you think are important - including perhaps even more modules than come with the standard installation.

comment:8 Changed 10 years ago by nirs@…

  • Resolution wontfix deleted
  • Status changed from closed to reopened

Python 2.5 port should be exactly the same as Python distributed from http://python.org. If you want to create a lightweight version, create a python-lite port that does not include these dependencies, and add proper docs how to get the missing components.

comment:9 Changed 10 years ago by nox@…

...or just add proper docs on how to get the missing component in this port.

comment:10 Changed 10 years ago by mww@…

  • Resolution set to wontfix
  • Status changed from reopened to closed

So what are the dependencies of the "same as the Python distribution"? Depends a lot on what exactly the configure-script finds, so the only deterministic solution is to provide everything.

People coming from e.g. FreeBSD would expect the python port to be exactly like it is - the FreeBSD ports also have a 'Python-lite standard' installation.

Also you totally ingored the fact that a "python-full" + python (like it is currently) would be compatible while your "python-lite" and "python-full" is NOT. You can only install one at a time, while you can "upgrade" from the current port to a "full" port.

If you're missing the docs: This is an open project, so feel free to join and provide them. It was not clear to me that people w/o a FreeBSD background would not find out themselves whats the current state.

comment:11 Changed 10 years ago by nirs@…

Well this is the problem, not many users have FreeBSD background and do not expect to get python25-lite and additional ports. Any user that use previous MacPorts versions expect a full Python port.

What the problem in having the same port as python24? Did anyone complained about 2.4 being too big?

I already submitted a patch to add docs to the port 2 month ago.

If you think that the docs are missing, why did you close this bug?

comment:12 Changed 10 years ago by mww@…

  • Milestone changed from Port Bugs to Port Enhancements
  • Priority changed from High to Normal

I think this is a misunderstanding: "docs" is the abbreviation for "documentation" and I assumed you wanted the behavior of python25 documented, which is btw. the same as python 2.4 (see py-bsddb, py-tkinter, etc.). I assumed you wanted to add some lines to the wiki or have a post-install message about modules that are a good idea to install, too.

Anyway, to come to a solution: You want a "full" size port while I think a slim port with extras is more reasonable. We can get both what we want if you just create a "python25-with-extras" port (I dont care about the name). If I just bloat the main port, it is not possible anymore to have a slim installation (because an additional python25-lite port would conflict with the python25 port).

So why dont you just create a "python25-full" port so everybody can get what they want?

comment:13 Changed 10 years ago by nirs@…

Python 2.5 is documented now, but was not documented when this bug was reported. However, the docs are not accessible enough. You need online docs, accessible from the ports tool.

I'm not sure why python2.5 can not work like python24, which does not require separate ports to get basic functionality like md5 to work. I'm also not convinced that there is a need for slim python25 port. but I don't have time to research this myself now.

About practical solution in your direction, how about this:

  1. Rename python25 to python25-core
  2. Add python25 port that depends on python25-core and all those py25-xxx dependencies

I should have time to work on this in 2-3 of weeks.

comment:14 Changed 9 years ago by matt macports@…

  • Resolution wontfix deleted
  • Status changed from closed to reopened

+1 for nirs's python25 / python25-core split

This makes it much easier for end-users to "port install python25" and be able to follow standard instructions for installing software not packaged in MacPorts. Packagers on the other hand have a deeper understanding of the dependencies of their app and can choose python25-core + the few extra packages they need instead of pulling in all of the Python libs.

comment:15 Changed 9 years ago by justin1@…

This is going get a lot more attention soon because of Google's App Engine. The GAE development server forbids importing C extensions, so _hashlib is blocked with MacPorts, causing lots of problems with the server.

comment:16 Changed 9 years ago by raimue (Rainer Müller)

  • Summary changed from BUG: ImportError when importing sha on Python 2.5.1 to Split python25 into python25-core and modules

comment:17 in reply to: ↑ description Changed 9 years ago by sasha@…

I'm attempting to use virtualenv under python25. I followed the instructions in this ticket and installed py25-hashlib. But virtualenv --no-site-packages errors out with the error bellow.

I believe the reason is the the copy of python executable copied to the new environment by the virtualenv script only has access to hashlib.py that's part of the crippled python25 port and not to the hashlib that's installed as part of the py25-hashlib port.

is there a way around this?

New python executable in ./bin/python
Installing setuptools...........
  Complete output from command ./bin/python -c "#!python
\"\"\"Bootstrap setuptoo...




" /opt/local/Library/Frameworks/...5.egg:
  Traceback (most recent call last):
  File "<string>", line 267, in <module>
  File "<string>", line 226, in main
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/virtualenv-1.2-py2.5.egg/support-files/setuptools-0.6c8-py2.5.egg/setuptools/command/easy_install.py", line 21, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/virtualenv-1.2-py2.5.egg/support-files/setuptools-0.6c8-py2.5.egg/setuptools/package_index.py", line 3, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/urllib2.py", line 91, in <module>
    import hashlib
ImportError: No module named hashlib
----------------------------------------
...Installing setuptools...done.
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.5/bin/virtualenv", line 8, in <module>
    load_entry_point('virtualenv==1.2', 'console_scripts', 'virtualenv')()
  File "/opt/local/lib/python2.5/site-packages/virtualenv-1.2-py2.5.egg/virtualenv.py", line 389, in main
    unzip_setuptools=options.unzip_setuptools)
  File "/opt/local/lib/python2.5/site-packages/virtualenv-1.2-py2.5.egg/virtualenv.py", line 638, in create_environment
    install_setuptools(py_executable, unzip=unzip_setuptools)
  File "/opt/local/lib/python2.5/site-packages/virtualenv-1.2-py2.5.egg/virtualenv.py", line 282, in install_setuptools
    cwd=cwd)
  File "/opt/local/lib/python2.5/site-packages/virtualenv-1.2-py2.5.egg/virtualenv.py", line 450, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command ./bin/python -c "#!python
\"\"\"Bootstrap setuptoo...




" /opt/local/Library/Frameworks/...5.egg failed with error code 1

Replying to nirs@…:

$ python2.5
Python 2.5.1 (r251:54863, Jul 29 2007, 14:28:00) 
[GCC 4.0.1 (Apple Computer, Inc. build 5367)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sha
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/local/lib/python2.5/sha.py", line 6, in <module>
    from hashlib import sha1 as sha
  File "/opt/local/lib/python2.5/hashlib.py", line 133, in <module>
    md5 = __get_builtin_constructor('md5')
  File "/opt/local/lib/python2.5/hashlib.py", line 60, in __get_builtin_constructor
    import _md5
ImportError: No module named _md5
>>> 

sha is important core module, many apps will break without it.

There was no problem with Python 2.5.0.

Environment: Mac OS X 10.4.8

comment:18 Changed 9 years ago by casbon@…

It's worth mentioning that this split creates problems if you use 'python -S'.

From the manpage:

       -S     Disable  the import of the module site and the site-dependent manipulations of sys.path that it
              entails.

So if you have software that invokes itself through 'python -S' it will not get any modules outside of the built python distribution. The times I have seen this, it is to prevent picking up any local customisations. Unfortunately, if the software uses hashlib which it is fair to assume *should* be there, it will fail, and there is no way of fixing it using macports. You have to manually copy the hashlib module from where macports puts it to where python expects it.

-1 for this split. Keep it as it's shipped.

comment:19 Changed 9 years ago by nirsof@…

The ticket name is wrong - it should be Rename python to python-core, add python port that depends on all py25-xxx ports.

The current python port is already split - and this is the problem. A user that install python expect it to contain everything installed by the regular python shipped by python.org.

comment:20 Changed 9 years ago by nirsof@…

  • Cc nirsof@… added

Cc Me!

comment:21 Changed 9 years ago by mf2k (Frank Schima)

  • Cc macsforever2000@… added

Cc Me!

comment:22 Changed 9 years ago by mf2k (Frank Schima)

  • Port python25 added
  • Type changed from defect to enhancement

comment:23 Changed 9 years ago by bryan@…

  • Cc bryan@… added

Cc Me!

comment:24 Changed 8 years ago by akitada@…

  • Cc akitada@… added

Cc Me!

comment:25 Changed 8 years ago by blb@…

  • Cc blb@… added

Cc Me!

comment:26 Changed 8 years ago by skymoo (Adam Mercer)

  • Cc ram@… added

Cc Me!

comment:27 Changed 8 years ago by free.as.in.speech@…

  • Cc free.as.in.speech@… added

Cc Me!

comment:28 Changed 8 years ago by anthony.theocharis@…

  • Cc anthony.theocharis@… added

Cc Me!

comment:29 Changed 8 years ago by arthurk@…

  • Cc arthurk@… added

Cc Me!

comment:30 Changed 8 years ago by jezdez@…

Any news when this brain dead error is going to be fixed?

Changed 8 years ago by jezdez@…

Patch to make the python25 port to behave as expected, e.g. like python24, python26, Apple's Python and python.org's Python.

comment:31 Changed 8 years ago by jezdez@…

That patch is a result of a discussion on IRC about #18567 that made me wonder why Python isn't build with hashlib etc.

Changed 8 years ago by jezdez@…

New patch that refers to http://lists.macosforge.org/pipermail/macports-dev/2009-April/008236.html and only removes the tk/tcl dependency.

Changed 8 years ago by jezdez@…

Updated patch for r49645

comment:32 Changed 8 years ago by anonymous

  • Milestone Port Enhancements deleted

Milestone Port Enhancements deleted

Changed 8 years ago by arthurk@…

Changed 8 years ago by arthurk@…

comment:33 Changed 8 years ago by arthurk@…

"python25-reenable-libs.4.diff" removes the dependencies on core module ports and updates python25 to enable core modules.

Changed 8 years ago by arthurk@…

enabled and added dep on tk

Changed 8 years ago by arthurk@…

fixed ncurses dep on python25 port

comment:34 Changed 8 years ago by jmroot (Joshua Root)

There's no point rev bumping all the (former) dependents. The dependency relationship will still stay in the registry because of #13054.

Changed 8 years ago by arthurk@…

Remove rev-bumping

comment:35 Changed 8 years ago by arthurk@…

Okay, I didn't know that. "python25-reenable-libs.7.diff" removes all the rev-bumping.

Changed 8 years ago by arthurk@…

comment:36 Changed 8 years ago by arthurk@…

The latest patch turns the core module ports into stubs.

comment:37 Changed 8 years ago by blb@…

py25-opengl and py25-yum-metadata-parser have 'port:port:' typos in dependencies, other than that it looks good to me; unless Markus says otherwise I think it definitely falls into maintainer timeout.

Changed 8 years ago by arthurk@…

Fixed 'port:port:' typos

comment:38 Changed 8 years ago by macports@…

how can I apply this patch ? is it safe to use it ? what happen if I run a safeupgrade ? cheers

comment:39 Changed 8 years ago by macports@…

  • Cc macports@… added

Cc Me!

comment:40 Changed 8 years ago by macports@…

  • Cc macports@… removed

Cc Me!

comment:41 Changed 8 years ago by macports@…

  • Cc macports@… added

Cc Me!

Changed 8 years ago by arthurk@…

Update patch to r51758

comment:42 Changed 8 years ago by arthurk@…

If there are no objections, I'm going to commit the patch next weekend.

comment:43 follow-up: Changed 8 years ago by mf2k (Frank Schima)

I'm against this patch because I need py25-tkinter based on tk 8.4. I am able to modify my local py25-tkinter portfile to manage this. See Ticket #14488. Unless someone can allow this in the new arrangement.

comment:44 in reply to: ↑ 43 ; follow-up: Changed 8 years ago by blb@…

Replying to macsforever2000@…:

I'm against this patch because I need py25-tkinter based on tk 8.4. I am able to modify my local py25-tkinter portfile to manage this. See Ticket #14488. Unless someone can allow this in the new arrangement.

Can you replace your local py25-tkinter with a local python25 which uses an 8.4 version of tk instead?

comment:45 in reply to: ↑ 44 Changed 8 years ago by mf2k (Frank Schima)

Replying to blb@…:

Replying to macsforever2000@…:

I'm against this patch because I need py25-tkinter based on tk 8.4. I am able to modify my local py25-tkinter portfile to manage this. See Ticket #14488. Unless someone can allow this in the new arrangement.

Can you replace your local py25-tkinter with a local python25 which uses an 8.4 version of tk instead?

After chatting with blb on irc, and looking at python26 with tkinter, i am convinced it will work fine for me. So i withdraw my objection!

comment:46 follow-up: Changed 8 years ago by raimue (Rainer Müller)

Your patch has trailing whitespace after a backslash at depends_lib in the Portfile for python25. This is fatal and not going to work.

comment:47 in reply to: ↑ 46 Changed 8 years ago by arthurk@…

Replying to raimue@…:

Your patch has trailing whitespace after a backslash at depends_lib in the Portfile for python25. This is fatal and not going to work.

Whoops! Thanks for spotting this. The .11 patch fixes this. The python25 port was the only one where this error occured.

Changed 8 years ago by arthurk@…

comment:48 Changed 8 years ago by arthurk@…

Checked in with r51955

comment:49 Changed 8 years ago by blb@…

  • Resolution set to fixed
  • Status changed from reopened to closed
Note: See TracTickets for help on using tickets.