Opened 14 years ago

Closed 12 years ago

Last modified 11 years ago

#11267 closed enhancement (fixed)

REVISION: Python25 readline variant and now builds as a framework

Reported by: dinosaur@… Owned by: raimue (Rainer Müller)
Priority: Normal Milestone:
Component: ports Version:
Keywords: Python readline Framework Cc: pmq@…, skymoo (Adam Mercer), mdickens@…, raimue (Rainer Müller)
Port:

Description

Here's an updated portfile for Python25 that will build it as a framework so it will now work with py-sip and PyQt4, and a variant that will allow readline support. Please integrate it into the tree.

Attachments (6)

Portfile (1.8 KB) - added by dinosaur@… 14 years ago.
Updated Portfile
patch (7.6 KB) - added by meissnem@… 14 years ago.
proposed patch
python25.tar (30.0 KB) - added by mdickens@… 13 years ago.
Updated tarball of Python25 port files, which works on 10.4 and 10.5, for no variants, +framework, +universal, and +framework+universal variants. See next entry for more info.
python252.tar.bz2 (6.5 KB) - added by mdickens@… 13 years ago.
Upgrade to Python 2.5.2, including Portfile and all updated files; provides the same functionality as the previous tarball.
python25.tar.bz (3.0 KB) - added by reiffert@… 13 years ago.
python25.tar.bz2 (3.0 KB) - added by reiffert@… 13 years ago.

Download all attachments as: .zip

Change History (42)

Changed 14 years ago by dinosaur@…

Attachment: Portfile added

Updated Portfile

comment:1 Changed 14 years ago by mww@…

Status: newassigned

what do we need the framework for? As far as I remembered, it brought more havoc than benefit with python24; for the readline part: I'd prefer a py25-readline port (or however we're gonna handle the stuff with python24 vs python25)

comment:2 Changed 14 years ago by gui-dos (Guido Soranzio)

You have forgotten to remove port:readline from the original depends_lib.

comment:3 Changed 14 years ago by mww@…

I'm currently working on a clean python25 port with py25-readline etc. - these ports will get commited when v1.4 gets final (1.4 will have support for python25 ports)

Changed 14 years ago by meissnem@…

Attachment: patch added

proposed patch

comment:4 Changed 14 years ago by meissnem@…

Please consider the patch I just attached. It adds two variants to the python25 port, +framework and +default_python. Both are entirely optional -- and without them, the port remains as it is today.

+default_python simply installs python2.5 as ${prefix}/bin/python -- useful if you don't use many modules (like me).

+framework builds and installs python2.5 as a framework. In order to facilitate frameworks, the py25-* modules need to remove the hard-coded "--prefix=${prefix}" line so modules are installed into the framework. Additionally, the pyexpat module needs to be built to get the toolbox glue working -- so I removed it from the list of modules to be excluded. No explicit dependency on port:expat is necessary since the source code for expat is included in the Python-2.5 tarball and is linked in statically.

comment:5 Changed 14 years ago by mww@…

Cc: pmq@… added; mww@… removed

btw. the py25-readline port exists now (as do some other py25-ports); I'm currently a bit short on time & ressources - perhaps pmq@ can have a look at the framework thing.

The thing about the "python" binary: This is a problem as other ports might stop working if they expect python 2.4 for the "python" binary. I'd rather say, create an alias in your .bashrc (alias python=/opt/local/bin/python2.5), if this is a solution for you.

comment:6 Changed 14 years ago by pipping@…

Milestone: Port Enhancements

comment:7 Changed 13 years ago by meissnem@…

at the very least, the "pyexpat" stuff included in this patch could be used to fix #11442.

comment:8 Changed 13 years ago by nox@…

Priority: ImportantHigh
Summary: UPDATE: Python25 readline variant and now builds as a frameworkREVISION: Python25 readline variant and now builds as a framework
Version: 1.3.2

comment:9 Changed 13 years ago by mdickens@…

There are 2 primary problems with the current python25 scripts and the py25-XXX ports that come from inside the python25 distro.

1: If the +framework variant is enabled (as per meissnem@…'s patch), then "/Applications/MacPython 2.5" is populated with some of the installed apps (instead of all of them in "/Applications/MacPorts/MacPython 2.5"). Reviewing the patches for python24, this can be fixed either with patches or with "post-patch" reinplace's ... for example, the following works well for me:

post-patch {
        reinplace "s|/Applications/MacPython|/Applications/MacPorts/MacPython|g\
" ${worksrcpath}/Mac/Makefile.in
        reinplace "s|/Applications/MacPython|/Applications/MacPorts/MacPython|g\
" ${worksrcpath}/Mac/IDLE/Makefile.in
        reinplace "s|/Applications/MacPython|/Applications/MacPorts/MacPython|g\
" ${worksrcpath}/Mac/PythonLauncher/Makefile.in
        reinplace "s|/Applications/MacPython|/Applications/MacPorts/MacPython|g\
" ${worksrcpath}/Mac/BuildScript/build-installer.py
        reinplace "s|2.4|2.5|g" ${worksrcpath}/Mac/Tools/Doc/setup.py
        reinplace "s|/Applications/MacPython|/Applications/MacPorts/MacPython|g\
" ${worksrcpath}/Mac/Tools/Doc/setup.py
}

2: Some of the py25-XXX modules from inside the python25 distro are disabled via the patch-setup.py file. This only disables the inclusion of the object (compiled .c) files in the python shared library; it does not prevent the inclusion of files that may use those Python modules in .../lib/python2.5/ (originals found in ${python25-distro}/Lib ). For whatever reason, Python2.5 seems to require the .py and .so files for a module to "import" to be in the same directory ... but that will -not- be the case for some of these modules. For example, hashlib is required by some of the files in ${python25-distro}/Lib , and these files cannot be disabled or not installed (beyond removing them all in post-destroot).

That said, none of the critical python25 components seem to rely upon any of the modules disabled by patch-setup.py . It should be possible in the py25-XXX Portfile's to extract the -related- .py file(s) in ${python25-distro}/Lib for the given module, and install them along with the .so created by setup.py .

For my needs, I'm just removing the patch-setup.py , then removing dependence on the py25-XXX modules that come from the python25 distro. This seems to be the "safest" way to get functionality.

comment:10 Changed 13 years ago by skymoo (Adam Mercer)

Cc: ram@… added

comment:11 Changed 13 years ago by mdickens@…

This ticket is getting too old. The addition of ticket:12803 to python25 (rev 4), means that creating a framework is no longer as easy as it was with the patch here. This ticket should probably be merged with ticket:12817 since it covers the framework install as well. The framework does work (almost) correctly if the dylib changes are negated. I think the dylib issue needs to be re-addressed more in the form of how 2.4.4 does it, which still provides framework compatibility. I do not yet have a Portfile that does all of this correctly, since it's gotten a bit complicated.

comment:12 Changed 13 years ago by mdickens@…

I finally figured out and resolved the issues between framework and dylib in 'configure' and 'Makefile.pre.in'. I'll attach a tarball of the Portfile and related files. These work for me for doing both +framework and without (just standard dylib) on 10.4 (not at the same time). I hope someone else is testing this on other Darwin installs (10.2, .3, .5)!

comment:13 Changed 13 years ago by mdickens@…

Could someone add me to the "cc:" list? I'd like to know what's going on with the Python25, +framework, and dylib issues.

comment:14 Changed 13 years ago by mdickens@…

I've added another ticket (ticket:13778) with a replacement for patch-setup.py, which supersedes the one included in the attached tarball (of 2007-12-28).

I can confirm that these files (from the tarball, and the new patch-setup.py) provide working python25 on Darwin 8 & 9, using either +Framework or not.

Please also note that the _OSA module will not compile or install on Darwin 9, since Apple removed the "debug" commands from the OSA Framework. This doesn't error out the install process, since it happens when executing setup.py (which doesn't check for those errors).

comment:15 Changed 13 years ago by pschumm@…

Thanks very much for your work on this. I need to use a frameworks build because -- among other things -- I need to use mod_wsgi and I can't seem to get it to compile against a non-frameworks Python. I downloaded the attachment you posted on 28-Dec-07 and replaced with the patch-setup.py from 01-Jan-08. The install went fine (OS X 10.4.11), but now I'm having problems because Python seems unable to find other modules which have been installed. For example, if I try to run ez_setup.py, I get the following error:

ImportError: No module named _md5

despite the fact that I have py25-hashlib installed (I get the same problem interactively, BTW). Is this a problem with the python25+frameworks port, or is this something that'll need to be changed in the other ports that depend on python25?

comment:16 in reply to:  15 Changed 13 years ago by mdickens@…

> ImportError: No module named _md5
> 
> despite the fact that I have py25-hashlib installed (I get the same problem interactively, BTW).  Is this a problem with the python25+frameworks port, or is this something that'll need to be changed in the other ports that depend on python25?

This might be related to ticket #13777. Make sure the PYTHONPATH environment variable (in your preferred shell) includes ${prefix}/lib/python25/site-packages, since that's where the py25-hashlib stuff will be placed. e.g. in bash:

export PYTHONPATH=/opt/local/lib/python25/site-packages

You can check this via:

python
>>> import sys
>>> sys.path

and the printout should include both ${prefix}/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5 and ${prefix}/lib/python25/site-packages

comment:17 Changed 13 years ago by mdickens@…

I've just attached a new tarball, which contains the changes needed to get the PYTHONPATH correct for the Framework install ; to include both ${Framework}/.../lib/python2.5/ and ${prefix}/lib/python2.5/site-packages/ . Checking the sys.path, these mods work as desired (I borrowed them from the python24 port, so this is no surprise). py25-hashlib and related ports should install and execute without needing the augment the PYTHONPATH in the Port environment (as was the case in python24).

Changed 13 years ago by mdickens@…

Attachment: python25.tar added

Updated tarball of Python25 port files, which works on 10.4 and 10.5, for no variants, +framework, +universal, and +framework+universal variants. See next entry for more info.

comment:18 Changed 13 years ago by mdickens@…

Changes to the tarball of 2008-01-06, from previous:

  • Instead of hard-coding "/opt/local" into setup.py, use "__PREFIX__" then replace it with the actual ${prefix} in the post-patch.
  • Corrected +universal install on 10.5: '-isysroot' does not work with gcc, but '-isystem' does. This corrects ticket #13701, if that was still an issue.

I've tested all combinations of variants (plain, +framework, +universal, +universal+framework) on both 10.4 and 10.5, and they all work as desired. While the resulting binaries using +universal are for "i386" and "ppc", it would not be difficult to add variants to do "ppc64" and "x86_64" (though, of course, 64-bit apps are not well-tested on OSX just yet). I will look into this variant, and see if the apps work.

comment:19 Changed 13 years ago by gjznituv@…

I replaced the tarball contents with the corresponding default macports python25 files.

Is there anything else I should do?

After reinstalling it I still get the same error for py25-wxpython (see ticket #13756 ).

comment:20 Changed 13 years ago by jmpp@…

Priority: HighNormal

comment:21 Changed 13 years ago by mdickens@…

Instructions for using the tarball, because others have asked for it:

0) Deactivate or uninstall any py25-YYY ports, and any other ports that rely upon them. If possible, uninstall all of these first and (re)install them afterwards (in step 6) just to make sure they all work with the "new" Python install.

1) Make sure 'port' is version 1.6.0 or newer; this takes care of some issues.

sudo port selfupdate

Optionally, also upgrade the other installed ports; while this shouldn't make any difference, it might and really can't hurt unless you need a specific version of some particular port.

sudo port upgrade installed

2) Make sure python25 is not installed, e.g.

sudo port -f uninstall python25

3) Download the tarball; let's say this is in ~/Downloads/python25.tar

4) Make the updated Portfile and related files available for 'port' to use:

pushd `port dir python25`
cd ..
sudo rm -rf python25
sudo tar xf ~/Downloads/python25.tar
popd

5) Install the new version of Python, with whatever variant you desire, e.g.:

sudo port install python25 +framework

6) Reactivate or (re)install any py25-YYY ports that were deactivated or uninstalled in step 0.

comment:22 in reply to:  19 Changed 13 years ago by mdickens@…

Replying to gjznituv@gmail.com:

I replaced the tarball contents with the corresponding default macports python25 files.

Is there anything else I should do?

After reinstalling it I still get the same error for py25-wxpython (see ticket #13756 ).

Please see my answer in ticket #13756, comment 5. A solution is to put a check into each pyXX-wxpython Portfile to make sure that the version of wxWidgets that's currently installed is compatible with the version of wxpython that is about to be installed. If not, tell the user what to do to get the correct version of wxWidgets, or something else useful ... not just a "random" compiler error message.

comment:23 Changed 13 years ago by jmroot (Joshua Root)

Cc: mdickens@… added

Adding cc as requested in comment:13.

Changed 13 years ago by mdickens@…

Attachment: python252.tar.bz2 added

Upgrade to Python 2.5.2, including Portfile and all updated files; provides the same functionality as the previous tarball.

comment:24 Changed 13 years ago by reiffert@…

Owner: changed from mww@… to reiffert@…
Status: assignednew

I'm currently working on a python25 framework installation as per default. Your help gets appreciated much.

comment:25 Changed 13 years ago by reiffert@…

Status: newassigned

The following problems got already fixed:

Change /Applications/MacPython 2.5 -> /Applications/MacPorts/MacPython 2.5

Symlinking Library/Frameworks/Python.framework Headers, Python, Resources and Versions/Current by the help of python_select. Thanks to raimue.

Symlinks lib/libpython2.5.dylib lib/python2.5 include/python2.5 into the framework

Currently I'm testing the port on various platform. Please contact me for this purpose.

Whats needs to be done after submitting: Fixing the prefix for all py25-* ports

Whats needs to be done for the next major macports release: Reflecting the new prefix into py25 Portgroup.

Kind regards Thomas

comment:26 Changed 13 years ago by reiffert@…

Cc: raimue@… added

Please test the atteched file python25.tar.bz2 and report back, thank you!

Changed 13 years ago by reiffert@…

Attachment: python25.tar.bz added

Changed 13 years ago by reiffert@…

Attachment: python25.tar.bz2 added

comment:27 Changed 13 years ago by mdickens@…

I applaud your efforts towards finally creating the framework Python install. Here are my Initial Note and Comments from a quick review and testing.

  • I'm concerned with sym-linking $prefix/lib/python2.5 and $prefix/include/python2.5 into the framework -- doing so is very non-standard. For the former, both paths ($prefix/Frameworks/.../lib/python2.5 and $prefix/lib/python2.5) can easily be made part of the default PYTHONPATH (before user additions via the environment variable) - this was done by my changes to setup.py - and Python is quite good about loading scripts and libraries from anywhere in its path - so there really is no reason -for- doing this. Reasons for avoiding it include:
    • The "upgrade" command will not work properly for already-installed py25-xxx ports that occupy $prefix/lib/python2.5 . Any of these currently installed ports must be first deactivated, then the new python upgraded, then those ports either activated or uninstalled/re-installed. At least in my testing, trying to activate those ports (individually, doing in a group didn't work) results in "Error: port activate failed: Not a directory", even though the port was successfully activated. While this might be a bug on 'port', it is off-putting to the end user.
    • It adds complexity to the install and could be confusing to end-users.
    • If it ain't broke, don't fix it: it's simple to allow for both of these paths to be part of the default PYTHONPATH, so why not leave things alone?
  • in "post-patch", there are two "Mac/IDLE/Makefile.in" ... only one is needed.
  • the manpages are not installed correctly, and do not work. I think the correct Portfile line is:
    	system "cd ${destroot}${prefix}/share/man/man1 && \
    		ln -sf ${prefix}/Library/Frameworks/Python.framework/Versions/2.5/share/man/man1/python.1.gz python2.5.2.1.gz"
    
  • variant +universal does not work for me on Mac OS X 10.5.2, when doing "make libpython2.5.dylib". It looks like someone added $(LDFLAGS) to the dylib Makefile rule. When doing universal, the "-arch" command will be added to LDFLAGS, which isn't compatible with Apple's libtool libtool can handle this without any further flags. Please remove $(LDFLAGS) to fix this problem. You can check the resulting .dylib file for universality by executing "otool -vf XXX.dylib".
    • Even removing the LDFLAGS, there is still an install error for me, during destroot. I don't have time to debug this further right now.
  • "destroot.target"
    • should not contain compiling or linking commands, such as "libpython2.5.dylib". This should be part of "build.target". Putting it here saves a line of Portfile code, but that 1 line really doesn't matter.
    • In the top-level Makefile, the target "frameworkinstall" is an alias for the target "install", which in turn does the "maninstall". Thus setting the "destroot.target" to "install" should be sufficient.
  • python_select:
    • Since "python_select" isn't part of Python, the port-activate note should probably read something like "To fully complete your installation and make python 2.5 the default, please run 'sudo python_select python25' using the separate port 'python_select'." .
    • Running "python_select --" results in errors.
    • after removing python25, the symlinks created by python_select are still around. This is not a good idea.
    • While I like the intent of python_select, does MacPorts really want to be distributing this level of new, untested, code that leaves issues to be resolved?
    • I think moving this "feature" into the Python25 (or other) Portfile would be the wisest way ... that way the end results are owned by a Port, and can be manipulated by said Port.

comment:28 Changed 13 years ago by mdickens@…

YA issue with the symlinks: I just uninstalled this revision of python25, after going through the process of deactivating and reactivating the py25-XXX ports I use. For some reason 'port' let me do this .. I don't think I did "port -f". Anyway, removing this test port removed the symlinks, which then removes my ability to properly uninstall the py25-XXX ports - there were residual files left in $prefix/Library/..../include and .../lib . I should -always- be able to properly uninstall ports, no matter the possibly odd state of whatever is installed. Using symlinks creates this possibility. This will be confusing, at best, to end-users.

comment:29 Changed 13 years ago by mdickens@…

I also see that python_select somehow gets ownership (in the sense of "port provides foo") of /opt/local/bin/{pydoc, python, pythonw} and such. While this is a good thing, it is still awkward when doing "sudo port uninstall python25" and the residual symlinks owned by python_select hanging around. There has to be a better way.

comment:30 Changed 13 years ago by reiffert@…

A roadmap for the python25 transition can be found here:

http://trac.macosforge.org/projects/macports/wiki/PythonFrameworkTransition

comment:31 Changed 12 years ago by sanchom (Sancho McCann)

Has there been any progress towards the framework build?

comment:32 Changed 12 years ago by raimue (Rainer Müller)

Owner: changed from reiffert@… to raimue@…
Status: assignednew

See this thread on macports-dev.

Also, I checked in a new version of python25 in the python-frameworks branch.

Reassigning to myself, because reiffert won't have time work on this anymore as reported on the thread linked above.

comment:33 Changed 12 years ago by raimue (Rainer Müller)

Status: newassigned

comment:34 Changed 12 years ago by mdickens@…

I applaud the effort to "do the right thing" with all Python installs and "python_select"; I think the end result will be fantastic. That said, it is taking a very long time - now around 6 months since I joined this discussion, and more than 1 year since this ticket was opened - to get what would otherwise be a "simple" framework variant added to this port. I would have greatly preferred to have had a revision early on that allowed for the framework variant, such as that provided by my tarball, and then working towards "doing the right thing" as time allows. This is still my preference because I need the Python 2.5 framework for my work ... hence could whoever is in charge please "First make it work, then make it work well - while being as backwards compatible as possible to the first iteration" (in the words of some other OSS project managers I know). Thanks for listening.

comment:35 Changed 12 years ago by raimue (Rainer Müller)

Resolution: fixed
Status: assignedclosed

Finally merged the new python25 port building as a framework to trunk in r37817.

comment:36 Changed 11 years ago by (none)

Milestone: Port Enhancements

Milestone Port Enhancements deleted

Note: See TracTickets for help on using tickets.