Opened 10 years ago

Closed 10 years ago

Last modified 9 years ago

#32090 closed defect (fixed)

python27 +universal: destroot fails if py27-xml is installed non-universal

Reported by: cdiehl@… Owned by: jyrkiwahlstedt
Priority: Normal Milestone:
Component: ports Version: 2.0.3
Keywords: Cc: Sam.Halliday@…, drkp (Dan Ports), vlsd (Vlad), ryanwilcox@…, deesto (John S. De Stefano Jr.)
Port: python27 py27-xml

Description

MacPorts 2.0.3
ProductName:	Mac OS X
ProductVersion:	10.7.2
BuildVersion:	11C74

$ sudo port installed | grep python
  mesa @7.11_1+python27+universal
  mesa @7.11_2+python27+universal (active)
  python24 @2.4.6_9
  python24 @2.4.6_9+universal (active)
  python26 @2.6.7_1 (active)
  python27 @2.7.2_1+universal (active)
  python31 @3.1.4_1 (active)
  python32 @3.2.2_0 (active)
  python_select @0.3_1 (active)
  xorg-libxcb @1.7_0+python27
  xorg-libxcb @1.7_0+python27+universal (active)
  xorg-xcb-proto @1.6_0+python27 (active)

Error happens during: upgrade outdated. Log file is attached.

Attachments (1)

main.log (1.7 MB) - added by cdiehl@… 10 years ago.
Log

Download all attachments as: .zip

Change History (19)

Changed 10 years ago by cdiehl@…

Attachment: main.log added

Log

comment:1 Changed 10 years ago by ryandesign (Ryan Schmidt)

Owner: changed from macports-tickets@… to jwa@…
Port: python27 added
Priority: HighNormal
Summary: Staging python27 into destroot failspython27: Staging into destroot fails

The log is incomplete. But it looks like you're building python27 universal for i386 and x86_64, and it's failing with this error:

:info:destroot ImportError: dlopen(/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_xmlplus/parsers/pyexpat.so, 2): no suitable image found.  Did find:
:info:destroot 	/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_xmlplus/parsers/pyexpat.so: mach-o, but wrong architecture

Of what architecture is the file /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_xmlplus/parsers/pyexpat.so? What port provided it? Presumably not one of the ones listed in "port rdeps python27". Try rebuilding the port that provides that file with the universal variant.

comment:2 Changed 10 years ago by cdiehl@…

$ lipo -info /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_xmlplus/parsers/pyexpat.so
Non-fat file: /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_xmlplus/parsers/pyexpat.so is architecture: x86_64

$ port rdeps python27

The following ports are dependencies of python27 @2.7.2_2:
  gettext
    libiconv
      gperf
    ncurses
      ncursesw
    expat
  zlib
  openssl
  sqlite3
    libedit
  db46
  gdbm
  bzip2
  python_select

comment:3 Changed 10 years ago by drkp (Dan Ports)

Cc: Sam.Halliday@… dports@… added

Has duplicate #32096

comment:4 Changed 10 years ago by drkp (Dan Ports)

Port: py27-xml added
Summary: python27: Staging into destroot failspython27 +universal: destroot fails if py27-xml is installed non-universal

I hit this problem when updating python27, too. It looks like the problem comes up because something in python27's build process uses the pyexpat module. The python27 port provides one version, in /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/pyexpat.so, but the py27-xml port also installs its own version into /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_xmlplus/parsers/pyexpat.so. The second one has a higher precedence.

If python27 is being built +universal, but py27-xml is installed non-universal, it will fail to load this module during destroot. The workaround I used was to install py27-xml +universal.

comment:5 Changed 10 years ago by alfredherrmann@…

this didn't help me, but: sudo port install python27 -universal

comment:6 Changed 10 years ago by jmroot (Joshua Root)

The real issue is that it's loading anything from the installed location at all. Presumably in the destroot phase it should only be looking in the work dir.

comment:7 Changed 10 years ago by foxtian@…

same problem,how make it ok?

comment:8 Changed 10 years ago by vlsd (Vlad)

I am having the same problem. I cannot install py27-xml +universal because it depends on python27, and I can't install python27 because it "depends" on py27-xml +universal. How do I go about fixing this cycle?

comment:9 Changed 10 years ago by vlsd (Vlad)

Cc: vloodo@… added

Cc Me!

comment:10 in reply to:  8 Changed 10 years ago by vlsd (Vlad)

Replying to vloodo@…:

I am having the same problem. I cannot install py27-xml +universal because it depends on python27, and I can't install python27 because it "depends" on py27-xml +universal. How do I go about fixing this cycle?

Fixed it by doing:

# sudo port -f deactivate py27-xml
# sudo port install python27 +universal
# sudo port install py27-xml +universal

comment:11 Changed 10 years ago by software@…

FYI,

I encountered a somewhat similar issue. While the problem is slightly different, the resolution posted by vloodo above works just as well in this case.

In my case, I wanted to replace python27 with python27 +ucs4. I had py27-xml installed at the time I was doing the replace. This lead to the following error during the destroot phase:

DYLD_FRAMEWORK_PATH=/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python27/python27/work/Python-2.7.2:  ../python.exe ./scripts/BuildApplet.py \
		--destroot "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python27/python27/work/destroot" \
		--python=/opt/local/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python \
		--output "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python27/python27/work/destroot/Applications/MacPorts/Python 2.7/Build Applet.app" \
		./scripts/BuildApplet.py
Traceback (most recent call last):
  File "./scripts/BuildApplet.py", line 156, in <module>
    main()
  File "./scripts/BuildApplet.py", line 36, in main
    buildapplet()
  File "./scripts/BuildApplet.py", line 123, in buildapplet
    progress=verbose, destroot=destroot)
  File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python27/python27/work/Python-2.7.2/Lib/plat-mac/buildtools.py", line 122, in process
    copy_codefragment, raw, others, filename, destroot)
  File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python27/python27/work/Python-2.7.2/Lib/plat-mac/buildtools.py", line 151, in process_common
    is_update, raw, others, filename, destroot)
  File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python27/python27/work/Python-2.7.2/Lib/plat-mac/buildtools.py", line 332, in process_common_macho
    builder.plist = plistlib.Plist.fromFile(plistname)
  File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python27/python27/work/Python-2.7.2/Lib/plistlib.py", line 345, in fromFile
    rootObject = readPlist(pathOrFile)
  File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python27/python27/work/Python-2.7.2/Lib/plistlib.py", line 78, in readPlist
    rootObject = p.parse(pathOrFile)
  File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python27/python27/work/Python-2.7.2/Lib/plistlib.py", line 401, in parse
    from xml.parsers.expat import ParserCreate
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_xmlplus/parsers/expat.py", line 4, in <module>
    from pyexpat import *
ImportError: dlopen(/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_xmlplus/parsers/pyexpat.so, 2): Symbol not found: _PyUnicodeUCS2_Decode
  Referenced from: /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_xmlplus/parsers/pyexpat.so
  Expected in: flat namespace
 in /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_xmlplus/parsers/pyexpat.so
make[1]: *** [install_BuildApplet] Error 1
make[1]: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python27/python27/work/Python-2.7.2/Mac'
make: *** [frameworkinstallapps] Error 2
make: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python27/python27/work/Python-2.7.2'

In short, destroot is calling some script called ./scripts/BuildApplet.py to compile .py to .pyc or .pyo files. (Building at the destroot phase is indeed ugly; you can blame that on Python).

This script somehow imports site-packages, including py27-xml. py27-xml apparently uses a pyexpat.so shared object which is compiled against Python's C API. This fails if the C API is changes between Python builds. In my case, it was looking for the _PyUnicodeUCS2_Decode symbol, but a wide Pythone build (+ucs4) will not contain the UCS2 stuff, hence pyexpat.so fails.

As for a solution -- it seems that making sure py27-xml is deactivated during the destroot phase of Python27 solves the problem, exactly as described by vloodo above:

Please let me know if I should file this as separate bug report.

comment:12 Changed 10 years ago by ned-deily (Ned Deily)

I see two separate issues here. The more important problem is that pyxx-xml is still being provided as a port and it shouldn't be. The upstream project has not been maintained in years since the most important subset of its functionality is now included in the Python standard library and there are more modern and maintained ports for added functionality, like lxml. The upstream project download page on SourceForge states Download out of date. Do not use. The package was also removed from Debian a few years ago. One of its problems is that it replaces the expat.so parser supplied with standard Python and the standard one is now more up-to-date, including a security fix that is being released in Python 3.2.3, 2.7.3, 2.6.8, and 3.1.5.

The second issue is that Build Applet is both a product of a Python 2 build and is used in the build to install itself (target install_BuildApplet in Mac/Makefile). In so doing, I think it's not quite careful enough to avoid mixing using an already installed Python 2 executable with the being-built standard library, which may have different architecture configure options (as in the cases noted in this ticket and elsewhere). That should be considered an upstream bug. But it's pretty minor and Build Applet has been removed in Python 3 so the only branch where it would be considered to be fixed would be 2.7 (since older releases are either retired or in security fix mode only).

comment:13 Changed 10 years ago by jmroot (Joshua Root)

Cf. #28832

comment:14 Changed 10 years ago by jmroot (Joshua Root)

Cc: ryanwilcox@… added

Cc maintainer of py-xml.

comment:15 Changed 10 years ago by jmroot (Joshua Root)

py*-xml deleted in r94528.

comment:16 Changed 10 years ago by jyrkiwahlstedt

Resolution: fixed
Status: newclosed

This ticket can be closed, then, according to the previous comments…

comment:17 Changed 9 years ago by mf2k (Frank Schima)

Cc: deesto@… added

Cc reporter of duplicate #36710.

comment:18 Changed 9 years ago by ryandesign (Ryan Schmidt)

Has duplicate #37019.

Note: See TracTickets for help on using tickets.