python27 +universal: destroot fails if py27-xml is installed non-universal

[cdiehl@…]

MacPorts 2.0.3
ProductName:	Mac OS X
ProductVersion:	10.7.2
BuildVersion:	11C74

$ sudo port installed | grep python
  mesa @7.11_1+python27+universal
  mesa @7.11_2+python27+universal (active)
  python24 @2.4.6_9
  python24 @2.4.6_9+universal (active)
  python26 @2.6.7_1 (active)
  python27 @2.7.2_1+universal (active)
  python31 @3.1.4_1 (active)
  python32 @3.2.2_0 (active)
  python_select @0.3_1 (active)
  xorg-libxcb @1.7_0+python27
  xorg-libxcb @1.7_0+python27+universal (active)
  xorg-xcb-proto @1.6_0+python27 (active)

Error happens during: upgrade outdated. Log file is attached.

[cdiehl@…]

Log

[ryandesign (Ryan Carsten Schmidt)]

The log is incomplete. But it looks like you're building python27 universal for i386 and x86_64, and it's failing with this error:

:info:destroot ImportError: dlopen(/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_xmlplus/parsers/pyexpat.so, 2): no suitable image found.  Did find:
:info:destroot 	/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_xmlplus/parsers/pyexpat.so: mach-o, but wrong architecture

Of what architecture is the file /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_xmlplus/parsers/pyexpat.so? What port provided it? Presumably not one of the ones listed in "port rdeps python27". Try rebuilding the port that provides that file with the universal variant.

[cdiehl@…]

$ lipo -info /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_xmlplus/parsers/pyexpat.so
Non-fat file: /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_xmlplus/parsers/pyexpat.so is architecture: x86_64

$ port rdeps python27

The following ports are dependencies of python27 @2.7.2_2:
  gettext
    libiconv
      gperf
    ncurses
      ncursesw
    expat
  zlib
  openssl
  sqlite3
    libedit
  db46
  gdbm
  bzip2
  python_select

[drkp (Dan Ports)]

Has duplicate #32096

[drkp (Dan Ports)]

I hit this problem when updating python27, too. It looks like the problem comes up because something in python27's build process uses the pyexpat module. The python27 port provides one version, in /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/pyexpat.so, but the py27-xml port also installs its own version into /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_xmlplus/parsers/pyexpat.so. The second one has a higher precedence.

If python27 is being built +universal, but py27-xml is installed non-universal, it will fail to load this module during destroot. The workaround I used was to install py27-xml +universal.

[alfredherrmann@…]

this didn't help me, but: sudo port install python27 -universal

[jmroot (Joshua Root)]

The real issue is that it's loading anything from the installed location at all. Presumably in the destroot phase it should only be looking in the work dir.

[foxtian@…]

same problem,how make it ok?

[vlsd (Vlad)]

I am having the same problem. I cannot install py27-xml +universal because it depends on python27, and I can't install python27 because it "depends" on py27-xml +universal. How do I go about fixing this cycle?

[vlsd (Vlad)]

Cc Me!

[vlsd (Vlad)]

Replying to vloodo@…:

I am having the same problem. I cannot install py27-xml +universal because it depends on python27, and I can't install python27 because it "depends" on py27-xml +universal. How do I go about fixing this cycle?

Fixed it by doing:

# sudo port -f deactivate py27-xml
# sudo port install python27 +universal
# sudo port install py27-xml +universal

[macfreek (Freek Dijkstra)]

FYI,

I encountered a somewhat similar issue. While the problem is slightly different, the resolution posted by vloodo above works just as well in this case.

In my case, I wanted to replace python27 with python27 +ucs4. I had py27-xml installed at the time I was doing the replace. This lead to the following error during the destroot phase:

DYLD_FRAMEWORK_PATH=/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python27/python27/work/Python-2.7.2:  ../python.exe ./scripts/BuildApplet.py \
		--destroot "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python27/python27/work/destroot" \
		--python=/opt/local/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python \
		--output "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python27/python27/work/destroot/Applications/MacPorts/Python 2.7/Build Applet.app" \
		./scripts/BuildApplet.py
Traceback (most recent call last):
  File "./scripts/BuildApplet.py", line 156, in <module>
    main()
  File "./scripts/BuildApplet.py", line 36, in main
    buildapplet()
  File "./scripts/BuildApplet.py", line 123, in buildapplet
    progress=verbose, destroot=destroot)
  File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python27/python27/work/Python-2.7.2/Lib/plat-mac/buildtools.py", line 122, in process
    copy_codefragment, raw, others, filename, destroot)
  File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python27/python27/work/Python-2.7.2/Lib/plat-mac/buildtools.py", line 151, in process_common
    is_update, raw, others, filename, destroot)
  File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python27/python27/work/Python-2.7.2/Lib/plat-mac/buildtools.py", line 332, in process_common_macho
    builder.plist = plistlib.Plist.fromFile(plistname)
  File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python27/python27/work/Python-2.7.2/Lib/plistlib.py", line 345, in fromFile
    rootObject = readPlist(pathOrFile)
  File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python27/python27/work/Python-2.7.2/Lib/plistlib.py", line 78, in readPlist
    rootObject = p.parse(pathOrFile)
  File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python27/python27/work/Python-2.7.2/Lib/plistlib.py", line 401, in parse
    from xml.parsers.expat import ParserCreate
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_xmlplus/parsers/expat.py", line 4, in <module>
    from pyexpat import *
ImportError: dlopen(/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_xmlplus/parsers/pyexpat.so, 2): Symbol not found: _PyUnicodeUCS2_Decode
  Referenced from: /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_xmlplus/parsers/pyexpat.so
  Expected in: flat namespace
 in /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_xmlplus/parsers/pyexpat.so
make[1]: *** [install_BuildApplet] Error 1
make[1]: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python27/python27/work/Python-2.7.2/Mac'
make: *** [frameworkinstallapps] Error 2
make: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python27/python27/work/Python-2.7.2'

In short, destroot is calling some script called ./scripts/BuildApplet.py to compile .py to .pyc or .pyo files. (Building at the destroot phase is indeed ugly; you can blame that on Python).

This script somehow imports site-packages, including py27-xml. py27-xml apparently uses a pyexpat.so shared object which is compiled against Python's C API. This fails if the C API is changes between Python builds. In my case, it was looking for the _PyUnicodeUCS2_Decode symbol, but a wide Pythone build (+ucs4) will not contain the UCS2 stuff, hence pyexpat.so fails.

As for a solution -- it seems that making sure py27-xml is deactivated during the destroot phase of Python27 solves the problem, exactly as described by vloodo above:

Please let me know if I should file this as separate bug report.

[ned-deily (Ned Deily)]

I see two separate issues here. The more important problem is that pyxx-xml is still being provided as a port and it shouldn't be. The upstream project has not been maintained in years since the most important subset of its functionality is now included in the Python standard library and there are more modern and maintained ports for added functionality, like lxml. The ​upstream project download page on SourceForge states Download out of date. Do not use. The package was also removed from Debian a few years ago. One of its problems is that it replaces the expat.so parser supplied with standard Python and the standard one is now more up-to-date, including ​a security fix that is being released in Python 3.2.3, 2.7.3, 2.6.8, and 3.1.5.

The second issue is that Build Applet is both a product of a Python 2 build and is used in the build to install itself (target install_BuildApplet in Mac/Makefile). In so doing, I think it's not quite careful enough to avoid mixing using an already installed Python 2 executable with the being-built standard library, which may have different architecture configure options (as in the cases noted in this ticket and elsewhere). That should be considered an upstream bug. But it's pretty minor and Build Applet has been removed in Python 3 so the only branch where it would be considered to be fixed would be 2.7 (since older releases are either retired or in security fix mode only).

[jmroot (Joshua Root)]

Cf. #28832

[jmroot (Joshua Root)]

Cc maintainer of py-xml.

[jmroot (Joshua Root)]

py*-xml deleted in r94528.

[jyrkiwahlstedt]

This ticket can be closed, then, according to the previous comments…

[mf2k (Frank Schima)]

Cc reporter of duplicate #36710.

[ryandesign (Ryan Carsten Schmidt)]

Has duplicate #37019.