Python >= 2.2 should allow for UCS-4 builds

[jarkko.laiho@…]

Since version 2.2, it has been possible to compile Python with the option --enable-unicode=ucs4, enabling Python Unicode strings to contain characters beyond the Basic Multilingual Plane (e.g. unichr(65535+1) would no longer result in a ValueError). Without that specific option, Python defaults to using the rather obsolete UCS-2 (resulting in a "narrow build").

This can be seen by issuing the following commands in the interactive interpreter:

>>> import sys
>>> sys.maxunicode
65535

The result is the same with both the version of Python (2.3) that ships with 10.4 Tiger and with python25 from MacPorts. I have not tested this with the ports of other Python versions.

In contrast, on my Gentoo Linux system, the following output is produced instead, since it (like in all modern Linux distributions) is a "wide build":

>>> import sys
>>> sys.maxunicode
1114111

This build option is a source of some confusion in various Python implementations and platforms.

wchar_t on Mac OS X is 32 bits, and PEP 261 states: "It is also proposed that one day --enable-unicode will just default to the width of your platforms wchar_t." UCS-4 builds of Python would seem to make sense on the Mac.

If not a new default option, could a variant be made for Python ports >= 2.2 to compile with UCS-4 support? Is there a reason not to go UCS-4?

[jann@…]

Mww, what do you think about this ?

[mww@…]

If this does not break compatibility in some way, I have no objections to add this to the default build options.

Are there any problems that might occur? (besides people using "clever tricks" that include provoking ValueError...)

[jann@…]

If the --enable-unicode=ucs4 option is specified, the PythonLauncher.app doesn't build anymore, see transcript. If you can provide a fix for this I will look at this again.

Traceback (most recent call last):
  File "./../../../Lib/plat-mac/bundlebuilder.py", line 921, in ?
    main()
  File "./../../../Lib/plat-mac/bundlebuilder.py", line 871, in main
    builder.plist = Plist.fromFile(arg)
  File "/opt/local/var/db/dports/build/_Volumes_Daten_SVN_macports_dports_lang_python24/work/Python-2.4.4/Lib/plat-mac/plistlib.py", line 341, in fromFile
    rootObject = readPlist(pathOrFile)
  File "/opt/local/var/db/dports/build/_Volumes_Daten_SVN_macports_dports_lang_python24/work/Python-2.4.4/Lib/plat-mac/plistlib.py", line 77, in readPlist
    rootObject = p.parse(pathOrFile)
  File "/opt/local/var/db/dports/build/_Volumes_Daten_SVN_macports_dports_lang_python24/work/Python-2.4.4/Lib/plat-mac/plistlib.py", line 397, in parse
    from xml.parsers.expat import ParserCreate
  File "/opt/local/lib/python2.4/site-packages/_xmlplus/parsers/expat.py", line 4, in ?
    from pyexpat import *
ImportError: Failure linking new module: /opt/local/lib/python2.4/site-packages/_xmlplus/parsers/pyexpat.so: Symbol not found: _PyUnicodeUCS2_DecodeUTF8
  Referenced from: /opt/local/lib/python2.4/site-packages/_xmlplus/parsers/pyexpat.so
  Expected in: flat namespace

make[2]: *** [PythonLauncher.app] Error 1
make[1]: *** [install_PythonLauncher] Error 2
make: *** [frameworkinstallapps] Error 2
Error: Target com.apple.destroot returned: shell command " cd "/opt/local/var/db/dports/build/_Volumes_Daten_SVN_macports_dports_lang_python24/work/Python-2.4.4" && make frameworkinstall maninstall DESTDIR=/opt/local/var/db/dports/build/_Volumes_Daten_SVN_macports_dports_lang_python24/work/destroot " returned error 2
Command output:         --iconfile=./PythonInterpreter.icns \
        --bundle-id=org.python.PythonLauncher \
        --resource=./PythonSource.icns \
        --resource=./PythonWSource.icns \
        --resource=./PythonCompiled.icns \
        --resource=./English.lproj \
        --resource=./PreferenceWindow.nib \
        --resource=./factorySettings.plist \
        --plist=./Info.plist \
        build
Traceback (most recent call last):
  File "./../../../Lib/plat-mac/bundlebuilder.py", line 921, in ?
    main()
  File "./../../../Lib/plat-mac/bundlebuilder.py", line 871, in main
    builder.plist = Plist.fromFile(arg)
  File "/opt/local/var/db/dports/build/_Volumes_Daten_SVN_macports_dports_lang_python24/work/Python-2.4.4/Lib/plat-mac/plistlib.py", line 341, in fromFile
    rootObject = readPlist(pathOrFile)
  File "/opt/local/var/db/dports/build/_Volumes_Daten_SVN_macports_dports_lang_python24/work/Python-2.4.4/Lib/plat-mac/plistlib.py", line 77, in readPlist
    rootObject = p.parse(pathOrFile)
  File "/opt/local/var/db/dports/build/_Volumes_Daten_SVN_macports_dports_lang_python24/work/Python-2.4.4/Lib/plat-mac/plistlib.py", line 397, in parse
    from xml.parsers.expat import ParserCreate
  File "/opt/local/lib/python2.4/site-packages/_xmlplus/parsers/expat.py", line 4, in ?
    from pyexpat import *
ImportError: Failure linking new module: /opt/local/lib/python2.4/site-packages/_xmlplus/parsers/pyexpat.so: Symbol not found: _PyUnicodeUCS2_DecodeUTF8
  Referenced from: /opt/local/lib/python2.4/site-packages/_xmlplus/parsers/pyexpat.so
  Expected in: flat namespace

make[2]: *** [PythonLauncher.app] Error 1
make[1]: *** [install_PythonLauncher] Error 2
make: *** [frameworkinstallapps] Error 2

[jarkko.laiho@…]

Testing with a local repository and a modified Portfile, python25 seems to build fine with --enable-unicode=UCS4, but since it defaults to --disable-framework it didn't even try to build PythonLauncher.

So I tried a combined UCS4/Framework build, which also completed just fine. PythonLauncher worked, Build Applet worked, but IDLE exited right after starting, briefly flashing its icon in the Dock. (When I say "worked", I mean the app started without errors.)

I lack the unix-dev knowledge needed to try and patch python25, let alone python24 with its large amount of existing patches. If no solution for python24 is found, how about going UCS4 in python25 and future versions? It would seem that python25 is an easier target for patching, since at least it builds, even if IDLE fails to run. But what do I know?

I leave the decision in the capable hands of the maintainer :-).

[akira@…]

+1 for adding ucs4 python support. On recent Linux distribution, python is built in ucs4 enabled