Python >= 2.2 should allow for UCS-4 builds

[jarkko.laiho@…]

Since version 2.2, it has been possible to compile Python with the option --enable-unicode=ucs4, enabling Python Unicode strings to contain characters beyond the Basic Multilingual Plane (e.g. unichr(65535+1) would no longer result in a ValueError). Without that specific option, Python defaults to using the rather obsolete UCS-2 (resulting in a "narrow build").

This can be seen by issuing the following commands in the interactive interpreter:

>>> import sys
>>> sys.maxunicode
65535

The result is the same with both the version of Python (2.3) that ships with 10.4 Tiger and with python25 from MacPorts. I have not tested this with the ports of other Python versions.

In contrast, on my Gentoo Linux system, the following output is produced instead, since it (like in all modern Linux distributions) is a "wide build":

>>> import sys
>>> sys.maxunicode
1114111

This build option is a source of some confusion in various Python implementations and platforms.

wchar_t on Mac OS X is 32 bits, and ​PEP 261 states: "It is also proposed that one day --enable-unicode will just default to the width of your platforms wchar_t." UCS-4 builds of Python would seem to make sense on the Mac.

If not a new default option, could a variant be made for Python ports >= 2.2 to compile with UCS-4 support? Is there a reason not to go UCS-4?

[roederja]

Mww, what do you think about this ?

[mww@…]

If this does not break compatibility in some way, I have no objections to add this to the default build options.

Are there any problems that might occur? (besides people using "clever tricks" that include provoking ValueError...)

[roederja]

If the --enable-unicode=ucs4 option is specified, the PythonLauncher.app doesn't build anymore, see transcript. If you can provide a fix for this I will look at this again.

Traceback (most recent call last):
  File "./../../../Lib/plat-mac/bundlebuilder.py", line 921, in ?
    main()
  File "./../../../Lib/plat-mac/bundlebuilder.py", line 871, in main
    builder.plist = Plist.fromFile(arg)
  File "/opt/local/var/db/dports/build/_Volumes_Daten_SVN_macports_dports_lang_python24/work/Python-2.4.4/Lib/plat-mac/plistlib.py", line 341, in fromFile
    rootObject = readPlist(pathOrFile)
  File "/opt/local/var/db/dports/build/_Volumes_Daten_SVN_macports_dports_lang_python24/work/Python-2.4.4/Lib/plat-mac/plistlib.py", line 77, in readPlist
    rootObject = p.parse(pathOrFile)
  File "/opt/local/var/db/dports/build/_Volumes_Daten_SVN_macports_dports_lang_python24/work/Python-2.4.4/Lib/plat-mac/plistlib.py", line 397, in parse
    from xml.parsers.expat import ParserCreate
  File "/opt/local/lib/python2.4/site-packages/_xmlplus/parsers/expat.py", line 4, in ?
    from pyexpat import *
ImportError: Failure linking new module: /opt/local/lib/python2.4/site-packages/_xmlplus/parsers/pyexpat.so: Symbol not found: _PyUnicodeUCS2_DecodeUTF8
  Referenced from: /opt/local/lib/python2.4/site-packages/_xmlplus/parsers/pyexpat.so
  Expected in: flat namespace

make[2]: *** [PythonLauncher.app] Error 1
make[1]: *** [install_PythonLauncher] Error 2
make: *** [frameworkinstallapps] Error 2
Error: Target com.apple.destroot returned: shell command " cd "/opt/local/var/db/dports/build/_Volumes_Daten_SVN_macports_dports_lang_python24/work/Python-2.4.4" && make frameworkinstall maninstall DESTDIR=/opt/local/var/db/dports/build/_Volumes_Daten_SVN_macports_dports_lang_python24/work/destroot " returned error 2
Command output:         --iconfile=./PythonInterpreter.icns \
        --bundle-id=org.python.PythonLauncher \
        --resource=./PythonSource.icns \
        --resource=./PythonWSource.icns \
        --resource=./PythonCompiled.icns \
        --resource=./English.lproj \
        --resource=./PreferenceWindow.nib \
        --resource=./factorySettings.plist \
        --plist=./Info.plist \
        build
Traceback (most recent call last):
  File "./../../../Lib/plat-mac/bundlebuilder.py", line 921, in ?
    main()
  File "./../../../Lib/plat-mac/bundlebuilder.py", line 871, in main
    builder.plist = Plist.fromFile(arg)
  File "/opt/local/var/db/dports/build/_Volumes_Daten_SVN_macports_dports_lang_python24/work/Python-2.4.4/Lib/plat-mac/plistlib.py", line 341, in fromFile
    rootObject = readPlist(pathOrFile)
  File "/opt/local/var/db/dports/build/_Volumes_Daten_SVN_macports_dports_lang_python24/work/Python-2.4.4/Lib/plat-mac/plistlib.py", line 77, in readPlist
    rootObject = p.parse(pathOrFile)
  File "/opt/local/var/db/dports/build/_Volumes_Daten_SVN_macports_dports_lang_python24/work/Python-2.4.4/Lib/plat-mac/plistlib.py", line 397, in parse
    from xml.parsers.expat import ParserCreate
  File "/opt/local/lib/python2.4/site-packages/_xmlplus/parsers/expat.py", line 4, in ?
    from pyexpat import *
ImportError: Failure linking new module: /opt/local/lib/python2.4/site-packages/_xmlplus/parsers/pyexpat.so: Symbol not found: _PyUnicodeUCS2_DecodeUTF8
  Referenced from: /opt/local/lib/python2.4/site-packages/_xmlplus/parsers/pyexpat.so
  Expected in: flat namespace

make[2]: *** [PythonLauncher.app] Error 1
make[1]: *** [install_PythonLauncher] Error 2
make: *** [frameworkinstallapps] Error 2

[jarkko.laiho@…]

Testing with a local repository and a modified Portfile, python25 seems to build fine with --enable-unicode=UCS4, but since it defaults to --disable-framework it didn't even try to build PythonLauncher.

So I tried a combined UCS4/Framework build, which also completed just fine. PythonLauncher worked, Build Applet worked, but IDLE exited right after starting, briefly flashing its icon in the Dock. (When I say "worked", I mean the app started without errors.)

I lack the unix-dev knowledge needed to try and patch python25, let alone python24 with its large amount of existing patches. If no solution for python24 is found, how about going UCS4 in python25 and future versions? It would seem that python25 is an easier target for patching, since at least it builds, even if IDLE fails to run. But what do I know?

I leave the decision in the capable hands of the maintainer :-).

[akira@…]

+1 for adding ucs4 python support. On recent Linux distribution, python is built in ucs4 enabled

[akira@…]

Attached a patch for adding UCS-4 support.

Python 2.5.2 (r252:60911, Aug 30 2008, 02:52:21) 
[GCC 4.0.1 (Apple Inc. build 5484)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.maxunicode
1114111
>>> if len(u'\U00010800') == 1:
...     print "UCS4"
... else:
...     print "UCS2"
... 
UCS4

[nox@…]

Cc Me!

[MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)]

Cc Me!

[MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)]

ucs4 variant added to python30 in r46197.

[MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)]

Since the ucs4 variant is now available in python26 and python30, can this ticket be closed, or should work continue on python25?

[(none)]

Milestone Port Enhancements deleted

[nox@…]

Why is it a variant?

[nox@…]

Any update on this? Why should it be a variant?

[jarkko.laiho@…]

Whoah, forgot this ticket existed. +1 for closing it, UCS-4 in python26/30 ought to be good enough.

[nox@…]

I didn't say we should close it, I asked why UCS-4 is a variant.

[jarkko.laiho@…]

nox, that's right, your message just caused a CC to be sent to my e-mail address, and my response was directed at the old comment by mcalhoun (I've somehow ignored the CCs sent between then and now) suggesting closing this ticket due to python26/30 having UCS-4 variants.

The passage of time has really made this ticket redundant, IMHO. Anyone in a real need of a UCS-4 build of Python from MacPorts has moved on to using python26 by now, so I don't think it's worth the trouble to make python24 or python25 UCS-4 builds. I originally opened this ticket, and the need having disappeared, would close it if I could.

Your concern about whether a variant is the correct solution for UCS-4 builds or not is probably valid, but I think you might consider opening a separate ticket for it; you'd probably get more responses that way. This ticket is basically dead (and good riddance ;-)

[blb@…]

Per Ronald Oussoren's comment in python's ​issue 5766, UCS4 and the Mac-specific stuff don't work right...A variant is definitely the better way to go.