MacPorts Python decimal module broken; Python installed from source works correctly

[josharian@…]

I noticed very unusual behavior in Python's decimal module, and reported it here: ​http://stackoverflow.com/questions/7590137/dividing-decimals-yields-invalid-results-in-python-2-5-to-2-7. It was tracked down to the MacPorts Python. I have since found simple reproduction instructions:

(1) Install MacPorts Python 2.7 (2.5 and 2.6 also work). Run:

python2.7 -m doctest /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/decimal.py

Result: 6 doctest failures, including simple things like:

Failed example:

print Decimal(3).sqrt()

Expected:

1.73205080756887729

Got:

-8.4467440737095516

(2) Install Python 2.7 from source (code from ​http://python.org/download/releases/2.7.2/, vanilla configure && make && make install). Run:

/usr/local/bin/python2.7 -m doctest /usr/local/lib/python2.7/decimal.py

All doctests pass.

The decimal.py modules are pure Python, and are identical, so the different must be in the compilation of Python itself.

For reference, my MacPorts Python version reports:

Python 2.7.2 (default, Sep 13 2011, 13:20:34) [GCC 4.2.1 Compatible Apple Clang 3.0 (tags/Apple/clang-211.9)] on darwin

My from-source Python version:

Python 2.7.2 (default, Sep 28 2011, 15:37:49) [GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.1.00)] on darwin

I'm running OS X 10.6.8. Please let me know what other details I can provide to help debug this.

[danielluke (Daniel J. Luke)]

As some additional data points, on 10.5.8/ppc and 10.7.1/x86_64, macports python2.6 and 2.7 pass the tests and work as expected for me.

[blb@…]

What's the result from running port -v installed python27? The decimal.py doctest works fine for me on 10.6.8, 64bit, no +universal...

Hmm, I note your MacPorts python says

Python 2.7.2 (default, Sep 13 2011, 13:20:34) [GCC 4.2.1 Compatible Apple Clang 3.0 (tags/Apple/clang-211.9)] on darwin 

Are you using Xcode 4? I still use 3.2.6, perhaps that could be it?

Also FYI, Cc (and assigned) need full email addresses (use port info <portname>).

[ned-deily (Ned Deily)]

There are known problems building Python with clang on OS X. See, for instance, ​http://bugs.python.org/issue11149

[josharian@…]

What's the result from running port -v installed python27? The decimal.py doctest works fine for me on 10.6.8, 64bit, no +universal...

$ port -v installed python27
The following ports are currently installed:
  python27 @2.7.1_3 platform='darwin 10' archs='x86_64'
  python27 @2.7.2_1 (active) platform='darwin 10' archs='x86_64'

Hmm, I note your MacPorts python says

Python 2.7.2 (default, Sep 13 2011, 13:20:34) [GCC 4.2.1 Compatible Apple Clang 3.0 (tags/Apple/clang-211.9)] on darwin 

Are you using Xcode 4? I still use 3.2.6, perhaps that could be it?

I am indeed using Xcode 4; that could definitely be it. Apple's made a *bunch* of compiler changes recently -- clang front-end, LLVM backend, etc.

I also have the gcc port installed (gcc44 @4.4.6_0+universal). If there's a good way to do so, I'd be happy to try reinstalling the Python port with different compilers, to try to narrow it down. Any pointers?

Also FYI, Cc (and assigned) need full email addresses (use port info <portname>).

Oops! Thanks. I just saw jwa in the Portfile and hoped that was enough. :)

[josharian@…]

There are known problems building Python with clang on OS X. See, for instance, ​http://bugs.python.org/issue11149

Thanks for the pointer. That seems like a very likely suspect.

An increasing number of people will be using clang over time. Perhaps it'd be worth (ugh) having a gcc port install as a dependency, and use that known good compiler to build Python?

[ned-deily (Ned Deily)]

I don't have the short-lived Xcode 4 package for OS X 10.6 but, if it is like the Xcode 4 package for 10.7, there should be 3 C compilers included: gcc-4.2, gcc-llvm, and clang. Modifying the port to do something like export CC=/usr/bin/gcc-4.2 would probably fix the problem. I don't know if that's the best way to fix this in general for MacPorts ports. You could also revert to the more standard Xcode 3 for 10.6.

[josharian@…]

Replying to nad@…:

I don't have the short-lived Xcode 4 package for OS X 10.6 but, if it is like the Xcode 4 package for 10.7, there should be 3 C compilers included: gcc-4.2, gcc-llvm, and clang. Modifying the port to do something like export CC=/usr/bin/gcc-4.2 would probably fix the problem.

I've been working on getting a build going using gcc-4.2. No luck yet (simple env var exports not doing it, my first variants attempt failed), but I will try more of the techniques at ​https://trac.macports.org/wiki/UsingTheRightCompiler tomorrow and report back.

I don't know if that's the best way to fix this in general for MacPorts ports.

Nor do I. I do suspect, however, that this will be an increasingly common problem in the short term...at least for anyone who uses the decimal module. Guess we might be rare, though. :)

You could also revert to the more standard Xcode 3 for 10.6.

Alas, I'm doing iOS development, so that's not really a viable option for me.

[blb@…]

Since dluke@ reports 10.7's build works fine (which uses clang) and apparently Xcode 4.0 & 4.1 cause MacPorts to default to llvm-gcc-4.2, perhaps avoiding llvm is the answer. If you try setting

configure.compiler gcc-4.2

in python27's Portfile, then install again, see if that one works. If that's the answer we'd probably want to copy how gmp does this​ since that'll limit it to just where llvm is used.

[josharian@…]

Since dluke@ reports 10.7's build works fine (which uses clang) and apparently Xcode 4.0 & 4.1 cause MacPorts to default to llvm-gcc-4.2, perhaps avoiding llvm is the answer. If you try setting

configure.compiler gcc-4.2

in python27's Portfile, then install again, see if that one works.

Awesome! Thanks, that did it. (FYI, I wasn't sure where precisely to put it in the Portfile, so I added it as a variant and used that variant.) Fix confirmed!

If that's the answer we'd probably want to copy how gmp does this​ since that'll limit it to just where llvm is used.

I trust I can leave this to someone who actually knows what they're doing? :)

[josharian@…]

Also reported upstream at ​http://bugs.python.org/issue13061

[josharian@…]

As of note: Per ​http://bugs.python.org/issue13061, I've confirmed that adding

configure.cppflags-append -fwrapv
configure.cflags-append -fwrapv
configure.cxxflags-append -fwrapv

is an alternative solution to the problem. I don't know which solution is preferable.

[ned-deily (Ned Deily)]

Note that a similar configure change is likely needed with all Python versions. There seem to be other unresolved issues with Python and clang optimization and python-dev hasn't yet thoroughly explored or tested builds with llvm-gcc, either. My recommendation would be to stick with gcc-4.2 (or gcc-4.0 on 10.5 and earlier) for all releases of python until further notice.

[jmroot (Joshua Root)]

#30031