Opened 3 months ago

Last modified 7 days ago

#59466 assigned defect

python27 +universal fails to build universal extension modules in standard library

Reported by: rmottola (Riccardo) Owned by: jmroot (Joshua Root)
Priority: Normal Milestone:
Component: ports Version:
Keywords: leopard snowleopard Cc: Raptor007 (Raptor007), fhgwright (Fred Wright)
Port: python27

Description

recently, ICU 65 was made to depend on python 2.7 so that it builds (and it did work(. On a different machine I also run upgrade of ICU, python 2.7 gets pulled in and is updated, but then build fails:

config.status: creating samples/Makefile
config.status: creating samples/date/Makefile
config.status: creating samples/cal/Makefile
config.status: creating samples/layout/Makefile
Not rebuilding data/rules.mk, assuming prebuilt data in data/in
Spawning Python to generate test/testdata/rules.mk...
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 163, in _run_module_as_main
    mod_name, _Error)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 111, in _get_module_details
    __import__(mod_name)  # Do not catch exceptions initializing package
  File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_icu/icu/work/icu/source-i386/python/icutools/databuilder/__init__.py", line 4, in <module>
    from collections import namedtuple
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/collections.py", line 20, in <module>
    from _collections import deque, defaultdict
ImportError: No module named _collections
configure: error: Python failed to run; see above error.
./runConfigureICU: ./configure failed
Command failed:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_icu/icu/work/icu/source-i386" && ./runConfigureICU MacOSX --prefix=/opt/local --disable-layoutex --disable-samples --enable-static 
Exit code: 1

is this still an ICU problem or actually an issue with python 2.7

Attachments (2)

main.log (1.6 MB) - added by rmottola (Riccardo) 3 months ago.
python27-leopard-intel-universal.log.txt.zip (119.7 KB) - added by kencu (Ken) 8 weeks ago.

Download all attachments as: .zip

Change History (18)

comment:1 Changed 3 months ago by jmroot (Joshua Root)

Keywords: leopard added
Port: python27 added
Summary: ICU 65 build failure with python on Leopardpython27: ImportError: No module named _collections

Seems to be python that's broken. First of all, if you start python2.7 interactively and run import collections do you get the same error? If so, the _collections module must have not been installed for some reason. That reason might be difficult to diagnose without access to a Leopard system, but the log from building python27 might help.

comment:2 Changed 3 months ago by jmroot (Joshua Root)

Owner: set to jmroot
Status: newassigned

comment:3 Changed 3 months ago by rmottola (Riccardo)

Joshua, indeed it is python acting up, so it has nothing to do with ICU. Here a transcript:

Artax:~ multix$ python2.7
Python 2.7.17 (default, Oct 23 2019, 23:39:52) 
[GCC 4.2.1 (Apple Inc. build 5577)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import collections
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/collections.py", line 20, in <module>
    from _collections import deque, defaultdict
ImportError: No module named _collections
>>> 

Executing the same command on my other 10.5 Laptop (only difference: x86 vs x86-64) does not have issues.

comment:4 Changed 3 months ago by rmottola (Riccardo)

I checked and even if python was build yesterday (check the build date in the previous comment) I have no build log for python27 in /opt/local/var/macports/logs/

Perhaps, since it was considered a successful build, it was removed automatically?

comment:5 Changed 3 months ago by jmroot (Joshua Root)

Yes, it would have been. You would have to do another build with sudo port destroot python27.

comment:6 Changed 3 months ago by rmottola (Riccardo)

I redid that, build log attached

Changed 3 months ago by rmottola (Riccardo)

Attachment: main.log added

comment:7 Changed 3 months ago by jmroot (Joshua Root)

Hmm, no obvious problems in the log, and _collections.so seems to have been put in the right place in the destroot. Let's check assumptions: does /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_collections.so exist on your system currently?

comment:8 Changed 2 months ago by rmottola (Riccardo)

no, it does not exist, instead a file called _collections_failed.so exists, which is not confidence inspiring.

the file is 23rd October 2019, so from one of the latest builds.

Actually, the majority of the files there are renamed "failed":

$ ls /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload
ColorPicker.so                  _Win.so                         _testcapi_failed.so
MacOS.so                        _bisect_failed.so               array_failed.so
Nav.so                          _bsddb_failed.so                audioop_failed.so
OSATerminology.so               _codecs_cn_failed.so            autoGIL_failed.so
Python-2.7.17-py2.7.egg-info    _codecs_hk_failed.so            binascii_failed.so
_AE.so                          _codecs_iso2022_failed.so       bz2_failed.so
_AH.so                          _codecs_jp_failed.so            cPickle_failed.so
_App.so                         _codecs_kr_failed.so            cStringIO_failed.so
_CF_failed.so                   _codecs_tw_failed.so            cmath_failed.so
_CG_failed.so                   _collections_failed.so          crypt_failed.so
_CarbonEvt.so                   _csv_failed.so                  datetime_failed.so
_Cm.so                          _ctypes_failed.so               dbm_failed.so
_Ctl.so                         _ctypes_test_failed.so          fcntl_failed.so
_Dlg.so                         _curses_failed.so               future_builtins_failed.so
_Drag.so                        _curses_panel_failed.so         gestalt.so
_Evt.so                         _elementtree_failed.so          grp_failed.so
_File.so                        _functools_failed.so            icglue.so
_Fm.so                          _hashlib_failed.so              itertools_failed.so
_Folder.so                      _heapq_failed.so                math_failed.so
_Help.so                        _hotshot_failed.so              mmap_failed.so
_IBCarbon.so                    _io_failed.so                   nis_failed.so
_Icn.so                         _json_failed.so                 operator_failed.so
_Launch_failed.so               _locale_failed.so               parser_failed.so
_List.so                        _lsprof_failed.so               pyexpat_failed.so
_Menu.so                        _multibytecodec_failed.so       readline_failed.so
_Mlte.so                        _multiprocessing_failed.so      resource_failed.so
_OSA.so                         _random_failed.so               select_failed.so
_Qd.so                          _scproxy_failed.so              strop_failed.so
_Qdoffs.so                      _sha256_failed.so               syslog_failed.so
_Qt.so                          _sha512_failed.so               termios_failed.so
_Res.so                         _socket_failed.so               time_failed.so
_Scrap.so                       _sqlite3_failed.so              unicodedata_failed.so
_Snd.so                         _ssl_failed.so                  zlib_failed.so
_TE.so                          _struct_failed.so

comment:9 Changed 2 months ago by jmroot (Joshua Root)

So it looks like if you just uninstall and then install again it will be fixed. Maybe some temporary problem when you built previously? I'd like to fix it so that won't happen again, but it's hard to do that without the ability to reproduce the problem.

comment:10 Changed 2 months ago by rmottola (Riccardo)

the issue... is worse. I uninstalled all python versions, install cmake and first python gets pulled in, built and installed (and has fine shared objects) then it does the same process again, but in +universal (it is a 64bit machine and I enabled that) and now the modules are broken. When I did the test you asked, I did not add +universal...

now, scrolling back in the console I see messages like this:

building '_sha256' extension
/usr/bin/gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -pipe -Os -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.17/Mac/Include -I. -IInclude -I./Include -I/opt/local/include -I/opt/local/include/db48 -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.17/Include -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.17 -c /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.17/Modules/sha256module.c -o build/temp.macosx-10.5-x86_64-2.7/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.17/Modules/sha256module.o
/usr/bin/gcc-4.2 -bundle -undefined dynamic_lookup -L/opt/local/lib -Wl,-headerpad_max_install_names -L/opt/local/lib/db48 build/temp.macosx-10.5-x86_64-2.7/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.17/Modules/sha256module.o -L/opt/local/lib -L/opt/local/lib/db48 -o build/lib.macosx-10.5-x86_64-2.7/_sha256.so
*** WARNING: renaming "_sha256" since importing it failed: dlopen(build/lib.macosx-10.5-x86_64-2.7/_sha256.so, 2): no suitable image found.  Did find:
        build/lib.macosx-10.5-x86_64-2.7/_sha256.so: mach-o, but wrong architecture

and this is repeated the same for every .so

comment:11 Changed 2 months ago by jmroot (Joshua Root)

Summary: python27: ImportError: No module named _collectionspython27 +universal fails to build universal extension modules in standard library

comment:12 Changed 8 weeks ago by kencu (Ken)

Same thing happens for me, trying to build python27 +universal on 10.5 Intel. Log attached.

Changed 8 weeks ago by kencu (Ken)

comment:13 Changed 8 weeks ago by jmroot (Joshua Root)

The -arch flags are clearly being lost somewhere along the way, but I have no immediate idea where or why.

comment:14 Changed 3 weeks ago by Raptor007 (Raptor007)

Cc: Raptor007 added

comment:15 Changed 7 days ago by jmroot (Joshua Root)

Cc: fhgwright added
Keywords: snowleopard added

Duplicate #59947 shows this affects 10.6 as well (but apparently not always).

comment:16 Changed 7 days ago by fhgwright (Fred Wright)

It may be that this isn't *directly* OS-dependent at all. I encountered this trying to build icu +universal, which thought (incorrectly) that it needed python27 +universal. In the case of icu, the python27 dependency seems to be because it needs Python >=2.7 to build, which isn't met by the Apple Python until 10.7. Of course propagating +universal to a *build* dependency is completely silly, but that's what base currently seems to do.

Since it's probably unusual for users to specifically request +universal for Python, it's likely that it mainly arises from gratuitous universality in build procedures, explaining why the impact isn't wider.

Note: See TracTickets for help on using tickets.