Opened 4 years ago

Closed 3 years ago

#59466 closed defect (fixed)

python27 +universal fails to build universal extension modules in standard library

Reported by: rmottola (Riccardo) Owned by: jmroot (Joshua Root)
Priority: Normal Milestone:
Component: ports Version:
Keywords: leopard snowleopard Cc: Raptor007 (Raptor007), fhgwright (Fred Wright), devernay (Frédéric Devernay)
Port: python27

Description

recently, ICU 65 was made to depend on python 2.7 so that it builds (and it did work(. On a different machine I also run upgrade of ICU, python 2.7 gets pulled in and is updated, but then build fails:

config.status: creating samples/Makefile
config.status: creating samples/date/Makefile
config.status: creating samples/cal/Makefile
config.status: creating samples/layout/Makefile
Not rebuilding data/rules.mk, assuming prebuilt data in data/in
Spawning Python to generate test/testdata/rules.mk...
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 163, in _run_module_as_main
    mod_name, _Error)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 111, in _get_module_details
    __import__(mod_name)  # Do not catch exceptions initializing package
  File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_icu/icu/work/icu/source-i386/python/icutools/databuilder/__init__.py", line 4, in <module>
    from collections import namedtuple
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/collections.py", line 20, in <module>
    from _collections import deque, defaultdict
ImportError: No module named _collections
configure: error: Python failed to run; see above error.
./runConfigureICU: ./configure failed
Command failed:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_icu/icu/work/icu/source-i386" && ./runConfigureICU MacOSX --prefix=/opt/local --disable-layoutex --disable-samples --enable-static 
Exit code: 1

is this still an ICU problem or actually an issue with python 2.7

Attachments (2)

main.log (1.6 MB) - added by rmottola (Riccardo) 4 years ago.
python27-leopard-intel-universal.log.txt.zip (119.7 KB) - added by kencu (Ken) 4 years ago.

Download all attachments as: .zip

Change History (22)

comment:1 Changed 4 years ago by jmroot (Joshua Root)

Keywords: leopard added
Port: python27 added
Summary: ICU 65 build failure with python on Leopardpython27: ImportError: No module named _collections

Seems to be python that's broken. First of all, if you start python2.7 interactively and run import collections do you get the same error? If so, the _collections module must have not been installed for some reason. That reason might be difficult to diagnose without access to a Leopard system, but the log from building python27 might help.

comment:2 Changed 4 years ago by jmroot (Joshua Root)

Owner: set to jmroot
Status: newassigned

comment:3 Changed 4 years ago by rmottola (Riccardo)

Joshua, indeed it is python acting up, so it has nothing to do with ICU. Here a transcript:

Artax:~ multix$ python2.7
Python 2.7.17 (default, Oct 23 2019, 23:39:52) 
[GCC 4.2.1 (Apple Inc. build 5577)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import collections
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/collections.py", line 20, in <module>
    from _collections import deque, defaultdict
ImportError: No module named _collections
>>> 

Executing the same command on my other 10.5 Laptop (only difference: x86 vs x86-64) does not have issues.

comment:4 Changed 4 years ago by rmottola (Riccardo)

I checked and even if python was build yesterday (check the build date in the previous comment) I have no build log for python27 in /opt/local/var/macports/logs/

Perhaps, since it was considered a successful build, it was removed automatically?

comment:5 Changed 4 years ago by jmroot (Joshua Root)

Yes, it would have been. You would have to do another build with sudo port destroot python27.

comment:6 Changed 4 years ago by rmottola (Riccardo)

I redid that, build log attached

Changed 4 years ago by rmottola (Riccardo)

Attachment: main.log added

comment:7 Changed 4 years ago by jmroot (Joshua Root)

Hmm, no obvious problems in the log, and _collections.so seems to have been put in the right place in the destroot. Let's check assumptions: does /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_collections.so exist on your system currently?

comment:8 Changed 4 years ago by rmottola (Riccardo)

no, it does not exist, instead a file called _collections_failed.so exists, which is not confidence inspiring.

the file is 23rd October 2019, so from one of the latest builds.

Actually, the majority of the files there are renamed "failed":

$ ls /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload
ColorPicker.so                  _Win.so                         _testcapi_failed.so
MacOS.so                        _bisect_failed.so               array_failed.so
Nav.so                          _bsddb_failed.so                audioop_failed.so
OSATerminology.so               _codecs_cn_failed.so            autoGIL_failed.so
Python-2.7.17-py2.7.egg-info    _codecs_hk_failed.so            binascii_failed.so
_AE.so                          _codecs_iso2022_failed.so       bz2_failed.so
_AH.so                          _codecs_jp_failed.so            cPickle_failed.so
_App.so                         _codecs_kr_failed.so            cStringIO_failed.so
_CF_failed.so                   _codecs_tw_failed.so            cmath_failed.so
_CG_failed.so                   _collections_failed.so          crypt_failed.so
_CarbonEvt.so                   _csv_failed.so                  datetime_failed.so
_Cm.so                          _ctypes_failed.so               dbm_failed.so
_Ctl.so                         _ctypes_test_failed.so          fcntl_failed.so
_Dlg.so                         _curses_failed.so               future_builtins_failed.so
_Drag.so                        _curses_panel_failed.so         gestalt.so
_Evt.so                         _elementtree_failed.so          grp_failed.so
_File.so                        _functools_failed.so            icglue.so
_Fm.so                          _hashlib_failed.so              itertools_failed.so
_Folder.so                      _heapq_failed.so                math_failed.so
_Help.so                        _hotshot_failed.so              mmap_failed.so
_IBCarbon.so                    _io_failed.so                   nis_failed.so
_Icn.so                         _json_failed.so                 operator_failed.so
_Launch_failed.so               _locale_failed.so               parser_failed.so
_List.so                        _lsprof_failed.so               pyexpat_failed.so
_Menu.so                        _multibytecodec_failed.so       readline_failed.so
_Mlte.so                        _multiprocessing_failed.so      resource_failed.so
_OSA.so                         _random_failed.so               select_failed.so
_Qd.so                          _scproxy_failed.so              strop_failed.so
_Qdoffs.so                      _sha256_failed.so               syslog_failed.so
_Qt.so                          _sha512_failed.so               termios_failed.so
_Res.so                         _socket_failed.so               time_failed.so
_Scrap.so                       _sqlite3_failed.so              unicodedata_failed.so
_Snd.so                         _ssl_failed.so                  zlib_failed.so
_TE.so                          _struct_failed.so

comment:9 Changed 4 years ago by jmroot (Joshua Root)

So it looks like if you just uninstall and then install again it will be fixed. Maybe some temporary problem when you built previously? I'd like to fix it so that won't happen again, but it's hard to do that without the ability to reproduce the problem.

comment:10 Changed 4 years ago by rmottola (Riccardo)

the issue... is worse. I uninstalled all python versions, install cmake and first python gets pulled in, built and installed (and has fine shared objects) then it does the same process again, but in +universal (it is a 64bit machine and I enabled that) and now the modules are broken. When I did the test you asked, I did not add +universal...

now, scrolling back in the console I see messages like this:

building '_sha256' extension
/usr/bin/gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -pipe -Os -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.17/Mac/Include -I. -IInclude -I./Include -I/opt/local/include -I/opt/local/include/db48 -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.17/Include -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.17 -c /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.17/Modules/sha256module.c -o build/temp.macosx-10.5-x86_64-2.7/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.17/Modules/sha256module.o
/usr/bin/gcc-4.2 -bundle -undefined dynamic_lookup -L/opt/local/lib -Wl,-headerpad_max_install_names -L/opt/local/lib/db48 build/temp.macosx-10.5-x86_64-2.7/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.17/Modules/sha256module.o -L/opt/local/lib -L/opt/local/lib/db48 -o build/lib.macosx-10.5-x86_64-2.7/_sha256.so
*** WARNING: renaming "_sha256" since importing it failed: dlopen(build/lib.macosx-10.5-x86_64-2.7/_sha256.so, 2): no suitable image found.  Did find:
        build/lib.macosx-10.5-x86_64-2.7/_sha256.so: mach-o, but wrong architecture

and this is repeated the same for every .so

comment:11 Changed 4 years ago by jmroot (Joshua Root)

Summary: python27: ImportError: No module named _collectionspython27 +universal fails to build universal extension modules in standard library

comment:12 Changed 4 years ago by kencu (Ken)

Same thing happens for me, trying to build python27 +universal on 10.5 Intel. Log attached.

Changed 4 years ago by kencu (Ken)

comment:13 Changed 4 years ago by jmroot (Joshua Root)

The -arch flags are clearly being lost somewhere along the way, but I have no immediate idea where or why.

comment:14 Changed 4 years ago by Raptor007 (Raptor007)

Cc: Raptor007 added

comment:15 Changed 4 years ago by jmroot (Joshua Root)

Cc: fhgwright added
Keywords: snowleopard added

Duplicate #59947 shows this affects 10.6 as well (but apparently not always).

comment:16 Changed 4 years ago by fhgwright (Fred Wright)

It may be that this isn't *directly* OS-dependent at all. I encountered this trying to build icu +universal, which thought (incorrectly) that it needed python27 +universal. In the case of icu, the python27 dependency seems to be because it needs Python >=2.7 to build, which isn't met by the Apple Python until 10.7. Of course propagating +universal to a *build* dependency is completely silly, but that's what base currently seems to do.

Since it's probably unusual for users to specifically request +universal for Python, it's likely that it mainly arises from gratuitous universality in build procedures, explaining why the impact isn't wider.

comment:17 Changed 4 years ago by devernay (Frédéric Devernay)

it seems like the lib-dynload modules are compiled twice, once with the arch flags and once without. The second time it is built seems to be during the install phase. 10.6 here, building on i386:

building 'itertools' extension
/opt/local/bin/clang-mp-3.4 -fno-strict-aliasing -fno-common -dynamic -arch x86_64 -arch i386 -pipe -Os -arch x86_64 -arch i386 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python27/python27/work/Python-2.7.17/Mac/Include -I. -IInclude -I./Include -I/opt/local/include -I/opt/local/include/db48 -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python27/python27/work/Python-2.7.17/Include -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python27/python27/work/Python-2.7.17 -c /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python27/python27/work/Python-2.7.17/Modules/itertoolsmodule.c -o build/temp.macosx-10.6-intel-2.7/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python27/python27/work/Python-2.7.17/Modules/itertoolsmodule.o
/opt/local/bin/clang-mp-3.4 -bundle -undefined dynamic_lookup -arch x86_64 -arch i386 -L/opt/local/lib -Wl,-headerpad_max_install_names -L/opt/local/lib/db48 -arch x86_64 -arch i386 build/temp.macosx-10.6-intel-2.7/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python27/python27/work/Python-2.7.17/Modules/itertoolsmodule.o -L/opt/local/lib -L/opt/local/lib/db48 -o build/lib.macosx-10.6-intel-2.7/itertools.so

...

building 'itertools' extension
/opt/local/bin/clang-mp-3.4 -fno-strict-aliasing -fno-common -dynamic -pipe -Os -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python27/python27/work/Python-2.7.17/Mac/Include -I. -IInclude -I./Include -I/opt/local/include -I/opt/local/include/db48 -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python27/python27/work/Python-2.7.17/Include -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python27/python27/work/Python-2.7.17 -c /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python27/python27/work/Python-2.7.17/Modules/itertoolsmodule.c -o build/temp.macosx-10.6-i386-2.7/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python27/python27/work/Python-2.7.17/Modules/itertoolsmodule.o
/opt/local/bin/clang-mp-3.4 -bundle -undefined dynamic_lookup -arch x86_64 -arch i386 -L/opt/local/lib -Wl,-headerpad_max_install_names -L/opt/local/lib/db48 -arch x86_64 -arch i386 build/temp.macosx-10.6-i386-2.7/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python27/python27/work/Python-2.7.17/Modules/itertoolsmodule.o -L/opt/local/lib -L/opt/local/lib/db48 -o build/lib.macosx-10.6-i386-2.7/itertools.so
ld: warning: ignoring file build/temp.macosx-10.6-i386-2.7/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python27/python27/work/Python-2.7.17/Modules/itertoolsmodule.o, file was built for unsupported file format which is not the architecture being linked (i386)
*** WARNING: renaming "itertools" since importing it failed: dynamic module does not define init function (inititertools)

@kencu maybe you could pin python27 to 2.7.16_2 on 10.5 and 10.6 (LeopardPorts and SnowLeopardPorts) until we have a solution? That version doesn't seem to be affected (checked the binary packages from http://packages.macports.org/python27/ and the modules dont have the _failed.so suffix). http://packages.macports.org/python27/python27-2.7.17_0+universal.darwin_10.i386-x86_64.tbz2 is broken.

Last edited 4 years ago by devernay (Frédéric Devernay) (previous) (diff)

comment:18 Changed 4 years ago by devernay (Frédéric Devernay)

Cc: devernay added

comment:19 Changed 3 years ago by jmroot (Joshua Root)

As far as I can tell, the python27 binary for 10.6 no longer has this issue:

% lipo -info ./python27-2.7.18_2+universal.darwin_10.i386-x86_64/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_collections.so
Architectures in the fat file: ./python27-2.7.18_2+universal.darwin_10.i386-x86_64/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_collections.so are: x86_64 i386 

Do you still see the issue on 10.5?

comment:20 Changed 3 years ago by jmroot (Joshua Root)

Resolution: fixed
Status: assignedclosed

No response; assuming fixed.

Note: See TracTickets for help on using tickets.