Opened 10 months ago

Last modified 5 months ago

#61545 new defect

python38 +universal fails to build on Big Sur (Apple Silicon Mac)

Reported by: Gregory-Gelfond (Gregory Gelfond) Owned by:
Priority: Normal Milestone:
Component: ports Version: 2.6.4
Keywords: bigsur arm64 Cc: hapaguy (Brian Kurt Fujikawa)
Port: python38

Description

sudo port install chez-scheme yields the following output:

The following dependencies will be installed: 
 icu
 libxml2
 python38
 xorg-libX11
 xorg-libpthread-stubs
 xorg-libxcb
 xorg-util-macros
 xorg-xcb-proto
 xorg-xtrans
Continue? [Y/n]: 
--->  Fetching archive for python38
--->  Attempting to fetch python38-3.8.6_0+universal.darwin_20.arm64-x86_64.tbz2 from https://packages.macports.org/python38
--->  Attempting to fetch python38-3.8.6_0+universal.darwin_20.arm64-x86_64.tbz2 from https://ywg.ca.packages.macports.org/mirror/macports/packages/python38/python38
--->  Attempting to fetch python38-3.8.6_0+universal.darwin_20.arm64-x86_64.tbz2 from https://lil.fr.packages.macports.org/python38
--->  Fetching distfiles for python38
--->  Attempting to fetch Python-3.8.6.tar.xz from https://distfiles.macports.org/python38
--->  Verifying checksums for python38                                          
--->  Extracting python38
--->  Applying patches to python38
--->  Configuring python38
--->  Building python38
Error: Failed to build python38: command execution failed
Error: See /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python38/python38/main.log for details.
Error: Follow https://guide.macports.org/#project.tickets to report a bug.
Error: Processing of port chez-scheme failed

The error appears to be with building python38.

This is strange since installing python38 with the +optimizations variant worked fine. Chez-scheme seems to be calling for a universal build variant. I am attaching the main.log for reference.

Attachments (1)

main.log (182.0 KB) - added by Gregory-Gelfond (Gregory Gelfond) 10 months ago.

Download all attachments as: .zip

Change History (19)

Changed 10 months ago by Gregory-Gelfond (Gregory Gelfond)

Attachment: main.log added

comment:1 Changed 10 months ago by jmroot (Joshua Root)

Keywords: bigsur arm64 added; chez-scheme python38 removed
Port: chez-scheme removed
Summary: chez-scheme fails to build on Big Sur (Apple Silicon Mac) due to Python38 dependencypython38 +universal fails to build on Big Sur (Apple Silicon Mac)

comment:3 Changed 10 months ago by jmroot (Joshua Root)

BTW, chez-scheme's dependencies are being installed with +universal because it's marked as only supporting i386 and x86_64 architectures. So it can only be built for x86_64 on an Apple Silicon machine and has to run with Rosetta 2.

comment:4 Changed 10 months ago by julesberry

I have the same problem with installing dependencies for R. I also can install python38 +optimizations, but R is requiring python38 +universal (python38-3.8.6_0+universal.darwin_20.arm64-x86_64.tbz2).

Similar problem (in the sense that +universal wont install) with port icu, which is discussed in ticket #45268 and #61704.

Is there a way around this or are we waiting on the developers of R to update?

comment:5 Changed 10 months ago by mohd-akram (Mohamed Akram)

Cc: mohd-akram removed

comment:6 Changed 10 months ago by kencu (Ken)

somebody is going to have to fix icu to build universal properly.

this has been a problem for years, but we got away without fixing it when universal was two flavors of Intel.

Now it is not.

comment:7 Changed 10 months ago by julesberry

I feel so stupid, but there’s a simple workaround for users ... compile everything under Rosetta 2 ! You can either run your Terminal from Rosetta 2. Or do commands as:

arch -x86_64 sudo port install R

It’s good too - code runs about twice as fast as 2 or 3 year old 13 inch equivalent MacBook Pro!

comment:8 Changed 9 months ago by arthurcnorman

I have just got a new macbook and am also hit by this. "port install python38 +universal" fails and on inspection most of its concept of "universal" goes back to the transition from ppc to intel and 32 bit to 64-bit. The reason this hits me is that I wish to build my own code as univrsal so want arm64 & x86_64 versions of libraries installeds, so I try eg the xorg libraries in universal mode and that picks up pything38 as a dependency and tries for a universal build there even though what architecture tools run on does not matter a lot to me. I can edit the python configure files and sources to get a version that builds, but not having dug that deep into macports before I do not know how to get that instralled so that macports believes it has a universal python that satisfies the depoendencies of other things... Eg when I go "port destroot" that seems to repeat earlier steps and wipes out my edits. Is there wither an explanation of how to make a local version of a package so I can patch it, or how I can get libraries that are to be build for universal to accept a non-universal version of python38?. If there is a sectio in TFM that I can R then a pointer towards it will be duly appreciated! Arthur

comment:9 Changed 9 months ago by kencu (Ken)

python38 is not working very well on arm64. python39 is better.

building anything +universal x86_64 and arm64 is very rudimentary just now. You can expect to help us fix many issues in so doing.

comment:10 Changed 9 months ago by arthurcnorman

It is very reasonable that the arm64+x86_64 +universal is still new and fragile. I am happy to try to identify issues and suggest fixes in the ones I try - but it would help to understand the protocol for testing and installing a modified version of a port without (eg) "port install" tending to revert it back to the official version that I wish to change. I am certain that those used to macports internals will view this as a dire newby question, but while I have a a decent number of years hacking autoconf scripts etc I have not looked at how they are driven my macports before, and when I have had troiuble with individual ports I have tended to fetch source and edit/configure/make as a separate activirt (eg alpine JUST need "#include <time.h>" stuck in at the front of half a dozen files to be able to build, but I do not know how to merge that in with the port system and pass it back as a proposed fix). Thanks and I am really rather impressed with how much you do have working and appearing effortless on Big Sur aarch64.

comment:11 Changed 9 months ago by kencu (Ken)

I decided to get started, with Michael, on seeing how this goes. So far:

% /opt/universal/bin/port -v installed
The following ports are currently installed:
  autoconf @2.69_5 (active) platform='darwin 20' archs='noarch' date='2021-01-09T21:44:39-0500'
  automake @1.16.3_0 (active) platform='darwin 20' archs='noarch' date='2021-01-09T21:44:46-0500'
  curl-ca-bundle @7.74.0_0 (active) platform='darwin 20' archs='noarch' date='2021-01-09T21:44:33-0500'
  gettext @0.19.8.1_2+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-09T21:35:04-0500'
  gperf @3.1_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-09T21:26:47-0500'
  libiconv @1.16_1+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-09T21:27:47-0500'
  libtool @2.4.6_11+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-09T21:45:02-0500'
  ncurses @6.2_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-09T21:26:16-0500'
  openssl @1.1.1i_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-09T21:43:05-0500'
  xz @5.2.5_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-09T21:36:56-0500'
  zlib @1.2.11_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-09T21:37:01-0500'

and we're just getting started.

BTW -- MacPorts has wonderful universal support in base and most ports. Homebrew has nil, for example, and as far as I know, no other pkg manager for MacOS has any either.

comment:12 Changed 9 months ago by arthurcnorman

Quite a lot of the ports survice +universal already. I had trouble with icu (but I am not quite certain if that was real or an accident and of the ports I tried that blocks

at-spi2-atk @2.38.0_0 at-spi2-core @2.38.0_0 atk @2.36.0_1 brotli @1.0.9_0 cairo @1.16.0_0+quartz+x11 gdk-pixbuf2 @2.42.2_0+x11 glib2 @2.58.3_1+x11 gnutls @3.6.15_0 gobject-introspection @1.60.2_4 libepoxy @1.5.4_1+python38 harfbuzz @2.7.4_0 harfbuzz-icu @2.7.4_0 graphite2 @1.3.13_1 gtk3 @3.24.23_0+x11 libarchive @3.5.1_0 libcerf @1.13_0 libxml2 @2.9.10_1 libxslt @1.1.34_4 p11-kit @0.23.21_0 pango @1.42.4_2+quartz+x11

that are tagged as depending on it. pythin38 is the other big blocker and the "38" is because that is what other ports seemsd to say the depended on, sof for instance

python38:

cyrus-sasl2 @2.1.27_2+kerberos fontconfig @2.13.1_1 freetype @2.10.4_0 gd2 @2.3.0_0+x11 xorg-libXt @1.2.0_0 xpm @3.5.13_0 kerberos5 @1.18.3_0 mesa @17.1.6_2+osmesa+python27 woff2 @1.0.2_0 Xft2 @2.3.3_0 xorg-libX11 @1.7.0_0

and a p[ile more X11 things clog there. jpeg also did not work first time and blocked tiff, webp etc. The only other one my scan showed up was osso-uuid and I do not know what tried to pull that in.

That is a short list compatred with the list of all things that worked. And my bet is that if building say xorg-libX11 knew it only depended on an executable version of python on whatev4er the current architecture and did not need to ask for a universal one then many more things woold sail through very happily. and harfbuzz-icu is maybe an issue and may block texlive-bin

It is great that on the m1 compilation is respactably fast! And let me repeat how pleased I am about all ths things that work, so this is not a complaint it is a report.

comment:13 Changed 9 months ago by kencu (Ken)

the system python on BigSur is universal, so it can be done...MacPorts does not make it easy for ports to just use that, however.

There is python27 installed as /usr/bin/python and python38 installed as /usr/bin/python3

Some ports may use python to just process scripts; there is a way to avoid a universal python there by telling the Portfile to skip the architecture check in that case for that port.

But others will need a real universal python, so that indeed seems to be the next hurdle.

icu has been difficult to build universal for a while...and then we will tackle gcc...

Last edited 9 months ago by kencu (Ken) (previous) (diff)

comment:14 Changed 9 months ago by kencu (Ken)

I hacked together a universal build of python39 here #62023 but it needs to be automated and then tested, as Ryan has found some messup with universal pythons that he referenced in another ticket #61282

comment:15 Changed 9 months ago by kencu (Ken)

Now all the way up to curl, and python39 as well:

% port -v installed | grep active
  autoconf @2.69_5 (active) platform='darwin 20' archs='noarch' date='2021-01-10T12:19:44-0800'
  automake @1.16.3_0 (active) platform='darwin 20' archs='noarch' date='2021-01-10T12:19:44-0800'
  bzip2 @1.0.8_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T09:41:38-0800'
  curl @7.74.0_0+ssl+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T13:24:31-0800'
  curl-ca-bundle @7.74.0_0 (active) platform='darwin 20' archs='noarch' date='2020-12-27T18:45:29-0800'
  db48 @4.8.30_4+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T12:20:23-0800'
  expat @2.2.10_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T09:41:39-0800'
  gdbm @1.18.1_1+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T12:21:16-0800'
  gettext @0.19.8.1_2+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T09:41:47-0800'
  glib2 @2.58.3_1+universal+x11 (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T12:52:45-0800'
  help2man @1.47.16_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T12:29:53-0800'
  libedit @20191231-3.1_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T09:41:51-0800'
  libffi @3.3_1+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T09:41:51-0800'
  libiconv @1.16_1+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T09:41:40-0800'
  libidn2 @2.3.0_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T12:48:14-0800'
  libpsl @0.21.1-20200817_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T12:53:17-0800'
  libtool @2.4.6_11+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T12:20:16-0800'
  libunistring @0.9.10_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T12:46:49-0800'
  lz4 @1.9.3_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T13:13:03-0800'
  ncurses @6.2_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T09:41:41-0800'
  openssl @1.1.1i_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T09:51:51-0800'
  p5.28-locale-gettext @1.70.0_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T12:29:47-0800'
  pcre @8.44_1+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T12:49:15-0800'
  perl5.28 @5.28.3_1+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T12:29:29-0800'
  perl5.30 @5.30.3_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T12:38:57-0800'
  pkgconfig @0.29.2_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T09:52:11-0800'
  python3_select @0.0_2 (active) platform='darwin 20' archs='noarch' date='2020-12-29T11:43:37-0800'
  python39 @3.9.1_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T12:09:42-0800'
  python_select @0.3_9 (active) platform='darwin 20' archs='noarch' date='2020-12-29T11:43:37-0800'
  readline @8.1.000_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T12:20:52-0800'
  sqlite3 @3.34.0_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T09:52:16-0800'
  texinfo @6.7_1+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T12:40:39-0800'
  the_silver_searcher @2.2.0_0 (active) platform='darwin 20' archs='x86_64' date='2021-01-10T09:58:02-0800'
  xz @5.2.5_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T09:52:20-0800'
  zlib @1.2.11_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T09:41:52-0800'
  zstd @1.4.8_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T13:21:26-0800'

MacPorts has settled on python39 as our current standard, so now that python39 can be built +universal, then we should just change anything that calls for python38 without a good reason (like libpsl) to switch to python39.

comment:16 Changed 8 months ago by arthurcnorman

With icu and python39 building in universal mode thinsg indeed go a lot further. So this is a report on what I have got going locally and what I seemed toi need to change. None of the changes I have made are in any respect deep so I hope that just comments here will help if bnobody has prodded these cases otherwise. (1) jpag showed checkdum failure on the download for me. When I just made a private Portfile wtaht wuoted the checksuims of what did get fetched it appeared to build. Note I have not tried USING anything yet - just having "port install X +universl" has felt like a triumph. (2) kerberos5 - just change to depend on python39 not 38 (3) create py39-libxml2 as a copy of py38/libxml2. But then python/libxml2.c and python/types.c have a bunch of cases where the C code goes

if PyXXX_Check(obj) {

where parens around the condition seem to be missing. When I inserted parens in half a doxen places things compiled! (4) xorg-libxcb: change to use pythin39 not 38. Thej LOTS of bits of xorg stuff build in universal mode. Hoorah!

(5) Then ossp-uuid is where I am currently stalled. The starting issue is that it uses autoconf does not use automake and that means that the autotools support files do not get refreshed very automatically - and its config.gues, .sub and various libtool files are there in versions from maybe 2008 such that they do not know about aarch64. I could copy in config.guess, config.sub and libtool.m4 and could make some progress, but right now I have libtool version disagreement woes and am muddles about whether this is down to ossp-uuid private copies or native mac libtoo vs macports or what. But that package is then a precondition that blocks quite a lot more.

(6) mesa and libexpoy (both wanted by other things) depend on py27-libxml2 has a build failure from within aetup-py atuff. I do not understand that.

YOu may be well ahead of me on all of these, so this is just encouragement and in case noting easy case and what seems the biggest blocker helps. Arthur

comment:17 Changed 7 months ago by hapaguy (Brian Kurt Fujikawa)

Cc: hapaguy added

comment:18 Changed 5 months ago by jmroot (Joshua Root)

Any improvement here with 3.8.10, which adds arm64 support upstream?

Note: See TracTickets for help on using tickets.