pypy @7.3.1_1 fails to build on 10.15.6: rpython not found

[ned-deily (Ned Deily)]

My MacPorts installation uses a non-default installation prefix so all ports have to be built from source. Trying to upgrade to the current pypy @7.3.1_1 (on macOS 10.15.6 with Xcode and CLT 12.0) fails with:

:debug:build system:  cd "/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/pypy-7.3.1/pypy/goal" && arch -x86_64 /opt/macports/lib/pypy-bootstrap/bin/pypy ../../rpython/bin/rpython --batch --verbose --cc=/usr/bin/clang --opt=jit --make-jobs=16 targetpypystandalone
:info:build IOError: [Errno 2] No such file or directory: '../../rpython/bin/rpython'

Looking at the work directory:

-rw-r--r--   1 macports  wheel  230 Sep 16 22:07 .macports.pypy.state
drwxr-xr-x   2 macports  wheel   64 Sep 16 22:07 .tmp
drwxr-xr-x   3 macports  wheel   96 Sep 16 22:07 pypy-7.3.1
drwxr-xr-x  25 macports  wheel  800 Apr  6 12:46 pypy-release-pypy2.7-v7.3.1-d044e299e8d67a20ced0286156007c51b17db2cb
lrwxr-xr-x   1 macports  wheel   68 Sep 16 22:07 pypy2.7-7.3.1 -> pypy-release-pypy2.7-v7.3.1-d044e299e8d67a20ced0286156007c51b17db2cb
r

pypy-7.3.1 only contains pypy/goal; the symlinked pypy2.7-7..3.1 contains rypthon (and lots of other stuff).

FWIW, I backed out the most recent pypy Portfile change (06e5e6b pypy: fetch using gitlab PortGroup) and was able to get a little further. The build then failed with C errors unable to find getpagesize (header file issue?):

:debug:build SDKROOT='/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk'
:info:build Executing:  cd "/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/archive/pypy/goal" && arch -x86_64 /opt/macports/lib/pypy-bootstrap/bin/pypy ../../rpython/bin/rpython --batch --verbose --cc=/usr/bin/clang --opt=jit --make-jobs=16 targetpypystandalone
:debug:build system:  cd "/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/archive/pypy/goal" && arch -x86_64 /opt/macports/lib/pypy-bootstrap/bin/pypy ../../rpython/bin/rpython --batch --verbose --cc=/usr/bin/clang --opt=jit --make-jobs=16 targetpypystandalone
:info:build [version:WARNING] Errors getting Mercurial information: Not running from a Mercurial repository!
:info:build [translation:info] 2.7.10 (3260adbeba4a, Apr 19 2016, 13:10:19)
:info:build [PyPy 5.1.0 with GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)]
:info:build [platform:msg] Set platform with 'host' cc=/usr/bin/clang, using cc='/usr/bin/clang', version='Unknown'
:info:build [translation:info] Translating target as defined by targetpypystandalone
:info:build [platform:Error] /opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/usession-unknown-0/platcheck_53.c:102:20: error: implicit declaration of function 'getpagesize' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
:info:build [platform:Error]         long int result = getpagesize();
:info:build [platform:Error]                           ^
:info:build [platform:Error] 1 error generated.
:info:build Traceback (most recent call last):
:info:build   File "../../rpython/bin/rpython", line 20, in <module>
:info:build     main()
:info:build   File "/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/pypy-release-pypy2.7-v7.3.1-d044e299e8d67a20ced0286156007c51b17db2cb/rpython/translator/goal/translate.py", line 217, in main
:info:build     targetspec_dic, translateconfig, config, args = parse_options_and_load_target()
:info:build   File "/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/pypy-release-pypy2.7-v7.3.1-d044e299e8d67a20ced0286156007c51b17db2cb/rpython/translator/goal/translate.py", line 155, in parse_options_and_load_target
:info:build     targetspec_dic = load_target(targetspec)
:info:build   File "/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/pypy-release-pypy2.7-v7.3.1-d044e299e8d67a20ced0286156007c51b17db2cb/rpython/translator/goal/translate.py", line 97, in load_target
:info:build     mod = __import__(specname)
:info:build   File "targetpypystandalone.py", line 6, in <module>
:info:build     from pypy.interpreter import gateway
:info:build   File "/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/pypy-release-pypy2.7-v7.3.1-d044e299e8d67a20ced0286156007c51b17db2cb/pypy/interpreter/gateway.py", line 19, in <module>
:info:build     from pypy.interpreter.eval import Code
:info:build   File "/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/pypy-release-pypy2.7-v7.3.1-d044e299e8d67a20ced0286156007c51b17db2cb/pypy/interpreter/eval.py", line 6, in <module>
:info:build     from pypy.interpreter.baseobjspace import W_Root
:info:build   File "/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/pypy-release-pypy2.7-v7.3.1-d044e299e8d67a20ced0286156007c51b17db2cb/pypy/interpreter/baseobjspace.py", line 13, in <module>
:info:build     from rpython.rlib.buffer import StringBuffer
:info:build   File "/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/pypy-release-pypy2.7-v7.3.1-d044e299e8d67a20ced0286156007c51b17db2cb/rpython/rlib/buffer.py", line 16, in <module>
:info:build     from rpython.rlib import rawstorage
:info:build   File "/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/pypy-release-pypy2.7-v7.3.1-d044e299e8d67a20ced0286156007c51b17db2cb/rpython/rlib/rawstorage.py", line 49, in <module>
:info:build     cpuname = detect_cpu.autodetect()
:info:build   File "/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/pypy-release-pypy2.7-v7.3.1-d044e299e8d67a20ced0286156007c51b17db2cb/rpython/jit/backend/detect_cpu.py", line 110, in autodetect
:info:build     return detect_model_from_host_platform()
:info:build   File "/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/pypy-release-pypy2.7-v7.3.1-d044e299e8d67a20ced0286156007c51b17db2cb/rpython/jit/backend/detect_cpu.py", line 84, in detect_model_from_host_platform
:info:build     from rpython.jit.backend.x86 import detect_feature as feature
::info:build [platform:Error]                           ^
:info:build [platform:Error] 1 error generated.
:info:build Traceback (most recent call last):
:info:build   File "../../rpython/bin/rpython", line 20, in <module>
:info:build     main()
:info:build   File "/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/pypy-release-pypy2.7-v7.3.1-d044e299e8d67a20ced0286156007c51b17db2cb/rpython/translator/goal/translate.py", line 217, in main
:info:build     targetspec_dic, translateconfig, config, args = parse_options_and_load_target()
:info:build   File "/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/pypy-release-pypy2.7-v7.3.1-d044e299e8d67a20ced0286156007c51b17db2cb/rpython/translator/goal/translate.py", line 155, in parse_options_and_load_target
:info:build     targetspec_dic = load_target(targetspec)
:info:build   File "/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/pypy-release-pypy2.7-v7.3.1-d044e299e8d67a20ced0286156007c51b17db2cb/rpython/translator/goal/translate.py", line 97, in load_target
:info:build     mod = __import__(specname)
:info:build   File "targetpypystandalone.py", line 6, in <module>
:info:build     from pypy.interpreter import gateway
:info:build   File "/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/pypy-release-pypy2.7-v7.3.1-d044e299e8d67a20ced0286156007c51b17db2cb/pypy/interpreter/gateway.py", line 19, in <module>
:info:build     from pypy.interpreter.eval import Code
:info:build   File "/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/pypy-release-pypy2.7-v7.3.1-d044e299e8d67a20ced0286156007c51b17db2cb/pypy/interpreter/eval.py", line 6, in <module>
:info:build     from pypy.interpreter.baseobjspace import W_Root
:info:build   File "/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/pypy-release-pypy2.7-v7.3.1-d044e299e8d67a20ced0286156007c51b17db2cb/pypy/interpreter/baseobjspace.py", line 13, in <module>
:info:build     from rpython.rlib.buffer import StringBuffer
:info:build   File "/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/pypy-release-pypy2.7-v7.3.1-d044e299e8d67a20ced0286156007c51b17db2cb/rpython/rlib/buffer.py", line 16, in <module>
:info:build     from rpython.rlib import rawstorage
:info:build   File "/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/pypy-release-pypy2.7-v7.3.1-d044e299e8d67a20ced0286156007c51b17db2cb/rpython/rlib/rawstorage.py", line 49, in <module>
:info:build     cpuname = detect_cpu.autodetect()
:info:build   File "/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/pypy-release-pypy2.7-v7.3.1-d044e299e8d67a20ced0286156007c51b17db2cb/rpython/jit/backend/detect_cpu.py", line 110, in autodetect
:info:build     return detect_model_from_host_platform()
:info:build   File "/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/pypy-release-pypy2.7-v7.3.1-d044e299e8d67a20ced0286156007c51b17db2cb/rpython/jit/backend/detect_cpu.py", line 84, in detect_model_from_host_platform
:info:build     from rpython.jit.backend.x86 import detect_feature as feature
:info:build   File "/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/pypy-release-pypy2.7-v7.3.1-d044e299e8d67a20ced0286156007c51b17db2cb/rpython/jit/backend/x86/detect_feature.py", line 4, in <module>
:info:build     from rpython.rlib.rmmap import alloc, free
:info:build   File "/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/pypy-release-pypy2.7-v7.3.1-d044e299e8d67a20ced0286156007c51b17db2cb/rpython/rlib/rmmap.py", line 171, in <module>
:info:build     includes=includes)
:info:build   File "/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/pypy-release-pypy2.7-v7.3.1-d044e299e8d67a20ced0286156007c51b17db2cb/rpython/rtyper/tool/rffi_platform.py", line 79, in getintegerfunctionresult
:info:build     return configure(CConfig)['RESULT']
:info:build   File "/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/pypy-release-pypy2.7-v7.3.1-d044e299e8d67a20ced0286156007c51b17db2cb/rpython/rtyper/tool/rffi_platform.py", line 215, in configure
:info:build     for name, result in zip(entries, results):
:info:build   File "/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/pypy-release-pypy2.7-v7.3.1-d044e299e8d67a20ced0286156007c51b17db2cb/rpython/rtyper/tool/rffi_platform.py", line 240, in configure_entries
:info:build     writer.path, eci, ignore_errors=ignore_errors))
:info:build   File "/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/pypy-release-pypy2.7-v7.3.1-d044e299e8d67a20ced0286156007c51b17db2cb/rpython/rtyper/tool/rffi_platform.py", line 743, in run_example_code
:info:build     output = build_executable_cache(files, eci, ignore_errors=ignore_errors)
:info:build   File "/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/pypy-release-pypy2.7-v7.3.1-d044e299e8d67a20ced0286156007c51b17db2cb/rpython/tool/gcc_cache.py", line 28, in build_executable_cache
:info:build     result = platform.execute(platform.compile(c_files, eci))
:info:build   File "/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/pypy-release-pypy2.7-v7.3.1-d044e299e8d67a20ced0286156007c51b17db2cb/rpython/translator/platform/__init__.py", line 57, in compile
:info:build     ofiles = self._compile_o_files(cfiles, eci, standalone)
:info:build   File "/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/pypy-release-pypy2.7-v7.3.1-d044e299e8d67a20ced0286156007c51b17db2cb/rpython/translator/platform/__init__.py", line 79, in _compile_o_files
:info:build     ofiles.append(self._compile_c_file(self.cc, cfile, compile_args))
:info:build   File "/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/pypy-release-pypy2.7-v7.3.1-d044e299e8d67a20ced0286156007c51b17db2cb/rpython/translator/platform/posix.py", line 42, in _compile_c_file
:info:build     cwd=str(cfile.dirpath()))
:info:build   File "/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/pypy-release-pypy2.7-v7.3.1-d044e299e8d67a20ced0286156007c51b17db2cb/rpython/translator/platform/__init__.py", line 143, in _execute_c_compiler
:info:build     self._handle_error(returncode, stdout, stderr, outname)
:info:build   File "/opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/pypy-release-pypy2.7-v7.3.1-d044e299e8d67a20ced0286156007c51b17db2cb/rpython/translator/platform/__init__.py", line 155, in _handle_error
:info:build     raise CompilationError(stdout, stderr)
:info:build CompilationError: CompilationError(err="""
:info:build     /opt/macports/var/macports/build/_opt_macports_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_pypy/pypy/work/usession-unknown-0/platcheck_53.c:102:20: error: implicit declaration of function 'getpagesize' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
:info:build             long int result = getpagesize();
:info:build                               ^
:info:build     1 error generated.
:info:build     """)

[danchr (Dan Villiom Podlaski Christiansen)]

Oops 😳 I assumed everything worked just because the checksums didn't change; I obviously should have tested it. Thanks for noticing!

[danchr (Dan Villiom Podlaski Christiansen)]

In 96ad8f76b66f73afe32532178f11e7c4cc584f33/macports-ports (master):

pypy: unbreak build failures caused by late invocation of gitlab.setup

Closes: #61193

[danchr (Dan Villiom Podlaski Christiansen)]

FWIW, I backed out the most recent pypy Portfile change (06e5e6b pypy: fetch using gitlab PortGroup) and was able to get a little further. The build then failed with C errors unable to find getpagesize (header file issue?):

I noticed this when building upstream directly; the root cause is this chunk in unistd.h:

/* Removed in Issue 6 */
#if !defined(_POSIX_C_SOURCE) || _POSIX_C_SOURCE < 200112L
int      getdtablesize(void) __POSIX_C_DEPRECATED(199506L);
int      getpagesize(void) __pure2 __POSIX_C_DEPRECATED(199506L);
char    *getpass(const char *) __POSIX_C_DEPRECATED(199506L);
#endif

Presumably Apple added that in a relatively recent update to Xcode, since the build ​succeeded earlier. Binary archives should be available, and used by default, though, so I'm not sure why it tried to build PyPy on your system?

[ned-deily (Ned Deily)]

Re binary archives: see first paragraph of description above. Some users will always have to build from source.

[danchr (Dan Villiom Podlaski Christiansen)]

Replying to ned-deily:

Re binary archives: see first paragraph of description above. Some users will always have to build from source.

Right, missed that; thanks!

[danchr (Dan Villiom Podlaski Christiansen)]

There were other issues as well; I've posted a ​draft PR to fix them. Feel free to try it out; I've started upgrades, and if they succeed, I'll push that change.

[danchr (Dan Villiom Podlaski Christiansen)]

In a4d10555f5ade7c31a430b6d4abeec67cfb33ba0/macports-ports (master):

pypy: fix build with latest Xcode

Refs: #61193

[ned-deily (Ned Deily)]

Verifying that pypy (and pypy3) @7.3.1_2 now build and install without error. Thanks for the quick turnaround!

[danchr (Dan Villiom Podlaski Christiansen)]

Replying to ned-deily:

Verifying that pypy (and pypy3) @7.3.1_2 now build and install without error.

Great; thanks!