opam @2.1.5 fails to build on Sonoma: sed: opam.install: No such file or directory

[barracuda156]

:info:build File "src/core/opamStubsTypes.ml", line 1:
:info:build Warning 70 [missing-mli]: Cannot find interface file.
:info:build File "src/solver/opamCudfSolverSig.ml", line 1:
:info:build Warning 70 [missing-mli]: Cannot find interface file.
:info:build File "src/tools/opam_installer.ml", line 1:
:info:build Warning 70 [missing-mli]: Cannot find interface file.
:info:build dune build --profile=release --root .   -- opam-installer.install opam.install
:info:build sed -f process.sed opam.install > processed-opam.install
:info:build sed: opam.install: No such file or directory
:info:build make: *** [processed-opam.install] Error 1
:info:build make: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_ocaml_opam/opam/work/opam-full-2.1.5'
:info:build Command failed:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_ocaml_opam/opam/work/opam-full-2.1.5" && /usr/bin/make -j1 -w lib-ext all 
:info:build Exit code: 2
:error:build Failed to build opam: command execution failed
:debug:build Error code: CHILDSTATUS 58690 2

[barracuda156]

Also fails on PowerPC for me now:

make[1]: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_ocaml_opam/opam/work/opam-full-2.1.5/src_ext'
dune build --profile=release --root .   -- opam-installer.install opam.install
File "_build/default/src/ocaml-flags-standard.sexp", line 1, characters 0-0:
Error: no s-expression found in input
File "_build/default/src/ocaml-flags-configure.sexp", line 1, characters 0-0:
Error: no s-expression found in input
File "_build/default/src/ocaml-context-flags.sexp", line 1, characters 0-0:
Error: no s-expression found in input
File "src_ext/mccs/src/dune", line 36, characters 0-121:
36 | (rule
37 |  (targets cxxflags.sexp clibs.sexp flags.sexp)
38 |  (deps context_flags.ml (universe))
39 |  (action
40 |   (run ocaml %{deps})))
Error: Rule failed to generate the following targets:
- src_ext/mccs/src/clibs.sexp
- src_ext/mccs/src/cxxflags.sexp
- src_ext/mccs/src/flags.sexp
File "src/format/opamInterpLexer.mll", line 1, character 0: syntax error.
File "src/format/opamLineLexer.mll", line 1, character 0: syntax error.
File "_build/default/src_ext/mccs/src/glpk/cflags.sexp", line 1, characters 0-0:
Error: no s-expression found in input
File "src_ext/opam-file-format/src/opamBaseParser.mly", line 0: unexpected end-of-file
File "src_ext/cudf/lib/cudf_822_lexer.mll", line 1, character 0: syntax error.
File "src_ext/opam-file-format/src/opamLexer.mll", line 1, character 0: syntax error.
File "src_ext/cudf/lib/cudf_822_parser.mly", line 0: unexpected end-of-file
File "src_ext/cudf/lib/cudf_type_lexer.mll", line 1, character 0: syntax error.
File "src_ext/cudf/lib/cudf_type_parser.mly", line 0: unexpected end-of-file
File "src_ext/ocamlgraph/src/dot_parser.mly", line 0: unexpected end-of-file
File "src_ext/ocamlgraph/src/gml.mll", line 1, character 0: syntax error.
File "src_ext/result/dune", line 11, characters 0-42:
11 | (rule
12 |  (copy# %{read:selected} result.ml))
Error: No rule found for src_ext/result
File "src_ext/ocamlgraph/src/dot_lexer.mll", line 1, character 0: syntax error.
File "src_ext/dose3/src/common/shell_lexer.mll", line 1, character 0: syntax error.
File "src_ext/base64/src/dune", line 7, characters 0-60:
7 | (rule
8 |  (copy %{read:../config/which-unsafe-file} unsafe.ml))
Error: No rule found for src_ext/base64/src
make: *** [build-opam] Error 1
make: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_ocaml_opam/opam/work/opam-full-2.1.5'
Command failed:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_ocaml_opam/opam/work/opam-full-2.1.5" && /usr/bin/make -j1 -w lib-ext all 
Exit code: 2
Error: Failed to build opam: command execution failed

[pmetzger (Perry E. Metzger)]

It works for me on Sonoma. I find this puzzling.

[pmetzger (Perry E. Metzger)]

(BTW, I have no PowerPC hardware nor has any been made by apple in a very long time. You can feel free to debug that if you like.)

[barracuda156]

Replying to pmetzger:

(BTW, I have no PowerPC hardware nor has any been made by apple in a very long time. You can feel free to debug that if you like.)

I solved the first error (it is the same as with Dune, so borrowing a fix from there works), but the got __Unwind_Resume undefined symbols. That often happens when C compiler is used to link a library instead of C++ one. However, I could not easily find where it is even set in Makefiles. Do you have an idea?

16. S. For Sonoma, I will clean everything and try again. But I installed everything default there. Puzzling.

[barracuda156]

Replying to pmetzger:

It works for me on Sonoma. I find this puzzling.

Same result, still fails. I have nothing custom Ocaml-related on this system, and pretty much everything is default (well, I have upgraded MPICH to use modern Fortran, but I am sure it is irrelevant for opam). So no ideas too.

[barracuda156]

Still fails on 14.2 with Xcode 15.1

[pmetzger (Perry E. Metzger)]

It is now failing for me. I can't explain why. The "sed" invocation in the Makefile doesn't even look like it could work in principle given that the file it's trying to access isn't in the directory where the build is happening. This is all quite puzzling. I don't know if I have time to work on it very soon, so please feel free to go after the problem.

[barracuda156]

Replying to pmetzger:

It is now failing for me. I can't explain why. The "sed" invocation in the Makefile doesn't even look like it could work in principle given that the file it's trying to access isn't in the directory where the build is happening. This is all quite puzzling. I don't know if I have time to work on it very soon, so please feel free to go after the problem.

Should we bring this to upstream?

[pmetzger (Perry E. Metzger)]

Yes, but try building it straight without MacPorts first just to exclude the possibility that it's our fault.

[barracuda156]

UPD for PowerPC (just to keep track): I got to the following:

File "src/client/dune", line 27, characters 15-23:
27 |   (name        opamMain)
                    ^^^^^^^^
Undefined symbols:
  "__Unwind_Resume", referenced from:
      __Z19call_mccs_protected6SolverPciP11CUDFproblem in libmccs_stubs.a(mccs_stubs.o)
      _call_solver in libmccs_stubs.a(mccs_stubs.o)
      __Z9ml2c_vpkgP16Virtual_packagesl in libmccs_stubs.a(mccs_stubs.o)
      __Z13ml2c_propertyP16Virtual_packagesPSt3mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEP12CUDFPropertySt4lessIS7_ESaISt4pairIKS7_S9_EEEl in libmccs_stubs.a(mccs_stubs.o)
      __Z16ml2c_propertydefP16Virtual_packagesl in libmccs_stubs.a(mccs_stubs.o)
      __Z12ml2c_packageP16Virtual_packagesPSt3mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEP12CUDFPropertySt4lessIS7_ESaISt4pairIKS7_S9_EEERil in libmccs_stubs.a(mccs_stubs.o)
      __Z20ml2c_propertydeflistB5cxx11P16Virtual_packagesl in libmccs_stubs.a(mccs_stubs.o)
      _gen_problem in libmccs_stubs.a(mccs_stubs.o)
      __ZN16Virtual_packages3getEPKc in libmccs_stubs.a(mccs_stubs.o)
      __ZN20CUDFVersionedPackageC4EPKci in libmccs_stubs.a(cudf_tools.o)
      __ZN18CUDFVirtualPackageC4EPKci in libmccs_stubs.a(cudf_tools.o)
      __Z19get_criteria_lambdaPcRjc in libmccs_stubs.a(mccscudf.o)
      __Z26get_criteria_property_namePcRj in libmccs_stubs.a(mccscudf.o)
      __Z36get_criteria_property_name_and_scopePcRjR11Count_scope in libmccs_stubs.a(mccscudf.o)
      __Z16process_criteriaPcRjbPSt6vectorIP17abstract_criteriaSaIS3_EE in libmccs_stubs.a(mccscudf.o)
      __Z9call_mccs6SolverPciP11CUDFproblemPP15abstract_solver in libmccs_stubs.a(mccscudf.o)
      __ZN9lp_solver13add_objectiveEv in libmccs_stubs.a(lp_solver.o)
      __Z20compute_reduced_CUDFP11CUDFproblem in libmccs_stubs.a(cudf_reductions.o)
      __ZNSt8_Rb_treeIySt4pairIKySt6vectorIP20CUDFVersionedPackageSaIS4_EEESt10_Select1stIS7_ESt4lessIyESaIS7_EE17_M_construct_nodeIJRKS7_EEEvPSt13_Rb_tree_nodeIS7_EDpOT_.isra.0 in libmccs_stubs.a(constraint_generation.o)
      __ZNSt8_Rb_treeIySt4pairIKySt6vectorIP20CUDFVersionedPackageSaIS4_EEESt10_Select1stIS7_ESt4lessIyESaIS7_EE7_M_copyILb0ENSD_11_Alloc_nodeEEEPSt13_Rb_tree_nodeIS7_ESI_PSt18_Rb_tree_node_baseRT0_.isra.0 in libmccs_stubs.a(constraint_generation.o)
      __ZNSt8_Rb_treeIySt4pairIKySt6vectorIP20CUDFVersionedPackageSaIS4_EEESt10_Select1stIS7_ESt4lessIyESaIS7_EE7_M_copyILb0ENSD_20_Reuse_or_alloc_nodeEEEPSt13_Rb_tree_nodeIS7_ESI_PSt18_Rb_tree_node_baseRT0_.isra.0 in libmccs_stubs.a(constraint_generation.o)
      __Z18preprocess_upgradeP11CUDFproblemRiRSt6vectorI14an_upgrade_setSaIS3_EE in libmccs_stubs.a(constraint_generation.o)
      __Z20generate_constraintsP11CUDFproblemR15abstract_solverR17abstract_combiner in libmccs_stubs.a(constraint_generation.o)
      __ZN11glpk_solver5solveEi in libmccs_stubs.a(glpk_solver.o)
      __ZN11glpk_solver13add_objectiveEv in libmccs_stubs.a(glpk_solver.o)
ld: symbol(s) not found
collect2: error: ld returned 1 exit status
File "_none_", line 1:
Error: Error while building custom runtime system
make: *** [build-opam] Error 1

And I think I know how to fix this now. Hopefully.

UPD. dune stupidly compiles C++ source with C compiler:

$ (cd _build/default/src_ext/mccs/src && /opt/local/bin/gcc-mp-13 -arch ppc -O2 -fno-strict-aliasing -fwrapv -pthread -pipe -Os -arch ppc -I . -DUSEGLPK -g -I /opt/local/lib/ocaml -I ../../cudf/lib -I ../../extlib/src -I glpk -I ../../stdlib-shims/src -o changed_criteria.o -c changed_criteria.cpp)
$ (cd _build/default/src_ext/mccs/src && /opt/local/bin/gcc-mp-13 -arch ppc -O2 -fno-strict-aliasing -fwrapv -pthread -pipe -Os -arch ppc -I . -DUSEGLPK -g -I /opt/local/lib/ocaml -I ../../cudf/lib -I ../../extlib/src -I glpk -I ../../stdlib-shims/src -o count_criteria.o -c count_criteria.cpp)
$ (cd _build/default/src_ext/mccs/src && /opt/local/bin/gcc-mp-13 -arch ppc -O2 -fno-strict-aliasing -fwrapv -pthread -pipe -Os -arch ppc -I . -DUSEGLPK -g -I /opt/local/lib/ocaml -I ../../cudf/lib -I ../../extlib/src -I glpk -I ../../stdlib-shims/src -o cudf_reductions.o -c cudf_reductions.cpp)
$ (cd _build/default/src_ext/mccs/src && /opt/local/bin/gcc-mp-13 -arch ppc -O2 -fno-strict-aliasing -fwrapv -pthread -pipe -Os -arch ppc -I . -DUSEGLPK -g -I /opt/local/lib/ocaml -I ../../cudf/lib -I ../../extlib/src -I glpk -I ../../stdlib-shims/src -o constraint_generation.o -c constraint_generation.cpp)
$ (cd _build/default/src_ext/mccs/src && /opt/local/bin/gcc-mp-13 -arch ppc -O2 -fno-strict-aliasing -fwrapv -pthread -pipe -Os -arch ppc -I . -DUSEGLPK -g -I /opt/local/lib/ocaml -I ../../cudf/lib -I ../../extlib/src -I glpk -I ../../stdlib-shims/src -o lexagregate_combiner.o -c lexagregate_combiner.cpp)
$ (cd _build/default/src_ext/mccs/src && /opt/local/bin/gcc-mp-13 -arch ppc -O2 -fno-strict-aliasing -fwrapv -pthread -pipe -Os -arch ppc -I . -DUSEGLPK -g -I /opt/local/lib/ocaml -I ../../cudf/lib -I ../../extlib/src -I glpk -I ../../stdlib-shims/src -o cudf_tools.o -c cudf_tools.cpp)
$ (cd _build/default/src_ext/mccs/src && /opt/local/bin/gcc-mp-13 -arch ppc -O2 -fno-strict-aliasing -fwrapv -pthread -pipe -Os -arch ppc -I . -DUSEGLPK -g -I /opt/local/lib/ocaml -I ../../cudf/lib -I ../../extlib/src -I glpk -I ../../stdlib-shims/src -o glpk_solver.o -c glpk_solver.cpp)
$ (cd _build/default/src_ext/mccs/src && /opt/local/bin/gcc-mp-13 -arch ppc -O2 -fno-strict-aliasing -fwrapv -pthread -pipe -Os -arch ppc -I . -DUSEGLPK -g -I /opt/local/lib/ocaml -I ../../cudf/lib -I ../../extlib/src -I glpk -I ../../stdlib-shims/src -o lp_solver.o -c lp_solver.cpp)
$ (cd _build/default/src_ext/mccs/src && /opt/local/bin/gcc-mp-13 -arch ppc -O2 -fno-strict-aliasing -fwrapv -pthread -pipe -Os -arch ppc -I . -DUSEGLPK -g -I /opt/local/lib/ocaml -I ../../cudf/lib -I ../../extlib/src -I glpk -I ../../stdlib-shims/src -o new_criteria.o -c new_criteria.cpp)
$ (cd _build/default/src_ext/mccs/src && /opt/local/bin/gcc-mp-13 -arch ppc -O2 -fno-strict-aliasing -fwrapv -pthread -pipe -Os -arch ppc -I . -DUSEGLPK -g -I /opt/local/lib/ocaml -I ../../cudf/lib -I ../../extlib/src -I glpk -I ../../stdlib-shims/src -o mccscudf.o -c mccscudf.cpp)

What could possibly go wrong.

UPD2. Oh, this is broken since 2018: ​https://github.com/ocaml/dune/issues/949

[barracuda156]

Upon setting DUNE_CONFIG__COPY_FILE=portable via build.env and disabling mccs (due to the issue above), I got to the same error as on Sonoma:

make[1]: Leaving directory `/opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_ocaml_opam/opam/work/opam-full-2.1.5/src_ext'
dune build --profile=release --root .   -- opam-installer.install opam.install
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
File "src/core/opamStubsTypes.ml", line 1:
Warning 70 [missing-mli]: Cannot find interface file.
File "src/tools/opam_installer.ml", line 1:
Warning 70 [missing-mli]: Cannot find interface file.
File "src/solver/opamCudfSolverSig.ml", line 1:
Warning 70 [missing-mli]: Cannot find interface file.
dune build --profile=release --root .   -- opam-installer.install opam.install
sed -f process.sed opam.install > processed-opam.install
sed: opam.install: No such file or directory
make: *** [processed-opam.install] Error 1
make: Leaving directory `/opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_ocaml_opam/opam/work/opam-full-2.1.5'
Command failed:  cd "/opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_ocaml_opam/opam/work/opam-full-2.1.5" && /usr/bin/make -j1 -w lib-ext all 
Exit code: 2

[barracuda156]

Opened an issue with upstream: ​https://github.com/ocaml/opam/issues/5770

[barracuda156]

In 509f11f95b972d688929b80369b0159d4e8b2e2e/macports-ports (master):

opam: fix bundled dune, fix build

Fixes: #68586