bazel build points to builder directory _opt_bblocal_var_buildworker_ports_build_ports_devel_bazel

[essandess (Steve Smith)]

I’m trying to build/update py-tensforflow-metadata and am hitting this (bizarre) bazel issue.

First, the build fails with the error:

:info:build Execution platform: @local_config_platform//:host
:info:build Use --sandbox_debug to see verbose messages from the sandbox
:info:build xcrun: error: can't exec '/opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_devel_bazel/bazel-3.7/work/bazelwrap/cc' (errno=No such file or directory)

Weird that bazel wants to look in a MacPorts build directory.

Searching for this string in ${worksrcpath}, it appears in the file created during the build stage:

${worksrcpath}/bazel_build/install/<hash>/embedded_tools/tools/osx/crosstool/wrapped_clang.cc:

  if (binary_name == "wrapped_clang_pp") {
    tool_name = "/opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_devel_bazel/bazel-3.7/work/bazelwrap/cxx";
  } else if (binary_name == "wrapped_clang") {
    tool_name = "/opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_devel_bazel/bazel-3.7/work/bazelwrap/cc";

This is not the file from ​https://github.com/bazelbuild/bazel/blob/master/tools/osx/crosstool/wrapped_clang.cc

To be clear, I am not building bazel-3.7 here. I am building py38-tensorflow-metadata, which uses bazel-3.7. I’ve attached the log file. It was produced with a build (please see attached Portfile) that adds bazel.build_opts-append --sandbox_debug, not that I see that helping a diagnosis.

FWIW, I looked in the binary /opt/local/libexec/bazel-3.7/bin/bazel for the string _opt_bblocal_var_buildworker_ports_build_ports_devel_bazel but it’s not there, at least not in plain text. I have no idea how this path is being introduced at the build stage.

[essandess (Steve Smith)]

main.log

[essandess (Steve Smith)]

Portfile for updated py-tensorflow-metadata

[ryandesign (Ryan Carsten Schmidt)]

The replacement of the unqualified "clang" and "clang++" in wrapped_clang.cc with their full paths is being performed by the bazel 1.0 portgroup. The bazel 1.0 portgroup also requests the use of the compiler_wrapper 1.0 portgroup, thus the full compiler paths are to temporary wrapper scripts that are only valid for the duration of the port build. The "bazel/bazel-3.7" part of the path you showed tells us this path was generated while building the bazel-3.7 subport of the bazel port; this path must be baked into the files installed by bazel-3.7 somewhere, though I also cannot find it.

Regardless what compiler path is baked into bazel-3.7, it is wrong that py-tensorflow-metadata is trying to use it. Ports must use the compiler MacPorts tells them to use. However, given all the other problems we already know about with bazel, it would not surprise me if bazel makes it impossible to use the right compiler. If that's the case, then bazel must not bake temporary compiler paths into itself; it should bake in paths to compilers that are guaranteed to exist, like /usr/bin/cc and /usr/bin/c++.

[cjones051073 (Chris Jones)]

As far as I was aware, the path to the compilers used to build bazel where never expose outside that builds, to ports using bazel. If this is now happening then something has changed to cause this...

[ryandesign (Ryan Carsten Schmidt)]

The bazel executable contains embedded zip-compressed data. It's in there.

With the bazel-3.7 @3.7.2_1 binary for Catalina:

$ cd /tmp
$ unzip /opt/local/bin/bazel-3.7
$ grep -C 2 'binary_name == "wrapped_clang"' embedded_tools/tools/osx/crosstool/wrapped_clang.cc
  if (binary_name == "wrapped_clang_pp") {
    tool_name = "clang++";
  } else if (binary_name == "wrapped_clang") {
    tool_name = "clang";
  } else {

After rebuilding it from source:

$ unzip /opt/local/bin/bazel-3.7
$ grep -C 2 'binary_name == "wrapped_clang"' embedded_tools/tools/osx/crosstool/wrapped_clang.cc
  if (binary_name == "wrapped_clang_pp") {
    tool_name = "/opt/local/var/macports/build/_Users_rschmidt_macports_macports-ports-ryandesign-fork_devel_bazel/bazel-3.7/work/bazelwrap/cxx";
  } else if (binary_name == "wrapped_clang") {
    tool_name = "/opt/local/var/macports/build/_Users_rschmidt_macports_macports-ports-ryandesign-fork_devel_bazel/bazel-3.7/work/bazelwrap/cc";
  } else {

The bazel-3.7 binary for macOS 10.15 was produced in April 2021 while the binary for macOS 12 x86_64 was made in November 2021.

There was a flurry of changes to the bazel 1.0 portgroup, starting with [5d9765ddd7d0c8f16d25beeca7be37fd74c42f1d/macports-ports] in early May 2021, that introduced the code that modified wrapped_clang.cc.

Since these changes to the bazel 1.0 portgroup have an effect on the files that get installed with bazel-3.7 (and maybe other bazel versions), the revisions of the bazel ports should be increased, either now or better yet after you've made further changes to mitigate the problem.

[telotortium (Robert Irelan)]

I've also encountered this as an end user.

I have

port installed | grep bazel
  bazel @4.2.2_0 (active)

and I'm trying to build ​Anki from source on MacOS following their instructions, but I'm receiving a similar error:

INFO: Analyzed target //qt:runanki (2 packages loaded, 262 targets configured).
INFO: Found 1 target...
ERROR: /private/var/tmp/_bazel_bytedance/9966f08c7f44483c2c460b2db85aad34/external/rules_rust/util/process_wrapper/BUILD.bazel:3:10: Compiling util/process_wrapper/process_wrapper.cc [for host] failed: (Aborted): wrapped_clang failed: error executing command external/local_config_cc/wrapped_clang '-D_FORTIFY_SOURCE=1' -fstack-protector -fcolor-diagnostics -Wall -Wthread-safety -Wself-assign -fno-omit-frame-pointer -g0 -O2 -DNDEBUG '-DNS_BLOCK_ASSERTIONS=1' ... (remaining 32 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
xcrun: error: can't exec '/opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_devel_bazel/bazel/work/bazelwrap/cc' (errno=No such file or directory)
Error in child process '/usr/bin/xcrun'. 71
Target //qt:runanki failed to build
Use --verbose_failures to see the command lines of failed build steps.
ERROR: /Users/bytedance/misc/build/anki/ts/graphs/BUILD.bazel:43:8 Bundling Javascript ts/graphs/index.ts [esbuild] failed: (Aborted): wrapped_clang failed: error executing command external/local_config_cc/wrapped_clang '-D_FORTIFY_SOURCE=1' -fstack-protector -fcolor-diagnostics -Wall -Wthread-safety -Wself-assign -fno-omit-frame-pointer -g0 -O2 -DNDEBUG '-DNS_BLOCK_ASSERTIONS=1' ... (remaining 32 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
INFO: Elapsed time: 4.444s, Critical Path: 3.29s
INFO: 44 processes: 32 internal, 12 darwin-sandbox.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully

[essandess (Steve Smith)]

This is also an issue on py*-tfx-bsl.

[essandess (Steve Smith)]

The issue appears to be that these reinplace lines in the bazel PG,

​https://github.com/macports/macports-ports/blob/17650ce5ce243e23518ef25d27f135e702cb9872/_resources/port1.0/group/bazel-1.0.tcl#L267-L268

are incompatible with these reinplace lines in the bazel Portfile,

​https://github.com/macports/macports-ports/blob/57e385306ceed921f829e0aa7a3bb29f9948ec83/devel/bazel/Portfile#L279-L280

The fix appears to be to be just keeping around an original version of the file ./tools/osx/crosstool/wrapped_clang.cc and copying this back in a post-build phase, or in post-destroot.

I'd post a PR for this, but I can't get bazel to build locally. Chris, is this correct? Would you please post a fix?

[cjones051073 (Chris Jones)]

Sorry, I've kinda given up on bazel myself. far to much of a PITA to deal with any more...

[mascguy (Christopher Nielsen)]

Replying to cjones051073:

Sorry, I've kinda given up on bazel myself. far to much of a PITA to deal with any more...

I'll try to help with this.

[essandess (Steve Smith)]

Replying to cjones051073:

Sorry, I've kinda given up on bazel myself. far to much of a PITA to deal with any more...

Sorry, understood. Unfortunately bazel is necessary to maintain tensorflow tools, so hopefully the number of PITA bazel interactions can be limited to just enough to get TF.

[essandess (Steve Smith)]

In d514b1a4d3aa3ae496f26ab5f2101486570e8c06/macports-ports (master):

PG bazel-1.0.tcl: Bugfix

• Fixes: #64442

[essandess (Steve Smith)]

In 94a406552293645e2b775cf17441ae5e65115455/macports-ports (master):

bazel: Bugfix, Update to version 6.1.1

• Fixes: #64442