py-tensorflow-data-validation: build fails on multiple macOS versions; bazel complains that 'rule is missing dependency declarations' for header files

[mascguy (Christopher Nielsen)]

Builds fail across Python variations for this port. But specifically attempted installing both the Python 3.8 and 3.9 versions, with similar errors. So the Python version doesn't appear to make a difference.

MacOS 10.14/Xcode 11.3.1:

:info:build ERROR: /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_python_py-tensorflow-data-validation/py38-tensorflow-data-validation/work/bazel_build/8f919143399337036c8563e76d591ca2/external/zlib_archive/BUILD.bazel:5:11: undeclared inclusion(s) in rule '@zlib_archive//:zlib':
:info:build this rule is missing dependency declarations for the following files included by 'zlib_archive/adler32.c':
:info:build   '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/limits.h'
:info:build   '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/cdefs.h'
:info:build   '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_symbol_aliasing.h'
[...]

On MacOS 10.15/Xcode 12.0:

:info:build ERROR: /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_python_py-tensorflow-data-validation/py38-tensorflow-data-validation/work/bazel_build/8f919143399337036c8563e76d591ca2/external/com_google_protobuf/BUILD:106:11: undeclared inclusion(s) in rule '@com_google_protobuf//:protobuf_lite':
:info:build this rule is missing dependency declarations for the following files included by 'com_google_protobuf/src/google/protobuf/arena.cc':
:info:build   '/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/pthread.h'
:info:build   '/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/string.h'
:info:build   '/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/_types.h'
[...]

Briefly discussed with @CJones in comments of issue:62725, but still a mystery at the moment.

[mascguy (Christopher Nielsen)]

Attached logs for Python 3.8 install attempts, for both MacOS 10.14 and 10.15.

[mascguy (Christopher Nielsen)]

This StackOverflow question provides some hints, but not a clear answer:

​https://stackoverflow.com/questions/43921911/how-to-resolve-bazel-undeclared-inclusions-error

[mascguy (Christopher Nielsen)]

Currently researching from the following starting point:

​https://www.google.com/search?num=50&q=bazel+rule+is+missing+dependency+declarations+macos

[mascguy (Christopher Nielsen)]

The following mentions setting environment variable BAZEL_SH, but the Bazel portgroup appears to be doing that:

​https://stackoverflow.com/questions/46614261/missing-dependencies-when-building-com-google-protobuf-protobuf-lite

​https://github.com/bazelbuild/bazel/issues/2852#issuecomment-295747929

[cjones051073 (Chris Jones)]

I've done some work on how the bazel PG, including how it handles SDKs etc. Could you update to have my changes, i.e. at least

​https://github.com/macports/macports-ports/commit/e3f85df38b56a9b75ee7222412a35852be82f350

and try again.

[mascguy (Christopher Nielsen)]

You bet, I'll switch to 10.14 and test this morning.

[mascguy (Christopher Nielsen)]

Hey Chris, the issue appears to be fixed, as a source build now succeeds on both MacOS 10.14 and 10.15. Thank you!

One issue of concern, though, relates to CCache: In the logs, I'm seeing the following set in the environment:

CCACHE_DIR=/opt/local/var/macports/build/.ccache

However, that's not correct, as I specify the location of the CCache store in macports.conf. So it looks like my override is somehow not being used. Is this a known issue...?

[mascguy (Christopher Nielsen)]

Another concern, is that part of the build is occurring during the destroot phase. Is that intentional?

[cjones051073 (Chris Jones)]

Replying to mascguy:

Hey Chris, the issue appears to be fixed, as a source build now succeeds on both MacOS 10.14 and 10.15. Thank you!

One issue of concern, though, relates to CCache: In the logs, I'm seeing the following set in the environment:

CCACHE_DIR=/opt/local/var/macports/build/.ccache

However, that's not correct, as I specify the location of the CCache store in macports.conf. So it looks like my override is somehow not being used. Is this a known issue...?

Nope, not known, until now.

The bazel PG computes the ccache dir using

​https://github.com/macports/macports-ports/blob/bd3701d5e452d4a88753e4ec9d0201b98ce84c26/_resources/port1.0/group/bazel-1.0.tcl#L144

I thought this would return the correct string but I guess not. Let me see if I can figure out how to query what is set in macports.conf...

[cjones051073 (Chris Jones)]

Replying to mascguy:

Another concern, is that part of the build is occurring during the destroot phase. Is that intentional?

what part of the build ?

[mascguy (Christopher Nielsen)]

Replying to cjones051073:

Replying to mascguy:

Another concern, is that part of the build is occurring during the destroot phase. Is that intentional?

what part of the build ?

I have no idea. Lol

But the buildbot logs are available for 10.14: ​https://build.macports.org/builders/ports-10.14_x86_64-builder/builds/85681/steps/install-port/logs/main.log

[cjones051073 (Chris Jones)]

​https://github.com/macports/macports-ports/commit/756928e796c80b70263007b2b22255bce70fd48d

[mascguy (Christopher Nielsen)]

Replying to cjones051073:

​https://github.com/macports/macports-ports/commit/756928e796c80b70263007b2b22255bce70fd48d

Thanks, that fixes the CCACHE_DIR issue

[mascguy (Christopher Nielsen)]

This has been fixed, closing.