pika @0.22.1: some failures with clang builds

[barracuda156]

It was failing before due to missing filesystem. After adding a setting to use legacysupport for Macports libcxx, it began failing on configure with atomics checks:

-- The CXX compiler identification is AppleClang 10.0.1.10010046
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/local/bin/mpicxx-mpich-mp - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Using C++17
-- CMake version: 3.24.4
-- pika version: 0.15.0
-- Nice threadlevel is disabled.
-- Stack traces are enabled.
-- Compiling with the native toolset
-- Performing Test PIKA_WITH_CXX11_ATOMIC - Failed
-- Performing Test PIKA_WITH_CXX11_ATOMIC (with -latomic) - Failed
-- Test failed, detailed output:

Change Dir: /opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_devel_pika/pika/work/build/CMakeFiles/config_tests/CMakeFiles/CMakeTmp

Run Build Command(s):/usr/bin/make -f Makefile cmTC_5c78e/fast && /Library/Developer/CommandLineTools/usr/bin/make  -f CMakeFiles/cmTC_5c78e.dir/build.make CMakeFiles/cmTC_5c78e.dir/build
Building CXX object CMakeFiles/cmTC_5c78e.dir/cxx11_std_atomic.cpp.o
/opt/local/bin/mpicxx-mpich-mp   -pipe -I/opt/local/libexec/boost/1.81/include -Os -DNDEBUG -isystem/opt/local/include/libcxx/v1 -nostdinc++ -isystem/opt/local/include/LegacySupport -I/opt/local/libexec/boost/1.81/include -I/opt/local/include -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -Werror   -arch x86_64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -mmacosx-version-min=10.14 -std=c++17 -MD -MT CMakeFiles/cmTC_5c78e.dir/cxx11_std_atomic.cpp.o -MF CMakeFiles/cmTC_5c78e.dir/cxx11_std_atomic.cpp.o.d -o CMakeFiles/cmTC_5c78e.dir/cxx11_std_atomic.cpp.o -c /opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_devel_pika/pika/work/pika-0.15.0/cmake/tests/cxx11_std_atomic.cpp
Linking CXX executable cmTC_5c78e
/opt/local/bin/cmake -E cmake_link_script CMakeFiles/cmTC_5c78e.dir/link.txt --verbose=1
/opt/local/bin/mpicxx-mpich-mp -pipe -I/opt/local/libexec/boost/1.81/include -Os -DNDEBUG -isystem/opt/local/include/libcxx/v1 -nostdinc++ -isystem/opt/local/include/LegacySupport -I/opt/local/libexec/boost/1.81/include -I/opt/local/include -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -Werror   -arch x86_64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -mmacosx-version-min=10.14 -Wl,-search_paths_first -Wl,-headerpad_max_install_names -L/opt/local/libexec/boost/1.81/lib -Wl,-headerpad_max_install_names -L/opt/local/lib -lMacportsLegacySupport -L/opt/local/lib/libcxx -Wl,-rpath,/opt/local/lib/libgcc -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk  CMakeFiles/cmTC_5c78e.dir/cxx11_std_atomic.cpp.o -o cmTC_5c78e  -latomic 
clang: error: argument unused during compilation: '-nostdinc++' [-Werror,-Wunused-command-line-argument]
make[1]: *** [cmTC_5c78e] Error 1
make: *** [cmTC_5c78e/fast] Error 2


CMake Error at cmake/pika_message.cmake:51 (message):
  pika needs support for C++11 std::atomic
Call Stack (most recent call first):
  cmake/pika_add_config_test.cmake:234 (pika_error)
  cmake/pika_add_config_test.cmake:298 (pika_add_config_test)
  cmake/pika_perform_cxx_feature_tests.cmake:15 (pika_check_for_cxx11_std_atomic)
  CMakeLists.txt:1076 (pika_perform_cxx_feature_tests)


-- Configuring incomplete, errors occurred!

It builds fine on 10.6.8 for ppc :)

[ryandesign (Ryan Carsten Schmidt)]

Seems like the solution would be either: don't use -Werror or: don't pass -nostdinc++ to a C compiler.

[ryandesign (Ryan Carsten Schmidt)]

Looks like -nostdinc++ is being added by the legacysupport 1.1 portgroup when legacysupport.use_mp_libcxx yes is set. It puts the flag into CPPFLAGS, which would get passed to both C and C++ compilers, so I'm not sure if that's right. Chris, you added the flag to fix this issue but in the issue description you suggested adding the flags to CXXFLAGS, not CPPFLAGS.

pika seems to be passing it to mpicxx-mpich-mp which sure sounds like a C++ compiler so you'd think that would be fine. You could try removing -Werror but if -nostdinc++ is needed to accomplish something, and it's being ignored, then presumably the build will still fail but with a different error. Or maybe mpicxx-mpich-mp is really calling a C compiler, in which case it doesn't need that flag.

[mascguy (Christopher Nielsen)]

Replying to ryandesign:

Looks like -nostdinc++ is being added by the legacysupport 1.1 portgroup when legacysupport.use_mp_libcxx yes is set. It puts the flag into CPPFLAGS, which would get passed to both C and C++ compilers, so I'm not sure if that's right. Chris, you added the flag to fix this issue but in the issue description you suggested adding the flags to CXXFLAGS, not CPPFLAGS.

If CXXFLAGS is the preferred way to do this, we can certainly change it. With the caveat that I'd want to test locally with the change - on an older macOS release, to ensure macports-libcxx comes into play - for everything that currently depends on it.

That being said, I'm always nervous about changing something like this, given that it doesn't cause issues for anything else. (At least, not that I'm currently aware of.)

Thoughts...?

[barracuda156]

In 44419fcaaf3a541fc4f8fe8d45c5ce3f41835346/macports-ports (master):

pika: update to 0.22.1, fix older x86 builds

Fixes two issues for some pre-Catalina Intel systems:
Configure breakage due to -nostdinc++ flag: #67446
And lack of support for aligned allocation in libc++.
It also adds a forgotten PG for backlists to work.

[barracuda156]

Replying to barracuda156:

Okay, so all systems fixed aside of 10.6 Intel with Clang:

Undefined symbols for architecture x86_64:
  "___cxa_thread_atexit", referenced from:
      pika::execution::detail::get_default_agent() in unity_0_cxx.cxx.o
      pika::detail::thread_name() in unity_0_cxx.cxx.o
      pika::threads::detail::topology::get_area_membind_nodeset(void const*, unsigned long) const in unity_0_cxx.cxx.o
      pika::threads::detail::topology::get_numa_domain(void const*) const in unity_0_cxx.cxx.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

[kencu (Ken)]

these flags in legacysupport 1.1:

append ls_cache_cppflags " -nostdinc++ -isystem${prefix}/include/libcxx/v1"

indeed should be only CXXFLAGS, not CPPFLAGS. Passing those to anything but the CXX compiler can only cause problems.

If there are builds that are not using CXXFLAGS for CXXFLAGS, but are using CPPFLAGS for CXXFLAGS, then those would have to be fixed to use the proper CXXFLAGS as they are discovered.

Fixing that will require some retooling in legacysupport 1.1, as there is currently no MACPORTS_LEGACY_SUPPORT_CXXFLAGS, and also there are a few other places I see that MACPORTS_LEGACY_SUPPORT_CPPFLAGS have been used, but those would need to be updated to use the does-not-yet-exist MACPORTS_LEGACY_SUPPORT_CXXFLAGS.

This error:

Undefined symbols for architecture x86_64:
  "___cxa_thread_atexit", referenced from:

indicates that the library containing that symbol has not been linked in. That symbol is in libc++abi.dylib on 10.6 (it's a special addition to support thread_local for 10.6 and less), and is supposed to be re-exported by libc++.dylib. One would have to examine the link line to see what is going wrong with that link.

[kencu (Ken)]

I'm a bit confused about the aligned allocation flag need.

Libcxx supports aligned allocations on all systems > 10.6 (ie that have posix_memalign) and possibly earlier systems with the additions I made to legacy support to add posix_memalign there.

I have not seen errors related to lack of aligned allocation in libcxx on older systems that I can recall.

There is a thing about a C11 aligned_alloc in clang:

​https://github.com/llvm/llvm-project/blob/0ed02621b9d87342daff0dcd9014319d4eeb2735/libcxx/include/__config#L939

#  if defined(__APPLE__)
#    if (defined(__ENVIRONMENT_MAC_OS_X_VERSION_MIN_REQUIRED__) &&                                                     \
         __ENVIRONMENT_MAC_OS_X_VERSION_MIN_REQUIRED__ < 101500)
#      define _LIBCPP_HAS_NO_C11_ALIGNED_ALLOC
#    endif

but that is supposed to be taken care of by the test in the block there...

so I'm not sure what is going on with the need to disable aligned allocations in the previous commit.

[barracuda156]

Replying to kencu:

I'm a bit confused about the aligned allocation flag need.

Libcxx supports aligned allocations on all systems > 10.6 (ie that have posix_memalign) and possibly earlier systems with the additions I made to legacy support to add posix_memalign there.

I did not dig into that deeply, to be honest, but I got an error on 10.8.5 (I think), complaining about that it is available on macOS 10.13+, so I borrowed the flag from folly port. I can try running the build without that flag and attach a log here, when I am back to the Intel machine (perhaps not today, but in a couple of days).

Re posix_memalign, I thought legacysupport provides it there? At least, AFAIK, it works on 10.6.

[kencu (Ken)]

Replying to barracuda156:

Re posix_memalign, I thought legacysupport provides it there? At least, AFAIK, it works on 10.6.

indeed, posix_memalign exists normally in the system on 10.6, and legacysupport emulates it on 10.4 and 10.5.