py36-isce2 fails to rebuild with Xcode 11.2

[EJFielding (Eric Fielding)]

My Mac running 10.14.4 recently had the Xcode version updated to 11.2 (my employer has a system that does automatic upgrades so I don't have a choice). I tried to install py36-isce2 with +gcc7 and it downloads and installs a binary package, but then MacPorts finds that the port is broken and tries to rebuild it. Presumably this is because of the Xcode version change. The rebuild fails because it is not able to find the required dependencies.

sudo port install py36-isce2 +gcc7
--->  Computing dependencies for py36-isce2
--->  Fetching archive for py36-isce2
--->  Attempting to fetch py36-isce2-2.3.2_0+gcc7.darwin_18.x86_64.tbz2 from https://packages.macports.org/py36-isce2
--->  Attempting to fetch py36-isce2-2.3.2_0+gcc7.darwin_18.x86_64.tbz2.rmd160 from https://packages.macports.org/py36-isce2
--->  Installing py36-isce2 @2.3.2_0+gcc7
--->  Activating py36-isce2 @2.3.2_0+gcc7
--->  Cleaning py36-isce2
--->  Scanning binaries for linking errors
--->  Found 79 broken files, matching files to ports
--->  Found 1 broken port, determining rebuild order
You can always run 'port rev-upgrade' again to fix errors.
The following ports will be rebuilt: py36-isce2 @2.3.2+gcc7
Continue? [Y/n]: Y
--->  Computing dependencies for py36-isce2
--->  Cleaning py36-isce2
--->  Scanning binaries for linking errors
--->  Found 79 broken files, matching files to ports
--->  Found 1 broken port, determining rebuild order
--->  Rebuilding in order
     py36-isce2 @2.3.2 +gcc7
--->  Computing dependencies for py36-isce2
--->  Fetching distfiles for py36-isce2
--->  Verifying checksums for py36-isce2
--->  Extracting py36-isce2
--->  Configuring py36-isce2
--->  Building py36-isce2
Error: Failed to build py36-isce2: command execution failed
Error: See /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_python_py-isce2/py36-isce2/main.log for details.
Error: rev-upgrade failed: Error rebuilding py36-isce2
Error: Follow https://guide.macports.org/#project.tickets to report a bug.
--->  Some of the ports you installed have notes:
  py36-isce2 has the following notes:
    After installation
    ------------------

    After installation and before using ISCE, do the following:

    1) Set environment variable
    ISCE_HOME=/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/isce .
    2) Append $ISCE_HOME/applications and $ISCE_HOME/bin to PATH.
    3) Stack processing scripts can be found under /opt/local/share/py36-isce2/stack


    Other important notes
    ----------------------

    1) GDAL is required for working with Radarsat2, Terrasar-X and Sentinel 1-A. Suggested variant of gdal is
    +expat+geos+hdf5+netcdf+postgresql95+sqlite3

    2) One can use isce2 with other packages like SpiceyPy, sarpy etc. These will need to be installed from source
    separately.

I attach the main log file.

[EJFielding (Eric Fielding)]

main log showing failure to build

[piyushrpt]

From the log file, you are still using the old xcode (sysroot variable). Look into running

xcode-select -s

to switch to the latest version.

You can test if the switch was mde by running

xcrun --show-sdk-path

and it should report something like

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk

[EJFielding (Eric Fielding)]

I found that my gcc7 installation was not working because it was linked to the MacOSX10.14.sdk in the Xcode directory, which is no longer available with Xcode 11.x.

I asked on the MacPorts mailing list and Chris Jones suggested:

You might need to manually force a CLT update, as its possible it is not being correctly updated. See

https://trac.macports.org/wiki/ProblemHotlist#reinstall-clt

This did not completely solve the problem because the gcc7 pre-compiled binary was still linked to the MacOSX10.14.sdk in the Xcode directory. I then rebuilt the entire gcc7 from source (which took over an hour). I also ran xcode-select -r to reset the Xcode path.

Now xcrun --show-sdk-path reports:

/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk

Now the gcc7 installation is working and I was able to get the GitHub source version of ISCE2 to compile (this was failing before as gcc was not able to compile anything that depends on the SDK). The rebuild of the py36-isce2 port still fails, however. I attach the new main.log file.

[EJFielding (Eric Fielding)]

main log showing failure to build after fixing gcc7

[piyushrpt]

The error seems to suggest that HDF5 was not found - this might be because it was built with a different compiler. We are working on getting isce2 to compile with clang. It works with clang and gfortran from gcc7. Needs couple of fixes to work with gfortran from gcc9 - and then the recommended way to be to just use clang for everything on the new macs.

[EJFielding (Eric Fielding)]

I found that I had the same problem with gcc9 under Xcode 11.2. The pre-built binary was linked to the MacOSX10.14.sdk in the Xcode directory that is no longer available. With the pre-built binary, I can't even compile the first scons config test:

gcc conftest_0.c -I/opt/local/Library/Frameworks/Python.framework/Versions/3.6/include/python3.6m /opt/local/include
In file included from /opt/local/lib/gcc9/gcc/x86_64-apple-darwin18/9.2.0/include-fixed/syslimits.h:7,
                 from /opt/local/lib/gcc9/gcc/x86_64-apple-darwin18/9.2.0/include-fixed/limits.h:34,
                 from /opt/local/Library/Frameworks/Python.framework/Versions/3.6/include/python3.6m/Python.h:11,
                 from conftest_0.c:2:
/opt/local/lib/gcc9/gcc/x86_64-apple-darwin18/9.2.0/include-fixed/limits.h:194:61: error: no include path in which to search for limits.h
  194 | #include_next <limits.h>  /* recurse down to the real one */
      |                                                             ^
In file included from /opt/local/Library/Frameworks/Python.framework/Versions/3.6/include/python3.6m/Python.h:25,
                 from conftest_0.c:2:
/opt/local/lib/gcc9/gcc/x86_64-apple-darwin18/9.2.0/include-fixed/stdio.h:78:10: fatal error: _stdio.h: No such file or directory
   78 | #include <_stdio.h>
      |          ^~~~~~~~~~
compilation terminated.

One correction: I am running MacOS 10.14.6 on this machine.

[piyushrpt]

Looks like you did not migrate to rebuild macports after the Xcode update - wiki:Migration

[EJFielding (Eric Fielding)]

I did not do the migration because I did not upgrade the OS. We never had to do a full reinstall of all ports for an Xcode upgrade before.

[piyushrpt]

You have to create symlinks for the 10.14sdk for this to work.

> gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/opt/local/libexec/gcc/x86_64-apple-darwin18/7.4.0/lto-wrapper
Target: x86_64-apple-darwin18
Configured with: /opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_lang_gcc7/gcc7/work/gcc-7.4.0/configure --prefix=/opt/local --build=x86_64-apple-darwin18 --enable-languages=c,c++,objc,obj-c++,lto,fortran --libdir=/opt/local/lib/gcc7 --includedir=/opt/local/include/gcc7 --infodir=/opt/local/share/info --mandir=/opt/local/share/man --datarootdir=/opt/local/share/gcc-7 --with-local-prefix=/opt/local --with-system-zlib --disable-nls --program-suffix=-mp-7 --with-gxx-include-dir=/opt/local/include/gcc7/c++/ --with-gmp=/opt/local --with-mpfr=/opt/local --with-mpc=/opt/local --with-isl=/opt/local --enable-stage1-checking --disable-multilib --enable-lto --enable-libstdcxx-time --with-build-config=bootstrap-debug --with-as=/opt/local/bin/as --with-ld=/opt/local/bin/ld --with-ar=/opt/local/bin/ar --with-bugurl=https://trac.macports.org/newticket --disable-tls --with-pkgversion='MacPorts gcc7 7.4.0_3' --with-sysroot=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk
Thread model: posix
gcc version 7.4.0 (MacPorts gcc7 7.4.0_3)

tells us where its looking for the SDK (see sysroot).

Creating the symlink and pointing it to the old SDK - /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk should solve the problem.

[EJFielding (Eric Fielding)]

Rebuilding gcc7 from source also works. Now it is looking for the 10.14 SDK in the CommandLineTools directory:

cpp -v
Using built-in specs.
COLLECT_GCC=cpp
Target: x86_64-apple-darwin18
Configured with: /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_gcc7/gcc7/work/gcc-7.4.0/configure --prefix=/opt/local --build=x86_64-apple-darwin18 --enable-languages=c,c++,objc,obj-c++,lto,fortran --libdir=/opt/local/lib/gcc7 --includedir=/opt/local/include/gcc7 --infodir=/opt/local/share/info --mandir=/opt/local/share/man --datarootdir=/opt/local/share/gcc-7 --with-local-prefix=/opt/local --with-system-zlib --disable-nls --program-suffix=-mp-7 --with-gxx-include-dir=/opt/local/include/gcc7/c++/ --with-gmp=/opt/local --with-mpfr=/opt/local --with-mpc=/opt/local --with-isl=/opt/local --enable-stage1-checking --disable-multilib --enable-lto --enable-libstdcxx-time --with-build-config=bootstrap-debug --with-as=/opt/local/bin/as --with-ld=/opt/local/bin/ld --with-ar=/opt/local/bin/ar --with-bugurl=https://trac.macports.org/newticket --disable-tls --with-pkgversion='MacPorts gcc7 7.4.0_3' --with-sysroot=/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk
Thread model: posix
gcc version 7.4.0 (MacPorts gcc7 7.4.0_3)
COLLECT_GCC_OPTIONS='-E' '-v' '-mmacosx-version-min=10.14.7' '-asm_macosx_version_min=10.14' '-mtune=core2'
 /opt/local/libexec/gcc/x86_64-apple-darwin18/7.4.0/cc1 -E -quiet -v -D__DYNAMIC__ - -fPIC -mmacosx-version-min=10.14.7 -mtune=core2
ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/opt/local/include"
ignoring nonexistent directory "/opt/local/lib/gcc7/gcc/x86_64-apple-darwin18/7.4.0/../../../../../x86_64-apple-darwin18/include"
ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/Library/Frameworks"
#include "..." search starts here:
#include <...> search starts here:
 /opt/local/lib/gcc7/gcc/x86_64-apple-darwin18/7.4.0/include
 /opt/local/lib/gcc7/gcc/x86_64-apple-darwin18/7.4.0/include-fixed
 /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/usr/include
 /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks
End of search list.

The py36-isce2 port still does not build because it is not finding the hdf5 libraries. The regular scons build of the ISCE2 source works and finds the hdf5 libraries, but something about the portfile is a little different.

Replying to piyushrpt:

You have to create symlinks for the 10.14sdk for this to work.

> gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/opt/local/libexec/gcc/x86_64-apple-darwin18/7.4.0/lto-wrapper
Target: x86_64-apple-darwin18
Configured with: /opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_lang_gcc7/gcc7/work/gcc-7.4.0/configure --prefix=/opt/local --build=x86_64-apple-darwin18 --enable-languages=c,c++,objc,obj-c++,lto,fortran --libdir=/opt/local/lib/gcc7 --includedir=/opt/local/include/gcc7 --infodir=/opt/local/share/info --mandir=/opt/local/share/man --datarootdir=/opt/local/share/gcc-7 --with-local-prefix=/opt/local --with-system-zlib --disable-nls --program-suffix=-mp-7 --with-gxx-include-dir=/opt/local/include/gcc7/c++/ --with-gmp=/opt/local --with-mpfr=/opt/local --with-mpc=/opt/local --with-isl=/opt/local --enable-stage1-checking --disable-multilib --enable-lto --enable-libstdcxx-time --with-build-config=bootstrap-debug --with-as=/opt/local/bin/as --with-ld=/opt/local/bin/ld --with-ar=/opt/local/bin/ar --with-bugurl=https://trac.macports.org/newticket --disable-tls --with-pkgversion='MacPorts gcc7 7.4.0_3' --with-sysroot=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk
Thread model: posix
gcc version 7.4.0 (MacPorts gcc7 7.4.0_3)

tells us where its looking for the SDK (see sysroot).

Creating the symlink and pointing it to the old SDK - /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk should solve the problem.

[EJFielding (Eric Fielding)]

I reported the gcc7 (and gcc9) prebuilt binary problem on those ports (#59665)