py38-scipy - vodemodule.c - fatal error: 'threads.h' not found

[dyne2meter]

Legacy system issue - MacOS 10.11.6

info:build extra options: '-msse -msse2 -msse3'
:info:build clang: build/src.macosx-10.11-x86_64-3.8/scipy/integrate/vodemodule.c
:info:build clang: build/src.macosx-10.11-x86_64-3.8/build/src.macosx-10.11-x86_64-3.8/scipy/integrate/fortranobject.c
:info:build In file included from build/src.macosx-10.11-x86_64-3.8/scipy/integrate/vodemodule.c:16:
:info:build In file included from build/src.macosx-10.11-x86_64-3.8/build/src.macosx-10.11-x86_64-3.8/scipy/integrate/fortranobject.h:13:
:info:build In file included from /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:4:
:info:build In file included from /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12:
:info:build In file included from /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1944:
:info:build /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: "Using deprecated NumPy API, disable it with "          "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]
:info:build #warning "Using deprecated NumPy API, disable it with " \
:info:build  ^
:info:build In file included from build/src.macosx-10.11-x86_64-3.8/build/src.macosx-10.11-x86_64-3.8/scipy/integrate/fortranobject.c:2:
:info:build In file included from build/src.macosx-10.11-x86_64-3.8/build/src.macosx-10.11-x86_64-3.8/scipy/integrate/fortranobject.h:13:
:info:build In file included from /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:4:
:info:build In file included from /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12:
:info:build In file included from /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1944:
:info:build /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: "Using deprecated NumPy API, disable it with "          "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]
:info:build #warning "Using deprecated NumPy API, disable it with " \
:info:build  ^
:info:build build/src.macosx-10.11-x86_64-3.8/scipy/integrate/vodemodule.c:86:10: fatal error: 'threads.h' file not found
:info:build #include <threads.h>
:info:build          ^
:info:build 1 warning and 1 error generated.
:info:build 1 warning generated.
:info:build Running from SciPy source directory.
:info:build /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/distutils/system_info.py:936: UserWarning: Specified path /usr/local/include/python3.8 is invalid.
:info:build   return self.get_paths(self.section, key)
:info:build /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/distutils/system_info.py:936: UserWarning: Specified path /usr/include/python3.8 is invalid.
:info:build   return self.get_paths(self.section, key)
:info:build /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/distutils/system_info.py:936: UserWarning: Specified path /opt/local/include/python3.8 is invalid.
:info:build   return self.get_paths(self.section, key)
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'float'
:info:build append_needs: unknown need 'float'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'float'
:info:build append_needs: unknown need 'float'
:info:build append_needs: unknown need 'float'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build ########### EXT COMPILER OPTIMIZATION ###########
:info:build Platform      : 
:info:build   Architecture: x64
:info:build   Compiler    : clang
:info:build CPU baseline  : 
:info:build   Requested   : 'min'
:info:build   Enabled     : SSE SSE2 SSE3
:info:build   Flags       : -msse -msse2 -msse3
:info:build   Extra checks: none
:info:build CPU dispatch  : 
:info:build   Requested   : 'max -xop -fma4'
:info:build   Enabled     : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2
:info:build   Generated   : none
:info:build CCompilerOpt._cache_write[796] : write cache to path -> /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_python_py-scipy/py38-scipy/work/scipy-scipy-f93ad28/build/temp.macosx-10.11-x86_64-3.8/ccompiler_opt_cache_ext.py
:info:build ########### CLIB COMPILER OPTIMIZATION ###########
:info:build Platform      : 
:info:build   Architecture: x64
:info:build   Compiler    : clang
:info:build CPU baseline  : 
:info:build   Requested   : 'min'
:info:build   Enabled     : SSE SSE2 SSE3
:info:build   Flags       : -msse -msse2 -msse3
:info:build   Extra checks: none
:info:build CPU dispatch  : 
:info:build   Requested   : 'max -xop -fma4'
:info:build   Enabled     : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2
:info:build   Generated   : none
:info:build CCompilerOpt._cache_write[796] : write cache to path -> /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_python_py-scipy/py38-scipy/work/scipy-scipy-f93ad28/build/temp.macosx-10.11-x86_64-3.8/ccompiler_opt_cache_clib.py
:info:build error: Command "/usr/bin/clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -pipe -Os -isysroot/ -I/opt/local/include -DHAVE_CBLAS -Ibuild/src.macosx-10.11-x86_64-3.8/build/src.macosx-10.11-x86_64-3.8/scipy/integrate -I/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/include -Ibuild/src.macosx-10.11-x86_64-3.8/numpy/distutils/include -I/opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8 -c build/src.macosx-10.11-x86_64-3.8/scipy/integrate/vodemodule.c -o build/temp.macosx-10.11-x86_64-3.8/build/src.macosx-10.11-x86_64-3.8/scipy/integrate/vodemodule.o -MMD -MF build/temp.macosx-10.11-x86_64-3.8/build/src.macosx-10.11-x86_64-3.8/scipy/integrate/vodemodule.o.d -msse -msse2 -msse3" failed with exit status 1
:info:build Command failed:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_python_py-scipy/py38-scipy/work/scipy-1.6.1" && /opt/local/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8 setup.py --no-user-cfg config_fc --fcompiler gnu95 --f77exec /opt/local/bin/gfortran-mp-10 --f77flags='-m64 -Os -fno-second-underscore' --f90exec /opt/local/bin/gfortran-mp-10 --f90flags='-m64 -Os -fno-second-underscore' config --cc /usr/bin/clang --include-dirs /opt/local/include --library-dirs /opt/local/lib build -j2 
:info:build Exit code: 1
:error:build Failed to build py38-scipy: command execution failed
:debug:build Error code: CHILDSTATUS 57990 1
:debug:build Backtrace: command execution failed
:debug:build     while executing
:debug:build "system {*}$notty {*}$nice $fullcmdstring"
:debug:build     invoked from within
:debug:build "command_exec build"
:debug:build     (procedure "portbuild::build_main" line 8)
:debug:build     invoked from within
:debug:build "$procedure $targetname"
:error:build See /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_python_py-scipy/py38-scipy/main.log for details.

[dyne2meter]

main.log

[jmroot (Joshua Root)]

Interestingly, it seems to have succeeded on both older and newer OS versions. Just not 10.11.

[michaelld (Michael Dickens)]

Strangely enough I see this issue on 11.3-beta4 (or 5; whatever the current latest is). I -might- have a fix. Testing it right now.

[cjones051073 (Chris Jones)]

Replying to jmroot:

Interestingly, it seems to have succeeded on both older and newer OS versions. Just not 10.11.

10.10 and older already use macports clang-9.0. 10.12 is new enough to have threads.h.

[cjones051073 (Chris Jones)]

Replying to michaelld:

Strangely enough I see this issue on 11.3-beta4 (or 5; whatever the current latest is). I -might- have a fix. Testing it right now.

I've already addressed it by just extending the compiler blacklist so 10.11 also uses MP's clang 9.0

[michaelld (Michael Dickens)]

Interesting & thanks for that fix for older OSX. I still have this issue as noted on 11.3 latest beta. It errors out on trying to include <thread.h> exactly as noted. Admittedly I'm using gcc-devel ... let me try with one of the other variants ... this might be a GCC issue!

[astroboylrx (Rixin Li)]

Replying to michaelld:

Interesting & thanks for that fix for older OSX. I still have this issue as noted on 11.3 latest beta. It errors out on trying to include <thread.h> exactly as noted. Admittedly I'm using gcc-devel ... let me try with one of the other variants ... this might be a GCC issue!

Have you found a work around now? I also ran into this issue and I'm using the +gcc9 variant. The log shows

:info:build error: Command "/opt/local/bin/gcc-mp-9 -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -DHAVE_CBLAS -Ibuild/src.macosx-11.2-x86_64-3.7/build/src.macosx-11.2-x86_64-3.7/scipy/integrate -I/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/numpy/core/include -Ibuild/src.macosx-11.2-x86_64-3.7/numpy/distutils/include -I/opt/local/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c build/src.macosx-11.2-x86_64-3.7/scipy/integrate/vodemodule.c -o build/temp.macosx-11.2-x86_64-3.7/build/src.macosx-11.2-x86_64-3.7/scipy/integrate/vodemodule.o -MMD -MF build/temp.macosx-11.2-x86_64-3.7/build/src.macosx-11.2-x86_64-3.7/scipy/integrate/vodemodule.o.d -msse -msse2 -msse3" failed with exit status 1

So I manually ran this compile command and found the error is due to

build/src.macosx-11.2-x86_64-3.7/scipy/integrate/vodemodule.c:86:10: fatal error: threads.h: No such file or directory

Okay, that's more clear and makes sense. I went to this vodemodule.c and found you can actually define __STDC_NO_THREADS__ to inform it that threads.h is not supported by the compiler. I tried the previous compile command with -D__STDC_NO_THREADS__ and the command finished without error. I thought I might be able to add this into the environment variable CFLAGS as a work around, so I did export CFLAGS="-D__STDC_NO_THREADS__". However, no luck.

I further found, in the build log of MacPorts, that the installation used --no-user-cfg while running setup.py build

:info:build Command failed:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_python_py-scipy/py37-scipy/work/scipy-1.6.2" && /opt/local/Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7 setup.py --no-user-cfg build -j8

So I again manually ran the build command without that: sudo /opt/local/Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7 setup.py build -j8, which again finished successfully.

The next step should just be something like python setup.py install. However, I don't know how port handles this process. Rerun port upgrade or port install didn't work b/c port did everything from a fresh restart.

Any suggestions would be greatly appreciated!

[astroboylrx (Rixin Li)]

Replying to michaelld:

Interesting & thanks for that fix for older OSX. I still have this issue as noted on 11.3 latest beta. It errors out on trying to include <thread.h> exactly as noted. Admittedly I'm using gcc-devel ... let me try with one of the other variants ... this might be a GCC issue!

Have you had a chance to look into this? Thanks in advance.

[ryandesign (Ryan Carsten Schmidt)]

Has duplicate #62875.

Looks like this numpy bug?

​https://github.com/numpy/numpy/issues/18179

[michaelld (Michael Dickens)]

Maybe ... I've wondered if some NumPy change impacted SciPy ... let me investigate that issue & related PRs. What I don't get yet is why on Darwin SciPy/NumPy is even trying to access GLIBC ... since we don't (and can't) use it all ... correct? If that's the case then I think setting some macro maybe like noted a few comments up __STDC_NO_THREADS__ might do the trick if we can get that into the Python setup.py script somehow ...

[michaelld (Michael Dickens)]

Ah... OK so the compiler being used here is some version of GCC, which does not define __STDC_NO_THREADS__. And since __GLIBC__ is also not defined then the 2nd macro combination succeeds & thus #include <threads.h> is the code ... which in theory should work since GCC should support it in C11 mode. Hmmm ...

Also regarding the issue noted in that NumPy link: ​A relevant PR has been merged into the current 1.20 release that we're using in MP & that's what the noted issue & related PR are doing, just for "master". Clearly there needs to be more consideration given for macOS / Xcode / MP with regard to this issue when using GCC as the compiler ... almost a cross-build, but not exactly.

[michaelld (Michael Dickens)]

If I do clang -std=c11 -dM -E - </dev/null | grep STDC (Xcode 12.3 build version 12C33 or MP Clang 11) I see among others #define __STDC_NO_THREADS__ 1 ... if I do the same with any MP GCC, I do not see that #define which implies that I should be able to #include <threads.h> when using -std=c11. But, when I create a program that does just that it always fails, regardless of whether using Clang or GCC and what std I select. Maybe GCC isn't being honest in its capabilities self-assessment? Anyway, forcing this macro to be defined seems like the way to go here, and that seems to be an easy tweak to the SciPy code ... will commit once I've verified it locally.

[michaelld (Michael Dickens)]

basic patch to fix C11 STDC Threads issue

[michaelld (Michael Dickens)]

I just attached a patch (-p0) that one can apply to the top-level work directory for py37+ SciPy ... it forces disabling C11 STDC THREADS, so that the fatal error: threads.h: No such file or directory goes away & SciPy builds cleanly. I think there has to be a better solution to fix the issue in SciPy, but in all of my playing around with the MP Portfile & settings I couldn't work out a better way ... except to patch the code in NumPy, which maybe really is the right way to go ... IDK!

For folks who want to give this patch a try, here's how I'd recommend you do it .. (1) download the patch; I'll assume it is downloaded under it's current name into ~/Downloads/. ...

sudo port clean py38-scipy
sudo port patch py38-scipy
pushd $(port work py38-scipy)/scipy-scipy*
patch -p0 < ~/Downloads/patch-add-STDC_NO_THREADS.diff
popd
sudo port install py38-scipy

and that should install cleanly now. You can also do upgrade, wait until the upgrade errors out, then apply the patch & reissue the update ... which should then work. For any patch, install, upgrade (etc) make sure the variants are what you want and are consistent, otherwise port will complain at you ;)

[michaelld (Michael Dickens)]

[crickets] ... I'm going to assume that "no response is good news" and go forth with merging this change as a temporary solution to getting SciPy installing again ... in general.

[michaelld (Michael Dickens)]

Related: During my testing I was regularly iterating over the "build" phase ... at some point I realized that the executed build command was different if I also had "port" go through "configure" ... so I look in the Portfile & there is a "pre-configure" phase that sets variables for "build" and "destroot" ... which just doesn't make sense! I split those into "pre-build" and "pre-destroot" and that seems to provide the necessary commands. If the port builds all the way through then the result is the same, but if doing stage by stage the results will be different the way the Portfile is currently setup.

I don't understand what is the value in setting later stage variables in an earlier stage. It breaks stage-by-stage building, and to me doesn't make any sense. Maybe I put that code there IDK ... either way, hoping someone can explain why "port" needs these split off that way ... or, if I can just go with my new way that allows stage-by-stage builds.

[michaelld (Michael Dickens)]

Fixed by ​https://github.com/macports/macports-ports/commit/d45376ea224ffa9184c6a0ecbcbdf024ee447f12