Opened 9 months ago

Closed 5 months ago

#62488 closed defect (fixed)

py38-scipy - vodemodule.c - fatal error: 'threads.h' not found

Reported by: dyne2meter Owned by: michaelld (Michael Dickens)
Priority: Normal Milestone:
Component: ports Version: 2.6.4
Keywords: elcapitan Cc: coffeyjj, cjones051073 (Chris Jones), mkuron (Michael Kuron), astroboylrx (Rixin Li), thermalecology
Port: py-scipy

Description

Legacy system issue - MacOS 10.11.6

info:build extra options: '-msse -msse2 -msse3'
:info:build clang: build/src.macosx-10.11-x86_64-3.8/scipy/integrate/vodemodule.c
:info:build clang: build/src.macosx-10.11-x86_64-3.8/build/src.macosx-10.11-x86_64-3.8/scipy/integrate/fortranobject.c
:info:build In file included from build/src.macosx-10.11-x86_64-3.8/scipy/integrate/vodemodule.c:16:
:info:build In file included from build/src.macosx-10.11-x86_64-3.8/build/src.macosx-10.11-x86_64-3.8/scipy/integrate/fortranobject.h:13:
:info:build In file included from /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:4:
:info:build In file included from /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12:
:info:build In file included from /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1944:
:info:build /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: "Using deprecated NumPy API, disable it with "          "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]
:info:build #warning "Using deprecated NumPy API, disable it with " \
:info:build  ^
:info:build In file included from build/src.macosx-10.11-x86_64-3.8/build/src.macosx-10.11-x86_64-3.8/scipy/integrate/fortranobject.c:2:
:info:build In file included from build/src.macosx-10.11-x86_64-3.8/build/src.macosx-10.11-x86_64-3.8/scipy/integrate/fortranobject.h:13:
:info:build In file included from /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:4:
:info:build In file included from /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12:
:info:build In file included from /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1944:
:info:build /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: "Using deprecated NumPy API, disable it with "          "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]
:info:build #warning "Using deprecated NumPy API, disable it with " \
:info:build  ^
:info:build build/src.macosx-10.11-x86_64-3.8/scipy/integrate/vodemodule.c:86:10: fatal error: 'threads.h' file not found
:info:build #include <threads.h>
:info:build          ^
:info:build 1 warning and 1 error generated.
:info:build 1 warning generated.
:info:build Running from SciPy source directory.
:info:build /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/distutils/system_info.py:936: UserWarning: Specified path /usr/local/include/python3.8 is invalid.
:info:build   return self.get_paths(self.section, key)
:info:build /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/distutils/system_info.py:936: UserWarning: Specified path /usr/include/python3.8 is invalid.
:info:build   return self.get_paths(self.section, key)
:info:build /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/distutils/system_info.py:936: UserWarning: Specified path /opt/local/include/python3.8 is invalid.
:info:build   return self.get_paths(self.section, key)
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'float'
:info:build append_needs: unknown need 'float'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'float'
:info:build append_needs: unknown need 'float'
:info:build append_needs: unknown need 'float'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'int'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build append_needs: unknown need 'double'
:info:build ########### EXT COMPILER OPTIMIZATION ###########
:info:build Platform      : 
:info:build   Architecture: x64
:info:build   Compiler    : clang
:info:build CPU baseline  : 
:info:build   Requested   : 'min'
:info:build   Enabled     : SSE SSE2 SSE3
:info:build   Flags       : -msse -msse2 -msse3
:info:build   Extra checks: none
:info:build CPU dispatch  : 
:info:build   Requested   : 'max -xop -fma4'
:info:build   Enabled     : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2
:info:build   Generated   : none
:info:build CCompilerOpt._cache_write[796] : write cache to path -> /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_python_py-scipy/py38-scipy/work/scipy-scipy-f93ad28/build/temp.macosx-10.11-x86_64-3.8/ccompiler_opt_cache_ext.py
:info:build ########### CLIB COMPILER OPTIMIZATION ###########
:info:build Platform      : 
:info:build   Architecture: x64
:info:build   Compiler    : clang
:info:build CPU baseline  : 
:info:build   Requested   : 'min'
:info:build   Enabled     : SSE SSE2 SSE3
:info:build   Flags       : -msse -msse2 -msse3
:info:build   Extra checks: none
:info:build CPU dispatch  : 
:info:build   Requested   : 'max -xop -fma4'
:info:build   Enabled     : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2
:info:build   Generated   : none
:info:build CCompilerOpt._cache_write[796] : write cache to path -> /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_python_py-scipy/py38-scipy/work/scipy-scipy-f93ad28/build/temp.macosx-10.11-x86_64-3.8/ccompiler_opt_cache_clib.py
:info:build error: Command "/usr/bin/clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -pipe -Os -isysroot/ -I/opt/local/include -DHAVE_CBLAS -Ibuild/src.macosx-10.11-x86_64-3.8/build/src.macosx-10.11-x86_64-3.8/scipy/integrate -I/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/include -Ibuild/src.macosx-10.11-x86_64-3.8/numpy/distutils/include -I/opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8 -c build/src.macosx-10.11-x86_64-3.8/scipy/integrate/vodemodule.c -o build/temp.macosx-10.11-x86_64-3.8/build/src.macosx-10.11-x86_64-3.8/scipy/integrate/vodemodule.o -MMD -MF build/temp.macosx-10.11-x86_64-3.8/build/src.macosx-10.11-x86_64-3.8/scipy/integrate/vodemodule.o.d -msse -msse2 -msse3" failed with exit status 1
:info:build Command failed:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_python_py-scipy/py38-scipy/work/scipy-1.6.1" && /opt/local/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8 setup.py --no-user-cfg config_fc --fcompiler gnu95 --f77exec /opt/local/bin/gfortran-mp-10 --f77flags='-m64 -Os -fno-second-underscore' --f90exec /opt/local/bin/gfortran-mp-10 --f90flags='-m64 -Os -fno-second-underscore' config --cc /usr/bin/clang --include-dirs /opt/local/include --library-dirs /opt/local/lib build -j2 
:info:build Exit code: 1
:error:build Failed to build py38-scipy: command execution failed
:debug:build Error code: CHILDSTATUS 57990 1
:debug:build Backtrace: command execution failed
:debug:build     while executing
:debug:build "system {*}$notty {*}$nice $fullcmdstring"
:debug:build     invoked from within
:debug:build "command_exec build"
:debug:build     (procedure "portbuild::build_main" line 8)
:debug:build     invoked from within
:debug:build "$procedure $targetname"
:error:build See /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_python_py-scipy/py38-scipy/main.log for details.

Attachments (2)

main.log.gz (59.4 KB) - added by dyne2meter 9 months ago.
main.log
patch-add-STDC_NO_THREADS.diff (2.6 KB) - added by michaelld (Michael Dickens) 7 months ago.
basic patch to fix C11 STDC Threads issue

Download all attachments as: .zip

Change History (22)

Changed 9 months ago by dyne2meter

Attachment: main.log.gz added

main.log

comment:1 Changed 9 months ago by mf2k (Frank Schima)

Cc: michaelld@… removed
Owner: set to michaelld
Port: py-scipy added; py38-scipy removed
Status: newassigned

comment:2 Changed 9 months ago by jmroot (Joshua Root)

Keywords: elcapitan added

Interestingly, it seems to have succeeded on both older and newer OS versions. Just not 10.11.

comment:3 Changed 9 months ago by michaelld (Michael Dickens)

Strangely enough I see this issue on 11.3-beta4 (or 5; whatever the current latest is). I -might- have a fix. Testing it right now.

comment:4 Changed 9 months ago by coffeyjj

Cc: coffeyjj added

comment:5 Changed 9 months ago by jmroot (Joshua Root)

Cc: cjones051073 added

comment:6 in reply to:  2 Changed 9 months ago by cjones051073 (Chris Jones)

Replying to jmroot:

Interestingly, it seems to have succeeded on both older and newer OS versions. Just not 10.11.

10.10 and older already use macports clang-9.0. 10.12 is new enough to have threads.h.

comment:7 in reply to:  3 Changed 9 months ago by cjones051073 (Chris Jones)

Replying to michaelld:

Strangely enough I see this issue on 11.3-beta4 (or 5; whatever the current latest is). I -might- have a fix. Testing it right now.

I've already addressed it by just extending the compiler blacklist so 10.11 also uses MP's clang 9.0

comment:8 Changed 9 months ago by michaelld (Michael Dickens)

Interesting & thanks for that fix for older OSX. I still have this issue as noted on 11.3 latest beta. It errors out on trying to include <thread.h> exactly as noted. Admittedly I'm using gcc-devel ... let me try with one of the other variants ... this might be a GCC issue!

comment:9 Changed 8 months ago by mkuron (Michael Kuron)

Cc: mkuron added

comment:10 Changed 8 months ago by astroboylrx (Rixin Li)

Cc: astroboylrx added

comment:11 in reply to:  8 Changed 8 months ago by astroboylrx (Rixin Li)

Replying to michaelld:

Interesting & thanks for that fix for older OSX. I still have this issue as noted on 11.3 latest beta. It errors out on trying to include <thread.h> exactly as noted. Admittedly I'm using gcc-devel ... let me try with one of the other variants ... this might be a GCC issue!

Have you found a work around now? I also ran into this issue and I'm using the +gcc9 variant. The log shows

:info:build error: Command "/opt/local/bin/gcc-mp-9 -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -DHAVE_CBLAS -Ibuild/src.macosx-11.2-x86_64-3.7/build/src.macosx-11.2-x86_64-3.7/scipy/integrate -I/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/numpy/core/include -Ibuild/src.macosx-11.2-x86_64-3.7/numpy/distutils/include -I/opt/local/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c build/src.macosx-11.2-x86_64-3.7/scipy/integrate/vodemodule.c -o build/temp.macosx-11.2-x86_64-3.7/build/src.macosx-11.2-x86_64-3.7/scipy/integrate/vodemodule.o -MMD -MF build/temp.macosx-11.2-x86_64-3.7/build/src.macosx-11.2-x86_64-3.7/scipy/integrate/vodemodule.o.d -msse -msse2 -msse3" failed with exit status 1

So I manually ran this compile command and found the error is due to

build/src.macosx-11.2-x86_64-3.7/scipy/integrate/vodemodule.c:86:10: fatal error: threads.h: No such file or directory

Okay, that's more clear and makes sense. I went to this vodemodule.c and found you can actually define __STDC_NO_THREADS__ to inform it that threads.h is not supported by the compiler. I tried the previous compile command with -D__STDC_NO_THREADS__ and the command finished without error. I thought I might be able to add this into the environment variable CFLAGS as a work around, so I did export CFLAGS="-D__STDC_NO_THREADS__". However, no luck.

I further found, in the build log of MacPorts, that the installation used --no-user-cfg while running setup.py build

:info:build Command failed:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_python_py-scipy/py37-scipy/work/scipy-1.6.2" && /opt/local/Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7 setup.py --no-user-cfg build -j8

So I again manually ran the build command without that: sudo /opt/local/Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7 setup.py build -j8, which again finished successfully.

The next step should just be something like python setup.py install. However, I don't know how port handles this process. Rerun port upgrade or port install didn't work b/c port did everything from a fresh restart.

Any suggestions would be greatly appreciated!

comment:12 in reply to:  8 Changed 7 months ago by astroboylrx (Rixin Li)

Replying to michaelld:

Interesting & thanks for that fix for older OSX. I still have this issue as noted on 11.3 latest beta. It errors out on trying to include <thread.h> exactly as noted. Admittedly I'm using gcc-devel ... let me try with one of the other variants ... this might be a GCC issue!

Have you had a chance to look into this? Thanks in advance.

comment:13 Changed 7 months ago by ryandesign (Ryan Schmidt)

Cc: thermalecology added

Has duplicate #62875.

Looks like this numpy bug?

https://github.com/numpy/numpy/issues/18179

comment:14 Changed 7 months ago by michaelld (Michael Dickens)

Maybe ... I've wondered if some NumPy change impacted SciPy ... let me investigate that issue & related PRs. What I don't get yet is why on Darwin SciPy/NumPy is even trying to access GLIBC ... since we don't (and can't) use it all ... correct? If that's the case then I think setting some macro maybe like noted a few comments up __STDC_NO_THREADS__ might do the trick if we can get that into the Python setup.py script somehow ...

comment:15 Changed 7 months ago by michaelld (Michael Dickens)

Ah... OK so the compiler being used here is some version of GCC, which does not define __STDC_NO_THREADS__. And since __GLIBC__ is also not defined then the 2nd macro combination succeeds & thus #include <threads.h> is the code ... which in theory should work since GCC should support it in C11 mode. Hmmm ...

Also regarding the issue noted in that NumPy link: A relevant PR has been merged into the current 1.20 release that we're using in MP & that's what the noted issue & related PR are doing, just for "master". Clearly there needs to be more consideration given for macOS / Xcode / MP with regard to this issue when using GCC as the compiler ... almost a cross-build, but not exactly.

comment:16 Changed 7 months ago by michaelld (Michael Dickens)

If I do clang -std=c11 -dM -E - </dev/null | grep STDC (Xcode 12.3 build version 12C33 or MP Clang 11) I see among others #define __STDC_NO_THREADS__ 1 ... if I do the same with any MP GCC, I do not see that #define which implies that I should be able to #include <threads.h> when using -std=c11. But, when I create a program that does just that it always fails, regardless of whether using Clang or GCC and what std I select. Maybe GCC isn't being honest in its capabilities self-assessment? Anyway, forcing this macro to be defined seems like the way to go here, and that seems to be an easy tweak to the SciPy code ... will commit once I've verified it locally.

Changed 7 months ago by michaelld (Michael Dickens)

basic patch to fix C11 STDC Threads issue

comment:17 Changed 7 months ago by michaelld (Michael Dickens)

I just attached a patch (-p0) that one can apply to the top-level work directory for py37+ SciPy ... it forces disabling C11 STDC THREADS, so that the fatal error: threads.h: No such file or directory goes away & SciPy builds cleanly. I think there has to be a better solution to fix the issue in SciPy, but in all of my playing around with the MP Portfile & settings I couldn't work out a better way ... except to patch the code in NumPy, which maybe really is the right way to go ... IDK!

For folks who want to give this patch a try, here's how I'd recommend you do it .. (1) download the patch; I'll assume it is downloaded under it's current name into ~/Downloads/. ...

sudo port clean py38-scipy
sudo port patch py38-scipy
pushd $(port work py38-scipy)/scipy-scipy*
patch -p0 < ~/Downloads/patch-add-STDC_NO_THREADS.diff
popd
sudo port install py38-scipy

and that should install cleanly now. You can also do upgrade, wait until the upgrade errors out, then apply the patch & reissue the update ... which should then work. For any patch, install, upgrade (etc) make sure the variants are what you want and are consistent, otherwise port will complain at you ;)

comment:18 Changed 7 months ago by michaelld (Michael Dickens)

[crickets] ... I'm going to assume that "no response is good news" and go forth with merging this change as a temporary solution to getting SciPy installing again ... in general.

comment:19 Changed 7 months ago by michaelld (Michael Dickens)

Related: During my testing I was regularly iterating over the "build" phase ... at some point I realized that the executed build command was different if I also had "port" go through "configure" ... so I look in the Portfile & there is a "pre-configure" phase that sets variables for "build" and "destroot" ... which just doesn't make sense! I split those into "pre-build" and "pre-destroot" and that seems to provide the necessary commands. If the port builds all the way through then the result is the same, but if doing stage by stage the results will be different the way the Portfile is currently setup.

I don't understand what is the value in setting later stage variables in an earlier stage. It breaks stage-by-stage building, and to me doesn't make any sense. Maybe I put that code there IDK ... either way, hoping someone can explain why "port" needs these split off that way ... or, if I can just go with my new way that allows stage-by-stage builds.

comment:20 Changed 5 months ago by michaelld (Michael Dickens)

Resolution: fixed
Status: assignedclosed
Note: See TracTickets for help on using tickets.