py-numpy: fails to build on macOS 10.14

[ra1nb0w]

Sorry if I bother you with another issue related to macOS 10.14 ;-)

it fails to build only on macOS 10.14

log at ​https://build.macports.org/builders/ports-10.14_x86_64-builder/builds/131767/steps/install-port/logs/stdio

[ra1nb0w]

Seems that the setup enable AVX512 without hardware support.

​https://github.com/numpy/numpy/issues/21806

[michaelld (Michael Dickens)]

Has duplicate #65400

[dyne2meter]

The update succeeds using variant -gfortran +gcc11 +openblas.

[dstrubbe (David Strubbe)]

Also succeeds with +gcc10 +openblas.

[mascguy (Christopher Nielsen)]

Michael, this is blocking downstream dependencies on 10.14, so we really need to try to fix this ASAP.

Thoughts as to how you want to proceed...?

[mascguy (Christopher Nielsen)]

Ah, just saw your latest comments on your PR for volk, and it's clear you're working on it. Never Mind! ;-)

​https://github.com/macports/macports-ports/pull/15244#issuecomment-1176312090

[michaelld (Michael Dickens)]

Yup working on it. My macOS 10.14 buildbot is churning on GCC11 right now ... even with 8 cores this takes hours. I might get to this issue later today; more likely is Monday, maybe over the weekend if I have time.

[mascguy (Christopher Nielsen)]

Replying to michaelld:

Yup working on it. My macOS 10.14 buildbot is churning on GCC11 right now ... even with 8 cores this takes hours. I might get to this issue later today; more likely is Monday, maybe over the weekend if I have time.

Hey Michael, is there anything we can do to help?

[mascguy (Christopher Nielsen)]

Currently reviewing the details of this, since it's blocking builds for py-pytorch - among other ports - on 10.14.

And a quick fix - at least to stop the bleeding - may be to simply blacklist Clang versions earlier than 11. If that works locally for 10.14, I'll commit the change.

This will give Michael more time to investigate a better option, without all of us breathing down his neck.

[Christopher Nielsen <mascguy@…>]

In 3dcd708af94761a0e313f4acdddf631cb4443a87/macports-ports (master):

py-numpy: expand clang blacklist to < 1100

• Initial fix, to stop the bleeding on 10.14

See: #65395

[mascguy (Christopher Nielsen)]

Queued up a rebuild for py-pytorch on 10.14. More will be needed, but that's one down.

[michaelld (Michael Dickens)]

Thanks for that (temporary?) fix, @mascguy. Got sick & am just recovering -- nothing serious, luckily. Hoping to investigate further later this week.

[mascguy (Christopher Nielsen)]

Replying to michaelld:

Thanks for that (temporary?) fix, @mascguy. Got sick & am just recovering -- nothing serious, luckily. Hoping to investigate further later this week.

No worries, and hope you feel better soon Michael!

[thomasrussellmurphy (Thomas Russell Murphy)]

On 10.14.6, I presumably had gcc12 get installed for the currently supported list of compilers. Now py-numpy is still failing for both 39 and 310 versions.

[michaelld (Michael Dickens)]

Building gcc12 at the moment, and then I'll finalize testing. I think the issue is that when using +gfortran the C/C++ compiler is AppleClang -- the Clang installed by Xcode, which for 10.14 the most recent is Apple clang version 11.0.0 (clang-1100.0.33.17) -- and AppleClang is notorious for not about feature set support correct. I think with -gfortran +gcc* we use GCC for all compilers, and GCC is much better about getting feature set support correct. If I'm correct then blocking use of specific AppleClang is the way to go for MP needs. NumPy should add better testing to verify feature set support, but that's outside the scope of what I am reasonably going to do. More soon!

[petrrr]

Replying to michaelld:

Building gcc12 at the moment, and then I'll finalize testing. I think the issue is that when using +gfortran the C/C++ compiler is AppleClang -- the Clang installed by Xcode, which for 10.14 the most recent is Apple clang version 11.0.0 (clang-1100.0.33.17) -- and AppleClang is notorious for not about feature set support correct. I think with -gfortran +gcc* we use GCC for all compilers, and GCC is much better about getting feature set support correct. If I'm correct then blocking use of specific AppleClang is the way to go for MP needs. NumPy should add better testing to verify feature set support, but that's outside the scope of what I am reasonably going to do. More soon!

I have tried to use +gcc10, +gcc11 and +gcc12, but could not build anyway. Do you need log for these builds?

[petrrr]

sudo port -v build py310-numpy -gfortran +gcc12

[mascguy (Christopher Nielsen)]

Replying to petrrr:

I have tried to use +gcc10, +gcc11 and +gcc12, but could not build anyway. Do you need log for these builds?

Just to confirm, have you updated your ports with sudo port selfupdate? I ask, as you'll definitely want the fix committed yesterday. (Not 100% sure whether it will affect builds with the gcc variants, but nonetheless...)

[michaelld (Michael Dickens)]

@petrrr : I see those errors too:

:info:build INFO: gcc-mp-12: numpy/core/src/common/npy_cpu_features.c
:info:build <stdin>:4930:20: error: invalid operand for instruction
:info:build         vfpclasspsz     $159, %zmm0, %k2
:info:build                               ^~~~~
:info:build <stdin>:6084:20: error: invalid operand for instruction
:info:build         vfpclasspdz     $159, %zmm0, %k2
:info:build                               ^~~~~

Same issue with GCC11 and GCC12. Seems like a GCC issue, but I'm no expert here!

[michaelld (Michael Dickens)]

@mascguy : I think my GCC* libgcc* ports are up to date as of this morning. I see there's YA tweak to some of those ("(lib)gcc12: migrate libgccjit to libgcc12, include versionless sym-links") ... not sure that will help here, but I'll remove all *gcc12 and reinstall them from scratch to see ...

[mascguy (Christopher Nielsen)]

Replying to michaelld:

@mascguy : I think my GCC* libgcc* ports are up to date as of this morning. I see there's YA tweak to some of those ("(lib)gcc12: migrate libgccjit to libgcc12, include versionless sym-links") ... not sure that will help here, but I'll remove all *gcc12 and reinstall them from scratch to see ...

Sorry for the confusion, I meant the update to py-numpy specifically. But if Clang doesn't come into play with the GCC variants, then it's a moot point! Lol

[petrrr]

Replying to mascguy:

Replying to petrrr:

I have tried to use +gcc10, +gcc11 and +gcc12, but could not build anyway. Do you need log for these builds?

Just to confirm, have you updated your ports with sudo port selfupdate? I ask, as you'll definitely want the fix committed yesterday. (Not 100% sure whether it will affect builds with the gcc variants, but nonetheless...)

I did some selfupdates (several) yesterday before the builds. However I now see that more fixes/changes on gcc12 and py-numpy came in. I'll rebuild and let you know.

[petrrr]

sudo port -v build py310-numpy -gfortran +gcc12 (build on 20/07/2022)

[petrrr]

sudo port -v build py310-numpy -gfortran +gcc11

[petrrr]

I had no success in building py310-numpy again this morning. I tested both +gcc12 and +gcc11. Logs are attached.

[michaelld (Michael Dickens)]

@petrrr: I did the same: updated *gcc12*, cleaned py*-numpy, then tried again. Exact same errors as before & what you're seeing. I think this is a compiler error, but it might be due to specifying flags that result in bad code on the target CPU (e.g., -msse3 then the CPU doesn't support it) -- probably not since I don't think that's how most compilers work, but throwing it out here just in case that resonates with someone.

[michaelld (Michael Dickens)]

Maybe @kencu has some ideas, if this is a compiler issue? Lots of work of recent on GCC10+ compilers!

[michaelld (Michael Dickens)]

OK so doing use_parallel_build=no, I reliably now see this:

:info:build creating build/temp.macosx-10.14-x86_64-cpython-310/build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath
:info:build INFO: compile options: '-DNPY_INTERNAL_BUILD=1 -DHAVE_NPY_CONFIG_H=1 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D_LARGEFILE64_SOURCE=1 -DHAVE_CBLAS -Ibuild/src.macosx-10.14-x86_64-3.10/numpy/core/src/multiarray -Ibuild/src.macosx-10.14-x86_64-3.10/numpy/core/src/common -Ibuild/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath -Inumpy/core/include -Ibuild/src.macosx-10.14-x86_64-3.10/numpy/core/include/numpy -Ibuild/src.macosx-10.14-x86_64-3.10/numpy/distutils/include -Ibuild/src.macosx-10.14-x86_64-3.10/numpy/core/src/npysort -Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/src/_simd -I/opt/local/Library/Frameworks/Python.framework/Versions/3.10/include/python3.10 -Ibuild/src.macosx-10.14-x86_64-3.10/numpy/core/src/common -Ibuild/src.macosx-10.14-x86_64-3.10/numpy/core/src/npymath -c'
:info:build extra options: '-msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2 -mavx512f -mno-mmx -mavx512cd -mavx512vl -mavx512bw -mavx512dq'
:info:build INFO: gcc-mp-12: build/src.macosx-10.14-x86_64-3.10/numpy/core/src/multiarray/argfunc.dispatch.avx512_skx.c
:info:build INFO: gcc-mp-12: build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_arithmetic.dispatch.avx512_skx.c
:info:build INFO: gcc-mp-12: build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_minmax.dispatch.avx512_skx.c
:info:build INFO: gcc-mp-12: build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_umath_fp.dispatch.avx512_skx.c
:info:build INFO: gcc-mp-12: build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_exponent_log.dispatch.avx512_skx.c
:info:build INFO: gcc-mp-12: build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_hyperbolic.dispatch.avx512_skx.c
:info:build <stdin>:4930:20: error: invalid operand for instruction
:info:build         vfpclasspsz     $159, %zmm0, %k2
:info:build                               ^~~~~
:info:build <stdin>:6084:20: error: invalid operand for instruction
:info:build         vfpclasspdz     $159, %zmm0, %k2
:info:build                               ^~~~~
:info:build error: Command "/opt/local/bin/gcc-mp-12 -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -arch x86_64 -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -DNPY_INTERNAL_BUILD=1 -DHAVE_NPY_CONFIG_H=1 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D_LARGEFILE64_SOURCE=1 -DHAVE_CBLAS -Ibuild/src.macosx-10.14-x86_64-3.10/numpy/core/src/multiarray -Ibuild/src.macosx-10.14-x86_64-3.10/numpy/core/src/common -Ibuild/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath -Inumpy/core/include -Ibuild/src.macosx-10.14-x86_64-3.10/numpy/core/include/numpy -Ibuild/src.macosx-10.14-x86_64-3.10/numpy/distutils/include -Ibuild/src.macosx-10.14-x86_64-3.10/numpy/core/src/npysort -Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/src/_simd -I/opt/local/Library/Frameworks/Python.framework/Versions/3.10/include/python3.10 -Ibuild/src.macosx-10.14-x86_64-3.10/numpy/core/src/common -Ibuild/src.macosx-10.14-x86_64-3.10/numpy/core/src/npymath -c build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_exponent_log.dispatch.avx512_skx.c -o build/temp.macosx-10.14-x86_64-cpython-310/build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_exponent_log.dispatch.avx512_skx.o -MMD -MF build/temp.macosx-10.14-x86_64-cpython-310/build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_exponent_log.dispatch.avx512_skx.o.d -msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2 -mavx512f -mno-mmx -mavx512cd -mavx512vl -mavx512bw -mavx512dq" failed with exit status 1
:info:build INFO: 
:info:build ########### EXT COMPILER OPTIMIZATION ###########
:info:build INFO: Platform      : 
:info:build   Architecture: x64
:info:build   Compiler    : gcc
:info:build CPU baseline  : 
:info:build   Requested   : 'min'
:info:build   Enabled     : SSE SSE2 SSE3
:info:build   Flags       : -msse -msse2 -msse3
:info:build   Extra checks: none
:info:build CPU dispatch  : 
:info:build   Requested   : 'max -xop -fma4'
:info:build   Enabled     : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_SKX AVX512_CLX AVX512_CNL AVX512_ICL
:info:build   Generated   : 
:info:build               : 
:info:build   SSE41       : SSE SSE2 SSE3 SSSE3
:info:build   Flags       : -msse -msse2 -msse3 -mssse3 -msse4.1
:info:build   Extra checks: none
:info:build   Detect      : SSE SSE2 SSE3 SSSE3 SSE41
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_unary_fp.dispatch.c
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_arithmetic.dispatch.c
:info:build               : 
:info:build   SSE42       : SSE SSE2 SSE3 SSSE3 SSE41 POPCNT
:info:build   Flags       : -msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2
:info:build   Extra checks: none
:info:build   Detect      : SSE SSE2 SSE3 SSSE3 SSE41 POPCNT SSE42
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/multiarray/argfunc.dispatch.c
:info:build               : 
:info:build   AVX2        : SSE SSE2 SSE3 SSSE3 SSE41 POPCNT SSE42 AVX F16C
:info:build   Flags       : -msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mavx2
:info:build   Extra checks: none
:info:build   Detect      : AVX F16C AVX2
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/multiarray/argfunc.dispatch.c
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_arithm_fp.dispatch.c
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_arithmetic.dispatch.c
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_minmax.dispatch.c
:info:build               : 
:info:build   (FMA3 AVX2) : SSE SSE2 SSE3 SSSE3 SSE41 POPCNT SSE42 AVX F16C
:info:build   Flags       : -msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2
:info:build   Extra checks: none
:info:build   Detect      : AVX F16C FMA3 AVX2
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_trigonometric.dispatch.c
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_exponent_log.dispatch.c
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_hyperbolic.dispatch.c
:info:build               : 
:info:build   AVX512F     : SSE SSE2 SSE3 SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2
:info:build   Flags       : -msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2 -mavx512f -mno-mmx
:info:build   Extra checks: AVX512F_REDUCE
:info:build   Detect      : AVX512F
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_arithm_fp.dispatch.c
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_arithmetic.dispatch.c
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_trigonometric.dispatch.c
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_exponent_log.dispatch.c
:info:build               : 
:info:build   AVX512_SKX  : SSE SSE2 SSE3 SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD
:info:build   Flags       : -msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2 -mavx512f -mno-mmx -mavx512cd -mavx512vl -mavx512bw -mavx512dq
:info:build   Extra checks: AVX512BW_MASK AVX512DQ_MASK
:info:build   Detect      : AVX512_SKX
:info:build               : numpy/core/src/npysort/x86-qsort.dispatch.cpp
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/multiarray/argfunc.dispatch.c
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_arithmetic.dispatch.c
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_minmax.dispatch.c
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_umath_fp.dispatch.c
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_exponent_log.dispatch.c
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_hyperbolic.dispatch.c
:info:build INFO: CCompilerOpt.cache_flush[857] : write cache to path -> /opt/local/var/macports/build/_Volumes_Common_MacPorts_ports_github_macports_python_py-numpy/py310-numpy/work/numpy-1.23.1/build/temp.macosx-10.14-x86_64-cpython-310/ccompiler_opt_cache_ext.py
:info:build INFO: 
:info:build ########### CLIB COMPILER OPTIMIZATION ###########
:info:build INFO: Platform      : 
:info:build   Architecture: x64
:info:build   Compiler    : gcc
:info:build CPU baseline  : 
:info:build   Requested   : 'min'
:info:build   Enabled     : SSE SSE2 SSE3
:info:build   Flags       : -msse -msse2 -msse3
:info:build   Extra checks: none
:info:build CPU dispatch  : 
:info:build   Requested   : 'max -xop -fma4'
:info:build   Enabled     : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_SKX AVX512_CLX AVX512_CNL AVX512_ICL
:info:build   Generated   : none
:info:build INFO: CCompilerOpt.cache_flush[857] : write cache to path -> /opt/local/var/macports/build/_Volumes_Common_MacPorts_ports_github_macports_python_py-numpy/py310-numpy/work/numpy-1.23.1/build/temp.macosx-10.14-x86_64-cpython-310/ccompiler_opt_cache_clib.py

Issuing the command noted to fail from ${worksrcpath}/numpy-1.23.1 indeed shows the same error. Appending -S to save the generated assembly code indeed shows the quoted lines.

[kencu (Ken)]

The error is definitely coming from the assembler.

gcc12 now forces a new macports-clang to be installed with it. cctools will use that as assembler if it is there (in most cases, depending on some specifics that have to be looked at in each case).

If this used to build, but now does not, it is possible the new clang is barfing on assembly some older clang accepted.

If this never built, could be other things too (configuring SIMD that isn't supported, etc).

[kencu (Ken)]

looking to see exactly what operands vfpclasspsz accepts, why the zmm0 might make it fail, and how support for that is turned on and off is a bit of a rabbit-hole indeed.

Deep into AVX512 semantics.

[petrrr]

I do not observe this problem anymore for a while. So has this been fix in a way that the ticket could be closed?

[ra1nb0w]

Seems fixed. If arise again feel free to re-open.