Opened 5 months ago

Last modified 2 weeks ago

#65395 assigned defect

py-numpy: fails to build on macOS 10.14

Reported by: ra1nb0w Owned by: michaelld (Michael Dickens)
Priority: Normal Milestone:
Component: ports Version:
Keywords: Cc: mascguy (Christopher Nielsen), blair (Blair Zajac), thomasrussellmurphy (Thomas Russell Murphy), hapaguy (Brian Kurt Fujikawa), kencu (Ken)
Port: py-numpy

Description

Sorry if I bother you with another issue related to macOS 10.14 ;-)

it fails to build only on macOS 10.14

log at https://build.macports.org/builders/ports-10.14_x86_64-builder/builds/131767/steps/install-port/logs/stdio

Attachments (3)

main.log (852.5 KB) - added by petrrr 4 months ago.
sudo port -v build py310-numpy -gfortran +gcc12
main.2.log (854.6 KB) - added by petrrr 4 months ago.
sudo port -v build py310-numpy -gfortran +gcc12 (build on 20/07/2022)
main.3.log (297.7 KB) - added by petrrr 4 months ago.
sudo port -v build py310-numpy -gfortran +gcc11

Download all attachments as: .zip

Change History (35)

comment:1 Changed 5 months ago by ra1nb0w

Seems that the setup enable AVX512 without hardware support.

https://github.com/numpy/numpy/issues/21806

comment:2 Changed 5 months ago by michaelld (Michael Dickens)

Has duplicate #65400

comment:3 Changed 5 months ago by mascguy (Christopher Nielsen)

Cc: mascguy added; michaelld removed
Owner: set to michaelld
Status: newassigned

comment:4 Changed 5 months ago by blair (Blair Zajac)

Cc: blair added

comment:5 Changed 5 months ago by dyne2meter

The update succeeds using variant -gfortran +gcc11 +openblas.

comment:6 Changed 5 months ago by dstrubbe (David Strubbe)

Also succeeds with +gcc10 +openblas.

comment:7 Changed 5 months ago by mascguy (Christopher Nielsen)

Cc: thomasrussellmurphy added

comment:8 Changed 5 months ago by mascguy (Christopher Nielsen)

Michael, this is blocking downstream dependencies on 10.14, so we really need to try to fix this ASAP.

Thoughts as to how you want to proceed...?

comment:9 Changed 5 months ago by mascguy (Christopher Nielsen)

Ah, just saw your latest comments on your PR for volk, and it's clear you're working on it. Never Mind! ;-)

https://github.com/macports/macports-ports/pull/15244#issuecomment-1176312090

comment:10 Changed 5 months ago by michaelld (Michael Dickens)

Yup working on it. My macOS 10.14 buildbot is churning on GCC11 right now ... even with 8 cores this takes hours. I might get to this issue later today; more likely is Monday, maybe over the weekend if I have time.

comment:11 Changed 5 months ago by hapaguy (Brian Kurt Fujikawa)

Cc: hapaguy added

comment:12 in reply to:  10 Changed 4 months ago by mascguy (Christopher Nielsen)

Replying to michaelld:

Yup working on it. My macOS 10.14 buildbot is churning on GCC11 right now ... even with 8 cores this takes hours. I might get to this issue later today; more likely is Monday, maybe over the weekend if I have time.

Hey Michael, is there anything we can do to help?

comment:13 Changed 4 months ago by mascguy (Christopher Nielsen)

Currently reviewing the details of this, since it's blocking builds for py-pytorch - among other ports - on 10.14.

And a quick fix - at least to stop the bleeding - may be to simply blacklist Clang versions earlier than 11. If that works locally for 10.14, I'll commit the change.

This will give Michael more time to investigate a better option, without all of us breathing down his neck.

comment:14 Changed 4 months ago by Christopher Nielsen <mascguy@…>

In 3dcd708af94761a0e313f4acdddf631cb4443a87/macports-ports (master):

py-numpy: expand clang blacklist to < 1100

  • Initial fix, to stop the bleeding on 10.14

See: #65395

comment:15 Changed 4 months ago by mascguy (Christopher Nielsen)

Queued up a rebuild for py-pytorch on 10.14. More will be needed, but that's one down.

comment:16 Changed 4 months ago by michaelld (Michael Dickens)

Thanks for that (temporary?) fix, @mascguy. Got sick & am just recovering -- nothing serious, luckily. Hoping to investigate further later this week.

comment:17 in reply to:  16 Changed 4 months ago by mascguy (Christopher Nielsen)

Replying to michaelld:

Thanks for that (temporary?) fix, @mascguy. Got sick & am just recovering -- nothing serious, luckily. Hoping to investigate further later this week.

No worries, and hope you feel better soon Michael!

comment:18 Changed 4 months ago by thomasrussellmurphy (Thomas Russell Murphy)

On 10.14.6, I presumably had gcc12 get installed for the currently supported list of compilers. Now py-numpy is still failing for both 39 and 310 versions.

comment:19 Changed 4 months ago by michaelld (Michael Dickens)

Building gcc12 at the moment, and then I'll finalize testing. I think the issue is that when using +gfortran the C/C++ compiler is AppleClang -- the Clang installed by Xcode, which for 10.14 the most recent is Apple clang version 11.0.0 (clang-1100.0.33.17) -- and AppleClang is notorious for not about feature set support correct. I think with -gfortran +gcc* we use GCC for all compilers, and GCC is much better about getting feature set support correct. If I'm correct then blocking use of specific AppleClang is the way to go for MP needs. NumPy should add better testing to verify feature set support, but that's outside the scope of what I am reasonably going to do. More soon!

comment:20 in reply to:  19 ; Changed 4 months ago by petrrr

Replying to michaelld:

Building gcc12 at the moment, and then I'll finalize testing. I think the issue is that when using +gfortran the C/C++ compiler is AppleClang -- the Clang installed by Xcode, which for 10.14 the most recent is Apple clang version 11.0.0 (clang-1100.0.33.17) -- and AppleClang is notorious for not about feature set support correct. I think with -gfortran +gcc* we use GCC for all compilers, and GCC is much better about getting feature set support correct. If I'm correct then blocking use of specific AppleClang is the way to go for MP needs. NumPy should add better testing to verify feature set support, but that's outside the scope of what I am reasonably going to do. More soon!

I have tried to use +gcc10, +gcc11 and +gcc12, but could not build anyway. Do you need log for these builds?

Changed 4 months ago by petrrr

Attachment: main.log added

sudo port -v build py310-numpy -gfortran +gcc12

comment:21 in reply to:  20 ; Changed 4 months ago by mascguy (Christopher Nielsen)

Replying to petrrr:

I have tried to use +gcc10, +gcc11 and +gcc12, but could not build anyway. Do you need log for these builds?

Just to confirm, have you updated your ports with sudo port selfupdate? I ask, as you'll definitely want the fix committed yesterday. (Not 100% sure whether it will affect builds with the gcc variants, but nonetheless...)

comment:22 Changed 4 months ago by michaelld (Michael Dickens)

@petrrr : I see those errors too:

:info:build INFO: gcc-mp-12: numpy/core/src/common/npy_cpu_features.c
:info:build <stdin>:4930:20: error: invalid operand for instruction
:info:build         vfpclasspsz     $159, %zmm0, %k2
:info:build                               ^~~~~
:info:build <stdin>:6084:20: error: invalid operand for instruction
:info:build         vfpclasspdz     $159, %zmm0, %k2
:info:build                               ^~~~~

Same issue with GCC11 and GCC12. Seems like a GCC issue, but I'm no expert here!

comment:23 Changed 4 months ago by michaelld (Michael Dickens)

@mascguy : I think my GCC* libgcc* ports are up to date as of this morning. I see there's YA tweak to some of those ("(lib)gcc12: migrate libgccjit to libgcc12, include versionless sym-links") ... not sure that will help here, but I'll remove all *gcc12 and reinstall them from scratch to see ...

comment:24 in reply to:  23 Changed 4 months ago by mascguy (Christopher Nielsen)

Replying to michaelld:

@mascguy : I think my GCC* libgcc* ports are up to date as of this morning. I see there's YA tweak to some of those ("(lib)gcc12: migrate libgccjit to libgcc12, include versionless sym-links") ... not sure that will help here, but I'll remove all *gcc12 and reinstall them from scratch to see ...

Sorry for the confusion, I meant the update to py-numpy specifically. But if Clang doesn't come into play with the GCC variants, then it's a moot point! Lol

comment:25 in reply to:  21 Changed 4 months ago by petrrr

Replying to mascguy:

Replying to petrrr:

I have tried to use +gcc10, +gcc11 and +gcc12, but could not build anyway. Do you need log for these builds?

Just to confirm, have you updated your ports with sudo port selfupdate? I ask, as you'll definitely want the fix committed yesterday. (Not 100% sure whether it will affect builds with the gcc variants, but nonetheless...)

I did some selfupdates (several) yesterday before the builds. However I now see that more fixes/changes on gcc12 and py-numpy came in. I'll rebuild and let you know.

Changed 4 months ago by petrrr

Attachment: main.2.log added

sudo port -v build py310-numpy -gfortran +gcc12 (build on 20/07/2022)

Changed 4 months ago by petrrr

Attachment: main.3.log added

sudo port -v build py310-numpy -gfortran +gcc11

comment:26 Changed 4 months ago by petrrr

I had no success in building py310-numpy again this morning. I tested both +gcc12 and +gcc11. Logs are attached.

comment:27 Changed 4 months ago by michaelld (Michael Dickens)

@petrrr: I did the same: updated *gcc12*, cleaned py*-numpy, then tried again. Exact same errors as before & what you're seeing. I think this is a compiler error, but it might be due to specifying flags that result in bad code on the target CPU (e.g., -msse3 then the CPU doesn't support it) -- probably not since I don't think that's how most compilers work, but throwing it out here just in case that resonates with someone.

comment:28 Changed 4 months ago by michaelld (Michael Dickens)

Cc: kencu added

Maybe @kencu has some ideas, if this is a compiler issue? Lots of work of recent on GCC10+ compilers!

comment:29 Changed 4 months ago by michaelld (Michael Dickens)

OK so doing use_parallel_build=no, I reliably now see this:

:info:build creating build/temp.macosx-10.14-x86_64-cpython-310/build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath
:info:build INFO: compile options: '-DNPY_INTERNAL_BUILD=1 -DHAVE_NPY_CONFIG_H=1 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D_LARGEFILE64_SOURCE=1 -DHAVE_CBLAS -Ibuild/src.macosx-10.14-x86_64-3.10/numpy/core/src/multiarray -Ibuild/src.macosx-10.14-x86_64-3.10/numpy/core/src/common -Ibuild/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath -Inumpy/core/include -Ibuild/src.macosx-10.14-x86_64-3.10/numpy/core/include/numpy -Ibuild/src.macosx-10.14-x86_64-3.10/numpy/distutils/include -Ibuild/src.macosx-10.14-x86_64-3.10/numpy/core/src/npysort -Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/src/_simd -I/opt/local/Library/Frameworks/Python.framework/Versions/3.10/include/python3.10 -Ibuild/src.macosx-10.14-x86_64-3.10/numpy/core/src/common -Ibuild/src.macosx-10.14-x86_64-3.10/numpy/core/src/npymath -c'
:info:build extra options: '-msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2 -mavx512f -mno-mmx -mavx512cd -mavx512vl -mavx512bw -mavx512dq'
:info:build INFO: gcc-mp-12: build/src.macosx-10.14-x86_64-3.10/numpy/core/src/multiarray/argfunc.dispatch.avx512_skx.c
:info:build INFO: gcc-mp-12: build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_arithmetic.dispatch.avx512_skx.c
:info:build INFO: gcc-mp-12: build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_minmax.dispatch.avx512_skx.c
:info:build INFO: gcc-mp-12: build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_umath_fp.dispatch.avx512_skx.c
:info:build INFO: gcc-mp-12: build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_exponent_log.dispatch.avx512_skx.c
:info:build INFO: gcc-mp-12: build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_hyperbolic.dispatch.avx512_skx.c
:info:build <stdin>:4930:20: error: invalid operand for instruction
:info:build         vfpclasspsz     $159, %zmm0, %k2
:info:build                               ^~~~~
:info:build <stdin>:6084:20: error: invalid operand for instruction
:info:build         vfpclasspdz     $159, %zmm0, %k2
:info:build                               ^~~~~
:info:build error: Command "/opt/local/bin/gcc-mp-12 -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -arch x86_64 -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -DNPY_INTERNAL_BUILD=1 -DHAVE_NPY_CONFIG_H=1 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D_LARGEFILE64_SOURCE=1 -DHAVE_CBLAS -Ibuild/src.macosx-10.14-x86_64-3.10/numpy/core/src/multiarray -Ibuild/src.macosx-10.14-x86_64-3.10/numpy/core/src/common -Ibuild/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath -Inumpy/core/include -Ibuild/src.macosx-10.14-x86_64-3.10/numpy/core/include/numpy -Ibuild/src.macosx-10.14-x86_64-3.10/numpy/distutils/include -Ibuild/src.macosx-10.14-x86_64-3.10/numpy/core/src/npysort -Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/src/_simd -I/opt/local/Library/Frameworks/Python.framework/Versions/3.10/include/python3.10 -Ibuild/src.macosx-10.14-x86_64-3.10/numpy/core/src/common -Ibuild/src.macosx-10.14-x86_64-3.10/numpy/core/src/npymath -c build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_exponent_log.dispatch.avx512_skx.c -o build/temp.macosx-10.14-x86_64-cpython-310/build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_exponent_log.dispatch.avx512_skx.o -MMD -MF build/temp.macosx-10.14-x86_64-cpython-310/build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_exponent_log.dispatch.avx512_skx.o.d -msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2 -mavx512f -mno-mmx -mavx512cd -mavx512vl -mavx512bw -mavx512dq" failed with exit status 1
:info:build INFO: 
:info:build ########### EXT COMPILER OPTIMIZATION ###########
:info:build INFO: Platform      : 
:info:build   Architecture: x64
:info:build   Compiler    : gcc
:info:build CPU baseline  : 
:info:build   Requested   : 'min'
:info:build   Enabled     : SSE SSE2 SSE3
:info:build   Flags       : -msse -msse2 -msse3
:info:build   Extra checks: none
:info:build CPU dispatch  : 
:info:build   Requested   : 'max -xop -fma4'
:info:build   Enabled     : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_SKX AVX512_CLX AVX512_CNL AVX512_ICL
:info:build   Generated   : 
:info:build               : 
:info:build   SSE41       : SSE SSE2 SSE3 SSSE3
:info:build   Flags       : -msse -msse2 -msse3 -mssse3 -msse4.1
:info:build   Extra checks: none
:info:build   Detect      : SSE SSE2 SSE3 SSSE3 SSE41
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_unary_fp.dispatch.c
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_arithmetic.dispatch.c
:info:build               : 
:info:build   SSE42       : SSE SSE2 SSE3 SSSE3 SSE41 POPCNT
:info:build   Flags       : -msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2
:info:build   Extra checks: none
:info:build   Detect      : SSE SSE2 SSE3 SSSE3 SSE41 POPCNT SSE42
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/multiarray/argfunc.dispatch.c
:info:build               : 
:info:build   AVX2        : SSE SSE2 SSE3 SSSE3 SSE41 POPCNT SSE42 AVX F16C
:info:build   Flags       : -msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mavx2
:info:build   Extra checks: none
:info:build   Detect      : AVX F16C AVX2
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/multiarray/argfunc.dispatch.c
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_arithm_fp.dispatch.c
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_arithmetic.dispatch.c
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_minmax.dispatch.c
:info:build               : 
:info:build   (FMA3 AVX2) : SSE SSE2 SSE3 SSSE3 SSE41 POPCNT SSE42 AVX F16C
:info:build   Flags       : -msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2
:info:build   Extra checks: none
:info:build   Detect      : AVX F16C FMA3 AVX2
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_trigonometric.dispatch.c
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_exponent_log.dispatch.c
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_hyperbolic.dispatch.c
:info:build               : 
:info:build   AVX512F     : SSE SSE2 SSE3 SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2
:info:build   Flags       : -msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2 -mavx512f -mno-mmx
:info:build   Extra checks: AVX512F_REDUCE
:info:build   Detect      : AVX512F
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_arithm_fp.dispatch.c
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_arithmetic.dispatch.c
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_trigonometric.dispatch.c
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_exponent_log.dispatch.c
:info:build               : 
:info:build   AVX512_SKX  : SSE SSE2 SSE3 SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD
:info:build   Flags       : -msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2 -mavx512f -mno-mmx -mavx512cd -mavx512vl -mavx512bw -mavx512dq
:info:build   Extra checks: AVX512BW_MASK AVX512DQ_MASK
:info:build   Detect      : AVX512_SKX
:info:build               : numpy/core/src/npysort/x86-qsort.dispatch.cpp
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/multiarray/argfunc.dispatch.c
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_arithmetic.dispatch.c
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_minmax.dispatch.c
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_umath_fp.dispatch.c
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_exponent_log.dispatch.c
:info:build               : build/src.macosx-10.14-x86_64-3.10/numpy/core/src/umath/loops_hyperbolic.dispatch.c
:info:build INFO: CCompilerOpt.cache_flush[857] : write cache to path -> /opt/local/var/macports/build/_Volumes_Common_MacPorts_ports_github_macports_python_py-numpy/py310-numpy/work/numpy-1.23.1/build/temp.macosx-10.14-x86_64-cpython-310/ccompiler_opt_cache_ext.py
:info:build INFO: 
:info:build ########### CLIB COMPILER OPTIMIZATION ###########
:info:build INFO: Platform      : 
:info:build   Architecture: x64
:info:build   Compiler    : gcc
:info:build CPU baseline  : 
:info:build   Requested   : 'min'
:info:build   Enabled     : SSE SSE2 SSE3
:info:build   Flags       : -msse -msse2 -msse3
:info:build   Extra checks: none
:info:build CPU dispatch  : 
:info:build   Requested   : 'max -xop -fma4'
:info:build   Enabled     : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_SKX AVX512_CLX AVX512_CNL AVX512_ICL
:info:build   Generated   : none
:info:build INFO: CCompilerOpt.cache_flush[857] : write cache to path -> /opt/local/var/macports/build/_Volumes_Common_MacPorts_ports_github_macports_python_py-numpy/py310-numpy/work/numpy-1.23.1/build/temp.macosx-10.14-x86_64-cpython-310/ccompiler_opt_cache_clib.py

Issuing the command noted to fail from ${worksrcpath}/numpy-1.23.1 indeed shows the same error. Appending -S to save the generated assembly code indeed shows the quoted lines.

comment:30 Changed 4 months ago by kencu (Ken)

The error is definitely coming from the assembler.

gcc12 now forces a new macports-clang to be installed with it. cctools will use that as assembler if it is there (in most cases, depending on some specifics that have to be looked at in each case).

If this used to build, but now does not, it is possible the new clang is barfing on assembly some older clang accepted.

If this never built, could be other things too (configuring SIMD that isn't supported, etc).

comment:31 Changed 4 months ago by kencu (Ken)

looking to see exactly what operands vfpclasspsz accepts, why the zmm0 might make it fail, and how support for that is turned on and off is a bit of a rabbit-hole indeed.

Deep into AVX512 semantics.

comment:32 Changed 2 weeks ago by petrrr

I do not observe this problem anymore for a while. So has this been fix in a way that the ticket could be closed?

Note: See TracTickets for help on using tickets.