Fails to build Python313 on Sequoia

[mouse07410 (Mouse)]

Apple Silicon M2 Max running MacOS Sequoia 15.2. Xcode-16.2.

In short:

:debug:build build phase started at Thu Jan  2 19:30:39 EST 2025
:notice:build --->  Building python313
:debug:build Executing org.macports.build (python313)
:debug:build Environment: 
:debug:build CC_PRINT_OPTIONS='YES'
:debug:build CC_PRINT_OPTIONS_FILE='/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python313/python313/work/.CC_PRINT_OPTIONS'
:debug:build CPATH='/opt/local/include'
:debug:build DEVELOPER_DIR='/Applications/Xcode.app/Contents/Developer'
:debug:build LIBRARY_PATH='/opt/local/lib'
:debug:build MACOSX_DEPLOYMENT_TARGET='15.0'
:debug:build SDKROOT='/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX15.sdk'
:debug:build SOURCE_DATE_EPOCH='1733567849'
:info:build Executing:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python313/python313/work/Python-3.13.1" && /usr/bin/make -j4 -w all 
:debug:build system:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python313/python313/work/Python-3.13.1" && /usr/bin/make -j4 -w all 
:info:build make: Entering directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python313/python313/work/Python-3.13.1'
:info:build Running code to generate profile data (this can take a while):
:info:build # First, we need to create a clean build with profile generation
:info:build # enabled.
:info:build /Applications/Xcode.app/Contents/Developer/usr/bin/make profile-gen-stamp
:info:build make[1]: Entering directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python313/python313/work/Python-3.13.1'
:info:build make[1]: `profile-gen-stamp' is up to date.
:info:build make[1]: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python313/python313/work/Python-3.13.1'
:info:build # Next, run the profile task to generate the profile information.
:info:build LLVM_PROFILE_FILE="/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python313/python313/work/Python-3.13.1/code-%p.profclangr" DYLD_FRAMEWORK_PATH=/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python313/python313/work/Python-3.13.1 ./python.exe -m test --pgo --timeout=
:info:build Using random seed: 1733567849
:info:build 0:00:00 load avg: 3.89 Run 44 tests sequentially in a single process
:info:build 0:00:00 load avg: 3.89 [ 1/44] test_array
:info:build 0:00:00 load avg: 3.98 [ 2/44] test_base64
:info:build 0:00:00 load avg: 3.98 [ 3/44] test_binascii
:info:build 0:00:00 load avg: 3.98 [ 4/44] test_binop
:info:build 0:00:00 load avg: 3.98 [ 5/44] test_bisect
:info:build 0:00:00 load avg: 3.98 [ 6/44] test_bytes
:info:build 0:00:01 load avg: 3.98 [ 7/44] test_bz2
:info:build 0:00:02 load avg: 3.98 [ 8/44] test_cmath
:info:build 0:00:02 load avg: 3.98 [ 9/44] test_codecs
:info:build 0:00:03 load avg: 3.98 [10/44] test_collections
:info:build 0:00:03 load avg: 3.98 [11/44] test_complex
:info:build 0:00:03 load avg: 3.98 [12/44] test_dataclasses
:info:build 0:00:04 load avg: 3.98 [13/44] test_datetime
:info:build 0:00:06 load avg: 4.14 [14/44] test_decimal
:info:build -------------------------------------------------- NOTICE --------------------------------------------------
:info:build test_decimal may generate "malloc can't allocate region"
:info:build warnings on macOS systems. This behavior is known. Do not
:info:build report a bug unless tests are also failing.
:info:build See https://github.com/python/cpython/issues/85100
:info:build ------------------------------------------------------------------------------------------------------------
:info:build 0:00:08 load avg: 4.14 [15/44] test_difflib
:info:build 0:00:08 load avg: 4.14 [16/44] test_embed
:info:build 0:00:12 load avg: 4.05 [17/44] test_float
:info:build 0:00:12 load avg: 4.05 [18/44] test_fstring
:info:build 0:00:14 load avg: 4.05 [19/44] test_functools
:info:build 0:00:14 load avg: 4.05 [20/44] test_generators
:info:build 0:00:14 load avg: 4.05 [21/44] test_hashlib
:info:build test test_hashlib failed
:info:build 0:00:14 load avg: 4.05 [22/44] test_heapq -- test_hashlib failed (1 error)
:info:build 0:00:15 load avg: 4.05 [23/44] test_int
:info:build 0:00:15 load avg: 3.88 [24/44] test_itertools
:info:build 0:00:16 load avg: 3.88 [25/44] test_json
:info:build 0:00:18 load avg: 3.88 [26/44] test_long
:info:build 0:00:19 load avg: 3.88 [27/44] test_lzma
:info:build 0:00:19 load avg: 3.88 [28/44] test_math
:info:build 0:00:20 load avg: 3.73 [29/44] test_memoryview
:info:build 0:00:21 load avg: 3.73 [30/44] test_operator
:info:build 0:00:21 load avg: 3.73 [31/44] test_ordered_dict
:info:build 0:00:21 load avg: 3.73 [32/44] test_patma
:info:build 0:00:21 load avg: 3.73 [33/44] test_pickle
:info:build 0:00:23 load avg: 3.73 [34/44] test_pprint
:info:build 0:00:23 load avg: 3.73 [35/44] test_re
:info:build 0:00:23 load avg: 3.73 [36/44] test_set
:info:build 0:00:25 load avg: 3.67 [37/44] test_sqlite3
:info:build 0:00:25 load avg: 3.67 [38/44] test_statistics
:info:build 0:00:27 load avg: 3.67 [39/44] test_str
:info:build 0:00:28 load avg: 3.67 [40/44] test_struct
:info:build 0:00:28 load avg: 3.67 [41/44] test_tabnanny
:info:build 0:00:29 load avg: 3.67 [42/44] test_time
:info:build 0:00:31 load avg: 3.78 [43/44] test_xml_etree
:info:build 0:00:31 load avg: 3.78 [44/44] test_xml_etree_c
:info:build Total duration: 31.8 sec
:info:build Total tests: run=9,394 skipped=198
:info:build Total test files: run=44/44 failed=1
:info:build Result: FAILURE
:info:build make: *** [profile-run-stamp] Error 2
:info:build make: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python313/python313/work/Python-3.13.1'
:info:build Command failed:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python313/python313/work/Python-3.13.1" && /usr/bin/make -j4 -w all 
:info:build Exit code: 2
:error:build Failed to build python313: command execution failed
:debug:build Error code: CHILDSTATUS 32425 2
:debug:build Backtrace: command execution failed
:debug:build     while executing
:debug:build "system {*}$notty {*}$callback {*}$nice $fullcmdstring"
:debug:build     invoked from within
:debug:build "command_exec -callback portprogress::target_progress_callback build"
:debug:build     (procedure "portbuild::build_main" line 10)
:debug:build     invoked from within
:debug:build "$procedure $targetname"
:error:build See /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python313/python313/main.log for details.

Full main.log is attached.

Also, I'm rather surprised it doesn't pull the binaries from a buildbot - or are there no buildbots for Sequoia on Apple Silicon?

[mouse07410 (Mouse)]

main.log

[kencu (Ken)]

I had no trouble:

% uname -a
Darwin MacBookPro 24.2.0 Darwin Kernel Version 24.2.0: Fri Dec  6 19:01:59 PST 2024; root:xnu-11215.61.5~2/RELEASE_ARM64_T6000 arm64

macOS 15.2 24C101 arm64
Xcode 16.2 16C5032a

% port -v installed python313
The following ports are currently installed:
  python313 @3.13.1_0+lto+optimizations (active) requested_variants='' platform='darwin 24' archs='arm64' date='2025-01-01T23:11:47-0800'

[ryandesign (Ryan Carsten Schmidt)]

Replying to mouse07410:

:info:build 0:00:14 load avg: 4.05 [21/44] test_hashlib
:info:build test test_hashlib failed
:info:build 0:00:14 load avg: 4.05 [22/44] test_heapq -- test_hashlib failed (1 error)

:info:build Total test files: run=44/44 failed=1
:info:build Result: FAILURE

I assume the failure of the test_hashlib test is the reason why the build failed. Did this test generate any additional log file within the work directory that we could examine? If so please attach it.

Also, I'm rather surprised it doesn't pull the binaries from a buildbot - or are there no buildbots for Sequoia on Apple Silicon?

There is no buildbot worker yet for macOS 15 arm64.

[mascguy (Christopher Nielsen)]

Replying to ryandesign:

There is no buildbot worker yet for macOS 15 arm64.

Ryan, what is the ETA on that?

[kencu (Ken)]

also, mouse, it appears you have no Command Line Tools installed, only Xcode.

That is also an untested scenario for macports, and can generate odd spurious errors and result in nonstandard builds of all kinds of ports.

[mouse07410 (Mouse)]

Darwin MacBookPro_MIT 24.2.0 Darwin Kernel Version 24.2.0: Fri Dec  6 18:56:34 PST 2024; root:xnu-11215.61.5~2/RELEASE_ARM64_T6020 arm64

Similar but different - RELEASE_ARM64_T6020 vs. RELEASE_ARM64_T6000. Don't know if it matters, of course.

it appears you have no Command Line Tools installed, only Xcode

Correct - because CLT appear to cause problems with other ports, and with other toolchains (e.g., Haskell).

Are you saying that I should install CLT and try again???

I assume the failure of the test_hashlib test is the reason why the build failed. Did this test generate any additional log file within the work directory that we could examine?

I'd love to - where should I look? I confess that the directory structure is a bit complicated for a layman.

[mouse07410 (Mouse)]

OK, not sure how useful the following is, but just in case:

$ cd /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python313/python313/work/
$ ll
total 8208
drwxr-xr-x    7 macports  wheel      224 Jan  3 10:50 ./
drwxr-xr-x    3 macports  wheel       96 Jan  3 10:50 ../
-rw-r--r--    1 macports  wheel  4068110 Jan  3 11:02 .CC_PRINT_OPTIONS
drwxr-xr-x    3 macports  wheel       96 Jan  3 11:02 .home/
-rw-r--r--    1 macports  wheel      268 Jan  3 10:58 .macports.python313.state
drwxr-xr-x    2 macports  wheel       64 Jan  3 11:02 .tmp/
drwxr-xr-x  181 macports  wheel     5792 Jan  3 11:03 Python-3.13.1/
$ cat .macports.python313.state 
version: 2
checksum: 8945e1b4a2e330995b674c7215a7e0e6d6f5f2c81ce919516072fce6544f4135
variant: +optimizations
variant: +lto
target: org.macports.fetch
target: org.macports.checksum
target: org.macports.extract
target: org.macports.patch
target: org.macports.configure
$ 

I'm also uploading /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python313/python313/work/Python-3.13.1/config.log on the off-chance that it could be useful.

[mouse07410 (Mouse)]

config.log

[mouse07410 (Mouse)]

/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python313/python313/work/Python-3.13.1/build/lib.macosx-15.0-arm64-3.13/_sysconfigdatadarwin_darwin.py

[kencu (Ken)]

Replying to mouse07410:

it appears you have no Command Line Tools installed, only Xcode

Correct - because CLT appear to cause problems with other ports, and with other toolchains (e.g., Haskell).

Are you saying that I should install CLT and try again???

I'm not aware of a Haskell issue with the CLTs (but could be something I am not aware of, certainly).

There was an issue with upgrading the CLTs that I sorted out was fixed by blowing out the old installation and reinstalling the CLTs, and Josh narrowed down to one particular folder that had to be erased, but that was 4 months ago.

It's not a long term option to stay without the CLTs installed.

I would do this, given your current situation:

sudo rm -rf /Library/Developer/CommandLineTools

then install the latest released CLTs that exist (no developer betas please).

And then go from there.

[kencu (Ken)]

You see, there are many places where someone coding a Portfile or PortGroup may assume that the CLTs are installed, with their SDKs, and then code to that. And there are ports that "burn in" their paths to SDKs and similar, and then when they don't exist, weird crap can happen.

So it's best to be on a very clean, very standard, unmodified, unsymlinked-to-a-USB-drive, unhacked, garden variety system that matches in every way another person's system or the buildbot (once it exists for current arm64). No modifications to anything.

So that way when you get a build error, we probably all get a build error, and we can sort it out.

[mouse07410 (Mouse)]

Update

Installing CLT for Xcode-16.2 and re-running the install made no difference (frankly, as I expected):

$ sudo rm -rf /Library/Developer/CommandLineTools
$ [installed CLT 16.2 via Finder/Installer]
$ sudo port clean python313
$ sudo port -v install python313
.  .  .
0:00:15 load avg: 12.70 [21/44] test_hashlib
test test_hashlib failed
0:00:15 load avg: 12.70 [22/44] test_heapq -- test_hashlib failed (1 error)
0:00:15 load avg: 12.70 [23/44] test_int
0:00:16 load avg: 12.70 [24/44] test_itertools
.  .  .
0:00:32 load avg: 10.33 [44/44] test_xml_etree_c

Total duration: 33.0 sec
Total tests: run=9,394 skipped=198
Total test files: run=44/44 failed=1
Result: FAILURE
make: *** [profile-run-stamp] Error 2
make: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python313/python313/work/Python-3.13.1'
Command failed:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python313/python313/work/Python-3.13.1" && /usr/bin/make -j4 -w all 
Exit code: 2
Error: Failed to build python313: command execution failed
Error: See /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python313/python313/main.log for details.
Error: Follow https://guide.macports.org/#project.tickets if you believe there is a bug.
Error: Processing of port python313 failed

Needless to say, stupid Python does not tell why the test failed, and there's no detailed log anywhere around the build directory that I looked.

[mouse07410 (Mouse)]

Update 2

$ port -v installed python313
The following ports are currently installed:
  python313 @3.13.1_0 (active) requested_variants='-lto-optimizations' platform='darwin 24' archs='arm64' date='2025-01-03T12:32:09-0500'
$

(self-explanatory)

I'll try to figure which one of the two defaults caused the problem.

Update 3

Here's your answer:

$ port -v installed python313
The following ports are currently installed:
  python313 @3.13.1_0+lto (active) requested_variants='-optimizations' platform='darwin 24' archs='arm64' date='2025-01-03T16:47:34-0500'
$

Something that +optimizations does, conflicts with Apple Silicon M2 Max (or something else on M2/Sequoia/Xcode-16.2 libraries).

[kencu (Ken)]

Or (considerably more likely) there is still something wrong on your system.

So far as we know, nobody else has this problem anywhere, and googling for it shows nothing to me. Python is extensively tested upstream, and by many users.

Port stats show many installations:

​https://ports.macports.org/port/python313/stats/

Homebrew has it's python3.13 installed with optimizations on:

​https://github.com/Homebrew/homebrew-core/blob/90efd58b0cd379a32147ae43fccb768e165d36eb/Formula/p/python@3.13.rb

and there are over 300,000 installations of that.

​https://formulae.brew.sh/formula/python@3.13

So I would again look more closely at your system:

1. does your Xcode match your CLTs? If no, make them match.
2. is there anything in /usr/local? If so remove it.
3. have you selected anything with "port select"? if so disable it all.
4. have you monkey around with symlinks in your SDKs folder? If so, fix them.
5. have you modified any of the builds of any of your ports manuallly -- forced a compiler, editied a Portfile? If so, undo that.
6. have any of the port you installed when you had no CLTs installed been installed in an "incorrect" fashion because there were no CLTs? Anything that burns in a path to an SDK or compiler setting (PERL, python, etc, etc, etc) are suspect.
7. have you set up anything in your PATH besides default MacPorts settings? Are you modifying anything at all in or your .zshrc?
8. are you using any non-default options? Any DEVEL ports? LIBRESSL instead of OPENSSL? Etc etc etc.

If ALL of that indicates you have a 100% clean system, then all it can come down to after all that is an interaction with some port you have installed that most of us would not have installed that is hosing the build.

[mouse07410 (Mouse)]

All the ports installed on this system

[kencu (Ken)]

one other thing that came to mind is sometimes, parallel building can result in otherwise hard-to-spot errors. That can be toggled off in the portfile. I can't see how that could cause the test failure you are seeing, but ... it's one more thing.

[kencu (Ken)]

if you enter this into chatgpt, it will give you a few more ideas:

"python 0:00:15 load avg: 12.70 [21/44] test_hashlib test test_hashlib failed"

[ryandesign (Ryan Carsten Schmidt)]

Replying to mascguy:

Replying to ryandesign:

There is no buildbot worker yet for macOS 15 arm64.

Ryan, what is the ETA on that?

I have no ETA for you right now.

Replying to mouse07410:

I assume the failure of the test_hashlib test is the reason why the build failed. Did this test generate any additional log file within the work directory that we could examine?

I'd love to - where should I look? I confess that the directory structure is a bit complicated for a layman.

I'm completely unfamiliar with python's build system so I can't guide you, beyond that it would be somewhere under the work directory.

[mouse07410 (Mouse)]

So far as we know, nobody else has this problem anywhere, and googling for it shows nothing to me. Python is extensively tested upstream, and by many users.

What about this: ​https://trac.macports.org/ticket/71738 ? While the ticket is rather poorly formed, the problem it indicates is the same. Actually, I discovered the problem with python313 that way too - meson failed to upgrade. I didn't plan to install python313 at all.

does your Xcode match your CLTs? If no, make them match.

Of course it does. Xcode-16.2 and CLT-16.2. Mind you, the results with and without CLT installed are exactly the same - test failure in "test_hashlib".

is there anything in /usr/local? If so remove it.

Sorry, that's absolutely impractical - /usr/local houses AFS, HSM (not SoftHSM), and other similar (necessary for my work) stuff, which Macports doesn't provide. One thing I can say about it - it does not collide with Macports-installed ports/stuff.

Also (see below) - removing /usr/local/bin and other "extraneous" dirs from the PATH changed nothing (same error/failure).

have you selected anything with "port select"? if so disable it all.

Currently, I've python311, python312, and (now) python313 installed, with "port select python python312" Similarly, with GCC and Clang. Are you suggesting disabling them all, reverting to MacOS system python and Xcode gcc???

Regardless, I disabled python-related selects:

$ port select python none
$ port select python3 none

It did not help.

have you monkey around with symlinks in your SDKs folder? If so, fix them.

No. (And there was no need.)

have you modified any of the builds of any of your ports manuallly -- forced a compiler, editied a Portfile? If so, undo that.

Absolutely not. Most of these things are above my competence anyway.

have any of the port you installed when you had no CLTs installed been installed in an "incorrect" fashion because there were no CLTs? Anything that burns in a path to an SDK or compiler setting (PERL, python, etc, etc, etc) are suspect.

Hmm, not sure. It is possible that many of the ports upgraded after this machine moved to Sequoia, were built without CLT. But since those ports do work (e.g., python312, clang-19.1.6, etc.) - why would they be suspect?

have you set up anything in your PATH besides default MacPorts settings?

Yes, absolutely! Rust toolchains, Haskell toolchains, Java toolchains, quite a few crypto stuff, etc. etc. This machine is for doing stuff besides carrying ports...

However, even after I set PATH to /opt/local/bin:/usr/bin:/bin:/sbin::, I get the same failure.

Are you modifying anything at all in or your .zshrc?

Nothing really.

are you using any non-default options?

The only non-default options for Macports are:

• setting Web proxy (otherwise it won't reach the servers);
• limiting the number of parallel executions to 4 (yes, I have reasons for that).

Any DEVEL ports? LIBRESSL instead of OPENSSL? Etc etc etc.

Oh no. None of the above.

If ALL of that indicates you have a 100% clean system, then all it can come down to after all that is an interaction with some port you have installed that most of us would not have installed that is hosing the build.

Sorry, I can't have a "100% clean system", as wouldn't be of any use to either me or my employer. Ports get installed on my machine to assist doing my work, they don't serve any purpose on their own.

And yes - it is quite possible that there's a clash with some of the other installed ports. I'm attaching the list of the installed ports (turns out, the number is 731).

I'm completely unfamiliar with python's build system so I can't guide you, beyond that it would be somewhere under the work directory.

Alas, there's nothing related to the tests output in the build directory. :-(

Addition

I decided to try the upstream code - downloaded the source for python-3.13.1, and built it. When I turn optimizations on - same error, and here are the details:

$ ./python.exe Lib/test/test_hashlib.py
..	fetching http://www.pythontest.net/hashlib/blake2b.txt ...
..	fetching http://www.pythontest.net/hashlib/blake2s.txt ...
..      fetching http://www.pythontest.net/hashlib/sha3_224.txt ...
..	fetching http://www.pythontest.net/hashlib/sha3_256.txt ...
..	fetching http://www.pythontest.net/hashlib/sha3_384.txt ...
..	fetching http://www.pythontest.net/hashlib/sha3_512.txt ...
.....	fetching http://www.pythontest.net/hashlib/shake_128.txt ...
.	fetching http://www.pythontest.net/hashlib/shake_256.txt ...
.............................
======================================================================
ERROR: test_algorithms_available (__main__.HashLibTestCase.test_algorithms_available)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/ur20980/src/Python-3.13.1/Lib/hashlib.py", line 160, in __hash_new
    return _hashlib.new(name, data, **kwargs)
           ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
_hashlib.UnsupportedDigestmodError: [digital envelope routines] unsupported

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ur20980/src/Python-3.13.1/Lib/test/test_hashlib.py", line 223, in test_algorithms_available
    digest = hashlib.new(name, usedforsecurity=False)
  File "/Users/ur20980/src/Python-3.13.1/Lib/hashlib.py", line 166, in __hash_new
    return __get_builtin_constructor(name)(data)
           ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/Users/ur20980/src/Python-3.13.1/Lib/hashlib.py", line 123, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type whirlpool

----------------------------------------------------------------------
Ran 78 tests in 9.911s

FAILED (errors=1, skipped=2)

It seems to be asking for weird unsupported hashes, such as "whirlpool" and "md4".

The above finally gave me the necessary clue.

Answer/Solution

My (Macports-installed!) OpenSSL-3.4.0 had extra providers that I added, including OQS provider, PKCS11 provider, and (drum-roll!) legacy provider. This provider, apparently, interfered with the hashlib test. Disabling it alleviated the problem:

$ openssl list -providers
Providers:
  default
    name: OpenSSL Default Provider
    version: 3.4.0
    status: active
  oqs
    name: OpenSSL OQS Provider
    version: 0.8.1-dev
    status: active
  pkcs11
    name: PKCS#11 Provider
    version: 3.4.0
    status: active
$ port -v installed python313
The following ports are currently installed:
  python313 @3.13.1_0+lto+optimizations (active) requested_variants='' platform='darwin 24' archs='arm64' date='2025-01-04T17:54:40-0500'
$

Conclusion

python313 port depends on OpenSSL-3 port, and requires that the legacy provider is not enabled.

[kencu (Ken)]

OK, good.

Let's see if anyone can reproduce your python build failure with the openssl +legacy enabled. If yes, then there we go. We can set up a check for that in the Portfile.

All those other things you do -- the things you absolutely need in /usr/local, etc -- all those things are increasing your chances of more troubles in the future, esp with no buildbot to bail you out by giving you clean builds of things.

[kencu (Ken)]

Bad news.

I reinstalled openssl3 +legacy on my system:

% port -v installed openssl3 |grep active
  openssl3 @3.4.0_0+legacy (active) requested_variants='+legacy' platform='darwin 24' archs='x86_64' date='2025-01-04T16:52:37-0800'

and then rebuilt python 313 without any troubles at all. Hashlib had no issues (Sequoia Intel):

LLVM_PROFILE_FILE="/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python313/python313/work/Python-3.13.1/code-%p.profclangr" DYLD_FRAMEWORK_PATH=/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python313/python313/work/Python-3.13.1 ./python.exe -m test --pgo --timeout=
Using random seed: 1733567849
0:00:00 load avg: 9.47 Run 44 tests sequentially in a single process
0:00:00 load avg: 9.47 [ 1/44] test_array
0:00:01 load avg: 9.47 [ 2/44] test_base64
0:00:02 load avg: 9.47 [ 3/44] test_binascii
0:00:02 load avg: 9.47 [ 4/44] test_binop
0:00:02 load avg: 9.47 [ 5/44] test_bisect
0:00:02 load avg: 9.47 [ 6/44] test_bytes
0:00:07 load avg: 8.79 [ 7/44] test_bz2
0:00:15 load avg: 7.75 [ 8/44] test_cmath
0:00:15 load avg: 7.75 [ 9/44] test_codecs
0:00:18 load avg: 7.75 [10/44] test_collections
0:00:20 load avg: 7.21 [11/44] test_complex
0:00:20 load avg: 7.21 [12/44] test_dataclasses
0:00:21 load avg: 7.21 [13/44] test_datetime
0:00:27 load avg: 6.95 [14/44] test_decimal
----------------------------------------------------------------- NOTICE -----------------------------------------------------------------
test_decimal may generate "malloc can't allocate region"
warnings on macOS systems. This behavior is known. Do not
report a bug unless tests are also failing.
See https://github.com/python/cpython/issues/85100
------------------------------------------------------------------------------------------------------------------------------------------
0:00:34 load avg: 6.63 [15/44] test_difflib
0:00:35 load avg: 6.18 [16/44] test_embed
0:00:55 load avg: 4.90 [17/44] test_float
0:00:55 load avg: 4.90 [18/44] test_fstring
0:01:00 load avg: 4.59 [19/44] test_functools
0:01:01 load avg: 4.59 [20/44] test_generators
0:01:01 load avg: 4.59 [21/44] test_hashlib
0:01:03 load avg: 4.59 [22/44] test_heapq
0:01:04 load avg: 4.59 [23/44] test_int
0:01:05 load avg: 4.70 [24/44] test_itertools
0:01:09 load avg: 4.70 [25/44] test_json
0:01:17 load avg: 4.37 [26/44] test_long
0:01:21 load avg: 4.10 [27/44] test_lzma
0:01:22 load avg: 4.10 [28/44] test_math
0:01:25 load avg: 4.01 [29/44] test_memoryview
0:01:26 load avg: 4.01 [30/44] test_operator
0:01:26 load avg: 4.01 [31/44] test_ordered_dict
0:01:27 load avg: 4.01 [32/44] test_patma
0:01:27 load avg: 4.01 [33/44] test_pickle
0:01:36 load avg: 3.70 [34/44] test_pprint
0:01:36 load avg: 3.70 [35/44] test_re
0:01:38 load avg: 3.70 [36/44] test_set
0:01:44 load avg: 3.57 [37/44] test_sqlite3
0:01:45 load avg: 3.52 [38/44] test_statistics
0:01:53 load avg: 3.48 [39/44] test_str
0:01:56 load avg: 3.28 [40/44] test_struct
0:01:57 load avg: 3.28 [41/44] test_tabnanny
0:01:59 load avg: 3.28 [42/44] test_time
0:02:01 load avg: 3.18 [43/44] test_xml_etree
0:02:03 load avg: 3.18 [44/44] test_xml_etree_c

Total duration: 2 min 4 sec
Total tests: run=9,394 skipped=198
Total test files: run=44/44
Result: SUCCESS

[kencu (Ken)]

Replying to mouse07410:

is there anything in /usr/local? If so remove it.

Sorry, that's absolutely impractical - /usr/local houses AFS, HSM (not SoftHSM), and other similar (necessary for my work) stuff, which Macports doesn't provide. One thing I can say about it - it does not collide with Macports-installed ports/stuff.

you know, you simply can't be sure it doesn't collide.

Almost all compilers will search first for headers and libraries in /usr/local, and you simply can't stop them from doing that.

At the very least, when you're plugging away at build errors, you can move /usr/local/* out of the way, eg

sudo mv /usr/local/include /usr/local/include-disabled
sudo mv /usr/local/lib /usr/local/lib-disabled

and then they won't be picked up against your will.

[mouse07410 (Mouse)]

@kencu, no, not bad news - irrelevant news.

Because I've no problem with legacy provider and python313 on my Intel Macs either. It manifests only on Apple Silicon (I have M2 Max chip, not sure how other HW versions would behave).

Re. potential impact from other installed things - I hear you. But it is what it is. And it doesn't seem to interfere (much 😉) with Macports (so far).

[kencu (Ken)]

Ah no, it is EXTREMELY relevant! Because now nobody is going to set things up so that openssl3 can't be installed with the legacy variant when we want python313.

And it points clever people in the right direction to potentially fix this problem, assuming anyone else does repro this on an arm Mac (so far, it's still only your arm system -- but we'll see...)

[mouse07410 (Mouse)]

I say it's on the new ARM64 (Apple Silicon) Macs - and very likely on the old PPC Macs.

Luckily (pleasantly surprised!), Intel Macs don't seem to be affected.

[kencu (Ken)]

Man, my M1 Mac is ripping fast.

Anyway, more bad news for you.

I reinstalled openssl +legacy, and rebuilt python313, and had no errors at all.

% uname -a
Darwin MacBookPro 24.2.0 Darwin Kernel Version 24.2.0: Fri Dec  6 19:01:59 PST 2024; root:xnu-11215.61.5~2/RELEASE_ARM64_T6000 arm64

% port -v installed | grep openssl | grep active
  openssl @3_21 (active) requested_variants='' platform='darwin 24' archs='arm64' date='2025-01-04T21:31:49-0800'
  openssl3 @3.4.0_0+legacy (active) requested_variants='+legacy' platform='darwin 24' archs='arm64' date='2025-01-04T21:31:43-0800'

# Next, run the profile task to generate the profile information.
LLVM_PROFILE_FILE="/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python313/python313/work/Python-3.13.1/code-%p.profclangr" DYLD_FRAMEWORK_PATH=/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python313/python313/work/Python-3.13.1 ./python.exe -m test --pgo --timeout=
Using random seed: 1735919212
0:00:00 load avg: 5.27 Run 44 tests sequentially in a single process
0:00:00 load avg: 5.27 [ 1/44] test_array
0:00:00 load avg: 5.27 [ 2/44] test_base64
0:00:00 load avg: 5.27 [ 3/44] test_binascii
0:00:00 load avg: 5.27 [ 4/44] test_binop
0:00:00 load avg: 5.27 [ 5/44] test_bisect
0:00:00 load avg: 5.27 [ 6/44] test_bytes
0:00:02 load avg: 4.93 [ 7/44] test_bz2
0:00:02 load avg: 4.93 [ 8/44] test_cmath
0:00:02 load avg: 4.93 [ 9/44] test_codecs
0:00:03 load avg: 4.93 [10/44] test_collections
0:00:03 load avg: 4.93 [11/44] test_complex
0:00:03 load avg: 4.93 [12/44] test_dataclasses
0:00:04 load avg: 4.93 [13/44] test_datetime
0:00:07 load avg: 4.61 [14/44] test_decimal
------------------------------------------------------- NOTICE ------------------------------------------------------
test_decimal may generate "malloc can't allocate region"
warnings on macOS systems. This behavior is known. Do not
report a bug unless tests are also failing.
See https://github.com/python/cpython/issues/85100
---------------------------------------------------------------------------------------------------------------------
0:00:08 load avg: 4.61 [15/44] test_difflib
0:00:09 load avg: 4.61 [16/44] test_embed
0:00:13 load avg: 4.32 [17/44] test_float
0:00:13 load avg: 4.32 [18/44] test_fstring
0:00:15 load avg: 4.32 [19/44] test_functools
0:00:15 load avg: 4.32 [20/44] test_generators
0:00:15 load avg: 4.32 [21/44] test_hashlib
0:00:15 load avg: 4.32 [22/44] test_heapq
0:00:16 load avg: 4.06 [23/44] test_int
0:00:16 load avg: 4.06 [24/44] test_itertools
0:00:17 load avg: 4.06 [25/44] test_json
0:00:19 load avg: 4.06 [26/44] test_long
0:00:20 load avg: 4.06 [27/44] test_lzma
0:00:21 load avg: 3.81 [28/44] test_math
0:00:22 load avg: 3.81 [29/44] test_memoryview
0:00:22 load avg: 3.81 [30/44] test_operator
0:00:22 load avg: 3.81 [31/44] test_ordered_dict
0:00:22 load avg: 3.81 [32/44] test_patma
0:00:22 load avg: 3.81 [33/44] test_pickle
0:00:24 load avg: 3.81 [34/44] test_pprint
0:00:25 load avg: 3.81 [35/44] test_re
0:00:25 load avg: 3.81 [36/44] test_set
0:00:27 load avg: 3.59 [37/44] test_sqlite3
0:00:27 load avg: 3.59 [38/44] test_statistics
0:00:30 load avg: 3.59 [39/44] test_str
0:00:30 load avg: 3.59 [40/44] test_struct
0:00:31 load avg: 3.38 [41/44] test_tabnanny
0:00:31 load avg: 3.38 [42/44] test_time
0:00:33 load avg: 3.38 [43/44] test_xml_etree
0:00:33 load avg: 3.38 [44/44] test_xml_etree_c

Total duration: 34.2 sec
Total tests: run=9,394 skipped=198
Total test files: run=44/44
Result: SUCCESS

So far, it's still only your system that shows this issue. Unless you're holding out for an M2 being different enough from my M1 to cause the error (a real long shot).

[kencu (Ken)]

the older 10.5 Macs build python313 just fine, with a one-line tiny define.

Tiger was slightly more complicated, but also fixed.

[kencu (Ken)]

@MacBookPro Python-3.13.1 % sudo ./python.exe Lib/test/test_hashlib.py
Password:
...	fetching http://www.pythontest.net/hashlib/blake2b.txt ...
..	fetching http://www.pythontest.net/hashlib/blake2s.txt ...
.............ss.................	fetching http://www.pythontest.net/hashlib/sha3_224.txt ...
..	fetching http://www.pythontest.net/hashlib/sha3_256.txt ...
..	fetching http://www.pythontest.net/hashlib/sha3_384.txt ...
..	fetching http://www.pythontest.net/hashlib/sha3_512.txt ...
.....	fetching http://www.pythontest.net/hashlib/shake_128.txt ...
.	fetching http://www.pythontest.net/hashlib/shake_256.txt ...
.............................
----------------------------------------------------------------------
Ran 78 tests in 10.605s

OK (skipped=2)

[mouse07410 (Mouse)]

Hmm, interesting.

While my OpenSSL was Macports-installed, I did not choose +legacy, but simply added/enabled the provider in openssl.cnf. The provider seems to be functional - at least it responds to queries...

Are you saying that not doing

$ port install openssl +legacy

was a mistake? It seems that the only difference +legacy makes is adding the appropriate lines in openssl.cnf. I installed OpenSSL with

$ sudo port install openssl3

and I see that the legacy provider itself is installed and seems well:

$ ll /opt/local/libexec/openssl3/lib/ossl-modules
total 5320
drwxr-xr-x   6 root  admin      192 Dec 19 11:34 ./
drwxr-xr-x  21 root  admin      672 Jan  1 22:44 ../
-rwxr-xr-x   1 root  admin   850608 Mar 10  2024 gostprov.dylib*
-rwxr-xr-x   1 root  admin   112016 Dec 17 13:08 legacy.dylib*
-rwxr-xr-x   1 root  admin  1408384 Jan  4 00:28 oqsprovider.dylib*
-rwxr-xr-x   1 root  admin   423632 Dec 19 11:34 pkcs11.dylib*
$ openssl list -digest-algorithms | grep legacy  
  { 1.3.36.3.2.1, RIPEMD, RIPEMD-160, RIPEMD160, RMD160 } @ legacy
$ 
$ openssl dgst -ripemd160 -hex ~/.zshrc
RIPEMD-160(/Users/ur20980/.zshrc)= c637580564c07502082cca805b075e776d5c77f1
$

On the other hand,

$ openssl dgst -digest -list
Supported digests:
-blake2b512                -blake2s256                -gost-mac                 
-gost-mac-12               -kuznyechik-ctr-acpkm-omac -kuznyechik-mac           
-magma-mac                 -md4                       -md5                      
-md5-sha1                  -md_gost12_256             -md_gost12_512            
-md_gost94                 -mdc2                      -ripemd                   
-ripemd160                 -rmd160                    -sha1                     
-sha224                    -sha256                    -sha3-224                 
-sha3-256                  -sha3-384                  -sha3-512                 
-sha384                    -sha512                    -sha512-224               
-sha512-256                -shake128                  -shake256                 
-sm3                       -ssl3-md5                  -ssl3-sha1                
-streebog256               -streebog512               -whirlpool                

$ openssl dgst -whirlpool -hex ~/.zshrc
Error setting digest
40024EF101000000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:355:Global default library context, Algorithm (whirlpool : 108), Properties ()
40024EF101000000:error:03000086:digital envelope routines:evp_md_init_internal:initialization error:crypto/evp/digest.c:271:
$ 

legacy provider doesn't seem to include whirlpool, but test_hashlib tried to use it, according to the log I got.

Unless you're holding out for an M2 being different enough from my M1 to cause the error (a real long shot).

I don't know - and I agree that it would be a really long shot.

Update

I re-installed OpenSSL-3 with +legacy, and, unfortunately, the results are the same regarding announced but failing algorithms:

$ port installed -v openssl3
The following ports are currently installed:
  openssl3 @3.4.0_0+legacy+rfc3779 (active) requested_variants='+legacy+rfc3779' platform='darwin 24' archs='arm64' date='2025-01-05T09:53:51-0500'
$ openssl version
OpenSSL 3.4.0 22 Oct 2024 (Library: OpenSSL 3.4.0 22 Oct 2024)
$ openssl list -digest-algorithms | grep whirlpool
  whirlpool
$ openssl dgst -whirlpool -hex ~/.zshrc
Error setting digest
40024EF101000000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:355:Global default library context, Algorithm (whirlpool : 108), Properties ()
40024EF101000000:error:03000086:digital envelope routines:evp_md_init_internal:initialization error:crypto/evp/digest.c:271:
$ 

On the other hand, OpenSSL-3 that I build myself from the sources (I track - and occasionally contribute to - master branch), works fine with the same configuration:

$ openssl3 version                      
OpenSSL 3.5.0-dev  (Library: OpenSSL 3.5.0-dev )
$ openssl3 list -providers
Providers:
  default
    name: OpenSSL Default Provider
    version: 3.5.0
    status: active
  gost
    name: OpenSSL GOST Provider
    status: active
  legacy
    name: OpenSSL Legacy Provider
    version: 3.4.0
    status: active
  oqs
    name: OpenSSL OQS Provider
    version: 0.8.1-dev
    status: active
  pkcs11
    name: PKCS#11 Provider
    version: 3.4.0
    status: active
$ openssl3 dgst -whirlpool -hex ~/.zshrc
WHIRLPOOL(/Users/ur20980/.zshrc)= 2714940724676c37e299b639c811c678faa93b581c75fc5e85c78f50939835de117a855146248a61e7a7f04cd980f97e20ebc699ee53275b15608a53c41e33ac
$ 

[kencu (Ken)]

this issue turned out to be due to a problem with openssl3 that occurs under rare circumstances when certain 3rdvparty providers have been manually enabled in a certain order.

That openssl3 issue is described and is being followed it this ticket #71760 and has already been fixed upstream

there is nothing to fix in python313