openssl3 @3.2.0_0+universal may have broken PRNG on High Sierra and older

[fhgwright (Fred Wright)]

After today's upgrade outdated, I found that ssh stopped working:

MacPro:~ fw$ ssh -4 MacPro
PRNG is not seeded

This is related to the new openssl3. Based on online threads, it seems that the typical cause on Linux systems is a problem with /dev/random and/or /dev/urandom, but that doesn't seem to be the case here, and it survives relogging in as well as rebooting.

It can be seen directly with openssl rand:

--->  Activating openssl3 @3.2.0_0+universal
MacPro:~ fw$ openssl rand -hex 8
10835E74FF7F0000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:342:Global default library context, Algorithm (CTR-DRBG : 0), Properties (<null>)
10835E74FF7F0000:error:12000090:random number generator:rand_new_drbg:unable to fetch drbg:crypto/rand/rand_lib.c:655:

Whereas:

--->  Activating openssl3 @3.1.4_1+universal
MacPro:~ fw$ openssl rand -hex 8
655a13b962ccc7df

The openssh upgrade build also failed with a PRNG-related error, but succeeded after rolling back openssl3 to 3.1.4_1. This is really bizarre, since I don't see why it should need random numbers during the build. Perhaps it's a "nonreproducible build" feature. :-)

This is on 10.9. It must not be entirely consistent, though, since I see an updated openssh package for darwin13 on the servers later than the openssl3 package.

It looks like OpenSSL 3.2.0 has some new paranoia regarding entropy, which may not be working properly on the Mac.

[thetrial (alabay)]

I have the same issue under 10.12.6. All my ssh-connections are broken now. How to fix that?!

[fhgwright (Fred Wright)]

As long as you have the previous version of openssl3 installed, just activate it and ssh will work again.

[thetrial (alabay)]

On one machine I can do that, on the other a reclaim has swiped through. Copy the binary? Or would this break more? I guess it’s much more important to fix that. What a mess!

[thetrial (alabay)]

BTW, could you please reassign that ticket to larryv? I can’t do that, but he is the maintainer.

I have restored the openssl3 directory under libexec, so it works now, but this is quite a manual hack. Maybe I should stay away from using reclaim each time.

[RobK88]

Yes, openssl3 @3.2.0_0 breaks many ports (including ntp, git and openssh) due to insufficient entropy when openssl3 was built. I reported this earlier.

See ​https://trac.macports.org/ticket/68769 and ​https://trac.macports.org/ticket/68763

e.g. One now sees issues like this:

rob$ git pull origin master
fatal: unable to access 'https://github.com/RobK88/macports-ports.git/': Insufficient randomness

When you try to build openssh on Lion, the error is more apparennt:

configure: error: OpenSSH has no source of random numbers. Please configure OpenSSL with an entropy source or re-run configure using one of the --with-prngd-port or --with-prngd-socket options
Command failed:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_net_openssh/openssh/work/openssh-9.5p1" && ./configure --prefix=/opt/local --with-ssl-dir=/opt/local --sysconfdir=/opt/local/etc/ssh --with-privsep-path=/var/empty --with-md5-passwords --with-pid-dir=/opt/local/var/run --with-pam --mandir=/opt/local/share/man --with-zlib=/opt/local --without-kerberos5 --with-libedit --with-pie --without-xauth --without-ldns --with-audit=bsm --with-keychain=apple 
Exit code: 1

I reverted back to the previous version of openssl3, now everything works fine again.

[RobK88]

The fix for openssl3 might be in the configure error message generated for openssh:

Please configure OpenSSL with an entropy source or re-run configure using one of the --with-prngd-port or --with-prngd-socket options

[fhgwright (Fred Wright)]

In addition to the entropy issues, openssl3 @3.2.0 fails many of its tests, while openssl3 @3.1.4_1 passes all of them, and openssl3 @3.2.0 fails to build on 10.4 and 10.5 (except 10.5 i386), while openssl3 @3.1.4_1 builds for all platforms.

It's clear that openssl3 @3.2.0 isn't ready for prime-time. I'm currently testing a fix that rolls openssl3 back to 3.1.4, while adding openssl3-devel @3.2.0 for sorting out the issues. Going from 3.1.x to 3.2.x is a major update, requiring more thorough testing than a run-of-the-mill point release.

BTW, the revbump for openssh was definitely superfluous. I suspect the same for freeradius and p5-net-ssleay, though I can't easily check. The revbump for openssl added some new manpage symlinks, but otherwise didn't change anything. The Portfile comment about unconditionally revbumping these dependents is clearly misguided (except perhaps for openssl).

[neverpanic (Clemens Lang)]

Replying to fhgwright:

In addition to the entropy issues, openssl3 @3.2.0 fails many of its tests, while openssl3 @3.1.4_1 passes all of them, and openssl3 @3.2.0 fails to build on 10.4 and 10.5 (except 10.5 i386), while openssl3 @3.1.4_1 builds for all platforms.

10.4 and 10.5 are very very old. Expect breakage from time to time on systems that are as old as this. You're reporting this for Mavericks, that's now 10 versions ago. Mavericks was released in 2013. The last update for Mavericks was shipped in 2016, that's 7 years ago.

Is any of this happening on operating systems that are still supported? I haven't seen any indication of such breakage.

It's clear that openssl3 @3.2.0 isn't ready for prime-time. I'm currently testing a fix that rolls openssl3 back to 3.1.4, while adding openssl3-devel @3.2.0 for sorting out the issues. Going from 3.1.x to 3.2.x is a major update, requiring more thorough testing than a run-of-the-mill point release.

Please don't hold supported operating systems hostage because you're trying to run outdated software. I'm fine with a conditional downgrade to 3.1.4 on Mavericks and older, but I really really really don't want to maintain another version of OpenSSL and deal with the potential breakage that occurs if we have multiple OpenSSL versions available, just because of Mavericks.

Spend the time to investigate how this can be fixed on Mavericks instead.

BTW, the revbump for openssh was definitely superfluous. I suspect the same for freeradius and p5-net-ssleay, though I can't easily check. The revbump for openssl added some new manpage symlinks, but otherwise didn't change anything. The Portfile comment about unconditionally revbumping these dependents is clearly misguided (except perhaps for openssl).

I'm not sure about freeradius, I haven't read their source code. For openssh, I'm pretty sure the revbump is required, because OpenSSH used to have a check to ensure that the minor version it was built against is the minor version against it is running, even though binary compatibility was preserved between OpenSSL 3.1.4 and 3.2.0. Maybe they dropped this pointless check now, but it was definitely still there a few months ago when a colleague of mine checked for Fedora.

I personally verified the p5-net-ssleay bump is required. We saw build failures in ports that depend on p5-net-ssleay when we updated from 3.0 to 3.1.

Doing the revbump is on the safe side. Not doing it may lead to problems we don't immediately see.

[neverpanic (Clemens Lang)]

Can you determine whether your operating systems have the CCRandomGenerateBytes library function from Apple's CommonCrypto available?

[neverpanic (Clemens Lang)]

Also, what's the output of openssl version -r on your operating system?

[neverpanic (Clemens Lang)]

CCRNGStatus CCRandomGenerateBytes(void *bytes, size_t count)
__OSX_AVAILABLE_STARTING(__MAC_10_10, __IPHONE_8_0);

So not on anything older than 10.10 (which 10.9 is). OpenSSL also doesn't use this on anything older than 10.12.

getentropy(2) is only available on 10.12 or newer, so that's out as well.

Can you try configure.args-append --with-rand-seed=devrandom to see if that helps?

[RobK88]

What do the openssl3 --with-prngd-port or --with-prngd-socket compile options do? Maybe one can choose another PRNG when openssl3 is being configured?

[RobK88]

@neverpanic - On my Mac running openssl @3.2.0 or @3.1.4 on Lion, here is the output you requested?:

bash-3.2$  openssl version -r
Seeding source: os-specific

FYI -- Here is a really good and recent article on entropy in openssl 3.0:

​https://lightshipsec.com/entropy-in-openssl-3-0/

According to the article:

By default, “os” is chosen and it is essentially an alias of “getrandom,devrandom“.

[thetrial (alabay)]

The output here under Sierra 10.12.6 with OpenSSL 3.1.4 24 Oct 2023 (Library: OpenSSL 3.1.4 24 Oct 2023) is the same – Seeding source: os-specific. So I’m quite above Mavericks 10.9, but PRNG is also broken with 3.2.0.

[jmroot (Joshua Root)]

Replying to thetrial:

So I’m quite above Mavericks 10.9, but PRNG is also broken with 3.2.0.

There are a few different symptoms being reported, so when you say "PRNG is also broken", what does that mean exactly? That openssl rand always fails?

[neverpanic (Clemens Lang)]

Replying to thetrial:

The output here under Sierra 10.12.6 with OpenSSL 3.1.4 24 Oct 2023 (Library: OpenSSL 3.1.4 24 Oct 2023) is the same – Seeding source: os-specific. So I’m quite above Mavericks 10.9, but PRNG is also broken with 3.2.0.

That's a helpful data point, because on 10.12, the getrandom(2) syscall should be used, so it isn't an issue with seeding.

[thetrial (alabay)]

With openssl3 @3.2.0_0+universal:

Trying to ssh: PRNG is not seeded.

Rand:

openssl rand -hex 8
C083D40F01000000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:342:Global default library context, Algorithm (CTR-DRBG : 0), Properties (<null>)
C083D40F01000000:error:12000090:random number generator:rand_new_drbg:unable to fetch drbg:crypto/rand/rand_lib.c:655:

With openssl3 @3.1.4_1+universal all is fine.

[neverpanic (Clemens Lang)]

Can somebody on an affected system provide the output for

openssl list -providers
openssl list -random-instances
openssl list -random-generators

[neverpanic (Clemens Lang)]

Replying to RobK88:

What do the openssl3 --with-prngd-port or --with-prngd-socket compile options do? Maybe one can choose another PRNG when openssl3 is being configured?

These aren't openssl options. I think they are openssh options, but that doesn't fix the problem in OpenSSL.

[jmroot (Joshua Root)]

Another interesting data point is that it seems fine on 10.6 (at least with an x86_64 only build; I do wonder if +universal makes a difference in some cases.) There are a number of differences there, including the use of gcc-4.2 rather than clang.

[thetrial (alabay)]

3.1.4:

openssl list -providers
Providers:
  default
    name: OpenSSL Default Provider
    version: 3.1.4
    status: active

openssl list -random-instances
primary:
  CTR-DRBG @ default
  state = ready
  cipher: 'AES-256-CTR'
public:
  CTR-DRBG @ default
  state = ready
  cipher: 'AES-256-CTR'
private:
  CTR-DRBG @ default
  state = ready
  cipher: 'AES-256-CTR'

Provided RNGs and seed sources:
  CTR-DRBG @ default
  HASH-DRBG @ default
  HMAC-DRBG @ default
  SEED-SRC @ default
  TEST-RAND @ default

3.2.0:

openssl list -providers
ERROR: Memory allocation

openssl list -random-instances
primary:
public:
private:

openssl list -random-generators
Provided RNGs and seed sources:

[neverpanic (Clemens Lang)]

OK, so maybe the issue is that OpenSSL fails to load the providers completely, and the random instances are just the first thing that would be needed from the providers, which aren't there?

Can you re-try OPENSSL_CONF=/dev/null openssl list -providers just to make sure that it isn't MacPorts' openssl.cnf file that somehow breaks this?

[thetrial (alabay)]

3.2.0:

OPENSSL_CONF=/dev/null openssl list -providers
ERROR: Memory allocation

[RobK88]

FYI -- On Lion and Mtn Lion, I get exactly the same output:

3.1.4

bash-3.2$ openssl list -providers
Providers:
  default
    name: OpenSSL Default Provider
    version: 3.1.4
    status: active

bash-3.2$ openssl list -random-instances
primary:
  CTR-DRBG @ default
  state = ready
  cipher: 'AES-256-CTR'
public:
  CTR-DRBG @ default
  state = ready
  cipher: 'AES-256-CTR'
private:
  CTR-DRBG @ default
  state = ready
  cipher: 'AES-256-CTR'

bash-3.2$ openssl list -random-generators
Provided RNGs and seed sources:
  CTR-DRBG @ default
  HASH-DRBG @ default
  HMAC-DRBG @ default
  SEED-SRC @ default
  TEST-RAND @ default

3.2.0

bash-3.2$ openssl list -providers
ERROR: Memory allocation

bash-3.2$ openssl list -random-instances
primary:
public:
private:

bash-3.2$ openssl list -random-generators
Provided RNGs and seed sources:

bash-3.2$ OPENSSL_CONF=/dev/null openssl list -providers
ERROR: Memory allocation

[kencu (Ken)]

on 10.8:

$ uname -a
Darwin localhost 12.6.0 Darwin Kernel Version 12.6.0: Wed Mar 18 16:23:48 PDT 2015; root:xnu-2050.48.19~1/RELEASE_X86_64 x86_64

building openssl3 with macports-clang-11:

sudo port -v -s install openssl3 configure.compiler=macports-clang-11

I get no errors:

$ openssl-3 list -providers
Providers:
  default
    name: OpenSSL Default Provider
    version: 3.2.0
    status: active

$ openssl-3 list -random-generators
Provided RNGs and seed sources:
  CTR-DRBG @ default
  HASH-DRBG @ default
  HMAC-DRBG @ default
  SEED-SRC @ default
  TEST-RAND @ default

[kencu (Ken)]

of course, that now likely means bootstrapping issues if the fix for older clangs is not obvious... have to see if that makes a loop or not.

sooner rather than later, clang-11-bootstrap will come to rescue us from those.

[RobK88]

Ken's workaround does not work on Lion:

bash-3.2$ sudo port upgrade openssl openssl3 configure.compiler=macports-clang-11
etc

bash-3.2$ port installed openssl3
The following ports are currently installed:
  openssl3 @3.1.4_0
  openssl3 @3.1.4_0+universal
  openssl3 @3.2.0_0+universal (active)

bash-3.2$ openssl-3 list -providers
ERROR: Memory allocation

bash-3.2$ openssl-3 list -random-generators
Provided RNGs and seed sources:

[RobK88]

I also tried Ken's workaround on my Mac running Mtn Lion. I built openssl3 @3.2.0_0 from source using clang-11.

For some reason, Ken's workaround does NOT work on my Mac running Mtn Lion.

I get the same errors such as:

bash-3.2$ openssl-3 list -providers
ERROR: Memory allocation

[kencu (Ken)]

Well, I also did one more thing - I reduced the optimization intensity from -Os to -O0, because I was planning to debug it.

configure.optflags -O0 -g

I tried that first with the default compiler, but still crashed -- but with clang-11 it just worked after I rebuilt it.

Let me go back and see exactly what does and doesn't fix it.

[thetrial (alabay)]

I can’t experiment with the machine here now because I replayed the necessary openssl3 directory under libexec, so I have openssl@3.1.4_1 … but Macports does not know that and thinks/claims I have 3.2.0_0. On my other machine I have both in the registry. Is there a way too manually add soomething? I could not replay the older version via git because the complete list of older ports (under devel? security?) was/is truncated :-/

[kencu (Ken)]

Yeah, all you need to do is build openssl3 with clang-11 and it's fine (for me).

This is on 10.7 now:

$ uname -a
Darwin Kens-107-VM-Parallels.local 11.4.2 Darwin Kernel Version 11.4.2: Thu Aug 23 16:25:48 PDT 2012; root:xnu-1699.32.7~1/RELEASE_X86_64 x86_64

$ openssl-3 list -providers
Providers:
  default
    name: OpenSSL Default Provider
    version: 3.2.0
    status: active

$ openssl-3 list -random-generators
Provided RNGs and seed sources:
  CTR-DRBG @ default
  HASH-DRBG @ default
  HMAC-DRBG @ default
  SEED-SRC @ default
  TEST-RAND @ default

Don't forget not to be tricked when you rebuild from source. You need to see it rebuild, and not just install or reinstall some previously-built version.

sudo port uninstall openssl3
sudo port -v -s install openssl3 configure.compiler=macports-clang-11

[thetrial (alabay)]

Only clang-11? I have 16 and 17. And maybe onboard 9. So one has to install explitely 11? I going to try that this evening or tomorrow. And I’m curious because of the +universal variant.

[kencu (Ken)]

probably 16 works. I didn't try it.

clang-17 is broken on older systems for exceptions and possibly other things, so you shouldn't use that.

[thetrial (alabay)]

Well, it didn’t work at my first attempt (with clang-11).

I then tried to uninstall the old 3.2.0 before retrying … and it broke my whole installation because it removed 3.4.1, too. I try to replay a backup of /opt. Annoying.

Something with this update is a biiig problem. And I’m not quite sure if it is just one or a multiple problem. There also this +universal thing.

On the other side … nothing kept unsolved at MacPorts. Maybe the bug or problem is found soon.

[kencu (Ken)]

looks like older ancient clangs miscompile openssl3.

we've seen that before many times, in openssl and other software. We'll see it many many many times in the future too.

the problem with blacklisting older clangs may be (as always) bootstrapping. Luckily we have clang-11-bootstrap which should work OK.

[neverpanic (Clemens Lang)]

So we can commit a stop-gap measure to blacklist older compilers for openssl3 – but the question is which versions?

We have seen reports of brokenness on 10.12. According to wiki:XcodeVersionInfo, the newest clang on 10.12 is Apple LLVM version 9.0.0 (clang-900.0.39.2). 10.13 has Apple LLVM version 10.0.0 (clang-1000.11.45.5), where we don't have data. 10.14 has Apple clang version 11.0.0 (clang-1100.0.33.17), which might work considering that clang-11 also seems to work as reported in this ticket.

Does somebody have a 10.13 system around where they could test whether its compiler also miscompiles OpenSSL?

The other question is whether somebody wants to debug what specifically is causing this issue, potentially using a git bisect between openssl-3.1.4 and openssl-3.2.0. Identifying the offending commit would allow us to fix this upstream, which would help us in making sure we don't have to keep working around this downstream. I'd do it myself, but running a 10.12 VM on arm64 seems to be almost impossible to do.

Considering that there is something fundamentally broken you should be able to identify this quickly by doing these steps in a clone of ​https://github.com/openssl/openssl.git:

• ./config
• make -j$(sysctl -n hw.ncpu)
• make test HARNESS_JOBS=$(sysctl -n hw.ncpu)
• Mark the build as git bisect good if all tests pass, or only a few tests fail, mark it as git bisect bad if many tests fail.

[thetrial (alabay)]

It did not work. I rebuilt with clang-16 the +universal variant … but the error remains. Im managed to build a non-universal 3.2.0 … and that seemed to work. I’m really clueless. I copied over my 3.1.4 directory … and guess I’m breaking my MP installation more and more :-(

[kencu (Ken)]

so a non-universal openssl3 built with clang-16 shows no errors, but when you built it +universal you got the error again?

That would be very weird.

I didn't try that as yet.

[RobK88]

I tried to reinstall openssl3@3.2.0_0 using clang-11 on Lion.

It build with no errors but openssl3 still does not work.

openssl-3 list -providers
ERROR: Memory allocation

etc

[RobK88]

I installed opensslw@3.2.0_0 on my Mac running High Sierra. (I have not tried building using clang-11)

It builds and installs but does NOT work. Same errors as with Lion and Mtn Lion:

openssl-3 list -providers
ERROR: Memory allocation

etc.

[kencu (Ken)]

Rob, did you do this again:

sudo port upgrade openssl openssl3 configure.compiler=macports-clang-11

If so, I am not sure that does anything to rebuild openssl3.

If you did something else, please excuse me. Or well, actually -- what exactly did you do? What did you see when you did it?

[RobK88]

ken - I am still building openssl3 on High Sierra using clang-11.

On Lion, below is the output you requested:

bash-3.2$ sudo port install -s openssl3 configure.compiler=macports-clang-11 
--->  Computing dependencies for openssl3
--->  Fetching distfiles for openssl3
--->  Verifying checksums for openssl3
--->  Extracting openssl3
--->  Configuring openssl3
--->  Building openssl3                                  
--->  Staging openssl3 into destroot                     
--->  Installing openssl3 @3.2.0_0                       
--->  Deactivating openssl3 @3.2.0_0+universal
--->  Cleaning openssl3
--->  Activating openssl3 @3.2.0_0
--->  Cleaning openssl3
--->  Updating database of binaries
--->  Scanning binaries for linking errors
--->  Found 4 broken files, matching files to ports      
--->  Found 1 broken port, determining rebuild order
You can always run 'port rev-upgrade' again to fix errors.
The following ports will be rebuilt: curl @8.4.0+ssl+universal
Continue? [Y/n]: Y
--->  Computing dependencies for curl
--->  Dependencies to be installed: openssl openssl3
--->  Deactivating openssl3 @3.2.0_0
--->  Cleaning openssl3
--->  Activating openssl3 @3.2.0_0+universal
--->  Cleaning openssl3
--->  Deactivating openssl @3_15
--->  Cleaning openssl
--->  Activating openssl @3_15+universal
--->  Cleaning openssl
--->  Cleaning curl
--->  Scanning binaries for linking errors
--->  No broken files found.                             
--->  No broken ports found.

bash-3.2$ openssl-3 list -providers
ERROR: Memory allocation

bash-3.2$ port installed openssl
The following ports are currently installed:
  openssl @3_14
  openssl @3_14+universal
  openssl @3_15
  openssl @3_15+universal (active)

bash-3.2$ port installed openssl3
The following ports are currently installed:
  openssl3 @3.1.4_0
  openssl3 @3.1.4_0+universal
  openssl3 @3.2.0_0
  openssl3 @3.2.0_0+universal (active)
bash-3.2$

[RobK88]

Success on High Sierra when building using clang-11 but only with the regular non-universal variant.

port installed openssl3
The following ports are currently installed:
  openssl3 @3.2.0_0 (active)
  openssl3 @3.2.0_0+universal

Grinchs-MacBook-Pro:~ grinch$ openssl-3 list -providers
Providers:
  default
    name: OpenSSL Default Provider
    version: 3.2.0
    status: active

[fhgwright (Fred Wright)]

Note that the failure on Sierra was reported here before the summary was changed to say "Mavericks and older".

While I understand the "holding hostage" concept, breaking critical programs like ssh and sshd on many platforms just to rush out (less than an hour after the upstream release) an update that almost nobody actually needs was hardly reasonable. There isn't even a "security fix" excuse, given that 3.1 is still fully supported upstream (as is 3.0, for that matter). And breaking sshd on headless systems can make them difficult to fix.

It looks like it was only sheer luck that the major upgrade from 3.0 to 3.1 went smoothly. 1.0->1.1 was incompatible. 1.1->3.0 was incompatible. 3.1->3.2 is currently majorly broken. 3.0->3.1 seems to have been the exception to the rule, and one shouldn't count on other major upgrades to go smoothly.

BTW, I've also seen test failures on 10.14 x86_64 and 14.x arm64 +universal. I guess folks who think Sonoma is too old to worry about can ignore the latter.

IMO, it would make sense to have subports for all the separately maintained upstream versions, similar to what's already done for languages (and for the 1.x versions). That would include not only openssl31 and openssl32, but also openssl30, not only for completeness, but also to make that version available for testing, even though it's unlikely to be needed as a port dependency. However, this is a significantly more complicated change than just a rollback plus a -devel for 3.2, and the priority should be undoing the damage ASAP.

I already have a fully tested commit for the latter, which I can submit as a PR if I'm not wasting my time. It would take the pressure off figuring out how to fix the multiple issues in 3.2

[RobK88]

It looks like opnssl@3.2.0 builds and runs fine on the latest macOS releases. The port breaks with older macOS releases.

If you want to keep and maintain just one openssl port, perhaps the simple solution is for the openssl port to install @3.2.0_0 on the latest macOS releases and instal #3.1.4_0 on older macOS releases. This has been done with other ports like go.

The downsides with this approach are possible security vulnerabilities for users of openssl on older Macs. If security vulnerabilities are found with @3.1.4 which are not fixed, then MacPosrts users of openssl may be exposed to security vulnerabilities. Often developers only fix security vulnerabilities in the latest releases and will not fix older releases.

[kencu (Ken)]

Replying to RobK88:

Success on High Sierra when building using clang-11 but only with the regular non-universal variant.

Ah ... I bet it also has something to do with that -DBROKEN_CLANG_ATOMICS that was added.

[kencu (Ken)]

(and that might be why it's showing up on 10.13 and less, because -DBROKEN_CLANG_ATOMICS is only used there, for universal builds.)

[RobK88]

Ken, my success with High Sierra was short lived. Even though openssl appears to work, it breaks other ports.

E.G. I am unable to build curl on High Sierra when openssl@3.2.0_0 was installed:

:info:configure configure: error: --with-openssl was given but OpenSSL could not be detected
:info:configure Command failed:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_net_curl/curl/work/curl-8.4.0-i386" && ./configure --prefix=/opt/local --disable-silent-rules --enable-ipv6 --without-gnutls --without-gssapi --without-libgsasl --without-librtmp --without-libssh2 --without-mbedtls --with-nghttp2=/opt/local --with-openssl --without-secure-transport --without-wolfssl --disable-ares --disable-ldap --disable-ldaps --with-brotli --with-libidn2=/opt/local --with-zlib=/opt/local --with-zstd ac_cv_prog_AWK=/usr/bin/awk --with-ca-bundle=/opt/local/share/curl/curl-ca-bundle.crt 
:info:configure Exit code: 1
:error:configure Failed to configure curl: consult /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_net_curl/curl/work/curl-8.4.0-i386/config.log
:error:configure Failed to configure curl: configure failure: command execution failed
:debug:configure Error code: NONE
:debug:configure Backtrace: configure failure: command execution failed
:debug:configure     while executing
:debug:configure "$procedure $targetname"
:error:configure See /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_net_curl/curl/main.log for details.

[RobK88]

P.S. I will be away from my old Macs for a few days. So I will not be to help. I hope others can find and fix the issue. Good luck!

[kencu (Ken)]

Ah, this build system is frustrating, as I dig deeper.....

I'm trying to build openssl3 with clang-11-bootstrap +universal to avoid the inevitable dependency loop, and guess what:

It won't link, because clang-11-bootstrap ain't actually universal and can't build anything universal:

$ nm -arch i386 /opt/local/libexec/clang-11-bootstrap/lib/clang/11.1.0/lib/darwin/libclang_rt.osx.a
nm: file: /opt/local/libexec/clang-11-bootstrap/lib/clang/11.1.0/lib/darwin/libclang_rt.osx.a does not contain architecture: i386

I am taking a break from this headache too -- I'm not sure how far to go back and start fixing things, to be honest.

[neverpanic (Clemens Lang)]

Replying to fhgwright:

Note that the failure on Sierra was reported here before the summary was changed to say "Mavericks and older".

I missed the one report about this, that's my fault. I have changed it to High Sierra now, which I believe accurately represents the situation unless I'm missing something?

While I understand the "holding hostage" concept, breaking critical programs like ssh and sshd on many platforms just to rush out (less than an hour after the upstream release) an update that almost nobody actually needs was hardly reasonable. There isn't even a "security fix" excuse, given that 3.1 is still fully supported upstream (as is 3.0, for that matter). And breaking sshd on headless systems can make them difficult to fix.

Homebrew has also already updated to 3.2. OpenSSL 3.2 also includes a fix for a timing side-channel in RSA PKCS#1v1.5 decryption that is exploitable using the Marvin Attack: ​https://people.redhat.com/~hkario/marvin/. The pull request that fixes this across all users is ​https://github.com/openssl/openssl/pull/13817, but was only merged into 3.2. Without this change, higher-level libraries such as python cryptography cannot implement RSA PKCS#1v1.5 decryption in a timing side channel free way.

It looks like it was only sheer luck that the major upgrade from 3.0 to 3.1 went smoothly. 1.0->1.1 was incompatible. 1.1->3.0 was incompatible. 3.1->3.2 is currently majorly broken. 3.0->3.1 seems to have been the exception to the rule, and one shouldn't count on other major upgrades to go smoothly.

3.1 to 3.2 isn't a major upgrade. The ABI is compatible. I'm not sure holding off on the update would have contributed to significantly broader test coverage, either.

BTW, I've also seen test failures on 10.14 x86_64 and 14.x arm64 +universal. I guess folks who think Sonoma is too old to worry about can ignore the latter.

Please stop the ad-hominem attacks.

Additionally, I did follow up on your comment saying that there are broken tests on 14.x by running the upstream test suite. I did not encounter test failures. Can you provide details?

IMO, it would make sense to have subports for all the separately maintained upstream versions, similar to what's already done for languages (and for the 1.x versions). That would include not only openssl31 and openssl32, but also openssl30, not only for completeness, but also to make that version available for testing, even though it's unlikely to be needed as a port dependency. However, this is a significantly more complicated change than just a rollback plus a -devel for 3.2, and the priority should be undoing the damage ASAP.

We've had that before, and I think it's an explicit non-goal. It locks us into not being able to update and stop maintaining the old port because some port still depends on it. Best example: The OpenSSL 1.1 port, which is now unmaintained, but we can't remove it because ports still use it.

I already have a fully tested commit for the latter, which I can submit as a PR if I'm not wasting my time. It would take the pressure off figuring out how to fix the multiple issues in 3.2

I don't think a general rollback on all systems is the way to go, as there are no issues I'm aware of in the range [current-3, current] (other than a PostgreSQL issue that is apparently on the PostgreSQL side, not in OpenSSL). I would take a pull request that rolls back older systems to 3.1.4.

[kencu (Ken)]

removing the block in the Portfile that sets BROKEN_CLANG_ATOMICS for all universal builds that include i386, and using a newer macport-clang like 16, builds a working openssl3 that works even when built +universal.

Now that there is a potential fix for clang-11-bootstrap to build universal ports, I'll test that out and if it can build universal indeed, then we can:

1. strip out the Portfile code that sets BROKEN_CLANG_ATOMICS as it is not needed I believe
2. force all older systems (exact systems TBA, but presumably 10.8 and less, and maybe newer ones) to build openssl3 with clang-11-bootstrap, to avoid dependency loops

[thetrial (alabay)]

Thanks Ken, I’m really excited.

[kencu (Ken)]

clang-11-bootstrap sadly seems to be unable to build openssl3 at present, even after the potential fixes in the clang-11-bootstrap PR.

some i386 atomic symbols are missing, for some reason, although many are there. At present I don’t fully understand why the symbols are missing.

so we remain out of luck for now.

[fhgwright (Fred Wright)]

Replying to neverpanic:

Replying to fhgwright:

Note that the failure on Sierra was reported here before the summary was changed to say "Mavericks and older".

I missed the one report about this, that's my fault. I have changed it to High Sierra now, which I believe accurately represents the situation unless I'm missing something?

Actually, I subsequently discovered that openssl rand also fails on High Sierra. The range of OSes where 3.2.0 works properly is continuing to shrink.

3.1 to 3.2 isn't a major upgrade. The ABI is compatible. I'm not sure holding off on the update would have contributed to significantly broader test coverage, either.

It's a major upgrade in the sense that it's a separate upstream release branch, with the prior branch still being fully maintained. It's recognized that there may be issues requiring some people to stick to the older version. ABI incompatibility isn't the only source of trouble. Even 3.0 is still maintained upstream, and 1.1 only stopped a few months ago.

Additionally, I did follow up on your comment saying that there are broken tests on 14.x by running the upstream test suite. I did not encounter test failures. Can you provide details?

There are failures in "port test" on 14.x arm64. I have logs, but that seems OT for this ticket.

IMO, it would make sense to have subports for all the separately maintained upstream versions, similar to what's already done for languages (and for the 1.x versions). That would include not only openssl31 and openssl32, but also openssl30, not only for completeness, but also to make that version available for testing, even though it's unlikely to be needed as a port dependency. However, this is a significantly more complicated change than just a rollback plus a -devel for 3.2, and the priority should be undoing the damage ASAP.

We've had that before, and I think it's an explicit non-goal. It locks us into not being able to update and stop maintaining the old port because some port still depends on it. Best example: The OpenSSL 1.1 port, which is now unmaintained, but we can't remove it because ports still use it.

Maintaining ports with few upstream changes isn't that hard. And ports don't exist solely to provide dependencies for other ports. Users may have other code that needs to build against other versions, if for no other reasons than testing. I'd volunteer as co-maintainer if that helps.

I don't think a general rollback on all systems is the way to go, as there are no issues I'm aware of in the range [current-3, current] (other than a PostgreSQL issue that is apparently on the PostgreSQL side, not in OpenSSL). I would take a pull request that rolls back older systems to 3.1.4.

​https://github.com/macports/macports-ports/pull/21603

The minimum OS version for 3.2.0 is now 10.14.

[thetrial (alabay)]

The minimum OS version for 3.2.0 is now 10.14.

Does that mean OS < 10.14 stay stuck or is there a future development or fix planned? And, in addition: On one of my systems 3.1.4 is uninstalled. Will it be reloaded or rebuilt?

[fhgwright (Fred Wright)]

Replying to thetrial:

The minimum OS version for 3.2.0 is now 10.14.

Does that mean OS < 10.14 stay stuck or is there a future development or fix planned?

That can change if and when 3.2 is fixed to work on older systems. And 3.1.4 isn't entirely "stuck" in the meantime, since 3.1 is still actively maintained upstream. 3.2 is a feature upgrade.

And, in addition: On one of my systems 3.1.4 is uninstalled. Will it be reloaded or rebuilt?

It will be reinstalled. The epoch bump ensures that the apparent downgrade in the version number is treated as an upgrade.

[fhgwright (Fred Wright)]

In dbf4acd621d2b7b73d980baa4559cd8b722a075e/macports-ports (master):

openssl3: Roll back broken v3.2.0 to v3.1.4 on OS <10.13

Version 3.2.0 has many issues on many older platforms.

Adds -devel version to facilitate testing 3.2.0 in such cases.

Although the epoch bump is only needed for the rollback cases, making
it conditional on the OS version would require that it remain so in
the future, so the unconditional epoch bump is preferable. The epoch
bump is the only change for macOS >= 10.14.

Revbumps the usual suspects.

Closes: #68763
Closes: #68766

TESTED:
Built, tested, and ran "openssl rand -hex 8" on 10.4-10.5 ppc,
10.4-10.6 i386, 10.5-10.15 x86_64, and 11.x-14.x arm64
The 3.1.4 version builds for all relevant platforms, and passes its
tests on all but 10.6-10.8.
The 3.2.0 -devel version builds for all but 10.4, failing its tests on
10.5-10.9 and 10.13 universal.
The 3.2.0 version fails its tests on 10.14 and 14.x.
The non-devel versions all pass "openssl rand".

[leon-matthews (Leon Matthews)]

Thank you all so very much for the fix! After this morning's update SSH is back up and running on my old MacBook Pro 17" with High Sierra. Much appreciated!

[thetrial (alabay)]

Yes, thank you very much also from my side.

On one machine I had the problem, that the opoenssl folder was emptied under libexec, so openssl could not be activated after the first attempt. I replayed a backup of that folder and tried to force activate, and then three openssl variants were presented and I could chose 3.1.4+universal (I guess those were the staged one). After that I just repeated a runthrough with MP, and it seems, everything is on its path again. I hope.

[fhgwright (Fred Wright)]

Replying to fhgwright:

Replying to thetrial:

And, in addition: On one of my systems 3.1.4 is uninstalled. Will it be reloaded or rebuilt?

It will be reinstalled. The epoch bump ensures that the apparent downgrade in the version number is treated as an upgrade.

However, out of paranoia I decided to test this, and found that upgrading the individual port failed to do this. See ​https://trac.macports.org/ticket/68820.

Replying to thetrial:

Yes, thank you very much also from my side.

On one machine I had the problem, that the opoenssl folder was emptied under libexec, so openssl could not be activated after the first attempt. I replayed a backup of that folder and tried to force activate, and then three openssl variants were presented and I could chose 3.1.4+universal (I guess those were the staged one). After that I just repeated a runthrough with MP, and it seems, everything is on its path again. I hope.

IIRC, you did some shuffling to try to restore proper operation, which may have put things in a state that port upgrade couldn't fix. In general, anything that gets the files in the tree inconsistent with the registry tends to confuse things, and AFAIK there are no automatic recovery tools for that sort of thing.

[neverpanic (Clemens Lang)]

So I'm about to commit the update to OpenSSL 3.2.1, and now I have to make a choice for the 3.1.x branch in the Portfile:

• Do I leave it at 3.1.4, potentially exposing users to known CVEs? (not the case this time since I did backport all CVE fixes that are in 3.1.5, but it'll be a relevant question for the next time)
• Do I update to 3.1.5, increasing my workload because I need to test two versions?
• Do I remove the if and update to 3.2.1, risking to break this again?

For this time, I'm going to put in the extra work to update to 3.1.5, but I'm not willing to do this forever for a few users that continue to run old operating systems.

Can one of you that has one of the affected machines please

• check whether this is still a problem with 3.2.1
• if it is, bisect the change that introduced the problem using the method I outlined in comment:41 so that we can look into getting this fixed upstream
• possibly narrow down the range of clang versions that miscompile this so we can blacklist them

I have no way of replicating the problem myself, or I would already have done these things.

[thetrial (alabay)]

I’m waiting for the update and later on will look for the things. This time I also should be able to reactivate the working ssl, if everything breaks. Hopefully.

[thetrial (alabay)]

Hm, at the moment it just pulled 3.1.5. How will I, later on, be able to test 3.2.1?

[neverpanic (Clemens Lang)]

Ideally by building it from source and running its testsuite as explained in comment:41:

Considering that there is something fundamentally broken you should be able to identify this quickly by doing these steps in a clone of ​​https://github.com/openssl/openssl.git:

• ./config
• make -j$(sysctl -n hw.ncpu)
• make test HARNESS_JOBS=$(sysctl -n hw.ncpu)

AFAIR there's also an openssl3-devel port on the affected systems that packages the latest version, so you could also try that.

[thetrial (alabay)]

If I’d build that from source – how can I remove that afterwards from my system?! I’m not that deep into it. A devel-port would have been nice but doesn’t exist.

[neverpanic (Clemens Lang)]

If you build it from source, it won't be installed anywhere, so there's nothing to remove. Just rm -rf the directory that contains the source code and it's gone.

I'm sure there are also other people on the Cc list of this ticket who have compiled software from source before and already know the rough steps how to do this. Maybe they want to help out?

A openssl3-devel port does exist according to the Portfile, but maybe the conditional way in which it is defined hides it from the PortIndex. In that case, cd "$(port dir openssl3)" && sudo port -v test -- subport=openssl3-devel should work.

[thetrial (alabay)]

Well, I downloaded the source (not cloned via git), built it and let the test run that you mentioned. Looots happened. And finally:

All tests successful.
Files=306, Tests=3596, 171 wallclock secs (18.05 usr  4.03 sys + 766.76 cusr 262.83 csys = 1051.67 CPU)
Result: PASS

But I’m not quite sure. Does this mean the +universal variant mentioned above would work? And, to be honest, I could not tell shot from the hip which compiler did that now by default :-)

I saved the terminal session, anonymized it and will attach it.

[thetrial (alabay)]

Building and testing openssl master 20240201.

[thetrial (alabay)]

BTW, when naively trying to start that ssl … it does not work because (obviously) the dylib is not under /usr

Rechnername user Downloads/openssl-master > /Users/user/Downloads/openssl-master/apps/openssl -V
dyld: Library not loaded: /usr/local/lib/libssl.3.dylib
  Referenced from: /Users/user/Downloads/openssl-master/apps/openssl
  Reason: image not found
[1]    26756 abort      /Users/user/Downloads/openssl-master/apps/openssl -V

[thetrial (alabay)]

Now I’m trying

cd "$(port dir openssl3)" && sudo port -v test -- subport=openssl3-devel

But, again, I’m not sure if that covers the +universal variant. Report will follow. Everything ran through, a second log will be attached (mp).

[neverpanic (Clemens Lang)]

If you want to test the apps/openssl, you can use util/wrap.pl apps/openssl. That should set up the required environment variables to make it use libssl.3.dylib from the build tree. Since openssl list -providers failed when this was broken, it would be helpful if you could try util/wrap.pl apps/openssl list -providers.

Your test does not cover the universal variant. I was under the impression that the problem was not specific to the universal variant. You can try the universal variant using sudo port clean openssl3 && cd "$(port dir openssl3)" && sudo port -v test -- +universal subport=openssl3.

[thetrial (alabay)]

I’ve done all checks with the MP openssl (not universal yet). Log is going to be attached. Now I build +universal and will repeat the tests.

[thetrial (alabay)]

Tests with MP openssl non-universal.

[thetrial (alabay)]

I just see … it builds the 3.1.5 as +universal … shouldn’t that have been subport=openssl3-devel? So I delete all and restart with

cd "$(port dir openssl3)" && sudo port -v test -- +universal subport=openssl3-devel

Hope that is the right way. Anyway … something is going on, seems this will build. Okay, now we have the failures. I’ll attach the log of the build. Plus the MP log.

[thetrial (alabay)]

MP logfile for build.

[thetrial (alabay)]

Try to build openssl +universal, fails.

[neverpanic (Clemens Lang)]

OK, so what we're seeing is that with +universal, openssl is configured twice, once for x86_64, and once for i686. Both build phases succeed. After that, there's a destroot phase, also run twice, which is supposed to merge the i686 and the x86_64 part of the binaries into one, and after that a test phase runs in the build directories for x86_64 and i686.

From your logs it is clear that the x86_64 test phase works fine if the i686 build doesn't happen. It also shows that the x86_64 test phase fails catastrophically if the i686 build happens in between. That's very unexpected.

I tried a +universal arm64 and x86_64 build, and could not reproduce that behavior. Both test phases pass just fine in that configuration.

As a consequence, my first advice would be: Don't build universal if you can avoid it. It costs you twice the time and twice the disk space, and you're really only running one of the slices anyway. If you can't avoid it, the next steps would be to debug what the differences between those build directories are before and after the destroot phase runs.

I did notice a couple of error messages in the log, which may or may not indicate and error. I'm not familiar enough with the muniversal PortGroup to say.

Command failed: /usr/bin/cmp -s "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-i386//opt/local/libexec/openssl3/bin/openssl" "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-x86_64//opt/local/libexec/openssl3/bin/openssl" && /bin/cp -v "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-i386//opt/local/libexec/openssl3/bin/openssl" "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-intel//opt/local/libexec/openssl3/bin" 
Exit code: 1
Command failed: /usr/bin/cmp -s "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-i386//opt/local/libexec/openssl3/include/openssl/configuration.h" "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-x86_64//opt/local/libexec/openssl3/include/openssl/configuration.h" && /bin/cp -v "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-i386//opt/local/libexec/openssl3/include/openssl/configuration.h" "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-intel//opt/local/libexec/openssl3/include/openssl" 
Exit code: 1
fatal error: /Library/Developer/CommandLineTools/usr/bin/lipo: can't figure out the architecture type of: /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-i386//opt/local/libexec/openssl3/include/openssl/configuration.h 
Command failed: /usr/bin/lipo -create "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-i386//opt/local/libexec/openssl3/include/openssl/configuration.h" "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-x86_64//opt/local/libexec/openssl3/include/openssl/configuration.h" -output "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-intel//opt/local/libexec/openssl3/include/openssl/configuration.h" 
Exit code: 1 
error: /Library/Developer/CommandLineTools/usr/bin/libtool: file: /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-i386//opt/local/libexec/openssl3/include/openssl/configuration.h is not an object file (not allowed in a library) 
error: /Library/Developer/CommandLineTools/usr/bin/libtool: file: /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-x86_64//opt/local/libexec/openssl3/include/openssl/configuration.h is not an object file (not allowed in a library) 
Command failed: /usr/bin/libtool "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-i386//opt/local/libexec/openssl3/include/openssl/configuration.h" "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-x86_64//opt/local/libexec/openssl3/include/openssl/configuration.h" -o "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-intel//opt/local/libexec/openssl3/include/openssl/configuration.h" 
Exit code: 1 
Command failed: test "`head -c2 /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-i386//opt/local/libexec/openssl3/include/openssl/configuration.h`" = '#!' 
Exit code: 1 

[…]

Command failed: /usr/bin/cmp -s "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-i386//opt/local/libexec/openssl3/lib/engines-3/capi.dylib" "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-x86_64//opt/local/libexec/openssl3/lib/engines-3/capi.dylib" && /bin/cp -v "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-i386//opt/local/libexec/openssl3/lib/engines-3/capi.dylib" "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-intel//opt/local/libexec/openssl3/lib/engines-3" 
Exit code: 1 
Command failed: /usr/bin/cmp -s "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-i386//opt/local/libexec/openssl3/lib/engines-3/loader_attic.dylib" "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-x86_64//opt/local/libexec/openssl3/lib/engines-3/loader_attic.dylib" && /bin/cp -v "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-i386//opt/local/libexec/openssl3/lib/engines-3/loader_attic.dylib" "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-intel//opt/local/libexec/openssl3/lib/engines-3" 
Exit code: 1 
Command failed: /usr/bin/cmp -s "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-i386//opt/local/libexec/openssl3/lib/engines-3/padlock.dylib" "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-x86_64//opt/local/libexec/openssl3/lib/engines-3/padlock.dylib" && /bin/cp -v "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-i386//opt/local/libexec/openssl3/lib/engines-3/padlock.dylib" "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-intel//opt/local/libexec/openssl3/lib/engines-3" 
Exit code: 1 
Command failed: /usr/bin/cmp -s "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-i386//opt/local/libexec/openssl3/lib/libcrypto.3.dylib" "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-x86_64//opt/local/libexec/openssl3/lib/libcrypto.3.dylib" && /bin/cp -v "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-i386//opt/local/libexec/openssl3/lib/libcrypto.3.dylib" "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-intel//opt/local/libexec/openssl3/lib" 
Exit code: 1 
Command failed: /usr/bin/cmp -s "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-i386//opt/local/libexec/openssl3/lib/libcrypto.a" "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-x86_64//opt/local/libexec/openssl3/lib/libcrypto.a" && /bin/cp -v "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-i386//opt/local/libexec/openssl3/lib/libcrypto.a" "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-intel//opt/local/libexec/openssl3/lib" 
Exit code: 1 
Command failed: /usr/bin/cmp -s "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-i386//opt/local/libexec/openssl3/lib/libssl.3.dylib" "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-x86_64//opt/local/libexec/openssl3/lib/libssl.3.dylib" && /bin/cp -v "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-i386//opt/local/libexec/openssl3/lib/libssl.3.dylib" "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-intel//opt/local/libexec/openssl3/lib" 
Exit code: 1 
Command failed: /usr/bin/cmp -s "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-i386//opt/local/libexec/openssl3/lib/libssl.a" "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-x86_64//opt/local/libexec/openssl3/lib/libssl.a" && /bin/cp -v "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-i386//opt/local/libexec/openssl3/lib/libssl.a" "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-intel//opt/local/libexec/openssl3/lib" 
Exit code: 1 
xinstall: mkdir /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-intel//opt/local/libexec/openssl3/lib/ossl-modules 
Command failed: /usr/bin/cmp -s "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-i386//opt/local/libexec/openssl3/lib/ossl-modules/legacy.dylib" "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-x86_64//opt/local/libexec/openssl3/lib/ossl-modules/legacy.dylib" && /bin/cp -v "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-i386//opt/local/libexec/openssl3/lib/ossl-modules/legacy.dylib" "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_devel_openssl3/openssl3-devel/work/destroot-intel//opt/local/libexec/openssl3/lib/ossl-modules" 
Exit code: 1 

Maybe those are normal due to how muniversal works, maybe they aren't. More debug output from that PortGroup would certainly be helpful to determine that.

[fhgwright (Fred Wright)]

I hadn't planned to comment until I'd had time to run all my own tests, which can probably happen this weekend, but:

Replying to neverpanic:

[...]

As a consequence, my first advice would be: Don't build universal if you can avoid it. It costs you twice the time and twice the disk space, and you're really only running one of the slices anyway. If you can't avoid it, the next steps would be to debug what the differences between those build directories are before and after the destroot phase runs.

Personally, I don't build +universal unless it's either for specifically testing +universal or because it's forced by dependents, if for no other reason than that the universal build is usually guaranteed not to be available as a precompiled binary. And if it uses muniversal, it typically takes a lot more than twice the build time, since there have to be two (or more) builds plus the expensive muniversal processing.

One of the biggest offenders is wine. Traditionally, wine could only run 32-bit Windows binaries when running 32-bit itself, which is probably why wine defaults to +universal. That inflicts the +universal requirement on its 279 recursive dependencies (including openssl3). Recent versions of wine have implemented an internal 32/64-bit bridge mechanism so that 64-bit wine can run 32-bit Windows binaries, but the wine port hasn't been updated in almost four years.

Another class of common offenders is toolchains such as gcc*, which can't manage to adequately separate target architectures from host architectures, so that they need to be built +universal themselves just to be able to build +universal targets. And all 13 versions ofgcc active here have openssl3 as a dependency.

I did notice a couple of error messages in the log, which may or may not indicate and error. I'm not familiar enough with the muniversal PortGroup to say.

[...]

I'm familiar enough with muniversal to know that it's an ugly and fragile kludge. :-)

[thetrial (alabay)]

Yes, wine is the killing gamechanger. I’m a conscious user of that and have this antique universal-dependency thatfor. This is the reason why I always keep an eye on these variants.

So, if I can help one can give me advices. I have the machine but am not so deepp into ist. Nevertheless I still manage to spot careless mistakes. For now I have removed all built stuff and wait what will come. Solving problems is exciting. Shall we reopen this case or rename it?

[fhgwright (Fred Wright)]

Replying to neverpanic:

If you build it from source, it won't be installed anywhere, so there's nothing to remove. Just rm -rf the directory that contains the source code and it's gone.

I'm sure there are also other people on the Cc list of this ticket who have compiled software from source before and already know the rough steps how to do this. Maybe they want to help out?

Although working in the source repo is often an easier way to track down bugs and come up with fixes, getting the right setup to do a MacPorts-style build in that context can take some work (and it's port-specific). It's also not the best way to do a "final test", anyway.

One can test various aspects of a port without actually installing or activating it, just by running in the destroot tree. There may be some gotchas, though (see below).

A openssl3-devel port does exist according to the Portfile, but maybe the conditional way in which it is defined hides it from the PortIndex. In that case, cd "$(port dir openssl3)" && sudo port -v test -- subport=openssl3-devel should work.

The conditional simply says that the -devel subport doesn't exist in the cases where it would be redundant. If it should exist for the current OS but doesn't appear in the index, that probably means that one is trying to use an index built for a different OS version, which is problematic in general and ought to be fixed. Once it is fixed, then the -devel subport should be usable without resorting to the index-bypass kludge.

Replying to neverpanic:

If you want to test the apps/openssl, you can use util/wrap.pl apps/openssl. That should set up the required environment variables to make it use libssl.3.dylib from the build tree. Since openssl list -providers failed when this was broken, it would be helpful if you could try util/wrap.pl apps/openssl list -providers.

In general, one can test in the destroot tree after a port destroot <port>, though sometimes additional setup beyond getting the right path may be needed. In the openssl3 case, there are two such issues:

1. It's necessary to use the libexec instances, since the normal instances are absolute symlinks to the installed locations, and hence useless for this purpose. I have no idea why they're absolute symlinks when relative symlinks would work better.

2. It's necessary to set DYLD_LIBRARY_PATH to point to the destroot/libexec lib dir, so that it uses the destrooted library rather than the installed one.

The setup in my generic test script is:

if [ "$PROG" == "" ]; then
  PROGCMD=""
else
  PROGCD="cd \$(port work $PORT)/destroot/$PREFIX"
  PROGCMD="$PROGCD && $PROG"
fi

With this, in many cases, all the port-specific test needs is:

PROG="bin/<command> <args>"

But for the openssl3 tests, it's:

PROG="\
   cd libexec/openssl3 \
   && DYLD_LIBRARY_PATH=./lib bin/openssl version \
   && DYLD_LIBRARY_PATH=./lib bin/openssl list -providers \
   && DYLD_LIBRARY_PATH=./lib bin/openssl rand -hex 8 \
   "

Your test does not cover the universal variant. I was under the impression that the problem was not specific to the universal variant. You can try the universal variant using sudo port clean openssl3 && cd "$(port dir openssl3)" && sudo port -v test -- +universal subport=openssl3.

I think you meant "specific to" rather than "not specific to".

That's not entirely the case. There are some -universal failures, but more +universal failures.

My test scripts use port destroot, followed by the three commands listed above, optionally followed by port test. For variants, I used "none", "exactly one", and "all" for the non-universal variants, then combined those with both +/-universal. Based on the evidence that muniversal doesn't play nicely with the test framework (at least on x86), I didn't bother with port test ... +universal on x86. Though there seems to be some flakiness in the test framework in general, which may be worse with many VMs running simultaneously.

I didn't bother making a modified Portfile to test 3.1.5 on 10.14+.

TL;DR: 3.1.5/3.2.1 is no better than 3.1.4/3.2.0.

The results are:

10.4 ppc:
3.1.5 -universal passes build/progs/tests
3.1.5 +universal fails build (broken zlib dependency)
3.2.1 build fails

10.5 ppc:
3.1.5 passes build/progs/tests
3.2.1 passes build/progs/tests

(10.4-10.5 ppc64 not available for testing)

10.4 i386:
3.1.5 passes build/progs/tests
3.2.1 build fails

(10.4 x86_64 not available for testing)

10.5 x86 (no test with +universal):
3.1.5 passes build/progs/tests
3.2.1 i386 passes build/progs/tests
3.2.1 x86_64 passes build/progs, tests hang

10.6 x86 (no test with +universal):
3.1.5 passes build/progs, fails tests
3.2.1 passes build/progs, fails tests

10.7-10.8 x86 (no test with +universal):
3.1.5 passes build/progs, fails tests
3.2.1 passes build, fails progs

10.9-10.13 x86 (no test with +universal):
3.1.5 passes build/progs/tests
3.2.1 -universal passes build/progs/tests
3.2.1 +universal passes build, fails progs

10.14 x86:
3.2.1 passes build/progs, fails tests

10.15-12.x x86:
3.2.1 passes build/progs/tests

(13.x-14.x x86 not available for testing)

11.x-12.x arm64:
3.2.1 passes build/progs/tests

13.x-14.x arm64:
3.2.1 passes build/progs, fails tests (tests flaky?)

[neverpanic (Clemens Lang)]

So where are we with this? I'm currently fixing yet another CVE (​https://github.com/macports/macports-ports/pull/23418), and because old OS versions are still stuck at 3.1.x, I have to do duplicate work because I need to identify and test the patch for 3.1.x as well. This is exactly the situation I wanted to avoid, because I'm forced to do additional work for older operating systems that are no longer supported by Apple.

Somebody needs to step up to replicate this problem with a git clone from the upstream sources, ideally in an automated fashion, so that it can be bisected to identify the exact commit that introduces it, reported upstream, and fixed. I cannot do this work because I don't have a machine that is affected by the problem.

If nobody steps up to do this work, I will eventually stop patching the openssl 3.1 for security issues, and eventually remove it, at the latest when it becomes unsupported on 2025-03-14 (see ​https://www.openssl.org/policies/releasestrat.html).

[fhgwright (Fred Wright)]

Umm, upstream published versions of that patch for all 3.x branches. There was also no rush to apply that patch at all, since upstream deemed it too low-severity to justify new releases. Since the effect is similar to a small memory leak, seeing an actual problem from it requires a server uptime which is humongous on 32-bit builds and astronomical on 64-bit builds.

Although not patching 3.1.5 at all would have been fine in this case, patching it without a revbump is inconsistent, so I just submitted PR 23478 to fix that.

Fixing the bug would certainly be desirable, though it's a nontrivial amount of work.

[RJVB (René Bertin)]

Replying to neverpanic:

So where are we with this?

I'm wondering too, in particular under which conditions OSSL 3.2.1 (and/or now 3.3.0) does NOT fail to build or work on "older" OS versions. The ticket has grown too long and I think a recap would be useful.

• does it actually work in non-universal builds if a recent enough compiler is used, and if so, on what OS versions?
• I see no mention of any testing with GCC (say 12 or 13); can using that compiler give a way out?

IMHO it would not be acceptable to remove the fallback for "legacy" systems given the 10.14 cut-off; that would leave a number of still perfectly usable generations of Apple hardware without functional OSSl3 at all. It would be far better to add a section to the OSSL3 Portfile that doesn't require significant maintenance, for instance one that turns the port into a dummy depending on something like port:openssl31 if the conditions are not united to guarantee a functional OSSL 3.2+ build. OpenSSL only depends on port:zlib (really?!) which itself evolves hardly, so the amount of maintenance for port:openssl31 should be negligible once upstream no longer provides updates, esp. if the port is kept specific to the Darwin versions it is intended for (and labels itself obsolete/replaced_by openssl3 on current versions).

Having an "old" but working OSSL install is always better than not having one, or having an even older one. Users of old hardware that can only run legacy OS versions are probably aware that they're exposing themselves to risks but it's not the responsibility of any port maintainer nor MacPorts as an institution to protect them from themselves.

Could it be that the failure in universal builds has something to do with e.g. a binary seed file containing data that is interpreted differently by 32 and 64 bit builds?

--- PS: adding a +tests variant speeds up the build significantly.

[RJVB (René Bertin)]

So ... OpenSSL 3.2.1 built with gcc12 on 10.9.5, non-universal x86_64 :

> DR=`port work openssl3`/destroot
> env DYDL_LIBRARY_PATH=${DR}/opt/local/libexec/openssl3/lib ${DR}/opt/local/libexec/openssl3/bin/openssl version
OpenSSL 3.2.1 30 Jan 2024 (Library: OpenSSL 3.2.1 30 Jan 2024)
> env DYDL_LIBRARY_PATH=${DR}/opt/local/libexec/openssl3/lib ${DR}/opt/local/libexec/openssl3/bin/openssl list -providers
Providers:
  default
    name: OpenSSL Default Provider
    version: 3.2.1
    status: active
  legacy
    name: OpenSSL Legacy Provider
    version: 3.1.5
    status: active
> env DYDL_LIBRARY_PATH=${DR}/opt/local/libexec/openssl3/lib ${DR}/opt/local/libexec/openssl3/bin/openssl list -random-generators
Provided RNGs and seed sources:
  CTR-DRBG @ default
  HASH-DRBG @ default
  HMAC-DRBG @ default
  SEED-SRC @ default
  TEST-RAND @ default

> /opt/local/libexec/openssl3/bin/openssl list -providers
Providers:
  default
    name: OpenSSL Default Provider
    version: 3.1.5
    status: active
  legacy
    name: OpenSSL Legacy Provider
    version: 3.1.5
    status: active
> /opt/local/libexec/openssl3/bin/openssl list -random-generators                                                  Provided RNGs and seed sources:
  CTR-DRBG @ default
  HASH-DRBG @ default
  HMAC-DRBG @ default
  SEED-SRC @ default
  TEST-RAND @ default

Two naggles using GCC:

• building with -g isn't currently possible (with GCC12) because of the assembly files: for those it will invoke $prefix/bin/as with --gstabs, which clang doesn't support.
• the Configure script still assumes that "GCC on Apple" means one of the pre-clang compilers that couldn't yet generate dependencies. As a result, makedepend will be used, increasing the build time immensely.

This patch fixes that:

> cat patch-avoid-makedepend.diff
diff --git orig.Configure Configure
index cca1ac8..a5fd98e 100755
--- orig.Configure
+++ Configure
@@ -1628,8 +1628,7 @@ if (!$disabled{makedepend}) {
         # If the makedepcmd attribute is defined, copy it.  If not, the
         # build files will have to fend for themselves.
         $config{makedepcmd} = $target{makedepcmd} if $target{makedepcmd};
-    } elsif (($predefined_C{__GNUC__} // -1) >= 3
-             && !($predefined_C{__APPLE_CC__} && !$predefined_C{__clang__})) {
+    } elsif (($predefined_C{__GNUC__} // -1) >= 3) {
         # We know that GNU C version 3 and up as well as all clang
         # versions support dependency generation, but Xcode did not
         # handle $cc -M before clang support (but claims __GNUC__ = 3)

EDIT:

> port -nok -v test `port-active-variants -echo openssl3` configure.optflags="-O3" configure.compiler=macports-gcc-12 configureccache=yes build.post_args=-wk
[...]
All tests successful.
Files=298, Tests=3590, 352 wallclock secs (31.55 usr  2.64 sys + 914.75 cusr 169.88 csys = 1118.82 CPU)
Result: PASS
952.372 user_cpu 174.236 kernel_cpu 6:00.05 total_time 312.9%CPU {153280512M 27267F 24894139R 319I 832O 11440k 89807w 887491c}

[neverpanic (Clemens Lang)]

I'm wondering too, in particular under which conditions OSSL 3.2.1 (and/or now 3.3.0) does NOT fail to build or work on "older" OS versions. The ticket has grown too long and I think a recap would be useful.

tl;dr: On old OS, when compiled +universal, openssl list -providers fails.

does it actually work in non-universal builds if a recent enough compiler is used, and if so, on what OS versions? I see no mention of any testing with GCC (say 12 or 13); can using that compiler give a way out?

I don't think non-universal builds were ever broken anywhere. The problem seems to mostly be +universal.

Could it be that the failure in universal builds has something to do with e.g. a binary seed file containing data that is interpreted differently by 32 and 64 bit builds?

Unlikely – that would cause either the x86_64 or the PPC slice to fail on its own, which doesn't seem to be happening.