Opened 8 years ago

Last modified 5 months ago

#51516 new enhancement

MacPorts should use a bundled copy of a newer libcurl and SSL library rather than the OS X version

Reported by: ryandesign (Ryan Carsten Schmidt) Owned by: macports-tickets@…
Priority: Normal Milestone: MacPorts Future
Component: base Version:
Keywords: Cc: udbraumann, mkae (Marko Käning), devernay (Frédéric Devernay), raimue (Rainer Müller), mojca (Mojca Miklavec), fhgwright (Fred Wright), dliessi (Davide Liessi), pkoshevoy (Pavel Koshevoy), bK4gYuRo, 1-61803, iEFdev, noloader (Jeffrey Walton), khepler, cooljeanius (Eric Gallager), MarcusCalhoun-Lopez (Marcus Calhoun-Lopez), iefdev (Eric F), evanmiller (Evan Miller), mascguy (Christopher Nielsen), catap (Kirill A. Korinsky), Dave-Allured (Dave Allured), jerryyhom, contextnerror, aeiouaeiouaeiouaeiouaeiouaeiou
Port:

Description

Much as we already use a bundled copy of a newer Tcl, MacPorts should use a bundled copy of a newer libcurl (and CA certificate bundle?) rather than the OS X version. On older systems, the OS X version of libcurl cannot connect to some https sites, including pypi (see #51509) and also https://distfiles.macports.org and https://packages.macports.org .

Change History (106)

comment:1 Changed 8 years ago by udbraumann

Cc: braumann@… added

Cc Me!

comment:2 Changed 7 years ago by ryandesign (Ryan Carsten Schmidt)

Has duplicate #52602.

comment:3 Changed 7 years ago by neverpanic (Clemens Lang)

I don't think MacPorts should be in the business of distributing a CA bundle. We shouldn't control who our users trust.

comment:4 Changed 7 years ago by raimue (Rainer Müller)

The issue is mostly not libcurl itself, but system's libcurl linked against system's OpenSSL 0.9.8, which does not support TLS >= 1.1. Therefore servers that require TLS >= 1.1 can no longer be reached (#51112, #52604). As of OS X 10.9 Mavericks, system's libcurl will use Secure Transport instead. Secure Transport supports TLS >= 1.1 as of OS X 10.8 Mountain Lion.

If we still wanted to support OS X <= 10.7, we would need both libcurl and a compatible SSL/TLS library in base.

comment:5 Changed 7 years ago by ryandesign (Ryan Carsten Schmidt)

Has duplicate #52604.

comment:6 in reply to:  3 Changed 7 years ago by ryandesign (Ryan Carsten Schmidt)

Replying to cal@…:

I don't think MacPorts should be in the business of distributing a CA bundle. We shouldn't control who our users trust.

I don't care how we arrange it, but MacPorts should at least be able to download from the MacPorts packages and distfiles servers via SSL. If the problem isn't trusting the certificates but rather the capabilities of the SSL library, then we need to bundle a newer SSL library.

comment:7 Changed 7 years ago by neverpanic (Clemens Lang)

Keep in mind that bundling an SSL library implies that we will release at least as often as this SSL library to fix any security problems with it. Personally, I'm not prepared to do that, and we should just fall back to non-SSL connections on systems that do not support modern SSL.

I doubt that we have the manpower to rescue platforms that Apple has abandoned. At some point we'll just have to admit that the system is no longer usable due to issues such as these.

comment:8 Changed 7 years ago by dbevans (David B. Evans)

I agree that bundling an SSL library could lead to a lot of extra cycles to update the bundle for security issues, etc. Also probably licensing issues with a binary bundle that includes OpenSSL or LibreSSL. Not sure about other SSL libraries.

To avoid these issues, could we set it up so a source build could be built with an external library other than the system ones? An interested user could then boot strap from an outdated binary Installer like this:

  • install old port binary using installer
  • build the SSL library of your choice using the just installed port binary
  • do a source build using the installed non-system SSL library using the appropriate configuration parameters ( say --with-openssl=${prefix}/lib )

This avoids binary licensing issues and updates for OpenSSL and the like would be handled by the normal port maintenance process.

The procedure would need to be well documented to avoid too many "I tried it but it didn't work" tickets.

Or is this too complicated for the average user to deal with? Just an idea.

comment:9 in reply to:  8 Changed 7 years ago by larryv (Lawrence Velázquez)

That sounds a bit complicated.

What are the downsides of downgrading to insecure connections on old platforms? Would we have to worry about tampering? We already verify checksums.

comment:10 Changed 7 years ago by ryandesign (Ryan Carsten Schmidt)

We already downgrade Leopard and earlier to http for packages.macports and distfiles.macports. r150758, r150765. Once we are properly mirroring distfiles before doing buildbot builds, that will probably be a sufficient fix. But it doesn't help old systems download from other sites that require https.

comment:11 in reply to:  10 ; Changed 7 years ago by neverpanic (Clemens Lang)

The problem with linking against a copy of OpenSSL in $prefix is that it will temporarily disappear while OpenSSL itself is upgraded, but that must not interrupt the upgrade process.

Replying to ryandesign@…:

Once we are properly mirroring distfiles before doing buildbot builds, that will probably be a sufficient fix. But it doesn't help old systems download from other sites that require https.

No matter what we do, that's a problem these platforms will always have, and will continue to do so. As I said, we cannot save abandoned platforms from dying.

comment:12 in reply to:  11 Changed 7 years ago by dbevans (David B. Evans)

Replying to cal@…:

The problem with linking against a copy of OpenSSL in $prefix is that it will temporarily disappear while OpenSSL itself is upgraded, but that must not interrupt the upgrade process.

Good point.

Replying to ryandesign@…:

Once we are properly mirroring distfiles before doing buildbot builds, that will probably be a sufficient fix. But it doesn't help old systems download from other sites that require https.

No matter what we do, that's a problem these platforms will always have, and will continue to do so. As I said, we cannot save abandoned platforms from dying.

Agreed. However, we can, perhaps, provide tools for those who are interested in using these platforms regardless of their antiquity. I'm not sure how many such people there are but they're quite enthusiastic and it would be a shame to shun them, altogether, from the MacPorts community. Best world, IMO, would be for them to form an ad hoc interest group within MacPorts so they could cooperatively work on these issues themselves.

So I'm suggesting that while we shouldn't commit (and haven't) to supporting officially anything older than 10.9, it wouldn't hurt to add a hook once in a while to allow them to do the work for themselves if they choose to do so. In this case, provide a configuration option for base to optionally build with external libraries rather than system ones to solve the SSL/TLS issues inherent in the old system libraries.

By the way, if one wanted to bundle something that is light weight with good protocol coverage and permissive licensing, wolfSSL/CyaSSL looks interesting. curl has supported it upstream as an alternative so openssl/gnutls for a while now.

comment:13 in reply to:  11 ; Changed 7 years ago by ryandesign (Ryan Carsten Schmidt)

Replying to cal@…:

Replying to ryandesign@…:

Once we are properly mirroring distfiles before doing buildbot builds, that will probably be a sufficient fix. But it doesn't help old systems download from other sites that require https.

No matter what we do, that's a problem these platforms will always have, and will continue to do so. As I said, we cannot save abandoned platforms from dying.

I don't understand why you would say that. The problem right now is caused by older systems have an old SSL implementation. The proposed fix is to provide a newer SSL implementation in MacPorts. That fixes the problem.

I agree that if we do this, we should release a new version of MacPorts when a new version of the SSL library is available, in case the new version of the SSL library fixes security vulnerabilities. I agree that our record to date of releasing new versions of MacPorts in a timely fashion is not good, which would be a reason not to make this change right now. However, I hope that once we move to GitHub, we can improve our processes and procedures to make it much easier to release new versions of MacPorts.

comment:14 Changed 7 years ago by ken-cunningham-webuse

Best world, IMO, would be for them to form an ad hoc interest group within MacPorts so they could cooperatively work on these issues themselves.

That might be a good idea. Even for example a separate trac stream for <10.9 issues would keep the main stream uncluttered, give a voice to those trying to keep these machines going, but keep the signal-to-noise ratio up for the main cadre of users. People who wanted to help out with older systems could.

Or maybe a separate stream would be the final death bell for these systems...

Last edited 7 years ago by ken-cunningham-webuse (previous) (diff)

comment:15 in reply to:  10 Changed 7 years ago by larryv (Lawrence Velázquez)

Replying to ryandesign@…:

Once we are properly mirroring distfiles before doing buildbot builds, that will probably be a sufficient fix. But it doesn't help old systems download from other sites that require https.

Other than the brief period before we mirror a new distfile, when would said systems not be able to fall back to the mirrors?

comment:16 in reply to:  13 Changed 7 years ago by neverpanic (Clemens Lang)

Replying to ryandesign@…:

I don't understand why you would say that. The problem right now is caused by older systems have an old SSL implementation. The proposed fix is to provide a newer SSL implementation in MacPorts. That fixes the problem.

Yes, that fixes the problem in MacPorts, but users on these platforms will continue to experience the same problems with other services that use TLS. So we can't fix the problem completely, but would still bundle an SSL library on platforms that already ship one that's perfectly fine.

I agree with devans here, we should provide instructions, warn users about the caveats and have them compile MacPorts against MacPorts curl.

comment:17 Changed 7 years ago by mkae (Marko Käning)

Cc: mk@… added

Cc Me!

comment:18 Changed 7 years ago by devernay (Frédéric Devernay)

Cc: frederic.devernay@… added

Cc Me!

comment:19 Changed 7 years ago by ken-cunningham-webuse

I gave this a shot today. Have curl installed in /opt/local

$ port -v installed | grep curl
  curl @7.50.3_0+ssl+universal (active) platform='darwin 10' archs='i386 x86_64'
  curl-ca-bundle @7.50.3_0 (active) platform='darwin 10' archs='noarch'

configured and installed macports from source, upgrading an existing installation

./configure --with-curlprefix=/opt/local
make
sudo make install

seems to work perfectly. The recent problem downloads, like hexchat and git, download without trouble now.

That's a 4-minute fix to an irritating problem.... unless you accidentally uninstall curl, or course...

Last edited 7 years ago by ken-cunningham-webuse (previous) (diff)

comment:20 Changed 7 years ago by ken-cunningham-webuse

although this setup as described is working well, I worried it was possibly too precarious to rely on itself like that in case of any unforeseen issues that might arise in future.

I decided to install a new copy of macports into /opt/bootstrap, installed curl in that installation, and then referenced the curl in /opt/bootstrap when reinstalling macports back into /opt/local.

comment:21 Changed 7 years ago by RJVB (René Bertin)

I think there might be another solution, one that doesn't require a hard dependency on curl.

If the curl function from Pextlib fails with an error that's likely to signal a TLS problem (and unlikely to have another cause), try again using the curl command (or wget). If that fails too and the command isn't found in ${prefix}/bin, instruct the user to install port:curl .

That'd be a 1-time investment to upgrade the Tcl function calling Pextlib::curl, but no cyclic or otherwise problematic dependencies will be introduced, and users don't have to follow more complex instructions than necessary.

comment:22 Changed 7 years ago by neverpanic (Clemens Lang)

Please, let's not have more 'if this doesn't work let's fall back to this other lesser-tested special case'.

Have you even looked at all the options supported in pextlib1.0/curl.c? Re-creating those for a command-line client is complicated, and in some instances, impossible.

comment:23 Changed 7 years ago by ryandesign (Ryan Carsten Schmidt)

The current discussion is centered around being able to reach servers running newer ssl. But the idea to use a bundled curl is much older, and is based on the idea that we should use the latest software, because it has more features and fixes bugs. That's the reason why we bundle tcl 8.5 now; we were able to remove workarounds for tcl 8.4 and start using features introduced in tcl 8.5. In the same way, the latest curl might offer some useful features not present in Tiger's curl, and there are workarounds in base for bugs in the version of curl included with Snow Leopard that we could remove if we bundled a newer version.

The problem of needing to also bundle a newer ssl library to support newer ssl sites, and to quickly release new MacPorts base versions to update the ssl library should any vulnerabilities be found, makes this idea less wonderful. On the one hand, having a recent but not completely up to date ssl library would be better for users of old OS versions that don't get security updates anymore; but on the other hand, it would be worse for users of new OS versions that do.

Maybe once we are on GitHub we will improve our testing and release process to the point where releasing a new version of MacPorts can be done very easily. If so, maybe then we can revisit this issue.

comment:24 Changed 7 years ago by raimue (Rainer Müller)

Cc: raimue added

comment:25 Changed 7 years ago by mojca (Mojca Miklavec)

Cc: mojca added

comment:26 Changed 7 years ago by fhgwright (Fred Wright)

Cc: fhgwright added

comment:27 Changed 7 years ago by dliessi (Davide Liessi)

Cc: dliessi added

comment:28 Changed 7 years ago by raimue (Rainer Müller)

In 7a428704/macports-ports:

git: add alternative download URL to master_sites

The main server at www.kernel.org has a hard requirement for TLS 1.2,
which is not available on older versions of macOS. This CDN mirror is
less restrictive and can be accessed using TLS 1.0.

Closes: #52602
Closes: #52604
Closes: #52836
See: #51516

comment:29 Changed 7 years ago by raimue (Rainer Müller)

Cc: udbraumann, , mkae, devernay, raimue, mojca, fhgwright, dliessiudbraumann, mkae, devernay, raimue, mojca, fhgwright, dliessi
Summary: MacPorts should use a bundled copy of a newer libcurl rather than the OS X versionMacPorts should use a bundled copy of a newer libcurl and SSL library rather than the OS X version

Has duplicate #46539.

comment:30 in reply to:  23 Changed 7 years ago by ryandesign (Ryan Carsten Schmidt)

Replying to ryandesign:

the idea to use a bundled curl is much older, and is based on the idea that we should use the latest software, because it has more features and fixes bugs.

#51045 is an example of such a curl bug.

comment:31 Changed 7 years ago by mf2k (Frank Schima)

Cc: pkoshevoy added

Has duplicate #53181.

comment:32 Changed 7 years ago by raimue (Rainer Müller)

Fetching Python distfiles from files.pythonhosted.org (as used by ports via master_sites pypi:) will require TLS 1.2 as of 2017-04-30. Support for TLS 1.0 and 1.1 will be disabled. https://pyfound.blogspot.de/2017/01/time-to-upgrade-your-python-tls-v12.html

TLS 1.2 is supported in Secure Transport as of OS X 10.8, which means fetching python ports will stop working on OS X <= 10.7 on this date.

comment:33 Changed 7 years ago by bK4gYuRo

Cc: bK4gYuRo added

comment:34 Changed 7 years ago by jduerstock (Jason Duerstock)

Since Macports locally stores the hashes of the source files anyway, can we ignore any HTTPS failures at least with respect to files downloaded with known hashes?

comment:35 in reply to:  34 Changed 7 years ago by neverpanic (Clemens Lang)

Replying to jduerstock:

Since Macports locally stores the hashes of the source files anyway, can we ignore any HTTPS failures at least with respect to files downloaded with known hashes?

Unfortunately we can't. The issue is not that the certificate presented by the servers would not verify (which we can ignore), but that all protocols implemented in older versions of libcurl (and the cryptography library it uses) shipped with older versions of macOS are no longer accepted by various servers. This means that all attempts to connect just fail. A workaround would be using an unencrypted channel to download the files, but a lot of servers nowadays redirect to SSL-only protocols.

comment:36 Changed 7 years ago by raimue (Rainer Müller)

We are already downgrading connections to our own mirrors to HTTP without SSL/TLS on older systems (see archive_sites.tcl). This would already help with many ports on these legacy systems, but our distfiles mirroring has not been brought back yet after we left macOS Forge.

comment:37 Changed 7 years ago by kencu (Ken)

Just FYI, the bootstrap method above has worked nicely on 10.4 up since I started using it last year. Haven't run into a fetch error since then on any system. I update the bootstrap install of Macports every few months, otherwise ignore it.

Last edited 6 years ago by kencu (Ken) (previous) (diff)

comment:38 Changed 6 years ago by 1-61803

Cc: 1-61803 added

comment:39 Changed 6 years ago by kencu (Ken)

One thing to note - when updating MacPorts versions, you will likely have to build it again from source rather than trust the sudo port selfupdate mechanism to do it properly for you. sudo port selfupdate does not presently appear to preserve configure settings like --with-curlprefix.

Last edited 2 years ago by kencu (Ken) (previous) (diff)

comment:40 Changed 6 years ago by yan12125 (Chih-Hsuan Yen)

Cc: yan12125 added

comment:41 Changed 6 years ago by iEFdev

Cc: iEFdev added

comment:42 in reply to:  23 Changed 6 years ago by ryandesign (Ryan Carsten Schmidt)

Replying to ryandesign:

But the idea to use a bundled curl is much older, and is based on the idea that we should use the latest software, because it has more features and fixes bugs.

Could we perhaps tackle this part -- bundle and link with a recent version of libcurl -- and postpone dealing with ssl issues? On older systems, we can link our bundled libcurl with the system's openssl library; on the newer systems that support it, we can use the --with-darwinssl configure flag to tell libcurl to use the macOS Secure Transport API.

comment:43 Changed 6 years ago by noloader (Jeffrey Walton)

Regarding this:

Much as we already use a bundled copy of a newer Tcl, MacPorts should use a bundled copy of a newer libcurl (and CA certificate bundle?) rather than the OS X version.

I copied cURL's cacert.pem to /System/Library/OpenSSL/certs/cacert.pem, which was previously empty. It did not resolve the issue.

It looks like its the old/downlevel version of OpenSSL. OpenSSL 0.9.8 lacks a lot of goodies, like elliptic curve support and TLS 1.2 support. It is certainly not following contemporary best practices.

Thanks for Ryan's help on a related ticket.

comment:44 Changed 6 years ago by ryandesign (Ryan Carsten Schmidt)

Cc: noloader added

Yes, I know that merely using a bundled curl that's still linked with the system's openssl will not solve all of the problems being discussed in this ticket. But I think we should still do it because it will solve some of the problems (e.g. comment:30).

comment:45 Changed 6 years ago by khepler

Cc: khepler added

comment:46 in reply to:  44 ; Changed 6 years ago by neverpanic (Clemens Lang)

Replying to ryandesign:

Yes, I know that merely using a bundled curl that's still linked with the system's openssl will not solve all of the problems being discussed in this ticket. But I think we should still do it because it will solve some of the problems (e.g. comment:30).

I think we all agree that we should not bundle an SSL library with MacPorts.

Now that the mirroring is fixed, I'm wondering whether we should just close this ticket as wontfix. You mentioned that you want a bundled libcurl in MacPorts for other reasons, but the cosmetic issue you linked in #51045 does in my opinion not warrant the additional effort of bundling libcurl (especially since it's fixed since two OS releases and we could add a simple workaround in base for that).

Do you have any other reasons for bundling a libcurl? If not, I would close this ticket as wontfix.

comment:47 in reply to:  46 ; Changed 6 years ago by ryandesign (Ryan Carsten Schmidt)

Replying to neverpanic:

Replying to ryandesign:

Yes, I know that merely using a bundled curl that's still linked with the system's openssl will not solve all of the problems being discussed in this ticket. But I think we should still do it because it will solve some of the problems (e.g. comment:30).

I think we all agree that we should not bundle an SSL library with MacPorts.

I do not agree yet. Older OSes need a newer ssl library to fetch from some https sites. So can't we provide a newer openssl or libressl on older systems, and use Apple's Secure Transport on newer ones? Yes there might be security vulnerabilities discovered in the ssl library we bundle, but isn't it likely that there are fewer vulnerabilities in it than in the old openssl version used on those systems?

Your objection in comment:3 was that we should not distribute a CA bundle. Would bundling an ssl library require us to include CA bundle? Doesn't the OS ship with one that we could use? Or is that part of the problem—the old OS's CA bundle doesn't contain the information needed to trust new sites?

Now that the mirroring is fixed, I'm wondering whether we should just close this ticket as wontfix. You mentioned that you want a bundled libcurl in MacPorts for other reasons, but the cosmetic issue you linked in #51045 does in my opinion not warrant the additional effort of bundling libcurl (especially since it's fixed since two OS releases and we could add a simple workaround in base for that).

Do you have any other reasons for bundling a libcurl? If not, I would close this ticket as wontfix.

#51045 was merely the first example of a curl bug that's fixed in newer versions that I found to link to this ticket. There's another cosmetic issue I found on Leopard that's been fixed. I'm sure there have been more bugs fixed in curl over the years.

The main difficulty or effort of bundling libcurl was originally that we didn't bundle any other software with MacPorts, so a method of doing that had to be invented. Now that we have created that method and already use it to bundle 4 libraries, it shouldn't be that difficult to bundle a few more libraries for modern ssl support or xz decompression (#52000).

comment:48 Changed 6 years ago by yan12125 (Chih-Hsuan Yen)

FWIW, I have a branch that bundles LibreSSL + libcurl with macports-base at https://github.com/yan12125/macports-base/tree/bundle-curl.

I choose LibreSSL just because it's based on autotools, while OpenSSL has a custom Perl-based configure system that once brought some headache to me. It should not be difficult to switch LibreSSL back to OpenSSL for macports-base.

comment:49 in reply to:  47 ; Changed 6 years ago by noloader (Jeffrey Walton)

Replying to ryandesign:

Replying to neverpanic:

Replying to ryandesign:

Yes, I know that merely using a bundled curl that's still linked with the system's openssl will not solve all of the problems being discussed in this ticket. But I think we should still do it because it will solve some of the problems (e.g. comment:30).

I think we all agree that we should not bundle an SSL library with MacPorts.

I do not agree yet. Older OSes need a newer ssl library to fetch from some https sites. So can't we provide a newer openssl or libressl on older systems, and use Apple's Secure Transport on newer ones? Yes there might be security vulnerabilities discovered in the ssl library we bundle, but isn't it likely that there are fewer vulnerabilities in it than in the old openssl version used on those systems?

Your objection in comment:3 was that we should not distribute a CA bundle. Would bundling an ssl library require us to include CA bundle? Doesn't the OS ship with one that we could use? Or is that part of the problem—the old OS's CA bundle doesn't contain the information needed to trust new sites?

Newer libraries with vulnerabilities seems the lesser of the two evils. Early 2000's tools with early CA lists don't help with security much. People will just disable the certificate checks, which puts them in a worse place.

And newer libraries that "just work" seems like a worthy usability and security goals. People won't have to do things like wget --no-check-certificate and curl --insecure.

Also, those old libraries and programs built on them are going to cause more trouble in the future because they can't do TLS 1.2. A TLS 1.2-only configuration is just common place nowadays. For example, GitHub recently made the engineering change: https://githubengineering.com/crypto-removal-notice/ .

After the GitHub change things like wget --no-check-certificate --tlsv1 and curl --insecure --tlsv1 simply will not work. It is technically impossible for some sites because of the site's configuration.

comment:50 in reply to:  49 Changed 6 years ago by noloader (Jeffrey Walton)

Replying to noloader:

...

Yes, I know that merely using a bundled curl that's still linked with the system's openssl will not solve all of the problems being discussed in this ticket. But I think we should still do it because it will solve some of the problems (e.g. comment:30).

I think we all agree that we should not bundle an SSL library with MacPorts.

I do not agree yet. Older OSes need a newer ssl library to fetch from some https sites. So can't we provide a newer openssl or libressl on older systems, and use Apple's Secure Transport on newer ones? Yes there might be security vulnerabilities discovered in the ssl library we bundle, but isn't it likely that there are fewer vulnerabilities in it than in the old openssl version used on those systems?

Your objection in comment:3 was that we should not distribute a CA bundle. Would bundling an ssl library require us to include CA bundle? Doesn't the OS ship with one that we could use? Or is that part of the problem—the old OS's CA bundle doesn't contain the information needed to trust new sites?

Newer libraries with vulnerabilities seems the lesser of the two evils. Early 2000's tools with early CA lists don't help with security much. People will just disable the certificate checks, which puts them in a worse place.

And newer libraries that "just work" seems like a worthy usability and security goals. People won't have to do things like wget --no-check-certificate and curl --insecure.

Also, those old libraries and programs built on them are going to cause more trouble in the future because they can't do TLS 1.2. A TLS 1.2-only configuration is just common place nowadays. For example, GitHub recently made the engineering change: https://githubengineering.com/crypto-removal-notice/ .

After the GitHub change things like wget --no-check-certificate --tlsv1 and curl --insecure --tlsv1 simply will not work. It is technically impossible for some sites because of the site's configuration.

This is worth mentioning... TLS 1.3 is in Last Call (LC) status on the TLS WG mailing list. I think the future trend will be sites supporting TLS 1.2 and TLS 1.3 as sites are reconfigured for the new standard. Also see Last Call: <draft-ietf-tls-tls13-24.txt> (The Transport Layer Security (TLS) Protocol Version 1.3) to Proposed Standard.

Without getting too bogged down in what version of cacert.pem to use, maybe the better engineering question would be, how can MacPorts ensure TLS 1.2 and TLS 1.3 "just work" for users. If you answer that question I believe the CA list question will follow without much effort.

comment:51 in reply to:  48 Changed 6 years ago by kencu (Ken)

Replying to yan12125:

FWIW, I have a branch that bundles LibreSSL + libcurl with macports-base at https://github.com/yan12125/macports-base/tree/bundle-curl.

Well that looks pretty fabulous! All that is missing is the CA certificates I guess. Well done.

Last edited 6 years ago by kencu (Ken) (previous) (diff)

comment:52 in reply to:  48 Changed 6 years ago by ryandesign (Ryan Carsten Schmidt)

Replying to yan12125:

FWIW, I have a branch that bundles LibreSSL + libcurl with macports-base at https://github.com/yan12125/macports-base/tree/bundle-curl.

I choose LibreSSL just because it's based on autotools, while OpenSSL has a custom Perl-based configure system that once brought some headache to me. It should not be difficult to switch LibreSSL back to OpenSSL for macports-base.

Yes thanks for this. There has been some spirited debate about the merits of openssl vs libressl, including pointing out that as of High Sierra, Apple is distributing libressl in macOS instead of openssl. I would be fine with using libressl in MacPorts instead of openssl.

I see that your branch deletes the MacPorts --with-curlprefix configure option. We probably want to keep that option for users who want to override it, but default it to using the bundled copy.

And as I mentioned above, on systems new enough to have a Secure Transport implementation that supports TLS 1.2, I'd like to use that, and only build and use the bundled libressl on older systems. /usr/bin/curl didn't start using Secure Transport until Mavericks, but it has been around a lot longer than that. I'd have to do some testing to figure out how far back it supports TLS 1.2. Ideally we would do a configure script test for Secure Transport's capabilities. The only check curl's own configure script does is for the existence of /System/Library/Frameworks/Security.framework, but that has existed since Mac OS X v10.0 so we'd need a more specific check than that.

comment:53 in reply to:  49 ; Changed 6 years ago by neverpanic (Clemens Lang)

Replying to ryandesign:

I do not agree yet. Older OSes need a newer ssl library to fetch from some https sites. So can't we provide a newer openssl or libressl on older systems, and use Apple's Secure Transport on newer ones? Yes there might be security vulnerabilities discovered in the ssl library we bundle, but isn't it likely that there are fewer vulnerabilities in it than in the old openssl version used on those systems?

I do not want to ship a library with potentially known and dangerous security vulnerabilities to users. If we use Apple's library and it does not work, the blame is on Apple, and users on old OSes cannot expect support from Apple. I do not want to create the impression that we support this setup.

Yes, there might be fewer problems, but if there are problems they would then be *our* problems. At the moment, they are not.

Your objection in comment:3 was that we should not distribute a CA bundle. Would bundling an ssl library require us to include CA bundle? Doesn't the OS ship with one that we could use? Or is that part of the problem—the old OS's CA bundle doesn't contain the information needed to trust new sites?

The latter, the CA bundle would eventually be too old for new certificates to validate.

#51045 was merely the first example of a curl bug that's fixed in newer versions that I found to link to this ticket. There's another cosmetic issue I found on Leopard that's been fixed. I'm sure there have been more bugs fixed in curl over the years.

Do you have specific tickets? I'm not convinced that we should bundle every library we use.

Replying to yan12125:

FWIW, I have a branch that bundles LibreSSL + libcurl with macports-base at https://github.com/yan12125/macports-base/tree/bundle-curl.

Thank you, that's helpful for users that want to run this at their own risk.

Replying to noloader:

Newer libraries with vulnerabilities seems the lesser of the two evils. Early 2000's tools with early CA lists don't help with security much. People will just disable the certificate checks, which puts them in a worse place.

Early 2000's tools are pretty much not going to give you any reasonable assumption of security at this point. I'd rather want users to realize that then to give them a false sense of security.

And newer libraries that "just work" seems like a worthy usability and security goals. People won't have to do things like wget --no-check-certificate and curl --insecure.

Also, those old libraries and programs built on them are going to cause more trouble in the future because they can't do TLS 1.2. A TLS 1.2-only configuration is just common place nowadays. For example, GitHub recently made the engineering change: https://githubengineering.com/crypto-removal-notice/ .

This is not the point of this ticket. Software installed by MacPorts will always use modern libraries and support modern crypto. This is about the libraries used by MacPorts itself. So either, your curl is MacPorts curl, which will work, or it is Apple's curl, which we will also not fix with this.

comment:54 Changed 4 years ago by yan12125 (Chih-Hsuan Yen)

Cc: yan12125 removed

comment:55 Changed 4 years ago by cooljeanius (Eric Gallager)

Cc: cooljeanius added

comment:56 Changed 4 years ago by MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)

Cc: MarcusCalhoun-Lopez added

comment:57 in reply to:  53 ; Changed 4 years ago by ryandesign (Ryan Carsten Schmidt)

Let me try one more time to explain my position. I still believe that we should bundle libcurl with MacPorts (either on all systems or only on older systems that don't get updates from Apple anymore), and that we should bundle a newer libressl or openssl with MacPorts (either only on older systems that don't get updates from Apple anymore or only on older systems that don't support newer TLS algorithms that many servers now require). On newer systems that still get updates from Apple we can either use our bundled curl and link it with Apple's security framework, or we can continue to use the system's libcurl as we do now.

To be concrete about it, at this time, I might propose that we use a bundled newer libcurl/libressl/openssl on OS X 10.11 El Capitan and earlier. I believe El Capitan is no longer receiving security updates as of the end of 2018. The problem I mentioned in #51045 was present in El Capitan's curl but not Sierra's, and also René recently encountered a KDE download server that could not be accessed from El Capitan's curl or earlier but worked with Sierra's. On Sierra and later, we could either use our bundled libcurl linked with the security framework or else use macOS libcurl.

Replying to neverpanic:

I do not want to ship a library with potentially known and dangerous security vulnerabilities to users.

To what library are you referring? libcurl? openssl? libressl? What are the specific known and dangerous security vulnerabilities that you're referring to?

I'm not certain what types of security vulnerabilities do or could exist in those libraries but even if there were any I'm not sure how serious that would be for us. We would only be using this libcurl for MacPorts tasks like downloading known files. We checksum our distfiles and sign our packages, so any vulnerability that caused curl to deliver data other than that which it should would be noticed by MacPorts immediately and the bad data would not be used. And we already don't consider the information we send or receive with curl to be confidential—we don't transmit private information and many of our mirrors already don't use https. As I have said before, it seems likely to me that bundling a newer libcurl/libressl/openssl will fix more problems than it causes.

Users often don't care or even know why something doesn't work, they just know it doesn't work and they blame the most obvious source. MacPorts is broken on 10.8 and earlier when downloading from many servers, and many users have reported this to us. There are some edge cases where our buildbot does not mirror distfiles immediately or at all, and sometimes it's just busy with other tasks or even offline; during those times, with no mirror to fall back on, users can encounter fetch errors. We've even had the problem on our own buildbot workers for older OS versions. We've identified the cause: outdated OS libcurl/openssl libraries and CA bundle. We have a fix: bundle newer ones. This would fix the bug, increase user happiness because things will just work for them, and free up the time MacPorts developers would otherwise spend responding to user bug reports about this problem.

Your objection in comment:3 was that we should not distribute a CA bundle. Would bundling an ssl library require us to include CA bundle? Doesn't the OS ship with one that we could use? Or is that part of the problem—the old OS's CA bundle doesn't contain the information needed to trust new sites?

The latter, the CA bundle would eventually be too old for new certificates to validate.

Ok, then we could bundle a new CA that MacPorts libcurl/libressl/openssl would use, again possibly only for older systems that won't get such an update from Apple. We could even think about a way to update it via selfupdate that would be separate from MacPorts base releases, if making a user wait for a new MacPorts base release might be undesirable. Or we could find a way to have MacPorts base optionally use the CA from the curl-ca-bundle port.

#51045 was merely the first example of a curl bug that's fixed in newer versions that I found to link to this ticket. There's another cosmetic issue I found on Leopard that's been fixed. I'm sure there have been more bugs fixed in curl over the years.

Do you have specific tickets?

As I recall, curl on Leopard would emit a spurious error message during some downloads. As far as I remember this did not affect the ability to download the files. I'm not able to find a ticket or other reference to it as I don't remember exactly what the error message was so I can't search for it now.

Obviously this bug and the previously mentioned one are trivial but they are a reminder that there have probably been many bugs fixed in curl since the versions bundled with old macOS versions which our users would benefit from if we bundled a newer version. The benefit to us is being able to remove code from MacPorts base that works around those old bugs, thus simplifying our code and making it easier to maintain. (I currently see 3 different workarounds for bugs or missing features in various versions of curl in our curl.c.) We were able to do the same when we bundled Tcl 8.5 and could remove the Tcl 8.4 workarounds we had, as I mentioned above years ago.

I'm not convinced that we should bundle every library we use.

I'm not trying to propose that. This ticket is only about bundling curl and libressl/openssl with MacPorts. I believe this will benefit us and our users and I am not convinced that the drawbacks outweigh that.

comment:58 Changed 4 years ago by ryandesign (Ryan Carsten Schmidt)

Even just bundling a newer curl on 10.8 and earlier would help. I'm discovering that the curl that comes with 10.8 and earlier is linked with openssl 0.9.8 while the curl on 10.9 and later is linked with the Security framework. openssl 0.9.8 seems to be able to do nothing newer than SSLv3. However if I compile the curl port with the +darwinssl variant to make it use the Security framework, then it is able to do TLSv1 on 10.6 and 10.7 and TLSv1.2 on 10.8. That's enough to let it get files from GitHub and SourceForge on 10.8 (and to connect to TLSv1 sites on 10.6 and 10.7 that it couldn't otherwise connect to) without putting us into the business of bundling encryption software like openssl.

comment:59 Changed 4 years ago by kencu (Ken)

Edit: this post missed the point for what you were after, bundling curl without openssl. Deleted.

Last edited 4 years ago by kencu (Ken) (previous) (diff)

comment:60 Changed 4 years ago by fhgwright (Fred Wright)

In the narrower case where the only issue is the certificate validation (and possibly too-old Apple root CAs), there could be a simpler fix. Currently, it's possible to get around such issues by setting fetch.ignore_sslcert. Aside from possibly using that on the command line, there are currently 11 ports and one PortGroup that set that option. It occurs to me that there could be a third value, 'auto', that maps to 'yes' or 'no' depending on circumstances.

The simplest definition of "circumstances" would be whether it's an OS version currently updated by Apple or not. It's reasonable to assume that versions still updated by Apple have sufficiently recent root CAs, while older OSes may not. If the only issue is whether the root CAs themselves have expired, then that could be checked directly, but if multiple certificates with different root CAs and different expirations are possible, then it's not so easy to check.

Doing without the certificate checking altogether isn't that awful, since distfiles need to pass checksum checks, anyway, but being a bit less heavy-handed in suppressing the check would be a plus. The 'auto' setting might even be a reasonable default.

It could also make a distinction between targeting a MacPorts distfile mirror or an upstream distfile source. In the former case, the certificate status could be known more accurately, while that isn't easily possible for arbitrary upstream sources.

comment:61 Changed 4 years ago by kencu (Ken)

I suppose using a bootstrap version of MacPorts in /opt/bootstrap to provide a full current version of curl and openssl that can be used by the main installation of MacPorts in ${prefix} is felt to be just beyond the capability of average users?

Fredowski has a quite beautiful shell script to automate the installation of MacPorts from source. It looks like that script, with slight modification, could accomplish the bootstrap and then full installation.

FYI I am using the /opt/bootstrap system on all systems up to 10.10 now, and it appears to be 100% bulletproof after many continuous years of use. Even if MacPorts changes to some kind of darwinssl system, I'll likely revert/override that because this system is just better.

comment:62 Changed 4 years ago by noloader (Jeffrey Walton)

I suppose using a bootstrap version of MacPorts in /opt/bootstrap to provide a full current version of curl and openssl that can be used by the main installation of MacPorts in ${prefix} is felt to be just beyond the capability of average users?

Yes (or is it No?). I think bootstrapping cURL is a good idea. You will need it on old PowerMacs.

In fact, for bootstrapping, I recommend you build Wget rather then cURL. Wget is easier to build because it has two dependencies - OpenSSL and Unistring. Bootstrapping Wget/OpenSSL/Unistring works back to at least OS X 10.5 on PowerMacs.

I eat my own dogfood: https://github.com/noloader/Build-Scripts/blob/master/bootstrap/bootstrap-wget.sh.

Wget has a minor annoyance due to dynamic linking. Sed fixes that with:

# Fix makefiles. No shared objects.
IFS="" find "$PWD" -iname 'Makefile' -print | while read -r file
do
    sed "s|-lssl|$LIBDIR/libssl.a|g" "$file" > "$file.fixed"
    mv "$file.fixed" "$file"
    sed "s|-lcrypto|$LIBDIR/libcrypto.a|g" "$file" > "$file.fixed"
    mv "$file.fixed" "$file"
    sed "s|-lunistring|$LIBDIR/libunistring.a|g" "$file" > "$file.fixed"
    mv "$file.fixed" "$file"
done

Once you bootstrap Wget you can fetch everything you need for a full-blown build of cURL: https://github.com/noloader/Build-Scripts/blob/master/build-curl.sh

Last edited 4 years ago by noloader (Jeffrey Walton) (previous) (diff)

comment:63 Changed 4 years ago by kencu (Ken)

wget is not useful for this issue, though.

You build MacPorts like this:

./configure --with-curlprefix=/opt/bootstrap

and there are no more SSL issues ever.

comment:64 Changed 4 years ago by kencu (Ken)

Oh, I see -- you mean use your build-curl.sh script instead of MacPorts to build curl.

Yes, perhaps. I just understand MacPorts so well, and it's so easy to use, I prefer it to a script.

comment:65 Changed 4 years ago by kencu (Ken)

that is a very fancy set of scripts you've put together there... thanks for all that work. I'll take a look at them in more detail.

comment:66 in reply to:  57 Changed 4 years ago by ryandesign (Ryan Carsten Schmidt)

Replying to ryandesign:

#51045 was merely the first example of a curl bug that's fixed in newer versions that I found to link to this ticket. There's another cosmetic issue I found on Leopard that's been fixed. I'm sure there have been more bugs fixed in curl over the years.

Do you have specific tickets?

As I recall, curl on Leopard would emit a spurious error message during some downloads. As far as I remember this did not affect the ability to download the files. I'm not able to find a ticket or other reference to it as I don't remember exactly what the error message was so I can't search for it now.

I saw it on Tiger just now:

--->  Fetching distfiles for guile
--->  Attempting to fetch guile-2.2.7.tar.gz from https://ftp.gnu.org/gnu/guile
--->  Attempting to fetch guile-2.2.7.tar.gz from ftp://ftp.gnu.org/old-gnu/guile
WAITCONNECT => PROTOCONNECT
--->  Attempting to fetch guile-2.2.7.tar.gz from http://mirror.facebook.net/gnu/guile
--->  Verifying checksums for guile

That unnecessary WAITCONNECT => PROTOCONNECT message comes from curl and later versions of curl don't print it anymore.

Version 0, edited 4 years ago by ryandesign (Ryan Carsten Schmidt) (next)

comment:67 Changed 4 years ago by iefdev (Eric F)

Cc: iefdev added

comment:68 Changed 3 years ago by evanmiller (Evan Miller)

Cc: evanmiller added

comment:69 Changed 2 years ago by mascguy (Christopher Nielsen)

Cc: mascguy added

comment:70 Changed 2 years ago by kencu (Ken)

has duplicate #64081

comment:71 Changed 2 years ago by catap (Kirill A. Korinsky)

Cc: catap added

comment:72 Changed 2 years ago by devernay (Frédéric Devernay)

Any chance of getting a proper fix for that? I'm tired of manually downloading github packages (#64081), especially now that most distfiles are fetched from github rather than macports mirrors.

I would even consider it a security issue at this point, since "All older versions [of openssl] (including 1.1.0, 1.0.2, 1.0.0 and 0.9.8) are now out of support and should not be used." - from https://www.openssl.org/source/

Any software that currently ships (including MacPorts) should not use any of these versions, and thus not use the system's curl on older systems.

comment:73 in reply to:  72 ; Changed 2 years ago by danielluke (Daniel J. Luke)

Replying to devernay:

I would even consider it a security issue at this point, since "All older versions [of openssl] (including 1.1.0, 1.0.2, 1.0.0 and 0.9.8) are now out of support and should not be used." - from https://www.openssl.org/source/

Any software that currently ships (including MacPorts) should not use any of these versions, and thus not use the system's curl on older systems.

The security issue is trying to run an old, unsupported macOS version that is no longer receiving patches from Apple. If you insist on doing that, you can work-around it. I don't think MacPorts should be in the business of providing OS patches after Apple stops, and I don't think it should be in the business of encouraging people to run obsolete OS versions.

comment:74 Changed 2 years ago by devernay (Frédéric Devernay)

you can work-around it

Unfortunately no, unless by "work-around" you mean buying new hardware. In which case I don't see the point in supporting any OS version (and thus hardware) that was obsoleted by Apple. And we might as well all use homebrew, which does exactly that.

And, by the way, a newer version of OpenSSL is not an OS patch. OpenSSL is a userland library.

comment:75 Changed 2 years ago by kencu (Ken)

it takes 4 minutes to rebuild and install macports from source against libcurl installed in /opt/local

it takes longer to install a macports into /opt/bootstrap and then rebuild macports in /opt/local against that, but it is more resilient.

comment:76 Changed 2 years ago by kencu (Ken)

see above:

comment:19

I first did this five years ago, and have on every older system since.

works nicely

Last edited 21 months ago by ryandesign (Ryan Carsten Schmidt) (previous) (diff)

comment:77 in reply to:  73 ; Changed 2 years ago by noloader (Jeffrey Walton)

Replying to danielluke:

Replying to devernay:

I would even consider it a security issue at this point, since "All older versions [of openssl] (including 1.1.0, 1.0.2, 1.0.0 and 0.9.8) are now out of support and should not be used." - from https://www.openssl.org/source/

Any software that currently ships (including MacPorts) should not use any of these versions, and thus not use the system's curl on older systems.

The security issue is trying to run an old, unsupported macOS version that is no longer receiving patches from Apple. If you insist on doing that, you can work-around it. I don't think MacPorts should be in the business of providing OS patches after Apple stops, and I don't think it should be in the business of encouraging people to run obsolete OS versions.

I don't think this position is correct. Macports specifically supports older versions of Apple hardware and software. Ken regularly works on support for the PowerPC machines, and I use the Macports support for the old hardware on my PowerMacs. The FAQ also includes discussions of older versions of OS X.

From the FAQ under "Why do I have to install xorg-* ports?" (wiki:FAQ#x11-libs):

This [x11 client] allows for more consistent support for older machines which are not as up to date (especially Tiger which doesn't have a pkg-config compatible X11 SDK). Most X11 packages in MacPorts are newer than those shipped by Apple and usually reflect the latest work from the ​XQuartz Project.

Last edited 21 months ago by ryandesign (Ryan Carsten Schmidt) (previous) (diff)

comment:78 in reply to:  77 Changed 2 years ago by udbraumann

Replying to noloader:

I don't think this position is correct. ...

I fully agree with noloader.

comment:79 in reply to:  76 ; Changed 2 years ago by devernay (Frédéric Devernay)

Replying to kencu:

see above:

comment:19

I first did this five years ago, and have on every older system since.

works nicely

Thanks a lot! That's a good solution.

The last phase of sudo make install (./src/images_to_archives.tcl) failed several times because some of my installed packages had wrong tarlists, and the error messages were not very informative. I had to reinstall these, and everything went fine afterwards. Maybe there's a way to skip that phase if it's not necessary?

Processing 10 of 317: lzo2-2.10_0+universal.darwin_10.i386-x86_64.tbz2
Command failed: /usr/bin/tar -cjf /opt/local-libc++/var/macports/software/lzo2/lzo2-2.10_0+universal.darwin_10.i386-x86_64.tbz2 -T /opt/local-libc++/var/macports/software/lzo2/tarlist > /opt/local-libc++/var/macports/software/lzo2/error.log 2>&1
Exit code: 1
command execution failed
    while executing
"system "$tarcmd -cjf $newlocation -T ${targetdir}/tarlist > ${targetdir}/error.log 2>&1""
    ("foreach" body line 78)
    invoked from within
"foreach installed $ilist {
    incr counter
    set iname [lindex $installed 0]
    set iversion [lindex $installed 1]
    set irevision [lindex $inst..."
    (file "./src/images_to_archives.tcl" line 36)
make: *** [install] Error 1

I wish the port command could issue a warning that it's using the system's outdated openssl/curl, and link to those special instructions to build from source with the --with-curlprefix option. Or at least make this a FAQ. Seeing the number of Cc: on this ticket, this seems to be affecting a number of users.

The real issues will come when even curl cannot be downloaded using the system's curl, but as long as pre-built packages are available (thanks again Ken) we're fine.

Last edited 21 months ago by ryandesign (Ryan Carsten Schmidt) (previous) (diff)

comment:80 in reply to:  79 Changed 2 years ago by noloader (Jeffrey Walton)

Replying to devernay:

... The real issues will come when even curl cannot be downloaded using the system's curl, but as long as pre-built packages are available (thanks again Ken) we're fine.

I think this is the linchpin of the problem. A modern cURL needs to be bootstrapped.

I find cURL is not the easiest program to build because it has so many dependencies. I recommend bootstrapping a downloader by building Wget. Wget has two dependencies - libunistring and OpenSSL. Once Wget is built and installed you can download anything, including a modern cURL.

I eat my own dogfood. I bootstrap with Wget in my collection of build scripts at https://github.com/noloader/Build-Scripts.

comment:81 in reply to:  77 Changed 2 years ago by danielluke (Daniel J. Luke)

Replying to noloader:

I don't think this position is correct. Macports specifically supports older versions of Apple hardware and software.

No, it doesn't Official support is much more limited than the number of versions we build/release for. Everything else is 'best effort' (everything more than the currently supported upstream releases is just whatever the volunteers what to support).

comment:82 Changed 2 years ago by evanmiller (Evan Miller)

As noted in #64189, if an installation links against /opt/local and a dependent library is later deactivated, the port program will become unusable and the MacPorts system will have to be fixed manually or reinstalled. So I would recommend the /opt/bootstrap route until something better is implemented here.

comment:83 Changed 2 years ago by kencu (Ken)

the /opt/bootstrap approach works, but for systems 10.6 to 10.8, changing macports.conf back to libstdc++ in /opt/bootstrap saves a lot of trouble, and in the case of 10.6, avoids overwriting /usr/lib/libc++.dylib by other installs of macports.

This is indeed a PITA....when I did it first 5 years ago I didn't imagine still needing it.

I support bundling some ssl as done above on older systems, and move on from this!

Throw in a dislaimer if you like.

Last edited 2 years ago by kencu (Ken) (previous) (diff)

comment:84 Changed 2 years ago by kencu (Ken)

Could also perhaps give instructions on how to use Jeffery's curl-building script mechanism in some prominent place...looks more lightweight than a whole /opt/bootstrap, and much simpler perhaps tp explain (I have not actually tried it, tho).

comment:85 in reply to:  19 Changed 2 years ago by barracuda156

Replying to ken-cunningham-webuse:

I gave this a shot today. Have curl installed in /opt/local

$ port -v installed | grep curl
  curl @7.50.3_0+ssl+universal (active) platform='darwin 10' archs='i386 x86_64'
  curl-ca-bundle @7.50.3_0 (active) platform='darwin 10' archs='noarch'

configured and installed macports from source, upgrading an existing installation

./configure --with-curlprefix=/opt/local
make
sudo make install

seems to work perfectly. The recent problem downloads, like hexchat and git, download without trouble now.

That's a 4-minute fix to an irritating problem.... unless you accidentally uninstall curl, or course...

Apparently failing for me.

36-9:MacPorts-2.7.1 arjuna$ port -v installed | grep curl
  curl @7.80.0_0+ssl (active) requested_variants='' platform='darwin 9' archs='ppc' date='2021-11-17T08:51:29+0800'
  curl-ca-bundle @7.80.0_0 (active) requested_variants='' platform='darwin 9' archs='noarch' date='2021-11-17T07:59:32+0800'
36-9:MacPorts-2.7.1 arjuna$  /opt/local/libexec/mpstats submit
Submitting data to https://ports.macports.org/statistics/submit/ ...
Error: SSL connect error
    while executing
"curl post "submission\[data\]=$json" $stats_url"

comment:86 Changed 2 years ago by kencu (Ken)

It doesn't work to just have curl installed.

You then need to rebuild macports against that newer curl.

Then the ssl stuff will work.

But it is fragile to have macports built against it's own installed ports, so the other method (/opt/bootstrap) will likely be favoured, until a proper fix is accepted at least.

comment:87 in reply to:  86 Changed 2 years ago by barracuda156

Replying to kencu:

It doesn't work to just have curl installed.

You then need to rebuild macports against that newer curl.

Then the ssl stuff will work.

But it is fragile to have macports built against it's own installed ports, so the other method (/opt/bootstrap) will likely be favoured, until a proper fix is accepted at least.

I did rebuild with:

36-9:~ arjuna$ cd /Users/arjuna/Downloads/MacPorts-2.7.1 
36-9:MacPorts-2.7.1 arjuna$ ./configure --with-curlprefix=/opt/local

and then make and sudo make install.

Will try bootstrap prefix.

comment:88 Changed 2 years ago by kencu (Ken)

Oh, well -- yes that is what usually works. I'm not completely sure then why it didn't fix the ssl transports for you!

comment:89 in reply to:  48 Changed 2 years ago by barracuda156

Replying to yan12125:

FWIW, I have a branch that bundles LibreSSL + libcurl with macports-base at https://github.com/yan12125/macports-base/tree/bundle-curl.

I choose LibreSSL just because it's based on autotools, while OpenSSL has a custom Perl-based configure system that once brought some headache to me. It should not be difficult to switch LibreSSL back to OpenSSL for macports-base.

This is outdated now? I see that repo is archived.

comment:90 Changed 21 months ago by Dave-Allured (Dave Allured)

Cc: Dave-Allured added

comment:91 Changed 21 months ago by jerryyhom

I have read through most comments. I guess the issue is whether someone is willing to put in the time/effort to link macports base with a modern curl, and maybe no one is willing. I have two comments.

  1. Ken (or others), you have manually compiled with a modern curl for several years. On old systems, what happens with expired certificates? Do you update them?
  1. As for linking with a modern curl, I think Ryan's comment:57 makes sense, which also follows Apple's recommendation about OpenSSL[1]. I would like to nudge this issue forward, but I am not familiar with macports base. Can someone comment how much development effort is required?

[1]: <https://developer.apple.com/library/archive/documentation/Security/Conceptual/cryptoservices/SecureNetworkCommunicationAPIs/SecureNetworkCommunicationAPIs.html#//apple_ref/doc/uid/TP40011172-CH13-SW1>

Last edited 21 months ago by ryandesign (Ryan Carsten Schmidt) (previous) (diff)

comment:92 in reply to:  91 Changed 21 months ago by ryandesign (Ryan Carsten Schmidt)

Replying to jerryyhom:

I guess the issue is whether someone is willing to put in the time/effort to link macports base with a modern curl,

A good amount of that time/effort was already put in in comment:48 though I had some follow-ups about changes I would have liked to see.

  1. Ken (or others), you have manually compiled with a modern curl for several years. On old systems, what happens with expired certificates? Do you update them?

Ken uses a separate MacPorts prefix to install a bootstrap copy of curl, which would use the certificates from the curl-ca-bundle port, which gets updated often. Ken would just have to selfupdate and upgrade outdated periodically in that bootstrap MacPorts prefix to receive those updates.

comment:93 Changed 21 months ago by ryandesign (Ryan Carsten Schmidt)

Cc: jerryyhom added

comment:94 Changed 20 months ago by jerryyhom

More than a week has passed without objection. I quickly re-read the history of comments and wonder if anyone still has objections? There were passionate arguments 3-6 years ago, but my feeling is that the discussion veered off on a subtle tangent and missed the point. My understanding is that bundling curl will allow macports on older systems to use modern ssl/tls. If I am missing the point, kindly share your knowledge. :)

Otherwise, I think this issue is mostly beneficial for macports and a shame that it has languished for six years. Most of the work has already been prototyped. Consider the time savings by those users working around the limitations. Do we need consensus or a willing macports base developer with time? How else can this issue be nudged forward?

comment:95 Changed 20 months ago by tomio-arisaka (Tomio Arisaka)

For example, Ghostscript cannot be built with MacPorts on both High-Sierra and Tiger.

On macOS High Sierra, "/usr/bin/curl" cannot download particular files due to the expired certificate.

$ /usr/bin/curl --version
curl 7.54.0 (x86_64-apple-darwin17.0) libcurl/7.54.0 LibreSSL/2.0.20 zlib/1.2.11 nghttp2/1.24.0
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz HTTP2 UnixSockets HTTPS-proxy 
$ 
$ /usr/bin/curl -LOR https://sourceforge.net/projects/gs-fonts/files/gs-fonts/6.0%20%28misc%2C%20AFPL%29/ghostscript-fonts-other-6.0.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   806    0   806    0     0   3317      0 --:--:-- --:--:-- --:--:--  3330
100   834    0   834    0     0   2071      0 --:--:-- --:--:-- --:--:--     0
100  1067    0  1067    0     0   1884      0 --:--:-- --:--:-- --:--:--  1884
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.
HTTPS-proxy has similar options --proxy-cacert and --proxy-insecure.

On Mac OS X Tiger PPC, "/usr/bin/curl" cannot download any files with SSL due to the old version of curl.

$ /usr/bin/curl --version
curl 7.16.3 (powerpc-apple-darwin8.0) libcurl/7.16.3 OpenSSL/0.9.7l zlib/1.2.3
Protocols: tftp ftp telnet dict ldap http file https ftps 
Features: GSS-Negotiate IPv6 Largefile NTLM SSL libz 
$ 
$ /usr/bin/curl -LOR https://sourceforge.net/projects/gs-fonts/files/gs-fonts/6.0%20%28misc%2C%20AFPL%29/ghostscript-fonts-other-6.0.tar.gz

curl: (35) error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
$ 
$ /usr/bin/curl -v -I https://sourceforge.net/projects/gs-fonts/files/gs-fonts/6.0%20%28misc%2C%20AFPL%29/ghostscript-fonts-other-6.0.tar.gz
* About to connect() to sourceforge.net port 443 (#0)
*   Trying 172.64.153.13... connected
* Connected to sourceforge.net (172.64.153.13) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: /usr/share/curl/curl-ca-bundle.crt
  CApath: none
* SSLv2, Client hello (1):
* error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
* Closing connection #0
curl: (35) error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

My simple resolution is to add fetch phase in order to use the latest version of curl:

if {${os.major} < 18} {
    depends_build-append    port:curl

    fetch {
        if { [file isfile "${distpath}/ghostscript-fonts-other-6.0.tar.gz"] == 0 } then {
            system -W ${distpath} "curl -LOR https://sourceforge.net/projects/gs-fonts/files/gs-fonts/6.0%20%28misc%2C%20AFPL%29/ghostscript-fonts-other-6.0.tar.gz"
        }
        ...
        ...
    }
}

comment:96 Changed 20 months ago by kencu (Ken)

that only is possible sometimes, depending on fetch details, and requires manual editing of each, so … ain’t likely gonna happen.

bundling curl may or may not happen eventually…influential.people don’t like macports being responsible for security updates (although we kinda already are…)

in the meantime, the /opt/bootstrap workaround above continues to work very well for those who use it, on all systems that show ssl issues, with zero changes in any portfiles or macports. So that is what I recommend.

Last edited 20 months ago by kencu (Ken) (previous) (diff)

comment:97 Changed 20 months ago by jerryyhom

Thanks for the feedback. Presumably, there are over a dozen people interested in this issue, and since I am not experienced with macports base, am genuinely interested to hear objections people might have. Otherwise, I wonder why this issue is stalled when most of the development work has been done.

Ken noted the issue of macports's security. I think the issue is more fundamental. Assuming macports wants to operate on older systems, TLS has become a necessary feature. The Apple guideline I referenced above even recommends bundling your own ssl lib instead of relying upon the builtin ssl lib.

I certainly appreciate that many macports maintainers have put in significant effort to keep operating on older systems. Setting aside the issue of TLS as a necessary feature, what are the security concerns? I am genuinely asking because I think we all agree security is important, but not all of us are security experts. Again, I'm hoping to nudge this issue forward with constructive dialogue.

comment:98 in reply to:  96 Changed 19 months ago by barracuda156

Replying to kencu:

in the meantime, the /opt/bootstrap workaround above continues to work very well for those who use it, on all systems that show ssl issues, with zero changes in any portfiles or macports. So that is what I recommend.

If the question is merely “If some users want this feature, there is a way to make it”, then yes, sure. But if Macports developers are interested to get more accurate statistics, then such solution is suboptimal, to say the least. Realistically speaking, pretty much no one gonna go that route, which we see from data collected. Then developers get a false impression that no one installs software, and support gets dropped. In the long run, everyone is worse off.

comment:99 Changed 19 months ago by kencu (Ken)

Maybe -- but you will not get all the important people to agree to bundle curl/openssl into the MacPorts build.

So either you convince the ones that won't to agree (good luck with that) or you accept the workaround (which I know you use) or -- you do a lot of work to come up with another method.

There is an acceptable real-world workaround that has been suggested by neverpanic in the mailing lists about a year ago. It involves having the relevant part of MacPorts use the system libcurl by default, but look for an use the alternate MacPorts libcurl first if it exists. You can find the mailing list discussion in MacPorts-dev a year ago or perhaps 18 months.

You would use the relevant dyld functions to look for and open the libcurl in ${prefix} if it exists, and then use the functions needed in there (as I recall they are slightly different), or if there is no libcurl in ${prefix} fall back to the one in the system and use that.

This solution is elegant, certainly correct, robust, and does not require macports to bundle libcurl (and thereby be responsible for security breaches in so doing). It is also time consuming, requires considerable programming expertise to accomplish, and will need to be maintained based on changes in libcurl in ${prefix} as it is updated.

As the /opt/bootstrap workaround works well enough for me, I won't be spending that time to do this. I doubt, in fact, that anyone with the needed skill set will, as -- well, life is short, time is limited, and this likely won't rise to the top of anyone's list, of those who can do it.

But you never know.

Last edited 19 months ago by kencu (Ken) (previous) (diff)

comment:100 Changed 19 months ago by contextnerror

Cc: contextnerror added

comment:101 Changed 17 months ago by aeiouaeiouaeiouaeiouaeiouaeiou

Cc: aeiouaeiouaeiouaeiouaeiouaeiou added

comment:102 Changed 5 months ago by catap (Kirill A. Korinsky)

Meanwhile we may have a much bigger issue with GitHub. On macOS 12.7 when I run s_client -showcerts -servername codeload.github.com -connect codeload.github.com:443 </dev/null I do have an output:

CONNECTED(00000005)
depth=0 C = US, ST = USA, L = New York, O = Optimization, OU = Optimization, CN = tomcat
verify error:num=18:self-signed certificate
verify return:1
depth=0 C = US, ST = USA, L = New York, O = Optimization, OU = Optimization, CN = tomcat
verify return:1
---
Certificate chain
 0 s:C = US, ST = USA, L = New York, O = Optimization, OU = Optimization, CN = tomcat
   i:C = US, ST = USA, L = New York, O = Optimization, OU = Optimization, CN = tomcat
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Aug 30 08:53:43 2019 GMT; NotAfter: Aug 25 08:53:43 2039 GMT
...

=> self-signed certificate won't pass verification. So, output is below expected but quite scary:

√ ~ % /opt/local/bin/curl -I https://codeload.github.com/   
curl: (60) SSL certificate problem: self-signed certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
? ~ % /usr/bin/curl -I https://codeload.github.com/
curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
? ~ %

comment:103 Changed 5 months ago by catap (Kirill A. Korinsky)

The issue is my internet connection, an ISP decided that codeload.github.com is bad web-site and redirected any DNS request to a page which says that it contains viruses and shouldn't be used.

comment:104 Changed 5 months ago by kencu (Ken)

just an observation…. the current curl port has become quite onerous with recursive deps on older systems lately.

I created a “minimalcurl” port that installs libcurl with vastly fewer deps to install into /opt/bootstrap to trim this up for my own use.

Perhaps if we asked nicely Ryan might feel such a subport of the curl port might be created.

comment:105 in reply to:  104 Changed 5 months ago by noloader (Jeffrey Walton)

Replying to kencu:

just an observation…. the current curl port has become quite onerous with recursive deps on older systems lately.

I created a “minimalcurl” port that installs libcurl with vastly fewer deps to install into /opt/bootstrap to trim this up for my own use.

Perhaps if we asked nicely Ryan might feel such a subport of the curl port might be created.

Wget is much easier to bootstrap than cURL. Wget has two minimal dependencies. Also see comment 62, https://trac.macports.org/ticket/51516#comment:62.

comment:106 Changed 5 months ago by kencu (Ken)

perhaps I'll try out your build_curl.sh script instead of using macports to install curl into /opt/bootstrap. I just trust the macports installation so far, as it's so reliable.

As you know, macports can't build against wget, so you need to install curl/libcurl somehow/somewhere to get macports fully functional, then build against it.

I wouldn't want to confuse people into thinking that installing wget will make macports work, as it won't.

Add: your Build-Scripts repo is very nicely done ... great work, Jeffrey!

Last edited 5 months ago by kencu (Ken) (previous) (diff)
Note: See TracTickets for help on using tickets.