Opened 4 years ago

Last modified 4 days ago

#51516 new enhancement

MacPorts should use a bundled copy of a newer libcurl and SSL library rather than the OS X version

Reported by: ryandesign (Ryan Schmidt) Owned by: macports-tickets@…
Priority: Normal Milestone: MacPorts Future
Component: base Version:
Keywords: Cc: udbraumann, mkae (Marko Käning), devernay (Frédéric Devernay), raimue (Rainer Müller), mojca (Mojca Miklavec), fhgwright (Fred Wright), dliessi (Davide Liessi), pkoshevoy (Pavel Koshevoy), bK4gYuRo, 1-61803, iEFdev (Eric F), noloader (Jeffrey Walton), khepler, cooljeanius (Eric Gallager), MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)
Port:

Description

Much as we already use a bundled copy of a newer Tcl, MacPorts should use a bundled copy of a newer libcurl (and CA certificate bundle?) rather than the OS X version. On older systems, the OS X version of libcurl cannot connect to some https sites, including pypi (see #51509) and also https://distfiles.macports.org and https://packages.macports.org .

Change History (59)

comment:1 Changed 4 years ago by udbraumann

Cc: braumann@… added

Cc Me!

comment:2 Changed 4 years ago by ryandesign (Ryan Schmidt)

Has duplicate #52602.

comment:3 Changed 4 years ago by neverpanic (Clemens Lang)

I don't think MacPorts should be in the business of distributing a CA bundle. We shouldn't control who our users trust.

comment:4 Changed 4 years ago by raimue (Rainer Müller)

The issue is mostly not libcurl itself, but system's libcurl linked against system's OpenSSL 0.9.8, which does not support TLS >= 1.1. Therefore servers that require TLS >= 1.1 can no longer be reached (#51112, #52604). As of OS X 10.9 Mavericks, system's libcurl will use Secure Transport instead. Secure Transport supports TLS >= 1.1 as of OS X 10.8 Mountain Lion.

If we still wanted to support OS X <= 10.7, we would need both libcurl and a compatible SSL/TLS library in base.

comment:5 Changed 4 years ago by ryandesign (Ryan Schmidt)

Has duplicate #52604.

comment:6 in reply to:  3 Changed 4 years ago by ryandesign (Ryan Schmidt)

Replying to cal@…:

I don't think MacPorts should be in the business of distributing a CA bundle. We shouldn't control who our users trust.

I don't care how we arrange it, but MacPorts should at least be able to download from the MacPorts packages and distfiles servers via SSL. If the problem isn't trusting the certificates but rather the capabilities of the SSL library, then we need to bundle a newer SSL library.

comment:7 Changed 4 years ago by neverpanic (Clemens Lang)

Keep in mind that bundling an SSL library implies that we will release at least as often as this SSL library to fix any security problems with it. Personally, I'm not prepared to do that, and we should just fall back to non-SSL connections on systems that do not support modern SSL.

I doubt that we have the manpower to rescue platforms that Apple has abandoned. At some point we'll just have to admit that the system is no longer usable due to issues such as these.

comment:8 Changed 4 years ago by dbevans (David B. Evans)

I agree that bundling an SSL library could lead to a lot of extra cycles to update the bundle for security issues, etc. Also probably licensing issues with a binary bundle that includes OpenSSL or LibreSSL. Not sure about other SSL libraries.

To avoid these issues, could we set it up so a source build could be built with an external library other than the system ones? An interested user could then boot strap from an outdated binary Installer like this:

  • install old port binary using installer
  • build the SSL library of your choice using the just installed port binary
  • do a source build using the installed non-system SSL library using the appropriate configuration parameters ( say --with-openssl=${prefix}/lib )

This avoids binary licensing issues and updates for OpenSSL and the like would be handled by the normal port maintenance process.

The procedure would need to be well documented to avoid too many "I tried it but it didn't work" tickets.

Or is this too complicated for the average user to deal with? Just an idea.

comment:9 in reply to:  8 Changed 4 years ago by larryv (Lawrence Velázquez)

That sounds a bit complicated.

What are the downsides of downgrading to insecure connections on old platforms? Would we have to worry about tampering? We already verify checksums.

comment:10 Changed 4 years ago by ryandesign (Ryan Schmidt)

We already downgrade Leopard and earlier to http for packages.macports and distfiles.macports. r150758, r150765. Once we are properly mirroring distfiles before doing buildbot builds, that will probably be a sufficient fix. But it doesn't help old systems download from other sites that require https.

comment:11 in reply to:  10 ; Changed 4 years ago by neverpanic (Clemens Lang)

The problem with linking against a copy of OpenSSL in $prefix is that it will temporarily disappear while OpenSSL itself is upgraded, but that must not interrupt the upgrade process.

Replying to ryandesign@…:

Once we are properly mirroring distfiles before doing buildbot builds, that will probably be a sufficient fix. But it doesn't help old systems download from other sites that require https.

No matter what we do, that's a problem these platforms will always have, and will continue to do so. As I said, we cannot save abandoned platforms from dying.

comment:12 in reply to:  11 Changed 4 years ago by dbevans (David B. Evans)

Replying to cal@…:

The problem with linking against a copy of OpenSSL in $prefix is that it will temporarily disappear while OpenSSL itself is upgraded, but that must not interrupt the upgrade process.

Good point.

Replying to ryandesign@…:

Once we are properly mirroring distfiles before doing buildbot builds, that will probably be a sufficient fix. But it doesn't help old systems download from other sites that require https.

No matter what we do, that's a problem these platforms will always have, and will continue to do so. As I said, we cannot save abandoned platforms from dying.

Agreed. However, we can, perhaps, provide tools for those who are interested in using these platforms regardless of their antiquity. I'm not sure how many such people there are but they're quite enthusiastic and it would be a shame to shun them, altogether, from the MacPorts community. Best world, IMO, would be for them to form an ad hoc interest group within MacPorts so they could cooperatively work on these issues themselves.

So I'm suggesting that while we shouldn't commit (and haven't) to supporting officially anything older than 10.9, it wouldn't hurt to add a hook once in a while to allow them to do the work for themselves if they choose to do so. In this case, provide a configuration option for base to optionally build with external libraries rather than system ones to solve the SSL/TLS issues inherent in the old system libraries.

By the way, if one wanted to bundle something that is light weight with good protocol coverage and permissive licensing, wolfSSL/CyaSSL looks interesting. curl has supported it upstream as an alternative so openssl/gnutls for a while now.

comment:13 in reply to:  11 ; Changed 4 years ago by ryandesign (Ryan Schmidt)

Replying to cal@…:

Replying to ryandesign@…:

Once we are properly mirroring distfiles before doing buildbot builds, that will probably be a sufficient fix. But it doesn't help old systems download from other sites that require https.

No matter what we do, that's a problem these platforms will always have, and will continue to do so. As I said, we cannot save abandoned platforms from dying.

I don't understand why you would say that. The problem right now is caused by older systems have an old SSL implementation. The proposed fix is to provide a newer SSL implementation in MacPorts. That fixes the problem.

I agree that if we do this, we should release a new version of MacPorts when a new version of the SSL library is available, in case the new version of the SSL library fixes security vulnerabilities. I agree that our record to date of releasing new versions of MacPorts in a timely fashion is not good, which would be a reason not to make this change right now. However, I hope that once we move to GitHub, we can improve our processes and procedures to make it much easier to release new versions of MacPorts.

comment:14 Changed 4 years ago by ken-cunningham-webuse

Best world, IMO, would be for them to form an ad hoc interest group within MacPorts so they could cooperatively work on these issues themselves.

That might be a good idea. Even for example a separate trac stream for <10.9 issues would keep the main stream uncluttered, give a voice to those trying to keep these machines going, but keep the signal-to-noise ratio up for the main cadre of users. People who wanted to help out with older systems could.

Or maybe a separate stream would be the final death bell for these systems...

Last edited 4 years ago by ken-cunningham-webuse (previous) (diff)

comment:15 in reply to:  10 Changed 4 years ago by larryv (Lawrence Velázquez)

Replying to ryandesign@…:

Once we are properly mirroring distfiles before doing buildbot builds, that will probably be a sufficient fix. But it doesn't help old systems download from other sites that require https.

Other than the brief period before we mirror a new distfile, when would said systems not be able to fall back to the mirrors?

comment:16 in reply to:  13 Changed 4 years ago by neverpanic (Clemens Lang)

Replying to ryandesign@…:

I don't understand why you would say that. The problem right now is caused by older systems have an old SSL implementation. The proposed fix is to provide a newer SSL implementation in MacPorts. That fixes the problem.

Yes, that fixes the problem in MacPorts, but users on these platforms will continue to experience the same problems with other services that use TLS. So we can't fix the problem completely, but would still bundle an SSL library on platforms that already ship one that's perfectly fine.

I agree with devans here, we should provide instructions, warn users about the caveats and have them compile MacPorts against MacPorts curl.

comment:17 Changed 4 years ago by mkae (Marko Käning)

Cc: mk@… added

Cc Me!

comment:18 Changed 4 years ago by devernay (Frédéric Devernay)

Cc: frederic.devernay@… added

Cc Me!

comment:19 Changed 4 years ago by ken-cunningham-webuse

I gave this a shot today. Have curl installed in /opt/local

$ port -v installed | grep curl
  curl @7.50.3_0+ssl+universal (active) platform='darwin 10' archs='i386 x86_64'
  curl-ca-bundle @7.50.3_0 (active) platform='darwin 10' archs='noarch'

configured and installed macports from source, upgrading an existing installation

./configure --with-curlprefix=/opt/local
make
sudo make install

seems to work perfectly. The recent problem downloads, like hexchat and git, download without trouble now.

That's a 4-minute fix to an irritating problem.... unless you accidentally uninstall curl, or course...

Last edited 4 years ago by ken-cunningham-webuse (previous) (diff)

comment:20 Changed 4 years ago by ken-cunningham-webuse

although this setup as described is working well, I worried it was possibly too precarious to rely on itself like that in case of any unforeseen issues that might arise in future.

I decided to install a new copy of macports into /opt/bootstrap, installed curl in that installation, and then referenced the curl in /opt/bootstrap when reinstalling macports back into /opt/local.

comment:21 Changed 4 years ago by RJVB (René Bertin)

I think there might be another solution, one that doesn't require a hard dependency on curl.

If the curl function from Pextlib fails with an error that's likely to signal a TLS problem (and unlikely to have another cause), try again using the curl command (or wget). If that fails too and the command isn't found in ${prefix}/bin, instruct the user to install port:curl .

That'd be a 1-time investment to upgrade the Tcl function calling Pextlib::curl, but no cyclic or otherwise problematic dependencies will be introduced, and users don't have to follow more complex instructions than necessary.

comment:22 Changed 4 years ago by neverpanic (Clemens Lang)

Please, let's not have more 'if this doesn't work let's fall back to this other lesser-tested special case'.

Have you even looked at all the options supported in pextlib1.0/curl.c? Re-creating those for a command-line client is complicated, and in some instances, impossible.

comment:23 Changed 4 years ago by ryandesign (Ryan Schmidt)

The current discussion is centered around being able to reach servers running newer ssl. But the idea to use a bundled curl is much older, and is based on the idea that we should use the latest software, because it has more features and fixes bugs. That's the reason why we bundle tcl 8.5 now; we were able to remove workarounds for tcl 8.4 and start using features introduced in tcl 8.5. In the same way, the latest curl might offer some useful features not present in Tiger's curl, and there are workarounds in base for bugs in the version of curl included with Snow Leopard that we could remove if we bundled a newer version.

The problem of needing to also bundle a newer ssl library to support newer ssl sites, and to quickly release new MacPorts base versions to update the ssl library should any vulnerabilities be found, makes this idea less wonderful. On the one hand, having a recent but not completely up to date ssl library would be better for users of old OS versions that don't get security updates anymore; but on the other hand, it would be worse for users of new OS versions that do.

Maybe once we are on GitHub we will improve our testing and release process to the point where releasing a new version of MacPorts can be done very easily. If so, maybe then we can revisit this issue.

comment:24 Changed 4 years ago by raimue (Rainer Müller)

Cc: raimue added

comment:25 Changed 4 years ago by mojca (Mojca Miklavec)

Cc: mojca added

comment:26 Changed 4 years ago by fhgwright (Fred Wright)

Cc: fhgwright added

comment:27 Changed 4 years ago by dliessi (Davide Liessi)

Cc: dliessi added

comment:28 Changed 4 years ago by raimue (Rainer Müller)

In 7a428704/macports-ports:

git: add alternative download URL to master_sites

The main server at www.kernel.org has a hard requirement for TLS 1.2,
which is not available on older versions of macOS. This CDN mirror is
less restrictive and can be accessed using TLS 1.0.

Closes: #52602
Closes: #52604
Closes: #52836
See: #51516

comment:29 Changed 4 years ago by raimue (Rainer Müller)

Cc: udbraumann, , mkae, devernay, raimue, mojca, fhgwright, dliessiudbraumann, mkae, devernay, raimue, mojca, fhgwright, dliessi
Summary: MacPorts should use a bundled copy of a newer libcurl rather than the OS X versionMacPorts should use a bundled copy of a newer libcurl and SSL library rather than the OS X version

Has duplicate #46539.

comment:30 in reply to:  23 Changed 4 years ago by ryandesign (Ryan Schmidt)

Replying to ryandesign:

the idea to use a bundled curl is much older, and is based on the idea that we should use the latest software, because it has more features and fixes bugs.

#51045 is an example of such a curl bug.

comment:31 Changed 4 years ago by mf2k (Frank Schima)

Cc: pkoshevoy added

Has duplicate #53181.

comment:32 Changed 4 years ago by raimue (Rainer Müller)

Fetching Python distfiles from files.pythonhosted.org (as used by ports via master_sites pypi:) will require TLS 1.2 as of 2017-04-30. Support for TLS 1.0 and 1.1 will be disabled. https://pyfound.blogspot.de/2017/01/time-to-upgrade-your-python-tls-v12.html

TLS 1.2 is supported in Secure Transport as of OS X 10.8, which means fetching python ports will stop working on OS X <= 10.7 on this date.

comment:33 Changed 3 years ago by bK4gYuRo

Cc: bK4gYuRo added

comment:34 Changed 3 years ago by jduerstock (Jason Duerstock)

Since Macports locally stores the hashes of the source files anyway, can we ignore any HTTPS failures at least with respect to files downloaded with known hashes?

comment:35 in reply to:  34 Changed 3 years ago by neverpanic (Clemens Lang)

Replying to jduerstock:

Since Macports locally stores the hashes of the source files anyway, can we ignore any HTTPS failures at least with respect to files downloaded with known hashes?

Unfortunately we can't. The issue is not that the certificate presented by the servers would not verify (which we can ignore), but that all protocols implemented in older versions of libcurl (and the cryptography library it uses) shipped with older versions of macOS are no longer accepted by various servers. This means that all attempts to connect just fail. A workaround would be using an unencrypted channel to download the files, but a lot of servers nowadays redirect to SSL-only protocols.

comment:36 Changed 3 years ago by raimue (Rainer Müller)

We are already downgrading connections to our own mirrors to HTTP without SSL/TLS on older systems (see archive_sites.tcl). This would already help with many ports on these legacy systems, but our distfiles mirroring has not been brought back yet after we left macOS Forge.

comment:37 Changed 3 years ago by kencu (Ken)

Just FYI, the bootstrap method above has worked nicely on 10.4 up since I started using it last year. Haven't run into a fetch error since then on any system. I update the bootstrap install of Macports every few months, otherwise ignore it.

Last edited 3 years ago by kencu (Ken) (previous) (diff)

comment:38 Changed 3 years ago by 1-61803

Cc: 1-61803 added

comment:39 Changed 3 years ago by kencu (Ken)

One thing to note - when updating MacPorts versions, you will likely have to build it again from source rather than trust the sudo port selfupdate mechanism to do it properly for you. sudo port selfupdated does not presently appear to preserve configure settings like --with-curlprefix.

comment:40 Changed 3 years ago by yan12125 (Chih-Hsuan Yen)

Cc: yan12125 added

comment:41 Changed 3 years ago by iEFdev (Eric F)

Cc: iEFdev added

comment:42 in reply to:  23 Changed 3 years ago by ryandesign (Ryan Schmidt)

Replying to ryandesign:

But the idea to use a bundled curl is much older, and is based on the idea that we should use the latest software, because it has more features and fixes bugs.

Could we perhaps tackle this part -- bundle and link with a recent version of libcurl -- and postpone dealing with ssl issues? On older systems, we can link our bundled libcurl with the system's openssl library; on the newer systems that support it, we can use the --with-darwinssl configure flag to tell libcurl to use the macOS Secure Transport API.

comment:43 Changed 3 years ago by noloader (Jeffrey Walton)

Regarding this:

Much as we already use a bundled copy of a newer Tcl, MacPorts should use a bundled copy of a newer libcurl (and CA certificate bundle?) rather than the OS X version.

I copied cURL's cacert.pem to /System/Library/OpenSSL/certs/cacert.pem, which was previously empty. It did not resolve the issue.

It looks like its the old/downlevel version of OpenSSL. OpenSSL 0.9.8 lacks a lot of goodies, like elliptic curve support and TLS 1.2 support. It is certainly not following contemporary best practices.

Thanks for Ryan's help on a related ticket.

comment:44 Changed 3 years ago by ryandesign (Ryan Schmidt)

Cc: noloader added

Yes, I know that merely using a bundled curl that's still linked with the system's openssl will not solve all of the problems being discussed in this ticket. But I think we should still do it because it will solve some of the problems (e.g. comment:30).

comment:45 Changed 3 years ago by khepler

Cc: khepler added

comment:46 in reply to:  44 ; Changed 2 years ago by neverpanic (Clemens Lang)

Replying to ryandesign:

Yes, I know that merely using a bundled curl that's still linked with the system's openssl will not solve all of the problems being discussed in this ticket. But I think we should still do it because it will solve some of the problems (e.g. comment:30).

I think we all agree that we should not bundle an SSL library with MacPorts.

Now that the mirroring is fixed, I'm wondering whether we should just close this ticket as wontfix. You mentioned that you want a bundled libcurl in MacPorts for other reasons, but the cosmetic issue you linked in #51045 does in my opinion not warrant the additional effort of bundling libcurl (especially since it's fixed since two OS releases and we could add a simple workaround in base for that).

Do you have any other reasons for bundling a libcurl? If not, I would close this ticket as wontfix.

comment:47 in reply to:  46 ; Changed 2 years ago by ryandesign (Ryan Schmidt)

Replying to neverpanic:

Replying to ryandesign:

Yes, I know that merely using a bundled curl that's still linked with the system's openssl will not solve all of the problems being discussed in this ticket. But I think we should still do it because it will solve some of the problems (e.g. comment:30).

I think we all agree that we should not bundle an SSL library with MacPorts.

I do not agree yet. Older OSes need a newer ssl library to fetch from some https sites. So can't we provide a newer openssl or libressl on older systems, and use Apple's Secure Transport on newer ones? Yes there might be security vulnerabilities discovered in the ssl library we bundle, but isn't it likely that there are fewer vulnerabilities in it than in the old openssl version used on those systems?

Your objection in comment:3 was that we should not distribute a CA bundle. Would bundling an ssl library require us to include CA bundle? Doesn't the OS ship with one that we could use? Or is that part of the problem—the old OS's CA bundle doesn't contain the information needed to trust new sites?

Now that the mirroring is fixed, I'm wondering whether we should just close this ticket as wontfix. You mentioned that you want a bundled libcurl in MacPorts for other reasons, but the cosmetic issue you linked in #51045 does in my opinion not warrant the additional effort of bundling libcurl (especially since it's fixed since two OS releases and we could add a simple workaround in base for that).

Do you have any other reasons for bundling a libcurl? If not, I would close this ticket as wontfix.

#51045 was merely the first example of a curl bug that's fixed in newer versions that I found to link to this ticket. There's another cosmetic issue I found on Leopard that's been fixed. I'm sure there have been more bugs fixed in curl over the years.

The main difficulty or effort of bundling libcurl was originally that we didn't bundle any other software with MacPorts, so a method of doing that had to be invented. Now that we have created that method and already use it to bundle 4 libraries, it shouldn't be that difficult to bundle a few more libraries for modern ssl support or xz decompression (#52000).

comment:48 Changed 2 years ago by yan12125 (Chih-Hsuan Yen)

FWIW, I have a branch that bundles LibreSSL + libcurl with macports-base at https://github.com/yan12125/macports-base/tree/bundle-curl.

I choose LibreSSL just because it's based on autotools, while OpenSSL has a custom Perl-based configure system that once brought some headache to me. It should not be difficult to switch LibreSSL back to OpenSSL for macports-base.

comment:49 in reply to:  47 ; Changed 2 years ago by noloader (Jeffrey Walton)

Replying to ryandesign:

Replying to neverpanic:

Replying to ryandesign:

Yes, I know that merely using a bundled curl that's still linked with the system's openssl will not solve all of the problems being discussed in this ticket. But I think we should still do it because it will solve some of the problems (e.g. comment:30).

I think we all agree that we should not bundle an SSL library with MacPorts.

I do not agree yet. Older OSes need a newer ssl library to fetch from some https sites. So can't we provide a newer openssl or libressl on older systems, and use Apple's Secure Transport on newer ones? Yes there might be security vulnerabilities discovered in the ssl library we bundle, but isn't it likely that there are fewer vulnerabilities in it than in the old openssl version used on those systems?

Your objection in comment:3 was that we should not distribute a CA bundle. Would bundling an ssl library require us to include CA bundle? Doesn't the OS ship with one that we could use? Or is that part of the problem—the old OS's CA bundle doesn't contain the information needed to trust new sites?

Newer libraries with vulnerabilities seems the lesser of the two evils. Early 2000's tools with early CA lists don't help with security much. People will just disable the certificate checks, which puts them in a worse place.

And newer libraries that "just work" seems like a worthy usability and security goals. People won't have to do things like wget --no-check-certificate and curl --insecure.

Also, those old libraries and programs built on them are going to cause more trouble in the future because they can't do TLS 1.2. A TLS 1.2-only configuration is just common place nowadays. For example, GitHub recently made the engineering change: https://githubengineering.com/crypto-removal-notice/ .

After the GitHub change things like wget --no-check-certificate --tlsv1 and curl --insecure --tlsv1 simply will not work. It is technically impossible for some sites because of the site's configuration.

comment:50 in reply to:  49 Changed 2 years ago by noloader (Jeffrey Walton)

Replying to noloader:

...

Yes, I know that merely using a bundled curl that's still linked with the system's openssl will not solve all of the problems being discussed in this ticket. But I think we should still do it because it will solve some of the problems (e.g. comment:30).

I think we all agree that we should not bundle an SSL library with MacPorts.

I do not agree yet. Older OSes need a newer ssl library to fetch from some https sites. So can't we provide a newer openssl or libressl on older systems, and use Apple's Secure Transport on newer ones? Yes there might be security vulnerabilities discovered in the ssl library we bundle, but isn't it likely that there are fewer vulnerabilities in it than in the old openssl version used on those systems?

Your objection in comment:3 was that we should not distribute a CA bundle. Would bundling an ssl library require us to include CA bundle? Doesn't the OS ship with one that we could use? Or is that part of the problem—the old OS's CA bundle doesn't contain the information needed to trust new sites?

Newer libraries with vulnerabilities seems the lesser of the two evils. Early 2000's tools with early CA lists don't help with security much. People will just disable the certificate checks, which puts them in a worse place.

And newer libraries that "just work" seems like a worthy usability and security goals. People won't have to do things like wget --no-check-certificate and curl --insecure.

Also, those old libraries and programs built on them are going to cause more trouble in the future because they can't do TLS 1.2. A TLS 1.2-only configuration is just common place nowadays. For example, GitHub recently made the engineering change: https://githubengineering.com/crypto-removal-notice/ .

After the GitHub change things like wget --no-check-certificate --tlsv1 and curl --insecure --tlsv1 simply will not work. It is technically impossible for some sites because of the site's configuration.

This is worth mentioning... TLS 1.3 is in Last Call (LC) status on the TLS WG mailing list. I think the future trend will be sites supporting TLS 1.2 and TLS 1.3 as sites are reconfigured for the new standard. Also see Last Call: <draft-ietf-tls-tls13-24.txt> (The Transport Layer Security (TLS) Protocol Version 1.3) to Proposed Standard.

Without getting too bogged down in what version of cacert.pem to use, maybe the better engineering question would be, how can MacPorts ensure TLS 1.2 and TLS 1.3 "just work" for users. If you answer that question I believe the CA list question will follow without much effort.

comment:51 in reply to:  48 Changed 2 years ago by kencu (Ken)

Replying to yan12125:

FWIW, I have a branch that bundles LibreSSL + libcurl with macports-base at https://github.com/yan12125/macports-base/tree/bundle-curl.

Well that looks pretty fabulous! All that is missing is the CA certificates I guess. Well done.

Last edited 2 years ago by kencu (Ken) (previous) (diff)

comment:52 in reply to:  48 Changed 2 years ago by ryandesign (Ryan Schmidt)

Replying to yan12125:

FWIW, I have a branch that bundles LibreSSL + libcurl with macports-base at https://github.com/yan12125/macports-base/tree/bundle-curl.

I choose LibreSSL just because it's based on autotools, while OpenSSL has a custom Perl-based configure system that once brought some headache to me. It should not be difficult to switch LibreSSL back to OpenSSL for macports-base.

Yes thanks for this. There has been some spirited debate about the merits of openssl vs libressl, including pointing out that as of High Sierra, Apple is distributing libressl in macOS instead of openssl. I would be fine with using libressl in MacPorts instead of openssl.

I see that your branch deletes the MacPorts --with-curlprefix configure option. We probably want to keep that option for users who want to override it, but default it to using the bundled copy.

And as I mentioned above, on systems new enough to have a Secure Transport implementation that supports TLS 1.2, I'd like to use that, and only build and use the bundled libressl on older systems. /usr/bin/curl didn't start using Secure Transport until Mavericks, but it has been around a lot longer than that. I'd have to do some testing to figure out how far back it supports TLS 1.2. Ideally we would do a configure script test for Secure Transport's capabilities. The only check curl's own configure script does is for the existence of /System/Library/Frameworks/Security.framework, but that has existed since Mac OS X v10.0 so we'd need a more specific check than that.

comment:53 in reply to:  49 ; Changed 2 years ago by neverpanic (Clemens Lang)

Replying to ryandesign:

I do not agree yet. Older OSes need a newer ssl library to fetch from some https sites. So can't we provide a newer openssl or libressl on older systems, and use Apple's Secure Transport on newer ones? Yes there might be security vulnerabilities discovered in the ssl library we bundle, but isn't it likely that there are fewer vulnerabilities in it than in the old openssl version used on those systems?

I do not want to ship a library with potentially known and dangerous security vulnerabilities to users. If we use Apple's library and it does not work, the blame is on Apple, and users on old OSes cannot expect support from Apple. I do not want to create the impression that we support this setup.

Yes, there might be fewer problems, but if there are problems they would then be *our* problems. At the moment, they are not.

Your objection in comment:3 was that we should not distribute a CA bundle. Would bundling an ssl library require us to include CA bundle? Doesn't the OS ship with one that we could use? Or is that part of the problem—the old OS's CA bundle doesn't contain the information needed to trust new sites?

The latter, the CA bundle would eventually be too old for new certificates to validate.

#51045 was merely the first example of a curl bug that's fixed in newer versions that I found to link to this ticket. There's another cosmetic issue I found on Leopard that's been fixed. I'm sure there have been more bugs fixed in curl over the years.

Do you have specific tickets? I'm not convinced that we should bundle every library we use.

Replying to yan12125:

FWIW, I have a branch that bundles LibreSSL + libcurl with macports-base at https://github.com/yan12125/macports-base/tree/bundle-curl.

Thank you, that's helpful for users that want to run this at their own risk.

Replying to noloader:

Newer libraries with vulnerabilities seems the lesser of the two evils. Early 2000's tools with early CA lists don't help with security much. People will just disable the certificate checks, which puts them in a worse place.

Early 2000's tools are pretty much not going to give you any reasonable assumption of security at this point. I'd rather want users to realize that then to give them a false sense of security.

And newer libraries that "just work" seems like a worthy usability and security goals. People won't have to do things like wget --no-check-certificate and curl --insecure.

Also, those old libraries and programs built on them are going to cause more trouble in the future because they can't do TLS 1.2. A TLS 1.2-only configuration is just common place nowadays. For example, GitHub recently made the engineering change: https://githubengineering.com/crypto-removal-notice/ .

This is not the point of this ticket. Software installed by MacPorts will always use modern libraries and support modern crypto. This is about the libraries used by MacPorts itself. So either, your curl is MacPorts curl, which will work, or it is Apple's curl, which we will also not fix with this.

comment:54 Changed 9 months ago by yan12125 (Chih-Hsuan Yen)

Cc: yan12125 removed

comment:55 Changed 9 months ago by cooljeanius (Eric Gallager)

Cc: cooljeanius added

comment:56 Changed 7 months ago by MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)

Cc: MarcusCalhoun-Lopez added

comment:57 in reply to:  53 Changed 7 months ago by ryandesign (Ryan Schmidt)

Let me try one more time to explain my position. I still believe that we should bundle libcurl with MacPorts (either on all systems or only on older systems that don't get updates from Apple anymore), and that we should bundle a newer libressl or openssl with MacPorts (either only on older systems that don't get updates from Apple anymore or only on older systems that don't support newer TLS algorithms that many servers now require). On newer systems that still get updates from Apple we can either use our bundled curl and link it with Apple's security framework, or we can continue to use the system's libcurl as we do now.

To be concrete about it, at this time, I might propose that we use a bundled newer libcurl/libressl/openssl on OS X 10.11 El Capitan and earlier. I believe El Capitan is no longer receiving security updates as of the end of 2018. The problem I mentioned in #51045 was present in El Capitan's curl but not Sierra's, and also René recently encountered a KDE download server that could not be accessed from El Capitan's curl or earlier but worked with Sierra's. On Sierra and later, we could either use our bundled libcurl linked with the security framework or else use macOS libcurl.

Replying to neverpanic:

I do not want to ship a library with potentially known and dangerous security vulnerabilities to users.

To what library are you referring? libcurl? openssl? libressl? What are the specific known and dangerous security vulnerabilities that you're referring to?

I'm not certain what types of security vulnerabilities do or could exist in those libraries but even if there were any I'm not sure how serious that would be for us. We would only be using this libcurl for MacPorts tasks like downloading known files. We checksum our distfiles and sign our packages, so any vulnerability that caused curl to deliver data other than that which it should would be noticed by MacPorts immediately and the bad data would not be used. And we already don't consider the information we send or receive with curl to be confidential—we don't transmit private information and many of our mirrors already don't use https. As I have said before, it seems likely to me that bundling a newer libcurl/libressl/openssl will fix more problems than it causes.

Users often don't care or even know why something doesn't work, they just know it doesn't work and they blame the most obvious source. MacPorts is broken on 10.8 and earlier when downloading from many servers, and many users have reported this to us. There are some edge cases where our buildbot does not mirror distfiles immediately or at all, and sometimes it's just busy with other tasks or even offline; during those times, with no mirror to fall back on, users can encounter fetch errors. We've even had the problem on our own buildbot workers for older OS versions. We've identified the cause: outdated OS libcurl/openssl libraries and CA bundle. We have a fix: bundle newer ones. This would fix the bug, increase user happiness because things will just work for them, and free up the time MacPorts developers would otherwise spend responding to user bug reports about this problem.

Your objection in comment:3 was that we should not distribute a CA bundle. Would bundling an ssl library require us to include CA bundle? Doesn't the OS ship with one that we could use? Or is that part of the problem—the old OS's CA bundle doesn't contain the information needed to trust new sites?

The latter, the CA bundle would eventually be too old for new certificates to validate.

Ok, then we could bundle a new CA that MacPorts libcurl/libressl/openssl would use, again possibly only for older systems that won't get such an update from Apple. We could even think about a way to update it via selfupdate that would be separate from MacPorts base releases, if making a user wait for a new MacPorts base release might be undesirable. Or we could find a way to have MacPorts base optionally use the CA from the curl-ca-bundle port.

#51045 was merely the first example of a curl bug that's fixed in newer versions that I found to link to this ticket. There's another cosmetic issue I found on Leopard that's been fixed. I'm sure there have been more bugs fixed in curl over the years.

Do you have specific tickets?

As I recall, curl on Leopard would emit a spurious error message during some downloads. As far as I remember this did not affect the ability to download the files. I'm not able to find a ticket or other reference to it as I don't remember exactly what the error message was so I can't search for it now.

Obviously this bug and the previously mentioned one are trivial but they are a reminder that there have probably been many bugs fixed in curl since the versions bundled with old macOS versions which our users would benefit from if we bundled a newer version. The benefit to us is being able to remove code from MacPorts base that works around those old bugs, thus simplifying our code and making it easier to maintain. (I currently see 3 different workarounds for bugs or missing features in various versions of curl in our curl.c.) We were able to do the same when we bundled Tcl 8.5 and could remove the Tcl 8.4 workarounds we had, as I mentioned above years ago.

I'm not convinced that we should bundle every library we use.

I'm not trying to propose that. This ticket is only about bundling curl and libressl/openssl with MacPorts. I believe this will benefit us and our users and I am not convinced that the drawbacks outweigh that.

comment:58 Changed 4 days ago by ryandesign (Ryan Schmidt)

Even just bundling a newer curl on 10.8 and earlier would help. I'm discovering that the curl that comes with 10.8 and earlier is linked with openssl 0.9.8 while the curl on 10.9 and later is linked with the Security framework. openssl 0.9.8 seems to be able to do nothing newer than SSLv3. However if I compile the curl port with the +darwinssl variant to make it use the Security framework, then it is able to do TLSv1 on 10.6 and 10.7 and TLSv1.2 on 10.8. That's enough to let it get files from GitHub and SourceForge on 10.8 (and to connect to TLSv1 sites on 10.6 and 10.7 that it couldn't otherwise connect to) without putting us into the business of bundling encryption software like openssl.

comment:59 Changed 4 days ago by kencu (Ken)

Edit: this post missed the point for what you were after, bundling curl without openssl. Deleted.

Last edited 4 days ago by kencu (Ken) (previous) (diff)
Note: See TracTickets for help on using tickets.