Mirroring of distfiles hosted on GitHub is failing

[jeff-kelley (j.y.bernier)]

(Please, see log in attached file since inline urls trigger the spam filter).

qpdf-12.3.0.tar.gz seems not have made its way to the repos, where latest version is 12.2.0

OS X version is Mavericks if this is relevant.

[jeff-kelley (j.y.bernier)]

Attachment rejected by the spam filter.

It shows a 404 error on all mirrors tried and ends in :

Error: Failed to fetch qpdf: The requested URL returned error: 404 Not Found

[jmroot (Joshua Root)]

​https://build.macports.org/builders/jobs-mirror/builds/1013778

See also thread starting here: ​https://lists.macports.org/pipermail/macports-dev/2026-January/046607.html

[jmroot (Joshua Root)]

A solution for this is in the works. Depending on how things go, it may take up to another few days.

[ryandesign (Ryan Carsten Schmidt)]

For now, I've manually recompiled the copy of MacPorts that's used to mirror distfiles using the --with-curlprefix option pointed to a newer curl. This fix might have to be redone after the automatic selfupdate that will happen when a new version of MacPorts is released.

Perhaps someone else could identify all the affected ports and ​schedule new builds on the affected portwatchers. I've already done openssl3 (which built successfully) and qpdf (which cannot be compiled).

Also, several of the buildmaster's steps were using Apple git to update their repositories which was failing for the same reason. I changed this to use a newer MacPorts git as well.

[fhgwright (Fred Wright)]

Here I also encountered this with cppcheck, p11-kit, and pinentry-mac, but that's based solely on what I have installed here. And I can't reschedule anything. :-)

[Dave-Allured (Dave Allured)]

Yes I have been seeing this with many of my port updates for the past week or so, for builders <= 10.13. IMO it is no longer worth discussing or listing them. I am quite looking forward to the "solution in the works". After that, a mass automated catch-up will probably be needed.

[fhgwright (Fred Wright)]

Apparently this is not a certificate issue as was originally assumed, but instead due to GitHub's raising the bar on TLS versions. That doesn't change the fact that the current MacPorts curl works. It does make it harder to determine the exact starting date of the trouble.

Replying to Dave-Allured:

Yes I have been seeing this with many of my port updates for the past week or so, for builders <= 10.13. IMO it is no longer worth discussing or listing them. I am quite looking forward to the "solution in the works". After that, a mass automated catch-up will probably be needed.

The "<=10.13" part sounds suspect, since earlier indications were that 10.12+ is OK. What I actually see now is that 10.11-10.12 fail while 10.13-10.14 work.

[Dave-Allured (Dave Allured)]

Fred, here is a 10.13 certificate fail from yesterday. 10.14 and higher were fetching okay. The target repo is gitlab.dkrz.de, not Github. That may explain the perceived difference in failed builder versions.

​https://build.macports.org/builders/ports-10.13_x86_64-builder/builds/275461

[Dave-Allured (Dave Allured)]

Builder fail, 10.13, port libaec

[Dave-Allured (Dave Allured)]

I believe that the overall mirroring problem is not Github-specific, as was just demonstrated by this libaec fail. I think that there is a fault in the current MacPorts mirror process, such that the first encountered certificate fail results in the port marked "mirror completed" in the ports database. This means that later successful fetches by more updated builders are blocked from completing the missing mirrors.

Admins, I hope that the solution in the works will include a fix for this logic error.

[fhgwright (Fred Wright)]

Replying to Dave-Allured:

Fred, here is a 10.13 certificate fail from yesterday. 10.14 and higher were fetching okay. The target repo is gitlab.dkrz.de, not Github. That may explain the perceived difference in failed builder versions.

​https://build.macports.org/builders/ports-10.13_x86_64-builder/builds/275461

That could be. The test I mentioned above only targeted GitHub.

Replying to Dave-Allured:

I believe that the overall mirroring problem is not Github-specific, as was just demonstrated by this libaec fail. I think that there is a fault in the current MacPorts mirror process, such that the first encountered certificate fail results in the port marked "mirror completed" in the ports database. This means that later successful fetches by more updated builders are blocked from completing the missing mirrors.

Admins, I hope that the solution in the works will include a fix for this logic error.

I'm not sure about that. Fetching libaec directly doesn't work on 10.9, and that doesn't involve tripping over a GitHub fetch. But GitHub isn't the only site with issues.

I created (via a somewhat crude method) a list of ports with version changes in the relevant time period. For many of these ports, fetching with --no-mirror still fails on 10.9 (sometimes with certificate errors and sometimes with protocol errors), but they all now work without --no-mirror, so the mirrors seem to have been corrected.

[jmroot (Joshua Root)]

Ryan's interim solution is in place for the Buildbot, and the GitHub Actions based system is now being used to update the distfiles on the public mirrors. New builds are in progress for ports updated since the end of December. There's more to do behind the scenes, but I think it's fair to call this fixed.

[Dave-Allured (Dave Allured)]

Good work, Josh, Ryan, et al. Thank you.

[jeff-kelley (j.y.bernier)]

Working again. Thanks.

[ryandesign (Ryan Carsten Schmidt)]

Replying to Dave-Allured:

I believe that the overall mirroring problem is not Github-specific, as was just demonstrated by this libaec fail. I think that there is a fault in the current MacPorts mirror process, such that the first encountered certificate fail results in the port marked "mirror completed" in the ports database. This means that later successful fetches by more updated builders are blocked from completing the missing mirrors.

There is no general fault in the mirroring process, except that mirroring has been happening on the buildmaster which runs OS X 10.11, and MacPorts uses Apple's curl by default, and Apple's curl on 10.11 uses SecureTransport which doesn't support TLS 1.3, so it cannot connect to GitHub anymore now that they've changed their server configuration to require TLS 1.3 (or whatever change they made).

This ticket was specific to GitHub in that its popularity meant something had to be done to work around it on our end urgently. Other servers may also be unable to be reached from old Apple curl versions depending on their server settings and the workaround should fix that as well.

[Dave-Allured (Dave Allured)]

except that mirroring has been happening on the buildmaster which runs OS X 10.11

Thanks for explaining. I had the wrong notion of where the mirroring was running from.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign:

For now, I've manually recompiled the copy of MacPorts that's used to mirror distfiles using the --with-curlprefix option pointed to a newer curl. This fix might have to be redone after the automatic selfupdate that will happen when a new version of MacPorts is released.

I did not survive the selfupdates to 2.12.0 and 2.12.1. I've redone the fix now with MacPorts 2.12.1.

Given how often and how widely we tell users to run selfupdate, it's surprising that it does not preserve the user's configure arguments.

[jmroot (Joshua Root)]

Replying to ryandesign:

Given how often and how widely we tell users to run selfupdate, it's surprising that it does not preserve the user's configure arguments.

That might be worth an enhancement ticket.

For the buildbot workers, my idea for a permanent solution is to set up a proxy that converts http to https to allow accessing the files mirrored by GitHub Actions.

[ryandesign (Ryan Carsten Schmidt)]

I've redone the fix now for MacPorts 2.12.2.