mirror_sites.tcl: trimming and updating

[iefdev (Eric F)]

As requested: A separate ticket for trimming and updating the ​mirror_sites.tcl

Continues the discussion from: #60509

Including the last parts/replies:

---

Replying to ryandesign:

Speaking of the list. Except from trimming it, I think it needs an update. For example…

• ​the PHP mirrors are gone, aren't they? All redirects to www.php.net now, and there's no ​mirror.php page anymore.

• For ​debian, they also have their default one: ​http://ftp.debian.org/debian/pool/main/ (ie without a country code in it).

• I also saw a ​dead link. There's no unix section there anymore: ​http://ftp.sunet.se/pub/

<snip />

And, most mirrors use https now, don't they? Same example: ​http://ftp.sunet.se/pub/ -> ​https://ftp.sunet.se/pub/ When/if trimming the list – https sites should be prioritized. Incase some of them uses redirects (->https) it could also make it faster.

ryandesign (Ryan Schmidt):

Please file separate tickets for the issues with the mirror list ...

kencu (Ken):

all the https links will likely fail on older systems about 10.9 and less unless users have followed my bootstrap procedure.

ryandesign (Ryan Schmidt):

Not necessarily.

[iEFdev]

Replying to kencu (Ken):

all the https links will likely fail on older systems about 10.9 and less unless users have followed my bootstrap procedure.

Could ${distfiles_scheme}:// be utilized (more) for that? Like ​here.

[ryandesign (Ryan Carsten Schmidt)]

Different servers implement different requirements for https, by virtue of the list of SSL/TLS protocols they support, the list of encryption algorithms they support, and/or whether the certificate authority through which their certificate is issued is recognized by the system. GitHub and SourceForge for example both implemented requirements a few years ago that surpass the abilities of /usr/bin/curl on OS X 10.8 and earlier, but not all servers are that restrictive, and some are more restrictive than that.

The distfiles_scheme and packages_scheme variables aren't great, because they presume that the decision about whether to use http or https is always the same on an OS version. It is not; it varies by server. So there should really be separate variables for each server, for those that offer http and https, if that https is not compatible with all systems.

We use these variables only for official MacPorts mirrors. For other servers, if they offer https, we have been switching unconditionally to https in the interest of privacy, even if that https does not support all OS versions. In that case older OS versions may try to connect to those servers, fail, and then download successfully from another server, such as our mirrors; that's fine. Figuring out which OS versions support https on each of hundreds of other servers would be too tedious.

As for the other issues, thanks for pointing them out. I wasn't aware of them. You're correct for example that ​the php mirrors are gone and we need to adapt MacPorts accordingly. Regarding the dead link, it's inevitable that servers will go way or that they will stop hosting some content. There is no automated process in place to keep mirror_sites.tcl up to date; it requires someone like you to notice a problem and for it to be fixed manually. If you'd like to submit a PR to fix any or all of this that would be great, otherwise I'm sure someone will get to it eventually.

[iEFdev]

Replying to ryandesign:

As for the other issues, thanks for pointing them out. I wasn't aware of them. You're correct for example that ​the php mirrors are gone and we need to adapt MacPorts accordingly. Regarding the dead link, it's inevitable that servers will go way or that they will stop hosting some content. There is no automated process in place to keep mirror_sites.tcl up to date; it requires someone like you to notice a problem and for it to be fixed manually. If you'd like to submit a PR to fix any or all of this that would be great, otherwise I'm sure someone will get to it eventually.

So, strip down the mirrors to just one (php).

Just made a quick test to see if one could do a status test:

$ for site in $(sed -n '18,30p' mirror_sites.tcl); do \
> echo -n "${site}: " && curl -I ${site} 2>/dev/null | grep HTTP; done
https://mirror.aarnet.edu.au/pub/apache/: HTTP/1.1 200 OK
https://archive.apache.org/dist/: HTTP/1.1 200 OK
https://www.apache.org/dist/: HTTP/1.1 302 Found
http://mirror.cc.columbia.edu/pub/software/apache/: HTTP/1.1 200 OK
http://mirror.facebook.net/apache/: HTTP/1.1 200 OK
http://www.gtlib.gatech.edu/pub/apache/: HTTP/1.1 200 OK
http://mirrors.ibiblio.org/apache/: HTTP/1.1 200 OK
ftp://ftp.infoscience.co.jp/pub/net/apache/dist/:      <--- err/timeout

http://mirror.internode.on.net/pub/apache/: HTTP/1.1 200 OK
http://apache.is.co.za/: HTTP/1.1 200 OK
https://www.mirrorservice.org/sites/ftp.apache.org/: HTTP/1.1 200 OK
http://apache.pesat.net.id/: HTTP/1.1 403 Forbidden
http://apache.mirror.rafal.ca/: HTTP/1.1 200 OK

A timeout err doesn't tell if the site's gone or not. If not obvious, perhaps better to keep them?

—

And it looks like mySQL have removed their ​mirror page as well. Couldn't find any info about it though. Tested a random few mirrors… and ~ half are gone. I don't know what to make of that. I can remove the bad ones, but there's no way to ​generate a new/updated list.

When clicking on a download link:

https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-8.0.20.zip
  ---> https://cdn.mysql.com//Downloads/MySQL-8.0/mysql-8.0.20.zip (via Finder info)

So, maybe just use https://dev.mysql.com/get/Downloads/ as a single mirror then?

Well, I just wanted to take a quick look. Another day.

[iEFdev]

Replying to ryandesign:

There is no automated process in place to keep mirror_sites.tcl up to date; it requires someone like you to notice a problem and for it to be fixed manually. If you'd like to submit a PR to fix any or all of this that would be great, otherwise I'm sure someone will get to it eventually.

Ok, I made this one: ​https://github.com/macports/macports-ports/pull/7257

[Eric F <3958664+iefdev@…>]

In f99cd49cdc409be79cbfe48a44cd489c9215f995/macports-ports (master):

Update mirror_sites.tcl (​https://github.com/macports/macports-ports/pull/7257)

Remove dead URLs; add new ones.

Closes: #60558
See: #60509
Closes: #25674