mirror-macports.sh runs daily but takes > 24 hours to complete

[ryandesign (Ryan Carsten Schmidt)]

The browser:trunk/base/portmgr/jobs/mirror_macports.sh​ script runs every day at 2:15 am on rsync-origin.macports.org, but because of the number of ports we have and because it tries to mirror each port for 8 different OS X versions (darwin 8 through darwin 15), it now takes more than 24 hours to complete.

When a file is not found or the checksum doesn't match, the fetch is attempted 8 times, one for each OS X version, wasting time and bandwidth.

It mirrors for each version of OS X separately because a small subset of ports declare different distfiles for different OS X versions. However the vast majority of ports don't do that.

I think it's also supposed to be mirroring for each variant, since some variants might add distfiles.

We should find a way to rewrite the script so that it gets a list of URLs to mirror for all permutations of the port, but only attempts to fetch from each URL once.

The script on the server has been updated quite a bit from the version currently in the repository, and there are changes in the repository that have not been reflected on the server yet.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign@…:

The browser:trunk/base/portmgr/jobs/mirror_macports.sh​ script runs every day at 2:15 am on rsync-origin.macports.org, but because of the number of ports we have and because it tries to mirror each port for 8 different OS X versions (darwin 8 through darwin 15), it now takes more than 24 hours to complete.

I cut it back to running only twice a week, on Wednesday and Saturday mornings.

When a file is not found or the checksum doesn't match, the fetch is attempted 8 times, one for each OS X version, wasting time and bandwidth.

It mirrors for each version of OS X separately because a small subset of ports declare different distfiles for different OS X versions. However the vast majority of ports don't do that.

I think it's also supposed to be mirroring for each variant, since some variants might add distfiles.

We should find a way to rewrite the script so that it gets a list of URLs to mirror for all permutations of the port, but only attempts to fetch from each URL once.

I'm working on a new script.

The script on the server has been updated quite a bit from the version currently in the repository, and there are changes in the repository that have not been reflected on the server yet.

[ryandesign (Ryan Carsten Schmidt)]

Really, we shouldn't need to mirror all ports' files very often. That's just a safeguard against what should be the primary mirroring method, which is mirroring a port's files right before the buildbot is going to build it.

[ryandesign (Ryan Carsten Schmidt)]

I have written a new mirroring script but have not deployed it yet because I haven't done a full test run of it yet.

[ryandesign (Ryan Carsten Schmidt)]

We no longer use mirror-macports.sh. We now have a mirror-distfiles command in mpbb which we use in our buildbot setup.