Opened 5 years ago

Closed 12 months ago

#50875 closed defect (wontfix)

mirror-macports.sh runs daily but takes > 24 hours to complete

Reported by: ryandesign (Ryan Schmidt) Owned by: admin@…
Priority: Normal Milestone:
Component: server/hosting Version:
Keywords: Cc: jmroot (Joshua Root), neverpanic (Clemens Lang)
Port:

Description

The browser:trunk/base/portmgr/jobs/mirror_macports.sh script runs every day at 2:15 am on rsync-origin.macports.org, but because of the number of ports we have and because it tries to mirror each port for 8 different OS X versions (darwin 8 through darwin 15), it now takes more than 24 hours to complete.

When a file is not found or the checksum doesn't match, the fetch is attempted 8 times, one for each OS X version, wasting time and bandwidth.

It mirrors for each version of OS X separately because a small subset of ports declare different distfiles for different OS X versions. However the vast majority of ports don't do that.

I think it's also supposed to be mirroring for each variant, since some variants might add distfiles.

We should find a way to rewrite the script so that it gets a list of URLs to mirror for all permutations of the port, but only attempts to fetch from each URL once.

The script on the server has been updated quite a bit from the version currently in the repository, and there are changes in the repository that have not been reflected on the server yet.

Change History (5)

comment:1 in reply to:  description Changed 5 years ago by ryandesign (Ryan Schmidt)

Replying to ryandesign@…:

The browser:trunk/base/portmgr/jobs/mirror_macports.sh script runs every day at 2:15 am on rsync-origin.macports.org, but because of the number of ports we have and because it tries to mirror each port for 8 different OS X versions (darwin 8 through darwin 15), it now takes more than 24 hours to complete.

I cut it back to running only twice a week, on Wednesday and Saturday mornings.

When a file is not found or the checksum doesn't match, the fetch is attempted 8 times, one for each OS X version, wasting time and bandwidth.

It mirrors for each version of OS X separately because a small subset of ports declare different distfiles for different OS X versions. However the vast majority of ports don't do that.

I think it's also supposed to be mirroring for each variant, since some variants might add distfiles.

We should find a way to rewrite the script so that it gets a list of URLs to mirror for all permutations of the port, but only attempts to fetch from each URL once.

I'm working on a new script.

The script on the server has been updated quite a bit from the version currently in the repository, and there are changes in the repository that have not been reflected on the server yet.

comment:2 Changed 5 years ago by ryandesign (Ryan Schmidt)

Really, we shouldn't need to mirror all ports' files very often. That's just a safeguard against what should be the primary mirroring method, which is mirroring a port's files right before the buildbot is going to build it.

comment:3 Changed 5 years ago by ryandesign (Ryan Schmidt)

I have written a new mirroring script but have not deployed it yet because I haven't done a full test run of it yet.

comment:4 Changed 4 years ago by ryandesign (Ryan Schmidt)

Owner: changed from admin@… to admin@…
Status: newassigned

comment:5 Changed 12 months ago by ryandesign (Ryan Schmidt)

Resolution: wontfix
Status: assignedclosed

We no longer use mirror-macports.sh. We now have a mirror-distfiles command in mpbb which we use in our buildbot setup.

Note: See TracTickets for help on using tickets.