Opened 2 years ago

#57343 new defect

mpbb mirror-distfiles repeatedly downloads distfiles with checksum mismatches

Reported by: ryandesign (Ryan Schmidt) Owned by: admin@…
Priority: Normal Milestone:
Component: buildbot/mpbb Version:
Keywords: Cc:


Consider this run of mpbb mirror-distfiles which ran after the qt58 port was modified to fix a typo. Upstream may have done a stealth update, because 31 of the subports' distfiles had checksum mismatches. But mpbb downloaded each files 12 times, because mpbb tries mirroring separately for each of our platform versions and archs. The 31 files totaled 339MB, downloaded 12 times is 4GB, which took 53 minutes, during which time no other builds could be started. And because we still haven't optimized our buildbot setup to trigger the mirror task only once per batch of commits, and instead we trigger a mirror task from each of the 12 portwatchers for each batch of commits, this 4GB 53-minute download was repeated 12 times.

The reason why we impersonate each platform while mirroring is so that we mirror files that a port only uses on a particular platform. But if a file fails to download or checksum properly for one platform, we should probably remember that and not try downloading that same file again for another platform during this mirroring run.

Change History (0)

Note: See TracTickets for help on using tickets.