Common Mirror Consolidations

[Eric.C.Seidel@…]

jkh and I were discussing the feature of having common mirrors (i.e. for gnu, sf, etc) stored away somewhere outside of each individual port file. Not something important, but something useful..

Best idea at the end was to provide function calls such as:

[lowest-ping [master-site "gnu"]]

which woudl not only load a list of all the "gnu" mirrors, but would also do a ping test to find the closest one.

Here is a transcript of the converstation which ensued:

[00:56] <MacDome> jkh: ok. it would also be nice if we could just say "gnu/${package-name}" and then maintain a list of gnu mirrors somewhere... [00:58] <jkh> MacDome: yeah, this is an infrastructural request we get a lot [00:58] <fkr> MacDome, yes, I have the same idea about sourceforge [00:58] <jkh> MacDome: I made a stab at implementing it but wimped out before I finished. [00:59] <jkh> MacDome: FreeBSD ports does this too - they give you a generic variable which expands to various master sites, like MASTER_SITE_GNU, MASTER_SITE_KDE, and so on. We should do the same. [00:59] <jkh> MacDome: in addition, we should always add the distfiles cache directory at www.opendarwin.org to the end [00:59] <fkr> and fallback on distfiles.opendarwin.org [00:59] <jkh> right, distfiles.opendarwin.org even [01:01] <jkh> MacDome: as you can see, there's tons of infrastructure work left to be done [01:01] <jkh> MacDome: and Kevin and Landon are like the marines racing to Tikrit - no time to stop and clean up the messes left behind along the way. ;-) [01:08] <MacDome> jkh: it would make most sense to me to have a master_sites.type or master_sites.common = gnu, then the master_sites main variable could still be "extra" mirrors. [01:08] jkh [01:08] <MacDome> jkh: well, that just came to me now... not sure if that's "best" but at least knee-jerk. [01:09] * MacDome goes off to file feature request. [01:09] <jkh> MacDome: that's one way [01:09] <jkh> MacDome: but that gives you no control over ordering [01:09] <jkh> you could also make these functions now that I think about it [01:10] <jkh> [master-site gnu] [01:10] <jkh> which would expand to n strings [01:10] * fkr takes a look at Bison and Analog in order to commit them. [01:10] <jkh> unlike a variable which can only expand to one [01:10] <jkh> then you can stick the function call any place which makes sense in your master_sites list [01:10] <jkh> before, after, whatever. Even make it conditional if need be. [01:12] <MacDome> do we ping all the mirrors first to test speed? or just take the first one? [01:12] <fkr> first one. [01:12] <fkr> and imho we should stick to that. [01:12] <MacDome> I guess that's good and bad. [01:12] <fkr> since then the Portfile author control the order. [01:13] <MacDome> but then mirrors don't really serve as mirrors [01:13] <MacDome> just backups. [01:13] <jkh> well, that's sort of why I'm liking more and more the idea of providing useful helper functions [01:13] <MacDome> a ping -c1 to each wouldn't cost much... [01:13] <jkh> since then the portfile author is free to use or not use them as they see fit for specific circumstances [01:14] <jkh> then you just wrap them together: [lowest-ping [master-site "gnu"]] would select a single site from a list of GNU mirrors based on how fast it was relative to the user using it [01:14] <jkh> and for GNU mirrors, that might make total sense given the topology the FSF has put together [01:15] <jkh> but not necessarily for other repositories [01:15] <jkh> some will have their own akamai-style load balancing, for example, or n mirrors which are all equally barfworthy and it just doesn't matter in their case. Heck, in that case you might want to do "[random [master-site "barforge"]]" :-)

[landonf@…]

master sites has been committed.