Opened 13 years ago

Closed 13 years ago

Last modified 12 years ago

#14891 closed enhancement (fixed)

PATCH: use the fastest mirror in fetch phase

Reported by: jmroot (Joshua Root) Owned by: macports-tickets@…
Priority: Normal Milestone: MacPorts 1.7.0
Component: base Version: 1.7.0
Keywords: Cc: raimue (Rainer Müller)
Port:

Description

Here's a patch which makes the fetch phase ping each of the candidate download sites, and then sort the list in ascending order of ping time. This is my first attempt at writing Tcl, so no doubt it could be done better, and I welcome any advice.

Some issues that need to be thought about:

  • We probably want to special-case some mirrors. We always want to try MacPorts svn last, and the primary site for each mirror group should be second-last.
  • Caching. Is it worth the effort? Maybe pinging every mirror every time is fine. When do we refresh cache entries?
  • User hints/overrides. If the user tells us that certain mirrors are good, we should always try them first when eligible.

Attachments (5)

sortsites.diff (1.6 KB) - added by jmroot (Joshua Root) 13 years ago.
sortsites2.diff (1.8 KB) - added by raimue (Rainer Müller) 13 years ago.
sortsites3.diff (2.6 KB) - added by jmroot (Joshua Root) 13 years ago.
sortsites4.diff (2.8 KB) - added by raimue (Rainer Müller) 13 years ago.
sortsites4.2.diff (2.8 KB) - added by raimue (Rainer Müller) 13 years ago.

Download all attachments as: .zip

Change History (16)

comment:1 Changed 13 years ago by wsiegrist@…

I tested with and without the patch on 2 random ports: less and php5.

less: the svn.macosforge.org sites won and were tried first, even though they dont have the files. You should probably exclude macosforge and macports sites from the ping test until we have mirroring going.

php5: it took longer to fetch because it waited up to 5s for each mirror it seems. Maybe try to find a winner with "-t1", then "-t5" to minimize the impact of downed mirrors.

Overall I like the idea though.

Changed 13 years ago by jmroot (Joshua Root)

Attachment: sortsites.diff added

comment:2 Changed 13 years ago by jmroot (Joshua Root)

Updated the patch to try svn.macports.org last, ping with only one packet, and use a 1-second timeout in the interest of speed. Slow mirrors won't be sorted in quite the correct order, but I don't think that that's a real problem. Working mirrors that just don't respond to ping will go last in any case, but again, things should still generally be better with the patch than without.

It would be ideal to run all the pings in parallel, but I don't know how to do that.

Changed 13 years ago by raimue (Rainer Müller)

Attachment: sortsites2.diff added

comment:3 Changed 13 years ago by raimue (Rainer Müller)

Added 'sortsites2.diff‘; an updated version of the patch which avoids hardcoding 'svn.macports.org', but uses the fallback_mirror_list where this host actually comes from.

comment:4 Changed 13 years ago by jmroot (Joshua Root)

Updated: don't re-ping hosts that have already been pinged. Saves a lot of time in cases where hosts appear in the list several times.

comment:5 Changed 13 years ago by jmroot (Joshua Root)

Updated again: ping all hosts in parallel.

comment:6 Changed 13 years ago by wsiegrist@…

Latest patch works well for me.

Changed 13 years ago by jmroot (Joshua Root)

Attachment: sortsites3.diff added

comment:7 Changed 13 years ago by jmroot (Joshua Root)

Updated: Recognise invalid hostnames like image_magick.veidrodis.com. (Sigh.) Check that the output from ping can be parsed as a number. Switched to using 3 ping packets and a 3-second timeout, since we effectively only have to wait for it once now that the pings run in parallel. Also simplified the code a bit.

Changed 13 years ago by raimue (Rainer Müller)

Attachment: sortsites4.diff added

Changed 13 years ago by raimue (Rainer Müller)

Attachment: sortsites4.2.diff added

comment:8 Changed 13 years ago by raimue (Rainer Müller)

Updated sortsites4.diff: If there is only one host (besides the fallbacks), don't ping or sort.

comment:9 Changed 13 years ago by raimue (Rainer Müller)

Cc: raimue@… added

This patch works just great. I encourage you to commit it, jmr!

comment:10 Changed 13 years ago by jmroot (Joshua Root)

Resolution: fixed
Status: newclosed

Committed in r35748. Thanks for the help!

comment:11 Changed 12 years ago by blb@…

Milestone: MacPorts base enhancementsMacPorts 1.7.0
Note: See TracTickets for help on using tickets.