package fetch does not honor preferred_hosts; fastly downloads throttled

[chrstphrchvz (Christopher Chavez)]

Specifying package sites for preferred_hosts in macports.conf has no effect (unless other sites are added to host_blacklist).

I would like preferred_hosts to be honored, because for me (using Google Fiber in San Antonio, TX), I can download from ywg.ca.packages.macports.org several times faster than from packages.macports.org (over 25Mbps versus 2.2Mbps); I do not care that the former has a longer ping time (60ms versus 10ms).

I want to think that this is not a difficult issue to fix if one knows where to look in the code.

[jmroot (Joshua Root)]

It seems to work fine for me. My pingtimes file shows 8.993 for packages.macports.org and 272.531 for ema.uk.packages.macports.org, and accordingly, packages are normally fetched from packages.macports.org. If I add ema.uk.packages.macports.org to preferred_hosts, I instead see this:

% sudo port -v archivefetch 7zip     
--->  Fetching archive for 7zip
--->  7zip-21.07_0.darwin_21.x86_64.tbz2 doesn't seem to exist in /opt/local/var/macports/incoming/verified
--->  Attempting to fetch 7zip-21.07_0.darwin_21.x86_64.tbz2 from https://ema.uk.packages.macports.org/7zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  990k  100  990k    0     0   350k      0  0:00:02  0:00:02 --:--:--  350k
--->  Attempting to fetch 7zip-21.07_0.darwin_21.x86_64.tbz2.rmd160 from https://ema.uk.packages.macports.org/7zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   512  100   512    0     0   1789      0 --:--:-- --:--:-- --:--:--  1790

[chrstphrchvz (Christopher Chavez)]

I currently cannot reproduce this issue. Please close this if it is believed to be invalid.

Note that other users have also observed slow speeds from fastly mirrors, as discussed recently on macports-dev: ​https://lists.macports.org/pipermail/macports-dev/2022-June/044314.html

[mascguy (Christopher Nielsen)]

Per Ryan's later comment, it sounds like we need to touch base with Fastly to validate our config/setup:

Yes, it has always been that slow for me as well, which is part of the reason why I had not formally announced the fact that we are using this CDN. I need to ask them if there is some mistake in our configuration that is causing this.

​https://lists.macports.org/pipermail/macports-dev/2022-June/044322.html

[mascguy (Christopher Nielsen)]

Ryan, any updates on this? Even if we can bump the speed up to 4 or 5 megabytes/sec, that would be a huge improvement!

[mascguy (Christopher Nielsen)]

Replying to mascguy:

Ryan, any updates on this? Even if we can bump the speed up to 4 or 5 megabytes/sec, that would be a huge improvement!

Alternatively, could we remove Fastly from the mix for now?

[ryandesign (Ryan Carsten Schmidt)]

I reached out to Fastly a few weeks ago. They were unable to replicate the slowness. I need to continue that conversation with them.

[mascguy (Christopher Nielsen)]

A few weeks ago, I blacklisted packages.macports.org, via macports.conf. And while that's not exactly ideal - if nothing else, mirroring to other sites is delayed vis-a-vis our primary site - it's a revelation: No more waiting endlessly for downloads, requiring me to interrupt via Ctrl-C [to select the next site].

I'm not suggesting it for everyone by any means, but it works for me...

[wryfi]

I consistently get terrible bandwidth from the fastly mirror. I noticed today that despite being located in California, I was downloading from an IP address in Indonesia. Something is definitely wrong with that. It's an embarrassment to both fastly and macports how slow the default mirror is. Not to mention how difficult it is to switch to a different mirror. I much prefer the design of MacPorts to the alternatives, but it's annoying how difficult (overcomplicated?) simple things like this often are.