nou.nc: "default" host for New Zealand: ping succeeds, HTTP times out

[ewen-naos-nz (Ewen McNeill)]

I'm based in New Zealand (.nz). At present it appears this is leading to nou.nc being the automatically selected mirror. However at least today nou.nc is pingable, but HTTP connections (at least from New Zealand) seem to just time out. (Which in turn is made worse by my ISP having a transparent HTTP proxy that fakes the connect and then hangs for a long time before it times out; but even testing via another ISP ping still succeeds and HTTP still fails to connect and/or return a result in a sensible amount of time.)

It's not clear to me how nou.nc is being chosen for New Zealand. While both New Caledonia and New Zealand are Pacific Islands they're a long distance apart, and the connections from New Caledonia to anywhere aren't amazing.

It's also not clear to me if nou.nc is even intended to be used from outside New Caledonia. If it isn't, perhaps it shouldn't be in the public list of mirrors?

Normally I'd expect Australia (eg, aarnet.au) is a much better default choice for New Zealand (it is physically and Internet-topology closer than New Caledonia). But oddly there seems to be a 160ms RTT jump on the NZ to aarnet.au path at present, via my normal ISP (but not via another test ISP). Eg, via my main ISP:

ewen@ashram:~$ ping -c 3 aarnet.au.packages.macports.org
PING mirror.aarnet.edu.au (202.158.214.106): 56 data bytes
64 bytes from 202.158.214.106: icmp_seq=0 ttl=51 time=181.136 ms
64 bytes from 202.158.214.106: icmp_seq=1 ttl=51 time=180.148 ms
64 bytes from 202.158.214.106: icmp_seq=2 ttl=51 time=179.894 ms

--- mirror.aarnet.edu.au ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 179.894/180.393/181.136/0.536 ms

And via another ISP in the same city (Wellington, NZ):

ewen@noc:~$ ping -c 3 aarnet.au.packages.macports.org
PING mirror.aarnet.edu.au (202.158.214.106) 56(84) bytes of data.
64 bytes from mirror.aarnet.edu.au (202.158.214.106): icmp_req=1 ttl=50 time=47.4 ms
64 bytes from mirror.aarnet.edu.au (202.158.214.106): icmp_req=2 ttl=50 time=47.0 ms
64 bytes from mirror.aarnet.edu.au (202.158.214.106): icmp_req=3 ttl=50 time=47.1 ms

--- mirror.aarnet.edu.au ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 47.022/47.226/47.496/0.320 ms

So possibly that's why nou.nc is being selected by the ping test -- it's more than 47ms away, but less than 180ms away:

ewen@ashram:~$ ping -c 3 nou.nc.packages.macports.org
PING pub.lagoon.nc (202.22.158.243): 56 data bytes
64 bytes from 202.22.158.243: icmp_seq=0 ttl=48 time=70.361 ms
64 bytes from 202.22.158.243: icmp_seq=1 ttl=48 time=67.023 ms
64 bytes from 202.22.158.243: icmp_seq=2 ttl=48 time=68.514 ms

--- pub.lagoon.nc ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 67.023/68.633/70.361/1.365 ms

via both ISPs.

Possibly some/all of the issue with nou.nc is that it's seeing a surprising amount of load due to aarnet.au having a non-trivial RTT latency to several paths, and thus not being selected for requests.

For now I'm attempting to work around this with ​host_blacklist in macports.conf. But it might be worth someone asking nou.nc if (a) they are actually allowing outside-country HTTP transfers and (b) if they want to be doing so, and (c) if they can actually cope with the load that gets diverted there when another mirror is "misbehaving".

Ewen

[ewen-naos-nz (Ewen McNeill)]

Relatedly, the ​configuration documentation seems vague as to what one actually puts in host_blacklist for instance to avoid a particular mirror. Eg nou.nc or the three variations on the name for rsync, distfiles and packages. For now I've listed all four, viz:

host_blacklist          nou.nc nou.nc.rsync.macports.org nou.nc.distfiles.macports.org nou.nc.packages.macports.org

which does seem to have worked to avoid nou.nc (and thus the timeouts).

Ewen

[ewen-naos-nz (Ewen McNeill)]

Heads-up email with pointer to this ticket sent to the email address listed for the nou.nc mirror on the Mirrors page.

FTR, with nou.nc blacklisted, New Zealand downloads currently come from the MaxCDN mirror (about 140ms away AFAICT, which means probably West-Coast North America).

Ewen

[ryandesign (Ryan Carsten Schmidt)]

MacPorts selects which distfiles and packages servers to use based on lowest ping time.

My understanding is that host_blacklist should contain the complete hostname(s) you want to blacklist. Not sure why that's not working for you.

The nou.nc server seems to be working for me, when I tested it just now from Texas.

Mirror issues should be reported to the administrator of the applicable mirror; thanks for doing so.

If you're curious which MaxCDN mirror server you're hitting, use dig packages.macports.org to find its IP address, then access its IP address in your web browser.

[ewen-naos-nz (Ewen McNeill)]

TL;DR: It appears nou.nc is now responding to HTTP requests again. I don't yet know why it wasn't before.

Thanks for the prompt response, and confirmation that host_blacklist should contain the full domain name.

The blacklist did work for me, in my "try all plausible options" list; I didn't do an A/B test of "short version" vs "long version" to figure out which was needed. Possibly the documentation and/or config file default could say "list the full domain name here".

The nou.nc also just worked for me now, from New Zealand. So, possibly someone was able to fix something, or possibly it's less loaded: so far I haven't heard back from the mirror admin either way yet.

Thanks for the "access MaxCDN node by IP to find out where it is" tip: You are hitting the MaxCDN Los Angeles Datacenter. So it is West Coast North America as I'd guessed from the RTT time.

Ewen

PS: I created a ticket in case others were experiencing what seemed to be a persistent problem (lasted at least an hour of me debugging it between doing other things). And because it wasn't obvious how to contact the mirror operator. ProTip: Turns out one needs to log into the MacPorts Wiki in order to see the email address to contact for the mirror operator. (Which I figured out only after I'd logged in and created the ticket... :-) )

[ewen-naos-nz (Ewen McNeill)]

Email from mirror operator, copied for reference of anyone else seeing this problem and searching for answers:

We had an outage on our NAS which has the files, the server is responding but cannot serve any files.

We are currently doing some maintenance on the NAS so it is not reliable at the moment.

So presumably there will be intermittent service for a bit until they're able to make the NAS reliable again.

Ewen

[ryandesign (Ryan Carsten Schmidt)]

Thanks for forwarding that to us. I've removed the nou.nc mirror for now in r148459. We can re-add it when it's reliable again.

[ewen-naos-nz (Ewen McNeill)]

Email last night from nou.nc Mirror operator:

NAS is now stable, but resyncing disk, so it will be a bit slow for 24hours at least

That "24 hours at least" is as of about 12 hours ago. So at a guess giving it another 12-24 hours should mean that the NAS disk resync is done.

Maybe look at enabling it again tonight, or tomorrow morning? (For whatever timezone that suits for doing that.) It is answering HTTP requests (with file contents) for me now.

Ewen

[tech@…]

Hi,

just letting you know that it is stable and synced if you want to close this :)

Regards

[ryandesign (Ryan Carsten Schmidt)]

Thanks. Mirror reinstated in r148837.