Opened 7 years ago

Closed 6 years ago

Last modified 6 years ago

#53347 closed defect (fixed)

distfile mirrors not automatically updating

Reported by: GregoryEAllen (Greg Allen) Owned by: admin@…
Priority: Normal Milestone:
Component: server/hosting Version:
Keywords: Cc: noloader (Jeffrey Walton), mojca (Mojca Miklavec), bK4gYuRo, l2dy (Zero King), cjones051073 (Chris Jones), iEFdev, ccorn, khepler, fvaccari, 1-61803, fracai
Port:

Description

Ryan Schmidt wrote:

We have not yet brought automatic distfile mirroring online on the new server after moving away from macOS forge.

Change History (25)

comment:1 Changed 7 years ago by ryandesign (Ryan Carsten Schmidt)

Cc: noloader added

Has duplicate #53423.

To explain: many sites now only offer downloads via https, and the versions of curl and openssl in Leopard and Tiger are too old to be able to communicate with modern https sites, so fetching the files from the original site fails. This wouldn't be a problem if we had our http distfile mirroring up and running, so we need to get it back online again.

comment:2 in reply to:  1 ; Changed 7 years ago by noloader (Jeffrey Walton)

Replying to ryandesign:

... To explain: many sites now only offer downloads via https, and the versions of curl and openssl in Leopard and Tiger are too old to be able to communicate with modern https sites, so fetching the files from the original site fails. This wouldn't be a problem if we had our http distfile mirroring up and running, so we need to get it back online again.

If it helps, I keep a handful of utilities up-to-date in /usr/local/bin as a fallback. They include cURL, OpenSSL, SCP, SSH and Wget. They all use static linking to avoid dll hell and versioning problems.

Maybe MacPorts could check for an updated (more recent?) version in /usr/local. Or, allow me to specify an override in /opt/local/etc/macports/macports.conf:

curl_bin     = /usr/local/bin/curl
openssl_bin  = /usr/local/bin/openssl
wget_bin     = /usr/local/bin/wget

comment:3 Changed 7 years ago by kencu (Ken)

be careful. /usr/local/include and /usr/local/lib are often on the search paths for macports compilers. Strange and hard-to-debug compiler and build errors can occur if the wrong include or lib files are found by macports compilers during the build. Builds can fail with weird errors that make no sense.

You might be best to put your own files somewhere like ~/bin where nothing will go looking for them. Up to you.

comment:4 in reply to:  2 Changed 7 years ago by raimue (Rainer Müller)

Replying to noloader:

If it helps, I keep a handful of utilities up-to-date in /usr/local/bin as a fallback. They include cURL, OpenSSL, SCP, SSH and Wget. They all use static linking to avoid dll hell and versioning problems.

This ticket is about getting infrastructure ready to do the automatic mirroring on the MacPorts buildbot instance. For the problems with HTTPS and an old libcurl, please see #51516.

comment:5 Changed 7 years ago by GregoryEAllen (Greg Allen)

I use distfiles because I work on networks that can't be directly connected to the internet, which is common in government research labs. Mirrors are fairly easy to set up and manage, but direct access is disallowed. I'm currently stuck with manually copying packages via "sneakernet".

comment:6 Changed 7 years ago by mojca (Mojca Miklavec)

Cc: mojca added

comment:7 Changed 7 years ago by bK4gYuRo

Cc: bK4gYuRo added

comment:8 Changed 7 years ago by l2dy (Zero King)

Cc: l2dy added

comment:9 Changed 6 years ago by kencu (Ken)

I'm looking to understand why distfile mirroring doesn't work at present. Was this feature implemented as part of MacPorts by using the portmirror:mirror_main function in /opt/local/libexec/macports/lib/port1.0/portmirror.tcl? If so, I don't immediately see why it is not working...

Or was there perhaps a specific buildbot with a current curl that mirrored it's distfiles to some main repository source mirror using a cron/rsync script outside of macports?

comment:10 Changed 6 years ago by jmroot (Joshua Root)

Running 'port mirror' works fine, it just isn't set up to happen automatically on distfiles.macports.org.

comment:11 Changed 6 years ago by kencu (Ken)

OK, so that part is working.

I take it there used to exist a cron job running a script on a similarly-named server that ran the whole list of portnames through port mirror portname every once in a while?

And we are missing either that script (it must still exist somewhere, I suspect), and the cron job entry? This is sounding encouraging...

comment:12 Changed 6 years ago by kencu (Ken)

Could it be that this cronjob is all we need on the distfiles.macports.org server?

30 * * * * /usr/local/bin/port -p mirror all 2>&1 >> /var/log/macports-mirror.log

Version 0, edited 6 years ago by kencu (Ken) (next)

comment:13 Changed 6 years ago by jmroot (Joshua Root)

No, that's not sufficient. Mirroring all ports takes too long, we aren't allowed to mirror some ports, and we have to make sure that other jobs don't update the copy of macports being used while mirroring is in progress. We also ideally want to mirror extra distfiles that may be added by variants.

The macports-infrastructure repo has the work-in-progress code.

comment:14 Changed 6 years ago by kencu (Ken)

Got it -- it would appear this is the script that needs to be hit by the cron job. macports-infrastructure/jobs/mirror_macports.sh.

Last edited 6 years ago by ryandesign (Ryan Carsten Schmidt) (previous) (diff)

comment:15 Changed 6 years ago by ryandesign (Ryan Carsten Schmidt)

The old macOS forge infrastructure mirrored distfiles in a Subversion post-commit hook, prior to sending the commit to the buildbot. Thus, the buildbot workers could download their distfiles from our mirror. This is what we want to achieve again, except now we're using GitHub, so we don't have a local post-commit hook script, and instead we get notifications from GitHub which trigger the buildbot. So we want to create a buildbot task that mirrors the files prior to scheduling the port-building task, or we want to have the port-building tasks depend on the mirroring task. Buildbot tasks occur in what the Buildbot project calls "builders", and that builder needs to be running on the server hosting the distfiles. We currently don't have a Buildbot builder running there, so I need to set one up, and decide what user to run that as. That builder also needs a copy of MacPorts and the ports tree, and for that copy of MacPorts and the ports tree to be kept up to date; I don't think we've yet solved how that happens.

comment:16 Changed 6 years ago by kencu (Ken)

It appears that most of this logic is in buildbot/master.cfg. There are entries in there that appear to reference the mirror stage.

if 'mirror' in config['deploy']:
    jobs_mirror_factory = util.BuildFactory()
    jobs_mirror_factory.addStep(steps.ShellCommand(
        command=util.WithProperties('port mirror %(portname)s'),
        name='mirror',
        description=['mirroring'],
        descriptionDone=['done']))

comment:17 Changed 6 years ago by cjones051073 (Chris Jones)

Cc: cjones051073 added

comment:18 Changed 6 years ago by iEFdev

Cc: iEFdev added

comment:19 Changed 6 years ago by ccorn

Cc: ccorn added

comment:20 Changed 6 years ago by khepler

Cc: khepler added

comment:21 Changed 6 years ago by fvaccari

Cc: fvaccari added

comment:22 Changed 6 years ago by 1-61803

Cc: 1-61803 added

comment:23 Changed 6 years ago by fracai

Cc: fracai added

comment:24 Changed 6 years ago by ryandesign (Ryan Carsten Schmidt)

Resolution: fixed
Status: newclosed

Automatic distfile mirroring is online on our buildbot now. It happens at commit time, so as ports are updated from now on, their files (and those of their dependencies) will be mirrored.

Existing ports that are not modified and that are not dependencies of modified ports will not be automatically mirrored. To partially address this, I will do some batch modifications of ports that I've been putting off until mirroring was fixed. I will also attempt to add the ability for a developer to force a port's files to be mirrored by the buildbot (without forcing the buildbot to build the port). I'll then investigate the feasibility of initiating a one-time mirroring of all ports.

comment:25 Changed 6 years ago by kencu (Ken)

Thanks very much to all for the effort involved in this. It is much appreciated.

Note: See TracTickets for help on using tickets.