Opened 19 months ago

Last modified 7 months ago

#58800 new enhancement

Buildbot should wait until the clock catches up to the Portfiles

Reported by: ryandesign (Ryan Schmidt) Owned by: admin@…
Priority: Normal Milestone:
Component: buildbot/mpbb Version:
Keywords: Cc: chrstphrchvz (Christopher Chavez)
Port:

Description

Apparently some of the buildbot workers' clocks run a little fast. This has happened a few times: a buildbot worker goes down due to a power outage or for some other reason needs to be restarted, and after it restarts its clock re-syncs with the time server and now time has moved a few minutes backwards. If builds were queued up and start immediately, the builds fail with a message:

Error: Portfile is from the future - check date and time of your system

It might be nice if the buildbot would wait until the clock catches up with the timestamp of the Portfile before trying to build, though it needs to check the timestamps of all dependencies to be truly effective. It's probably simpler if we record the timestamp when we sync the ports tree (by touching some file, or maybe there is already some file that gets touched whose timestamp we can use), and then wait until the clock is equal to or greater than that timestamp before allowing the build to proceed.

Change History (9)

comment:1 Changed 19 months ago by mojca (Mojca Miklavec)

This problem very often appears on Travis as well.

comment:2 Changed 19 months ago by ryandesign (Ryan Schmidt)

Oh. Can you show a log example of that?

For buildbot I was thinking it would be enough to record the timestamp during one run, so that the next run (which might be after a reboot with fixed clock) could check it. But on Travis, where the environment is thrown away after every build, we might need to actually inspect files in the ports tree.

I do have a script somewhere that finds the newest modification date of any file in a directory. Maybe that's what we would need to do.

Alternately, since time on Travis is precious, maybe our strategy should be to identify Portfiles newer than now and touch them to bring their modification time back to now. Then we can proceed with a build immediately.

Or maybe we need to revisit why MacPorts checks Portfile timestamps in the first place. If this affects buildbot and Travis maybe it affects some users too.

comment:3 Changed 19 months ago by jmroot (Joshua Root)

The PortIndex's mtime will always be that of the newest Portfile. Proceeding despite evidence of an incorrect system clock is a bad idea because many build systems also rely on timestamps to work correctly.

comment:4 Changed 19 months ago by chrstphrchvz (Christopher Chavez)

Recent example log from one of my PRs: https://paste.macports.org/246941cbb5ff

Would it be safe to synchronize the clock at the beginning of a job?

comment:5 Changed 19 months ago by chrstphrchvz (Christopher Chavez)

Cc: chrstphrchvz added

comment:6 Changed 19 months ago by ryandesign (Ryan Schmidt)

By "synchronize the clock" do you mean do whatever the operating system is supposed to be doing on its own, but is apparently not doing often enough, to get the current time from ntp and set the system clock to that value, or do you mean what I proposed above? If the former, do you know how we could do that?

comment:7 Changed 19 months ago by mojca (Mojca Miklavec)

I might be wrong, but I can imagine that once you fire up a "virtual machine", the OS might not have yet successfully ran the clock syncronisation via NTP before the build starts. Given how often we see random network issue (on more or less the same builder than the one experiencing issues with time; the one running xcode 7), it could even be that the initial NTP sync fails because of network issues.

Yes, I would retry to sync via NTP. That said, if the network is having issues, this could fail as well.

comment:8 Changed 19 months ago by jmroot (Joshua Root)

AFAICT macOS doesn't actually run ntpd, which would be why the clock drift happens in the first place. Pre-Mojave you could run ntpdate to do a one-off sync, but that's gone in the latest ntp, with better alternatives now existing: https://support.ntp.org/bin/view/Dev/DeprecatingNtpdate#Functionality_List

comment:9 Changed 7 months ago by chrstphrchvz (Christopher Chavez)

In f9d70325451b35a34e62d8525c721d1c60c12d92/macports-ports (master):

bootstrap.sh: force VM clock to sync over NTP

See: #58800

Note: See TracTickets for help on using tickets.