Opened 13 months ago

Last modified 12 months ago

#54800 new enhancement

Cache built dependency packages on Travis CI to speed up testing flow

Reported by: yan12125 (Chih-Hsuan Yen) Owned by: admin@…
Priority: Normal Milestone:
Component: server/hosting Version:
Keywords: Cc: mojca (Mojca Miklavec), l2dy (Zero King)
Port:

Description

Motivation

I just submitted a new port libfilezilla. Builds on Travis CI timed out after 48 minutes [1] while on buildbot it takes about 8 minutes only [2]. I believe what @raimue said [3] - time to build non-redistributable dependencies - is the key difference. I hope we can have a caching mechanism so that newcomers can know if their patches are OK or not quickly, and maintainers can spend less time on testing patches manually.

Proposal

Travis CI provides a simple caching mechannism [4]. If built packages under $PREFIX/var/macports/software are cached, there's no need to rebuild them every time.

Cached data are only accessible to Travis CI builders. There are no URLs to cached files in build logs. I guess it's not a "redistribution" as other people can't access them. Thus there are no licensing issues.

[1] https://travis-ci.org/macports/macports-ports/builds/273323633

[2] https://build.macports.org/builders/ports-10.12_x86_64-builder/builds/38131

[3] https://github.com/macports/macports-ports/pull/756#issuecomment-328345604

[4] https://docs.travis-ci.com/user/caching/

Change History (6)

comment:1 Changed 13 months ago by mojca (Mojca Miklavec)

Cc: mojca added

comment:2 Changed 13 months ago by mojca (Mojca Miklavec)

Cc: l2dy added

comment:3 Changed 13 months ago by l2dy (Zero King)

  1. $PREFIX is not writable by the user (https://docs.travis-ci.com/user/caching/#caches-and-read-permissions). But this is easy to workaround by moving files over and fixing permissions.
  2. In order to cache dependencies across PRs, we have to enable builds on the master branch. To avoid timeouts, we can only build ports changed in the last commit pushed. (When many unrelated commits are pushed at once, only one Travis build will be triggered.)
  3. We should not cache distributable packages and purge outdated cached packages to avoid making the cache too big. Extracting the cache (and maybe generating it) costs precious build time.

After the first pull request build is run, it creates a new pull request cache.

  1. We have to remove cache of ports to be tested in PRs.
Last edited 13 months ago by l2dy (Zero King) (previous) (diff)

comment:4 Changed 13 months ago by yan12125 (Chih-Hsuan Yen)

Thanks for comments. Here are some numbers for the caching speed. In a recent build of my project, it takes 19.37s to download and 11.05s to extract a 3554.06MB cache. Regenerating the cache takes 65.66s. Indeed huge caches bring problems.

comment:5 Changed 12 months ago by l2dy (Zero King)

Travis CI cache is not suitable for this (hard to implement, too many constraints).

Hosting these binaries on a private (limiting access to Travis CI VMs only) server technically works but I'm not a lawyer so...

comment:6 Changed 12 months ago by mojca (Mojca Miklavec)

That's not a question for lawyers. Webtheoretically have the same issue with buildbot in case a builder runs out of space. It would probably make sense to store packages on a private server both for Travis and building.

Note: See TracTickets for help on using tickets.