Cache built dependency packages on Travis CI to speed up testing flow

[yan12125 (Chih-Hsuan Yen)]

Motivation

I just submitted a new port libfilezilla. Builds on Travis CI timed out after 48 minutes [1] while on buildbot it takes about 8 minutes only [2]. I believe what @raimue said [3] - time to build non-redistributable dependencies - is the key difference. I hope we can have a caching mechanism so that newcomers can know if their patches are OK or not quickly, and maintainers can spend less time on testing patches manually.

Proposal

Travis CI provides a simple caching mechannism [4]. If built packages under $PREFIX/var/macports/software are cached, there's no need to rebuild them every time.

Cached data are only accessible to Travis CI builders. There are no URLs to cached files in build logs. I guess it's not a "redistribution" as other people can't access them. Thus there are no licensing issues.

[1] ​https://travis-ci.org/macports/macports-ports/builds/273323633

[2] ​https://build.macports.org/builders/ports-10.12_x86_64-builder/builds/38131

[3] ​https://github.com/macports/macports-ports/pull/756#issuecomment-328345604

[4] ​https://docs.travis-ci.com/user/caching/

[l2dy (Zero King)]

1. $PREFIX is not writable by the user (​https://docs.travis-ci.com/user/caching/#caches-and-read-permissions). But this is easy to workaround by moving files over and fixing permissions.
2. In order to cache dependencies across PRs, we have to enable builds on the master branch. To avoid timeouts, we can only build ports changed in the last commit pushed. (When many unrelated commits are pushed at once, only one Travis build will be triggered.)
3. We should not cache distributable packages and purge outdated cached packages to avoid making the cache too big. Extracting the cache (and maybe generating it) costs precious build time.

After the first pull request build is run, it creates a new pull request cache.

4. We have to remove cache of ports to be tested in PRs.

[yan12125 (Chih-Hsuan Yen)]

Thanks for comments. Here are some numbers for the caching speed. In a recent build of my project, it takes 19.37s to download and 11.05s to extract a 3554.06MB cache. Regenerating the cache takes 65.66s. Indeed huge caches bring problems.

[l2dy (Zero King)]

Travis CI cache is not suitable for this (hard to implement, too many constraints).

Hosting these binaries on a private (limiting access to Travis CI VMs only) server technically works but I'm not a lawyer so...

[mojca (Mojca Miklavec)]

That's not a question for lawyers. We theoretically have the same issue with buildbot in case a builder runs out of space. It would probably make sense to store packages on a private server both for Travis and building.