Reproducible Builds
Reproducible Builds have the goal to verify the required steps from the source code to the binary. Multiple open source software projects joined together to apply existing techniques to achieve deterministic builds in popular software distributions.
Software projects involved in Reproducible Builds met in Athens in December 2015. Detailed information can be found in Clemens' report.
Reproducible Builds in MacPorts
Port builds with MacPorts should be repeatable and predictable. That means that if a user installs a port today, and another user installs the port tomorrow, and the Portfile has not been changed in between, both users should get the same software (allowing of course for possible differences due to architecture or OS X version).
Compiler
Ports should use the right compiler.
Dependencies
When a software package has a mandatory dependency on another package, and that other package is missing, it's easy to see, because usually the configure phase will exit with an error explaining what was missing, and then that dependency can be added to the portfile. But some software packages have optional dependencies: they'll use the dependency if it's already installed, but won't complain if it's not installed; they'll just omit that functionality. These situations are easier for a maintainer to inadvertently overlook, and if so, some users who install the port will have the optional functionality and some won't, based on whether they happened to already have the optional dependency installed. If the optional dependency's library version number increases, that may even cause the port to be broken until it's rebuilt. Or if the port is distributable, then they may get the binary built by our buildbot, which doesn't have any extraneous ports installed, so any optional functionalities won't be there.
We don't want this, so when such a situation is discovered, either the dependency should be unconditionally added to the portfile and the revision increased, or the software should be informed to never use the optional functionality, even if the optional dependency is already installed. In the latter case, a variant could also be added to let the user choose whether to add the functionality or not.
Fetching the source from a repository
Wherever possible, ports should fetch from compressed distfiles, which our network of servers will automatically mirror. When this is not possible, a port might fetch directly from the developers' repository. When fetching from a repository, it is important to specify a specific revision or commit to fetch. It is not acceptable to fetch from the head or tip of the trunk or of a specific branch, because if two users install the same version of the port a day apart, and in between the developers commit a new revision to their repository, the two users will have different software, but no outward indication that this is the case. Also, the user who installed the port first will have no indication that new software is available, and even if they knew that a new version was available, they would have to forcibly rebuild the port to get it, since MacPorts would not consider the port outdated. Even that might not work: if the port is distributable, our buildbot will build a binary package the moment it is committed, and users might forevermore get that version, even if newer versions are available upstream. Finally, users who do manage to get newer commits are thus getting software that nobody at MacPorts has tested before; we always want at least one person (the MacPorts committer) to have verified that software at least builds and ideally works correctly and does not have any adverse side-effects (malware perhaps?) before making it available to other MacPorts users.
The solution is to always specify a specific known-good commit of the upstream repository to use. When new software is available upstream, the portfile can be updated to reference that new version.
Current reproducibility problems in MacPorts
At the moment, binary archives in MacPorts are generally not reproducible. The following issues are known and must be fixed to get reproducible binary archives.
- The order of entries in a binary archive depends on the filesystem order. Entries should be sorted before being added to the archives.
- The timestamps of files in a binary archive vary with the local system time. Timestamps for these files should be clamped to a specific maximum value, possibly specified by $SOURCE_DATE_EPOCH.
- The macOS linker includes a file's modification time when creating debug maps, unless the environment variable
ZERO_AR_DATE
is set (value doesn't matter). Search the ld64 source code forZERO_AR_DATE
insrc/ld/Options.cpp
for more information.