BuildBot Restructuring

Original Problem

When designing the new buildbot at the last MacPorts Meeting in 2016, we tried to achieve the following picture for each commit:

 +---------------+
 |    commit     |
 +---------------+
 |    prepare    |
 |   resources   |
 +---------------+
 |   port dep1   |
 +---------------+
 |   port dep2   |
 +---------------+
 |      ...      |
 +---------------+

Where port dep1, port dep 2, etc. are dependencies of the port actually changed in the commit and are scheduled in dependency order. We did not end up doing this, because it requires the ability to dynamically add steps, which buildbot 0.8 does not support yet. We would have had to switch to buildbot >= 0.9 for this to work.

We then attempted to emulate the same by introducing the portwatcher and triggering builds from the portwatcher on a separate portbuilder builder, to emulate the same picture:

 +--------------+
 |   commit     |
 +--------------+
 |   scheduler  |
 +--------------+
 |   prepare    |         +---------------+
 |  resources   |  ---->  |   port dep1   |
 +--------------+         +---------------+
                          |   port dep2   |
                          +---------------+
                          |   port dep3   |
                          +---------------+

Unfortunately this did not work either, because we could not get buildbot to schedule builds in the order we triggered them (#52766). To work around this, we introduced the mpbb install-dependencies step:

 +--------------+
 |   commit     |
 +--------------+
 |   prepare    |
 |  resources   |
 +--------------+         +---------------+
 |   scheduler  |  ---->  |  port1 deps   |
 +--------------+         +---------------+
                          |  port1 build  |
                          +---------------+
                          |  port2 deps   |
                          +---------------+
                          |  port2 build  |
                          +---------------+

Ryan's proposal

Ryan asked us to get rid of the separate portwatcher & portbuilder jobs and re-configure the remaining job to interleave the two types of actions. As we understood it, this was to solve the following problem:

• A commit for portA comes in
• portwatcher for this commit schedules a portbuilder job for portA
• This build takes a long time
• While the build is still running, a new commit for the same port arrives, which queues a portwatcher job
• Another commit for the same port arrives, which queues another portwatcher job
• When the portbuilder job finishes, a useless build is scheduled for portA.

He proposed the following to solve this:

 +---------------+
 |    commit1    |
 +---------------+
 |    prepare    |
 |   resources   |
 +---------------+
 |   scheduler1  |
 +---------------+
 +---------------+
 |   port1 dep1  |
 +---------------+
 +---------------+
 |   port1 dep2  |
 +---------------+
 +---------------+
 |    commit2    |
 +---------------+
 |    prepare    |
 |   resources   |
 +---------------+
 |   scheduler2  |
 +---------------+
 +---------------+
 |   port1 dep2  |
 +---------------+
 +---------------+
 |     port1     |
 +---------------+
 +---------------+
 |   port2 dep1  |
 +---------------+
 +---------------+
 |   port2 dep2  |
 +---------------+
 +---------------+
 |     port2     |
 +---------------+

Unfortunately that introduces the problem with the prepared shared resources (like the portindex, mpbb checkout, ports tree), because we really cannot change the portstree while we have build scheduled in a dependency order that was computed from the old ports tree. This would cause seemingly random and hard-to-debug problems if a follow-up commit changes dependencies of ports.

If we were to wait for the steps planned by the first scheduler to finish, we would end up in the same situation as outlined at the beginning, which was our initial approach, but only works with buildbot >= 0.9.

To avoid the problem outlined by Ryan, we could also just enable build merging on the portwatcher, which would avoid the spurious builds.

Alternative Solutions

We could also try setting up buildbot 1.0 and just fix the UI, rather than working around 0.8's limitations with hacks.

Also, #52766 is solved now, enabling us to schedule builds in-order, so we could also implement this approach again:

 +---------------+
 |    commit     |
 +---------------+
 |   scheduler   |
 +---------------+
 |    prepare    |         +---------------+
 |   resources   |  ---->  |   port dep1   |
 +---------------+         +---------------+
                           |   port dep2   |
                           +---------------+
                           |   port dep3   |
                           +---------------+

The approach with two separate builders is always affected by #53587 (a restart of the buildmaster will trigger pending portwatcher and portbuilder jobs at the same time), though.