mpbb takes too long to install dependencies

[ryandesign (Ryan Carsten Schmidt)]

It's dumb that mpbb takes 80 minutes to install the dependencies of a port that takes only a minute to build.

​https://build.macports.org/builders/ports-11_x86_64-builder/builds/25468

Yes, there are 280 dependencies. That's still 17 seconds per dependency, which is too much.

It's also dumb that the port has that many dependencies.

[jmroot (Joshua Root)]

Part of it is probably how many of the dependencies are being uninstalled between builds.

[ryandesign (Ryan Carsten Schmidt)]

In c8db5a3f36610a13f3c99915e32a6fa853982be9/mpbb (master):

More selectively uninstalling ports with only 1 dep

Don't uninstall all ports that have only 1 dependency. Only uninstall
those ports with 1 dependency if that dependency also has only 0 or 1
dependencies and so on.

See: #57464
See: #62621

[ryandesign (Ryan Carsten Schmidt)]

Here we have ​a pull request updating auto-multiple-choice-devel, a port with 554 dependencies. It has taken 4 hours so far to attempt to install all dependencies. The Azure Pipelines build terminated after ~3.5 hours but its install-dependencies.log is already 1GB in size and contains 9669 occurrences of "activate phase started at" which is wildly excessive. There should be 554 occurrences, one for each port, but for each of the 554 we appear to be activating and deactivating that port and its dependencies, so many ports keep getting activated and deactivated over and over. Why? Can we stop doing that? This wears out build machine SSDs (all of the buildbot SSDs bought in 2016 died in 4-5 years) and wastes time.

$ egrep '^(----> Installing dependency|--->  Activating|--->  Deactivating)' install-dependencies.log  | head -n 50
----> Installing dependency (1 of 554) 'gperf' with variants '' (requesting '')
--->  Activating gperf @3.1_0
--->  Deactivating gperf @3.1_0
----> Installing dependency (2 of 554) 'libiconv' with variants '' (requesting '')
--->  Activating libiconv @1.16_1
--->  Deactivating libiconv @1.16_1
----> Installing dependency (3 of 554) 'ncurses' with variants '' (requesting '')
--->  Activating ncurses @6.2_1
--->  Deactivating ncurses @6.2_1
----> Installing dependency (4 of 554) 'gettext' with variants '' (requesting '')
--->  Activating libiconv @1.16_1
--->  Activating ncurses @6.2_1
--->  Activating gettext @0.19.8.1_2
--->  Deactivating gettext @0.19.8.1_2
--->  Deactivating libiconv @1.16_1
--->  Deactivating ncurses @6.2_1
----> Installing dependency (5 of 554) 'm4' with variants '' (requesting '')
--->  Activating m4 @1.4.18_2
--->  Deactivating m4 @1.4.18_2
----> Installing dependency (6 of 554) 'xz' with variants '' (requesting '')
--->  Activating libiconv @1.16_1
--->  Activating ncurses @6.2_1
--->  Activating gettext @0.19.8.1_2
--->  Activating xz @5.2.5_0
--->  Deactivating xz @5.2.5_0
--->  Deactivating gettext @0.19.8.1_2
--->  Deactivating libiconv @1.16_1
--->  Deactivating ncurses @6.2_1
----> Installing dependency (7 of 554) 'autoconf' with variants '' (requesting '')
--->  Activating libiconv @1.16_1
--->  Activating ncurses @6.2_1
--->  Activating gettext @0.19.8.1_2
--->  Activating m4 @1.4.18_2
--->  Activating autoconf @2.71_1
--->  Deactivating autoconf @2.71_1
--->  Deactivating gettext @0.19.8.1_2
--->  Deactivating libiconv @1.16_1
--->  Deactivating m4 @1.4.18_2
--->  Deactivating ncurses @6.2_1
----> Installing dependency (8 of 554) 'automake' with variants '' (requesting '')
--->  Activating automake @1.16.3_0
--->  Deactivating automake @1.16.3_0
----> Installing dependency (9 of 554) 'libffi' with variants '' (requesting '')
--->  Activating libffi @3.3_1
--->  Deactivating libffi @3.3_1
----> Installing dependency (10 of 554) 'libtool' with variants '' (requesting '')
--->  Activating libtool @2.4.6_11
--->  Deactivating libtool @2.4.6_11
----> Installing dependency (11 of 554) 'bzip2' with variants '' (requesting '')
--->  Activating bzip2 @1.0.8_0

[ryandesign (Ryan Carsten Schmidt)]

These are two somewhat different cases. On the buildbot we keep all ports installed, and mpbb-install-dependencies prints "Already installed, nothing to do" and does nothing more for each direct dependency that's already installed. But on the CI build machines no ports are installed to begin with so it goes through the whole process of fetching the archive, installing and activating it (and its dependencies), then deactivating everything, for each direct dependency.

I expect the reason why it deactivates everything between dependencies is so that if any dependency was not already built, then that dependency will build from source with a clean slate, without any other random ports activated that might affect its build. However since #60935 was implemented, that should never be the case anymore. Packages for all dependencies should already exist by the time any build is attempted. So maybe we can streamline mpbb-install-dependencies so that if archive fetch fails, the entire build fails.

That might be too simple though. CI build machines sometimes lose access to the private packages server; in that case, it would be better to take extra time to build those nondistributable dependencies again in the PR rather than fail the PR. We could move "${option_prefix}/bin/port" -fp deactivate active so that it is only run right before we install a dependency from source, rather than right after installing any dependency. We should be able to determine whether fetching the archive succeeded to decide whether or not to then deactivate active. And if any dependency was built from source, we might have to deactivate again at the end of installing all dependencies to ensure none of the dependencies' build dependencies are present for the main port build.

It would be even better if we could deactivate just the ports that should not be active for the next build, rather than deactivate everything. For example, looking at the dependency list from auto-multiple-choice-devel above, suppose we were able to install and activate gperf, libiconv, and ncurses from binaries, and then the gettext package couldn't be fetched and needed to be built from source. Rather than deactivating gperf, libiconv, and ncurses, and then building gettext from source, which would reactivate libiconv and ncurses again, it would be better if we could just deactivate gperf. But I don't think we have a way to do that yet.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign:

CI build machines sometimes lose access to the private packages server

There might also be a chance that one of the port's distributable dependencies has been updated so recently that its package has not yet made it to the public packages server.

It would be even better if we could deactivate just the ports that should not be active for the next build [...] But I don't think we have a way to do that yet.

or would that be as simple as "${option_prefix}/bin/port" -fp deactivate active and not depof:${depname}?

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign:

Replying to ryandesign:

It would be even better if we could deactivate just the ports that should not be active for the next build [...] But I don't think we have a way to do that yet.

or would that be as simple as "${option_prefix}/bin/port" -fp deactivate active and not depof:${depname}?

Hmm, no because it needs to be recursive. And it's not "${option_prefix}/bin/port" -fp deactivate active and not rdepof:${depname} either because it would leave all ports' build dependencies active.

[jmroot (Joshua Root)]

Replying to ryandesign:

Packages for all dependencies should already exist by the time any build is attempted. So maybe we can streamline mpbb-install-dependencies so that if archive fetch fails, the entire build fails.

That might be too simple though.

There are other possibilities than the ones you already mentioned that make this unworkable even on the buildbot—the bugs that make builds sometimes run out of order, and noarch ports that are skipped because they will be built by another builder, but then are needed as dependencies for something else, to name two.

So a check for whether an archive is available for each dependency is definitely needed before skipping the deactivation before installation.

Something like the opposite of tools/dependencies.tcl would probably be needed to deactivate only the unneeded ports.

[mascguy (Christopher Nielsen)]

Large dependencies - specifically, ports consisting of many thousands of files, like the various texlive-related components - would also benefit from speedup of the file registration process (issue:56793).

[mascguy (Christopher Nielsen)]

Update: Josh's fix for issue:56793 will improve dependency installation times in some situations. And for the most extreme cases - ports that depend on huge components like texlive-fonts-extra - that fix is a game-changer! (Both for buildbot/CI jobs, as well as the user experience in-general.)

Net-Net: For ports like auto-multiple-choice, which fit into the latter category, we'll see overall dependency installation time reduced dramatically. Great stuff!

[jmroot (Joshua Root)]

In f83c5266f670e1334e6a3eb1591194c6c0efe593/mpbb (master):

mpbb dependency management improvements

• Only deactivate ports that are not needed for the next build, rather than always deactivating everything.
• Only deactivate ports before dependency installations if building from source (most installs will now be from an archive).
• Fail fast if a dependency is in the failcache, rather than continuing to check the rest of the deps. That does mean that multiple failed deps might have to be discovered one at a time by maintainers, but on the other hand the failcache checks can really add up and cause us to spend significant time on a port that we already know will fail.
• Replace much shell code with Tcl in order to take advantage of direct API access to avoid repeated work.
• Turn off verbose and debug output during the final activation step.

See: #62621

[jmroot (Joshua Root)]

The timing breakdown for the latest auto-multiple-choice build is now:

init took 1 seconds
deactivating unneeded ports took 61 seconds
calculating deps took 8 seconds
checking failcache took 16 seconds
installing deps took 2 seconds
activating deps took 2375 seconds

386 dependencies needed to be activated, of which 5 needed to be installed (from binary archives) first. ​https://build.macports.org/builders/ports-11_x86_64-builder/builds/69530

The build of auto-multiple-choice-devel right after that really sees the benefits, of course:

init took 2 seconds
deactivating unneeded ports took 89 seconds
calculating deps took 0 seconds
checking failcache took 0 seconds
installing deps took 0 seconds
activating deps took 0 seconds

​https://build.macports.org/builders/ports-11_x86_64-builder/builds/69531

[jmroot (Joshua Root)]

Replying to jmroot:

the bugs that make builds sometimes run out of order

A textbook example of this, BTW: ​https://build.macports.org/builders/ports-12_x86_64-builder/builds/27325

The build of gcc-devel with priority 7001.2 ran before the build of libgcc-devel with priority 7001.1. The good news at least is that the mpbb code seems to have handled this perfectly.