restore_ports.tcl: fail with a better message in case of dependency cycle

[JDLH (Jim DeLaHunt)]

I recently updated my macOS version from 10.11 El Capitan to macOS High Sierra 10.13.6. I am following the Migration instructions.

When I ran the restore_ports.tcl script, it failed with the error message:

Error: we appear to be stuck, exiting...
infinite loop
    while executing
"sort_ports $portList"
    invoked from within
"set operationList [sort_ports $portList]"
    (file "./restore_ports.tcl" line 285)

I did some investigation, adding diagnostic statements to restore_ports.tcl . My conclusion is that the underlying cause of the problem is that: a) some of the ports in my list had a mutual dependency cycle, and b) the script could not come up with an installation order for these two ports. Thus it issued the "infinite loop" error message. Details below.

My workaround was to edit my list of ports to restore. I looked at the ports involved in the cycle. One was clearly little used, I didn't even recognise it. I removed that from my list of ports. I also had hundreds of entries for old versions of ports which were no longer active; I removed all of them also. With this change, restore_ports.tcl was able to avoid the dependency cycle, and the restore is underway.

This problem is similar to #49210 'restore_ports.tcl script on fails on El Capitan with the error "infinite loop"'. That ticket did not come up with a clear solution to their problem, and it is closed and four years old. I think it's better to open a new ticket than to add this commentary on to that ticket. Hopefully others with this problem will find the workaround helpful, even before the underlying bug gets fixed.

Versions:
macOS High Sierra 10.13.6
Xcode 10.1 [but I think not involved in this bug]
restore_ports.tcl: ​https://github.com/macports/macports-contrib/raw/master/restore_ports/restore_ports.tcl (I believe commit 75135bd from Mar 24, 2015).
Ports forming dependency cycle: librsvg, adwaita-icon-theme

[JDLH (Jim DeLaHunt)]

Here is my diagnosis of the failure in restore_ports.tcl, and why I think it's due to a dependency cycle which the script can't handle.

The message "we appear to be stuck" comes from line 135, within proc sort_ports {portList}. This occurs when an iteration through $newList, the list of ports needing installation, results in no new ports being moved from $newList to the $operationList of ports to be installed.

I added some diagnostic code just after the line with the "we appear to be stuck" message:

            ui_error "newList($oldLen) is:"
            foreach port $newList {
            	foreach {active name variants} $port break
            	set deps [list]
				foreach dep $port_deps(${name},${variants}) {
					set installed "-"
					if {[info exists port_installed($dep)]} {
						set installed $port_installed($dep)
					}
					lappend deps [list $dep installed $installed]
				}
            	ui_error [list $name $variants $deps ]
            }

The result was a long list of ports which could not be added to the install list. Among them were:

...
Error: adwaita-icon-theme {} {{xz installed 1} {pkgconfig installed 1} {icon-naming-utils installed -} {autoconf installed 1} {automake installed 1} {libtool installed 1} {gtk3 installed 1} {librsvg installed 0}}
...
Error: librsvg {viewer +} {{xz installed 1} {pkgconfig installed 1} {rust installed 1} {cargo installed 1} {glib2 installed 1} {cairo installed 1} {pango installed 1} {libcroco installed 1} {libxml2 installed 1} {gdk-pixbuf2 installed 1} {vala installed 1} {gobject-introspection installed 1} {gtk3 installed 1} {python27 installed 1} {adwaita-icon-theme installed 0}}
...

This says that the script cannot add adwaita-icon-theme to the installation queue because it depends on librsvg, which is not able to join the queue, and that the script also cannot add librsvg to the queue, because it (supposedly) depends on adwaita-icon-theme, which is not able to join the queue. This is a dependency cycle. The script has no way of resolving this.

I verified that the script is able to remove items successfully at line 127.

I verified which dependencies the script was gathering for librsvg, by changing the main invocation in the final line to read,

#@@@ install_ports $operationList
ui_msg [list "deps of librsvg" [dependenciesForPort "librsvg" [list "viewer" "+"]]]

This gave me the output:

{deps of librsvg} {xz pkgconfig rust cargo glib2 cairo pango 
libcroco libxml2 gdk-pixbuf2 vala gobject-introspection gtk3 
python27 adwaita-icon-theme}

And, the mac ports system itself confirms that both gtk3 and adwaita-icon-theme are ports which librsvg depends on:

% port rdeps librsvg +viewer
The following ports are dependencies of librsvg @2.44.14_0+viewer:
  xz
...[many lines omitted]...
      urw-fonts
  gtk3
    atk
...[many lines omitted]...
    hicolor-icon-theme
  adwaita-icon-theme
    icon-naming-utils
      p5.28-xml-simple
        p5.28-xml-namespacesupport
        p5.28-xml-sax
          p5.28-xml-sax-base
        p5.28-xml-sax-expat
    librsvg

I no longer believe the problem is that the script gathers dependencies incorrectly. The problem is that the script appears unable to handle a dependency cycle, even though the underlying MacPorts system can.

A next step is to look into modifying the restore_ports.tcl script so that it can handle dependency cycles. I suspect that if it could detect that port A depended at the top level on port B, and port B on port A, that it could add an operation to install both of them together to the $operationList, and maybe that would succeed.

[JDLH (Jim DeLaHunt)]

Changed the title and the introduction of the ticket to withdraw my claim that the script gathers dependencies incorrectly. It appears to gather dependencies correctly. It appears unable, however, to deal with a dependency cycle. MacPorts itself appears to be able to handle the dependency cycle.

[ryandesign (Ryan Carsten Schmidt)]

Dependency cycles are not allowed in MacPorts and the behavior of MacPorts and any associated scripts is undefined if dependency cycles exist. Therefore, portfile authors must ensure that dependency cycles do not exist.

[JDLH (Jim DeLaHunt)]

Update: changed description of ports involved to say neutral "form cycle", instead of saying script listed dependencies wrong for one port. Added MacPorts version number.

[JDLH (Jim DeLaHunt)]

Sure enough, once I got past the problem of the ./restore_ports.tcl script failing with an infinite loop, I discovered that port install got stuck trying to install ports librsvg +viewer and adwaita-icon-theme. I believe this is due to a dependency cycle between those ports.

#59454 tracks the problem of a dependency cycle between librsvg +viewer and adwaita-icon-theme. Fixing that bug will resolve this ticket.

It would be nice if the restore_ports.tcl script would give better information than merely "infinite loop", in the presence of a dependency cycle. It could print an additional message, "This may be due to a dependency cycle in the ports list.", after the "infinite loop" message. It could print a list of ports in $newList and their dependencies, to help the user determine which ports formed the cycle. It could even add code to detect a cycle (though really, the base would be a better place to detect and report cyles, as mentioned in #59289).

[jmroot (Joshua Root)]

OK, so this ticket is for the suggested improvement to restore_ports.tcl.

[JDLH (Jim DeLaHunt)]

N.B. a fix for #59289 in MacPorts 2.7.2 might improve the behaviour of this bug. I haven't had a chance to try it, however.

[cooljeanius (Eric Gallager)]

I have a pull request open to address this; it doesn't fully fix it though: ​https://github.com/macports/macports-contrib/pull/10

[jmroot (Joshua Root)]

In c69a968e5127688358382c4fc9db8046f4e1b6ce/macports-contrib (master):

restore_ports.tcl: better reporting of dep cycles

Closes: #59445