Opened 14 years ago

Closed 12 years ago

#10827 closed defect (fixed)

BUG: 'port -f -R upgrade' rebuilds too much

Reported by: yaseppochi (Stephen J. Turnbull) Owned by: jmroot (Joshua Root)
Priority: Normal Milestone: MacPorts 1.7.0
Component: base Version: 1.4.0
Keywords: Cc: jmpp@…, jberry@…, kballard (Lily Ballard), boeyms@…
Port:

Description

Here's a log (partially completed, and of course I have no idea how much longer it's going to run) of port -f -R upgrade openssl. Both -f and -R are necessary AFAICT because port doesn't have versioned dependencies for at least some packages (eg gettext and libiconv) so upgrading those packages silently breaks others, even with -R, because dyld *does* track library versions, and the upgraded library won't link. This wouldn't be a problem---I'd just have an unupgraded package whose old version probably still works---except that port often has deactivated some library in the process of upgrading and doesn't reactivate it on abort due to build breakage. Thus -f is needed.

Net effect---I don't dare upgrade anything unless I can afford literally days of downtime.

On second thought, it looks like posting the log as text is too much work; I'll cross my fingers and hope there's an opportunity to attach it later.

Attachments (3)

dpsux.txt (36.1 KB) - added by yaseppochi (Stephen J. Turnbull) 14 years ago.
Log of port rebuilding the same package many times in a row
R-implies-n.diff (485 bytes) - added by jmroot (Joshua Root) 12 years ago.
make -R imply -n
upgrade-dependents-fix.diff (3.7 KB) - added by jmroot (Joshua Root) 12 years ago.
turn off -R while upgrading dependencies, and use the dep cache for dependents too

Download all attachments as: .zip

Change History (24)

Changed 14 years ago by yaseppochi (Stephen J. Turnbull)

Attachment: dpsux.txt added

Log of port rebuilding the same package many times in a row

comment:1 Changed 14 years ago by yaseppochi (Stephen J. Turnbull)

On my system, `port dependents gconf | uniq -c' returns

95 gnome-vfs depends on gconf

It seems to be adding another redundancy to the count every time gnome-vfs gets rebuilt (the command logged in dpsux.txt is still running...).

Port 1.400, updated from svn about a week ago.

comment:2 Changed 14 years ago by markd@…

Summary: port rebuilds too muchBUG: port rebuilds too much

comment:3 Changed 14 years ago by yaseppochi (Stephen J. Turnbull)

port 1.500

I gather that nobody cares, but FWIW, I just did

port -f -k upgrade git-core +doc

and among other things it went and started to install XFree86 (even though I have a perfectly usable self-built X.org installation from ancient days).

All I wanted to do was to rebuild git-core with the doc, leaving source in place so I can see why most of the docs don't seem to get installed....

comment:4 Changed 14 years ago by pipping@…

Milestone: MacPorts 1.5

comment:5 Changed 13 years ago by markd@…

We care, but upgrade is an area that needs a lot of work and it just hasn't been done yet. I believe that -n will not follow dependencies and improves the situation a bit.

comment:6 Changed 13 years ago by markd@…

Summary: BUG: port rebuilds too muchBUG: 'port -f upgrade' rebuilds too much

Changing summary just a bit.

comment:7 Changed 13 years ago by markd@…

For more examples of upgrade problems, see duplicate #10813.

comment:8 Changed 13 years ago by jmpp@…

Priority: BlockerImportant

comment:9 Changed 13 years ago by jmpp@…

Owner: changed from darwinports-bugs@… to macports-dev@…

comment:10 Changed 13 years ago by nox@…

Priority: ImportantHigh

comment:11 Changed 13 years ago by boeyms@…

Cc: stephen@… jmpp@… jberry@… eridius@… boeyms@… added

Cc'ing to reporter and coders with greater MacPorts-base-fu than I; I hope nobody minds.

In theory, there is a simple solution to this problem: for each of the ports to be upgraded, traverse its dependency graph depth-first, appending a port in the tree to a list once all of its dependencies have been visited or skipping the port if it has already been visited, and then upgrading the ports in order of appearance in the list. This would also enable fetching a port and its dependencies all at once, as well as enabling detection of circular dependencies (the search requires us to keep track of the dependency trail from the root of the graph to the current dependency, so if the end of trail appears in the body too, we have a loop).

My problem is that I can't figure out where in the code this ought to go. If nobody understands (or has the time) to write this from scratch, I'll try to write some Tcl-ish pseudocode that can hopefully be added with only minor modifications.

comment:12 Changed 13 years ago by frstan@…

it also downloads items multiple times due to the creation of multiple subdirs once each repition at least in some cases: upgrading libpng with -R produced: -> Fetching gimp2 DEBUG: Executing org.macports.fetch (gimp2) ---> gimp-2.4.1.tar.bz2 doesn't seem to exist in /opt/local/var/macports/distfiles/gimp2/gimp2/gimp2 I have seen as many as a dozen levels in other cases.

The repition seems to arise from the dependency register getting a new entry with each new install of an item while the preceeding uninstall (during a port -R upgrade foo) does not cause the existing entry to be deleted. Therefore that dependent will get rebuilt multiple times, and next time even more times.......

we badly need a utility to purge the dependentcy register of non-unique entries. port clean --reg_duplicates --all .

comment:13 Changed 13 years ago by jmpp@…

Milestone: MacPorts 1.5MacPorts base bugs
Priority: HighNormal

comment:14 Changed 12 years ago by jmroot (Joshua Root)

Cc: jmr@… added; stephen@… removed

For reference, the issue with duplicate dep map entries noted in comment:1 is #8763. The subdirectory creation noted in comment:12 is #11971.

The method proposed in comment:11 is probably the Right Thing, but I think that after the resolution of the other two tickets, this would be mostly solved by making -R imply -n. Upgrade normally applies recursively to dependencies, but if you're recursing in the other direction, you usually don't want to go back the other way as well. Would we then need an option to complement -n in case you really do want the both-ways recursion?

Changed 12 years ago by jmroot (Joshua Root)

Attachment: R-implies-n.diff added

make -R imply -n

comment:15 Changed 12 years ago by jmroot (Joshua Root)

Attached a one-liner to make -R imply -n.

comment:16 Changed 12 years ago by raimue (Rainer Müller)

I am against committing intermediate solutions without at least adding a comment there why -R should imply -n. This will just become the default and noone will ever remember why -R implies -n, even once the other bugs are fixed.

comment:17 Changed 12 years ago by jmroot (Joshua Root)

I think that #7361 being fixed should also help with the initial problem described in the summary.

comment:18 Changed 12 years ago by jmroot (Joshua Root)

Cc: jmr@… removed
Owner: changed from macports-tickets@… to jmr@…
Status: newassigned
Summary: BUG: 'port -f upgrade' rebuilds too muchBUG: 'port -f -R upgrade' rebuilds too much

OK, the proper fix isn't to make -R imply -n. It is to turn off -R for the dependencies of the requested port. We want to upgrade its dependents, not its dependencies' dependents. The attached patch does this, and can make a huge difference to the amount of upgrade checks (which are all actual rebuilds when using -f).

The other thing to do to eliminate the last of the unnecessary upgrading would be to use the depscache for dependents as well as dependencies.

comment:19 Changed 12 years ago by jmroot (Joshua Root)

Updated the patch to use the depscache for dependents. Now no port should be touched more than once.

Changed 12 years ago by jmroot (Joshua Root)

Attachment: upgrade-dependents-fix.diff added

turn off -R while upgrading dependencies, and use the dep cache for dependents too

comment:20 Changed 12 years ago by jmroot (Joshua Root)

Milestone: MacPorts base bugsMacPorts 1.6.1
Version: 1.4.0

Committed the fix to trunk in r37046. Thanks to Rainer for reviewing.

I think this should go into 1.6.1, so I'm leaving this ticket open until the fix is merged into the branch, or retargeted at a later release.

comment:21 Changed 12 years ago by raimue (Rainer Müller)

Resolution: fixed
Status: assignedclosed

Marking fixed, targeted for 1.7.0.

Note: See TracTickets for help on using tickets.