Opened 10 years ago

Closed 8 years ago

Last modified 8 years ago

#29223 closed defect (fixed)

if an install is interrupted, a later install of the same package can fail if version number has since changed.

Reported by: rogerdpack@… Owned by: neverpanic (Clemens Lang)
Priority: Normal Milestone: MacPorts 2.1.3
Component: base Version: 1.9.2
Keywords: Cc: ryandesign (Ryan Schmidt), nerdling (Jeremy Lavergne), drkp (Dan Ports), nonstop.server@…, cooljeanius (Eric Gallager)
Port:

Description (last modified by ryandesign (Ryan Schmidt))

kind of explained here:

https://gist.github.com/raw/933741/93ded0007937f70df19aeb3d77f523a9dbd5639c/gistfile1.txt

What happened is I did a $ sudo port install mplayer one day. I think I did a control+c to cancel it sometime in the middle. Next day, I did

$ sudo port install mplayer

again.

m4 is a dependency of mplayer. With the first one, the m4 install was interrupted. Now, however, you can see from the gist that it was saying

:debug:main Skipping completed org.macports.fetch (m4)

then attempting to build from:

/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_devel_m4/work/m4-1.4.16

$ ls /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_devel_m4/work/
.CC_PRINT_OPTIONS   .macports.m4.state  m4-1.4.15/  

So basically it appears to be skipping the fetch phase because 1.4.15 had already been fetched (and unpacked), then trying to go ahead and build 1.4.16, which of course fails. Less than ideal, wouldn't you agree?

10.6.7, xcode 3.2.6 Thanks. -r

Change History (86)

comment:1 Changed 10 years ago by ryandesign (Ryan Schmidt)

Description: modified (diff)

comment:2 in reply to:  description ; Changed 10 years ago by ryandesign (Ryan Schmidt)

Replying to rogerdpack@…:

Less than ideal, wouldn't you agree?

Of course, but I'm not seeing how it was possible for you to get into this situation. If, between the first and second attempt, you ran "sudo port sync", and the version of m4 was changed by this sync, then the m4 portfile was updated. Next time you tried to build m4, MacPorts would have noticed the portfile was newer than your build attempt, and thrown out the previous build attempt. So I cannot explain why it did not do so.

To fix the problem, clean m4 and try again ("sudo port clean m4").

comment:3 Changed 10 years ago by rogerdpack@…

I suppose I am hoping that when it does the fetch phase it can double check that its cached fetch is the right version.

I believe I did do a port selfupdate between installs, but that's it.

-r

comment:4 in reply to:  3 ; Changed 10 years ago by ryandesign (Ryan Schmidt)

Replying to rogerdpack@…:

I suppose I am hoping that when it does the fetch phase it can double check that its cached fetch is the right version.

I'm telling you that MacPorts already does this, by comparing the date the portfile was modified with the date the work directory was created.

I believe I did do a port selfupdate between installs, but that's it.

selfupdate includes sync.

comment:5 in reply to:  4 Changed 10 years ago by ryandesign (Ryan Schmidt)

Cc: ryandesign@… added

Replying to ryandesign@…:

Replying to rogerdpack@…:

I suppose I am hoping that when it does the fetch phase it can double check that its cached fetch is the right version.

I'm telling you that MacPorts already does this, by comparing the date the portfile was modified with the date the work directory was created.

Unfortunately, I've just realized at least one way this problem can occur, and have been able to reproduce a failure like yours on my own system.

It hinges on the following sequence of events (using m4 as the example port, though it would apply to any port):

  1. At time T1, you sync or selfupdate, getting a current set of ports (for example including m4 1.4.15)
  2. At time T2, we update a port (for example we update m4 to 1.4.16)
  3. At time T3, without selfupdating or syncing again, you begin installing e.g. m4 (1.4.15) and cancel the installation before it finishes, or it stops due to an error
  4. At time T4, you sync or selfupdate, which downloads our current set of portfiles, preserving the modification date/time of those changes. Now you have the new m4 1.4.16 Portfile, but its modification date/time is T2, whereas the creation date of your m4 port's work directory is T3
  5. At time T5, you try installing m4 again, which, since the work directory seems to be newer than the portfile, is retained instead of being cleaned up

We're going to have to think about the proper way of fixing this.

comment:6 Changed 10 years ago by ryandesign (Ryan Schmidt)

I looked at the code and it's actually using the mtime of the statefile, and not looking at the work directory itself, but that doesn't change the point I'm making.

One could say that the problem is that we preserve the mtime when we sync. When using rsync (the default for MacPorts), it's the -t flag that preserves mtime; the default rsync_options in macports.conf do include -t (they're -rtzv --delete-after). If the user has configured to sync with Subversion instead, Subversion by default does not preserve mtime, but the user can edit the Subversion configuration file to make it do so. (See use-commit-times in the Subversion book.)

One solution would therefore be to not preserve mtimes when we sync, by removing -t from the rsync_options, and for Subversion, by enforcing a config file that doesn't specify use-commit-times. However, I think preserving mtimes is nice for other reasons, so I'd like to find a fix that doesn't require us to destroy this piece of metadata.

What we really want to know, in the case of resuming an interrupted install, is: is the portfile we're now using newer than the portfile we were using when we started? In order to answer that question, we need to keep track of the mtime of the portfile we were using when we started. I'd put that in a line in the statefile. Then we can check if the mtime of the portfile is newer (or, heck: is different at all -- newer or older) than the mtime previously recorded in the statefile; if so, delete the work directory and start over.

comment:7 Changed 10 years ago by jmroot (Joshua Root)

Component: portsbase

Should probably just put the version and revision in the statefile as well.

comment:8 Changed 10 years ago by ryandesign (Ryan Schmidt)

Even better than mtimes, we could store / compare checksums (sha1?)

What would having the version or revision in the statefile do for us?

comment:9 in reply to:  6 ; Changed 10 years ago by raimue (Rainer Müller)

Replying to ryandesign@…:

One solution would therefore be to not preserve mtimes when we sync, by removing -t from the rsync_options, and for Subversion, by enforcing a config file that doesn't specify use-commit-times. However, I think preserving mtimes is nice for other reasons, so I'd like to find a fix that doesn't require us to destroy this piece of metadata.

What other reasons do you refer to? We would still have the $Id$ line indicating the time of commit at the top.

comment:10 in reply to:  8 ; Changed 10 years ago by jmroot (Joshua Root)

Replying to ryandesign@…:

What would having the version or revision in the statefile do for us?

It would tell us when we're building a different version than what's in the existing work dir.

comment:11 in reply to:  9 Changed 10 years ago by ryandesign (Ryan Schmidt)

Replying to raimue@…:

Replying to ryandesign@…:

I think preserving mtimes is nice for other reasons, so I'd like to find a fix that doesn't require us to destroy this piece of metadata.

What other reasons do you refer to? We would still have the $Id$ line indicating the time of commit at the top.

I just feel that it's a non-pointless piece of metadata and it would be nice for us to not destroy it. It is nice to be able to look at a file in a file browser and see when it was last modified, and to know that it was actually modified by someone at that time, and not merely downloaded from the rsync server.

The $Id$ line exists for Portfiles, but not necessarily for other files, like those in the files directory. Portfiles don't necessarily have the Id line either (or the svn:keywords property that keeps it updated), though they should.

Conversely, while I support preserving mtimes, they are also unreliable and might be destroyed by other processes (e.g. some kinds of backups...). Therefore it would be great not to rely on them.

So while I wanted to begin by pointing out that the way we're handling mtimes now is not sufficient for identifying a changed file, it is not necessary either; a checksum is the correct way to solve this problem. I don't see disadvantage to using a checksum; if you see any, please point them out.

comment:12 in reply to:  10 Changed 10 years ago by ryandesign (Ryan Schmidt)

Replying to jmr@…:

Replying to ryandesign@…:

What would having the version or revision in the statefile do for us?

It would tell us when we're building a different version than what's in the existing work dir.

Right, but the message we print is "Portfile changed since last build; discarding previous state." That is the condition we want to capture -- that the portfile has changed -- and I think using a checksum is the simplest way to check for that. We don't need to care whether the portfile's mtime or version or revision or anything else specifically in it has changed; it doesn't matter.

comment:13 Changed 10 years ago by ryandesign (Ryan Schmidt)

#29741 was probably caused by this problem.

comment:14 Changed 10 years ago by ryandesign (Ryan Schmidt)

#25864 was probably caused by this problem.

comment:15 Changed 10 years ago by ryandesign (Ryan Schmidt)

#15517 might have been caused by this problem.

comment:16 Changed 10 years ago by ryandesign (Ryan Schmidt)

#28088 might have been caused by this problem.

comment:17 Changed 10 years ago by ryandesign (Ryan Schmidt)

#16196 might have been caused by this problem.

comment:18 Changed 10 years ago by ryandesign (Ryan Schmidt)

#15476 was probably caused by this problem.

comment:19 Changed 10 years ago by ryandesign (Ryan Schmidt)

#4598 was probably caused by this problem.

comment:20 Changed 10 years ago by ryandesign (Ryan Schmidt)

#19838 was probably caused by this problem.

comment:21 Changed 10 years ago by ryandesign (Ryan Schmidt)

#20159 was probably caused by this problem.

comment:22 Changed 10 years ago by ryandesign (Ryan Schmidt)

#29508 was probably caused by this problem.

comment:23 Changed 10 years ago by ryandesign (Ryan Schmidt)

#27821 was probably caused by this problem.

comment:24 Changed 10 years ago by ryandesign (Ryan Schmidt)

#26668 was probably caused by this problem.

comment:25 Changed 10 years ago by ryandesign (Ryan Schmidt)

#24780 was probably caused by this problem.

comment:26 Changed 10 years ago by ryandesign (Ryan Schmidt)

comment:ticket:25872:5 was probably caused by this problem.

comment:27 Changed 10 years ago by ryandesign (Ryan Schmidt)

The last portion of the description of #16785 was probably caused by this problem.

comment:28 Changed 10 years ago by ryandesign (Ryan Schmidt)

comment:ticket:29805:6 was probably caused by this problem.

comment:29 Changed 10 years ago by nerdling (Jeremy Lavergne)

Cc: snc@… added

Cc Me!

comment:30 Changed 10 years ago by ryandesign (Ryan Schmidt)

comment:ticket:30056:2 was caused by this problem.

comment:31 Changed 10 years ago by ryandesign (Ryan Schmidt)

#29014 was caused by this problem.

comment:32 Changed 9 years ago by ryandesign (Ryan Schmidt)

#29843 was probably caused by this problem.

comment:33 Changed 9 years ago by ryandesign (Ryan Schmidt)

#30484 was caused by this problem.

comment:34 Changed 9 years ago by ryandesign (Ryan Schmidt)

#30606 was caused by this problem.

comment:35 Changed 9 years ago by ryandesign (Ryan Schmidt)

#30789 may have been caused by this problem.

comment:36 Changed 9 years ago by ryandesign (Ryan Schmidt)

#28699 may have been caused by this problem.

comment:37 Changed 9 years ago by ryandesign (Ryan Schmidt)

#31532 was probably caused by this issue.

comment:38 Changed 9 years ago by ryandesign (Ryan Schmidt)

#31258 was caused by this issue.

comment:39 Changed 9 years ago by ryandesign (Ryan Schmidt)

#31868 was caused by this problem.

comment:40 Changed 9 years ago by ryandesign (Ryan Schmidt)

#32255 was caused by this problem.

comment:41 Changed 9 years ago by ryandesign (Ryan Schmidt)

#32349 was caused by this problem.

comment:42 Changed 9 years ago by drkp (Dan Ports)

Cc: dports@… added

Cc Me!

comment:43 Changed 9 years ago by ryandesign (Ryan Schmidt)

comment:ticket:31652:3 was probably caused by this problem.

comment:44 Changed 9 years ago by ryandesign (Ryan Schmidt)

This post on the mailing list was caused by this problem.

comment:45 Changed 9 years ago by ryandesign (Ryan Schmidt)

#33579 was caused by this problem.

comment:46 Changed 9 years ago by ryandesign (Ryan Schmidt)

#33582 was caused by this problem.

comment:47 Changed 9 years ago by ryandesign (Ryan Schmidt)

#33605 was caused by this problem.

comment:48 Changed 9 years ago by nonstop.server@…

Cc: nonstop.server@… added

Cc Me!

comment:49 Changed 9 years ago by ryandesign (Ryan Schmidt)

comment:ticket:34292:3 and the original error message reported in #34291 were caused by this problem.

comment:50 in reply to:  2 Changed 9 years ago by nerdling (Jeremy Lavergne)

Replying to ryandesign@…:

Of course, but I'm not seeing how it was possible for you to get into this situation. If, between the first and second attempt, you ran "sudo port sync", and the version of m4 was changed by this sync, then the m4 portfile was updated. Next time you tried to build m4, MacPorts would have noticed the portfile was newer than your build attempt, and thrown out the previous build attempt. So I cannot explain why it did not do so.

Does this problem manifest only when using the ports tarball and not Portfiles? I'm wondering if the timestamps only get checked properly in one case and not the other.

comment:51 Changed 9 years ago by ryandesign (Ryan Schmidt)

To my knowledge, it doesn't matter how you got the portfile. As discussed above, the solution is to remove the timestamp checking, and to use checksum checking instead. I haven't looked into where in the code we'd need to do that exactly. But as the history of this ticket hopefully shows, fixing this issue would save us many tickets and much user time and aggravation.

comment:52 Changed 9 years ago by ryandesign (Ryan Schmidt)

Looks like #34410 was caused by this problem.

comment:53 Changed 9 years ago by ryandesign (Ryan Schmidt)

#34428 was caused by this issue.

comment:54 Changed 9 years ago by ryandesign (Ryan Schmidt)

#34446 was caused by this issue.

comment:55 Changed 9 years ago by ryandesign (Ryan Schmidt)

#34557 was caused by this issue.

comment:56 Changed 9 years ago by ryandesign (Ryan Schmidt)

#34936 may have been caused by this issue.

comment:57 Changed 9 years ago by ryandesign (Ryan Schmidt)

#35189 was caused by this issue.

comment:58 Changed 9 years ago by ryandesign (Ryan Schmidt)

#35280 was caused by this issue.

comment:59 Changed 9 years ago by ryandesign (Ryan Schmidt)

#35289 was caused by this issue.

comment:60 Changed 9 years ago by ryandesign (Ryan Schmidt)

#35315 was caused by this problem.

comment:61 Changed 8 years ago by ryandesign (Ryan Schmidt)

#29696 was probably caused by this issue.

comment:62 Changed 8 years ago by ryandesign (Ryan Schmidt)

#35373 was caused by this issue.

comment:63 Changed 8 years ago by ryandesign (Ryan Schmidt)

#35376 was caused by this issue.

comment:64 Changed 8 years ago by ryandesign (Ryan Schmidt)

#35446 was caused by this issue.

comment:65 Changed 8 years ago by neverpanic (Clemens Lang)

Milestone: MacPorts Future
Resolution: fixed
Status: newclosed

comment:66 Changed 8 years ago by ryandesign (Ryan Schmidt)

#35558 was caused by this issue.

comment:67 Changed 8 years ago by ryandesign (Ryan Schmidt)

#35578 was caused by this issue.

comment:68 Changed 8 years ago by neverpanic (Clemens Lang)

Dropping support for the old statefile format would cause some undesired effects in the transition phase: Archives previously built (or downloaded from the buildbot) still contain an old statefile and would thus not get used. Change r96399 fixes this by re-introducing support for reading older statefiles (where it uses the old mechanism to check whether the partial build is outdated).

r96400 contains a number of testcases to verify correct behavior of the statefile handling.

comment:69 Changed 8 years ago by neverpanic (Clemens Lang)

Merged into release_2_1 in r96466.

comment:70 Changed 8 years ago by neverpanic (Clemens Lang)

Resolution: fixed
Status: closedreopened

The current state of the statefile handling is still buggy:

  • port fetch $someport, followed by
  • port build $someport +variant

doesn't correctly restart fetch phase. This is because the variants check relied on the statefile being empty when no variants have been selected and implicitly assuming the empty variant was selected if it is not. Starting fetch without specifying a variant doesn't write a "variant: " line into the statefile, which is what the code currently checks for.

So, either, we always write a "variant: " line and ignore it when checking the selected variants, or we assume variants to be "" as soon as there's a "target: " line in the statefile.

comment:71 Changed 8 years ago by neverpanic (Clemens Lang)

Owner: changed from macports-tickets@… to cal@…
Status: reopenednew

comment:72 Changed 8 years ago by neverpanic (Clemens Lang)

Resolution: fixed
Status: newclosed

This should be fixed in r96538 by assuming variants to be "" if there's a line matching "target: .*" in the statefile but no lines matching "variant: .*" have been found.

ChangeLog updated in r96539, merged into release_2_1 in 96540.

comment:73 Changed 8 years ago by ryandesign (Ryan Schmidt)

#36012 was caused by this issue.

comment:74 Changed 8 years ago by ryandesign (Ryan Schmidt)

#36136 was caused by this issue.

comment:75 Changed 8 years ago by ryandesign (Ryan Schmidt)

#36300 was caused by this issue.

comment:76 Changed 8 years ago by ryandesign (Ryan Schmidt)

#36309 was caused by this issue.

comment:77 Changed 8 years ago by ryandesign (Ryan Schmidt)

#36407 was caused by this issue.

comment:78 Changed 8 years ago by ryandesign (Ryan Schmidt)

#36436 was caused by this issue.

comment:79 Changed 8 years ago by ryandesign (Ryan Schmidt)

#36471 was caused by this issue.

comment:80 Changed 8 years ago by ryandesign (Ryan Schmidt)

#34949 seems to have been caused by this issue.

comment:81 Changed 8 years ago by ryandesign (Ryan Schmidt)

#37243 was caused by this issue.

comment:82 Changed 8 years ago by jmroot (Joshua Root)

Milestone: MacPorts FutureMacPorts 2.1.3

comment:83 Changed 8 years ago by ryandesign (Ryan Schmidt)

#37760 was caused by this issue.

comment:84 Changed 8 years ago by cooljeanius (Eric Gallager)

Cc: egall@… added

Cc Me!

comment:85 Changed 8 years ago by ryandesign (Ryan Schmidt)

#38246 seems to have been caused by this issue.

comment:86 Changed 8 years ago by ryandesign (Ryan Schmidt)

#38860 seems to have been caused by this issue.

Note: See TracTickets for help on using tickets.