py39-numpy checksum error

[smeingast (Stefan Meingast)]

Just tried to install py39-numpy and noticed the checksums were not matching.

The Portfiles says: sha256 2113dd3af5b71fe7cfcb8866be2629aef90ad9a28eb6a8c43ebbe045b677f2ae

and I get (also verified that myself on the archive file)

sha256 79c0e27e27cf5c16558ae463906d29aeb0f26444dda21893039cfd197a31c849

[smeingast (Stefan Meingast)]

main.log

[ryandesign (Ryan Carsten Schmidt)]

The numpy-1.21.3.tar.gz we mirrored on October 20 has the checksums that are listed in the Portfile.

After that, numpy-1.21.3.tar.gz changed on GitHub so now the checksums don't match anymore.

The change between the old one and the new one is:

numpy-numpy-efe3349/numpy/_version.py

@@ -22 +22 @@
     # setup.py/versioneer.py will grep for the variable names, so they must
     # each be defined on a line of their own. _version.py will just call
     # get_keywords().
-    git_refnames = " (tag: v1.21.3, maintenance/1.21.x)"
+    git_refnames = " (tag: v1.21.3)"
     git_full = "d4d0584ccb508cad69288b504e6c8b04aa0d5caf"
     git_date = "2021-10-19 18:45:47 -0600"
     keywords = {"refnames": git_refnames, "full": git_full, "date": git_date}

We have seen this kind of problem before, for example in #60444 (which has a good explanation of why it happens) and in #41657. A set of git tags gets inserted into the generated source code, and if upstream changes which tags are available, as apparently happened for numpy between October 20 and today, then the generated source code changes, and the checksums along with it.

This is one of the reasons why relying on automatically-generated tarballs isn't always a great idea. If a project offers manually-generated distfiles, that can often be a better choice. numpy does offer manually-generated tarballs. There are more differences between those and the automatically-generated ones though. Michael would have to evaluate whether those differences matter for MacPorts or not.

If we switch to the release tarball or update the checksums for the auto-generated tarball without updating the version, it would need to be handled as a stealth update (dist_subdir ${name}/${version}_1). Instead I would probably handle it by preventing the use of the GitHub download location for now, since the change is of no significance to MacPorts (master_sites macports_distfiles) and when the next version of numpy is released, decide then whether to switch to the release tarball (github.tarball_from releases). One must bear in mind that the py-numpy port handles several different versions of numpy depending on python version, so these decisions will have to be coded carefully.

[michaelld (Michael Dickens)]

Thanks for the summary @ryandesign ... I will investigate moving to the release & doing the stealth update procedure. I didn't realize that the default github.tarball_from was -not- release ... very interesting!

[michaelld (Michael Dickens)]

hmnmm ... so it looks like the default -is- release ... not sure what's going on then in this Portfile ... let me see what I can figure out here ...

[michaelld (Michael Dickens)]

well I can fix the stealth update ... not sure about the tarball release style ... seems like the NumPy folks released the first version & then did an updated version sometime ... which by our definition is a stealth update & hence requires that specific change to the Portfile to handle it the most robustly.

[michaelld (Michael Dickens)]

In 22ae352f72772737a5e3a8f60d7926523a5d80dd/macports-ports (master):

py37+-numpy : stealth update

Closes: #63692

[ryandesign (Ryan Carsten Schmidt)]

The default value of github.tarball_from is tarball. It cannot be releases since not all projects have release distfiles and since changing the default now would break all the ports that depend on the existing default. Ports for projects that have release distfiles should usually use github.tarball_from releases. Ports for projects that do not have release distfiles should ususally use github.tarball_from archive.

I explained above that the differences were exclusively due to upstream removing a git tag, which has no effect on MacPorts, so switching to the new tarball was unnecessary as was increasing the revision, but it's too late to change it now.

[michaelld (Michael Dickens)]

Ah ... wow I really missed the boat there. ​Looking at the code, this setting is clearly -not- what I wrote ... thanks for clarifying & I will add a note to myself to swap that setting in with the next release. Totally makes sense, and will help avoid these stealth update.