github portgroup: offer the option to use archive URLs instead of tarball URLs

[fred.brunner@…]

Trying to update a Github-hosted project via Macports, the tarball pathname seems to have changed from 'tarball' to 'archive'. With the default option in place for the github.master_sites config parameter, attempted fetch of the tarball fails, if changed to 'archive' it works fine. I may be missing something about how this is supposed to work, but I have attached a diff file for the portgroup definition with this change in case it is useful.

[fred.brunner@…]

updated Githib portgroup template

[ryandesign (Ryan Carsten Schmidt)]

The portgroup works fine for me as is.

Making the suggested change seems to cause the checksums of existing ports' distfiles to change, which would necessitate changes to every port using the github portgroup, so before considering doing so I would want to understand the benefit.

[gorticus (Jason Mitchell)]

Replying to ryandesign@…:

The portgroup works fine for me as is.

Making the suggested change seems to cause the checksums of existing ports' distfiles to change, which would necessitate changes to every port using the github portgroup, so before considering doing so I would want to understand the benefit.

Perhaps I'm restating things that are already known, but from the discussion over on #46241 ...

While playing w/ the GitHub API, I discovered that the old API is redirected to a legacy API, e.g., the prior

https://github.com/user/repo/{tarball,zipball}/{branch,tag,sha}

is redirected to

https://codeload.github.com/user/repo/legacy.{tar.gz,zip}/{branch,tag,sha}

However, based on the file structure of the returned archive, it's not clear this is a drop-in replacement, i.e., the top level director is user-repo-sha/. So it seems likely that checksums using the existing would have to be revisited.

Alternatively, for GitHub'ers using the v3 API, perhaps just by default, could you grant github.tarball_from a new archive or v3 option, e.g.,

archive {
  github.master_sites ${github.homepage}/archive/${version}
}
default { ...

It may also be possible to query the repo via the API to determine what method it used to to provide distfiles, e.g., gitflow AVH Edition v3 items of interest:

"has_downloads": true,
"archive_url": "https://api.github.com/repos/petervanderdoes/gitflow/{archive_format}{/ref}",
"releases_url": "https://api.github.com/repos/petervanderdoes/gitflow/releases{/id}",
"downloads_url": "https://api.github.com/repos/petervanderdoes/gitflow/downloads",

unfortunately, querying downloads_url returns an empty set, e.g.,

$ curl -o- https://api.github.com/repos/petervanderdoes/gitflow/downloads
[

]

[ryandesign (Ryan Carsten Schmidt)]

I've never looked at any version of github API. I've just looked at the github web site to determine what URLs exist, and coded the github portgroup to match.

I may some day:

• add archive URL support to the github portgroup via a new option "github.tarball_from archive"
• add "github.tarball_from tags" to all ports currently relying on that being the default, along with a comment advising the portfile author to remove it and switch to archives with the next update
• change the default for github.tarball_from to archive

However, since "tags" and "archive" tarballs have exactly the same contents (with just a different enclosing directory name, and thus a different checksum), it is not a high priority for me.

Really, the most convincing argument I have right now for doing this is so that I will no longer need to tell people to stop submitting portfiles in which they have overridden the github portgroup's default master_sites with the archive URL, which has happened now more often than I can remember.

[gorticus (Jason Mitchell)]

git's interface is more stable than GitHub's API. I suspect we'll be a generation or two down the road in VCS tools before git clone is deprecated.

[ryandesign (Ryan Carsten Schmidt)]

Yes, but in MacPorts we prefer to use distfiles, because they can be mirrored and checksummed and don't have to be fetched every time.

[larryv (Lawrence Velázquez)]

Has duplicate #47895.

[larryv (Lawrence Velázquez)]

Cc Me!

[MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)]

In ec36db505d63f07eda197db1dbd8d3bcbd30a31c/macports-ports:

py-graphviz: add improvement reminder in comments

See ​https://lists.macports.org/pipermail/macports-dev/2017-August/036291.html
See #40518

[l2dy (Zero King)]

Replying to ryandesign:

However, since "tags" and "archive" tarballs have exactly the same contents (with just a different enclosing directory name, and thus a different checksum), it is not a high priority for me.

Really, the most convincing argument I have right now for doing this is so that I will no longer need to tell people to stop submitting portfiles in which they have overridden the github portgroup's default master_sites with the archive URL, which has happened now more often than I can remember.

One more argument: NeoMutt provides sha256 checksum and GPG signature for "archive" tarballs, which obviously can't be verified with "tags" tarballs. e.g. ​https://github.com/neomutt/neomutt/releases/tag/neomutt-20180323

[chrstphrchvz (Christopher Chavez)]

A PR to add this has been open for several months: ​https://github.com/macports/macports-ports/pull/2587

[herbygillot (Herby Gillot)]

The Github PR has been closed and merged, this ticket can be closed. You can now do:

github.tarball_from archive