Lots of golang ports are downloading dependencies at build time

[amake (Aaron Madlon-Kay)]

With the introduction of the Go modules system, it's very easy to accidentally make a port that downloads dependencies at build time.

I want to add GOPROXY=off GO111MODULE=off to build.env to prevent this, but there are many ports that currently fail in my testing.

Discover ports using the golang-1.0 portgroup (currently 85):

find . -name Portfile | xargs grep -l -E 'PortGroup +golang' | xargs -n 1 dirname | xargs -n 1 basename

Of those, ports that failed to build with GOPROXY=off GO111MODULE=off appended to build.env:

port	maintainer	status
annie	@l2dy,openmaintainer
aws-vault	@herbygillot,openmaintainer	​fixed
certigo	@herbygillot,openmaintainer
chezmoi	@herbygillot,openmaintainer
cloudmonkey	@herbygillot,openmaintainer
copilot	@herbygillot,openmaintainer
croc	@herbygillot,openmaintainer	​fixed
elvish	@herbygillot,openmaintainer	​fixed
evans	@herbygillot,openmaintainer
fzf	@cardi,openmaintainer
gitqlite	@herbygillot,openmaintainer
glow	@herbygillot,openmaintainer	​fixed
go-migrate	@herbygillot,openmaintainer
golangci-lint	@herbygillot,openmaintainer	Non-trivial: import path error
gore	@herbygillot,openmaintainer	​fixed
gotop	@i0ntempest,openmaintainer	​fixed
grpcurl	@herbygillot,openmaintainer	Non-trivial: depends on multiple modules housed in same repo but with different versions
hugo	@cardi,openmaintainer
ipfs	@sirn,openmaintainer
istioctl	@nickolaev,openmaintainer	due to #61184
jenkins-cli	@harens,openmaintainer
k9s	@breun,openmaintainer	Non-trivial: import path error
krew	@herbygillot,openmaintainer	​pending
kubergrunt	@herbygillot,openmaintainer	#61185; Non-trivial: import path error
kustomize	@breun,openmaintainer
micro	@herbygillot,openmaintainer	​pending
mole	@herbygillot,openmaintainer	​fixed
newreleases	@herbygillot,openmaintainer	​fixed
pulumi	@herbygillot,openmaintainer	Non-trivial: contains numerous dependencies from unsupported domains
rclone	@eborisch,openmaintainer
scw	@dgsb,openmaintainer
staticcheck	@herbygillot,openmaintainer	​fixed
syncthing	@lbschenkel,openmaintainer
tektoncd-cli	@herbygillot,openmaintainer	Non-trivial: contains numerous dependencies from unsupported domains
terragrunt	@mjrc,openmaintainer
trivy	@herbygillot,openmaintainer	Non-trivial: contains numerous dependencies from unsupported domains
uni	@herbygillot,openmaintainer	​fixed
up	@herbygillot,openmaintainer	​fixed
webify	@harens,openmaintainer	​fixed
wtfutil	@herbygillot,openmaintainer	Non-trivial: contains numerous dependencies from unsupported domains
yq	@herbygillot,openmaintainer	​fixed

I have not yet checked that each of these failures is from being unable to download dependencies, as opposed to e.g. something more esoteric about GO111MODULE.

[ryandesign (Ryan Carsten Schmidt)]

Herby, you may wish to know about this since you've been doing a lot of go-based ports.

[herbygillot (Herby Gillot)]

Thank you @ryandesign, @amake

I will look to transition more of the Go ports over time to using go.vendor. It will be cumbersome, but will give it a go.

[herbygillot (Herby Gillot)]

go2port goes not seem to work very well... the checksums that it generates for go.vendor don't always match what MacPorts sees.... so when attempting to build a Portfile derived from go2port, the build process immediately fails on checksum mismatches.

[amake (Aaron Madlon-Kay)]

go2port is very much a "best effort" tool :(

I just had a go at fixing the glow port ​here and noted the following tricky parts:

1. go2port 20200217 will duplicate some dependencies. I just fixed this and pushed an update to the port, so you should see it soon.
2. There were some checksum mismatches as you mentioned; surprisingly enough it seemed like GitHub sometimes served truncated tarballs, because deleting and re-downloading fixed it. This is quite troubling, but I'm not sure what I can do about it.
3. Simply pasting the generated go.vendors block is not enough; you also have to give the main distfile a filename. Until you do so, the checksum phase will fail in a way that makes it look like issue (2) above.

One issue you might run into is that custom domains are generally not supported by either go2port or the golang portgroup*. So far all the packages with custom domains I've seen have been mere redirects to a supported domain like github.com. If you encounter something else then it might be quite difficult to handle.

*Someone has ​offered a patch to support arbitrary domains in go2port but since the golang portgroup would also need a lot of work I haven't yet properly reviewed it.

[Aaron Madlon-Kay <amake@…>]

In 9876801804d06272278ebfdbb8512532440a0f83/macports-ports (master):

glow: don't fetch dependencies in build phase

See #61192

[herbygillot (Herby Gillot)]

I feel like instead of having go2port reinvent the wheel, we should download deps _exactly_ as Go would... and so perhaps the go mod download command could be of some use here. The only downside is that Go seems to prefer using .zip files, which won't match MacPort's preferred .tar.gz suffix used in most places.

Within a Go project's source tree, go mod download -json will print the manifest that Go would use to download the project's dependencies if that project is using Go modules. This can lead to a much more robust way for MacPorts to vendor the Go deps.

Sample output:

{
        "Path": "cloud.google.com/go",
        "Version": "v0.26.0",
        "Info": "/Users/herby/go/pkg/mod/cache/download/cloud.google.com/go/@v/v0.26.0.info",
        "GoMod": "/Users/herby/go/pkg/mod/cache/download/cloud.google.com/go/@v/v0.26.0.mod",
        "Zip": "/Users/herby/go/pkg/mod/cache/download/cloud.google.com/go/@v/v0.26.0.zip",
        "Dir": "/Users/herby/go/pkg/mod/cloud.google.com/go@v0.26.0",
        "Sum": "h1:e0WKqKTd5BnrG8aKH3J3h+QvEIQtSUcf2n5UZ5ZgLtQ=",
        "GoModSum": "h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw="
}
{
        "Path": "github.com/BurntSushi/toml",
        "Version": "v0.3.1",
        "Info": "/Users/herby/go/pkg/mod/cache/download/github.com/!burnt!sushi/toml/@v/v0.3.1.info",
        "GoMod": "/Users/herby/go/pkg/mod/cache/download/github.com/!burnt!sushi/toml/@v/v0.3.1.mod",
        "Zip": "/Users/herby/go/pkg/mod/cache/download/github.com/!burnt!sushi/toml/@v/v0.3.1.zip",
        "Dir": "/Users/herby/go/pkg/mod/github.com/!burnt!sushi/toml@v0.3.1",
        "Sum": "h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=",
        "GoModSum": "h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU="
}
...

[amake (Aaron Madlon-Kay)]

Some background:

I made go2port before the modern go module system existed, when there were several competing dependency managers (glide, gopkg, glock); it's modeled after other portfile generators like pypi2port and cpan2port. I made the golang-1.0 portgroup largely based on other portgroup idioms and the peco portfile.

I am not a golang guy. I just wanted to make and maintain ports for tools that happened to be in golang. go2port was my first (and last, so far) golang project. So it sounds like you have a much better understanding of the ecosystem than I do.

There is no mandate to use either go2port or the golang-1.0 portgroup. If you have better ideas that better meet MacPorts requirements then by all means we should move on.

The go mod download -json output sounds very useful, but I'm not seeing a way to use that *within* a portfile that lets MacPorts manage the fetching (which is what we're really concerned with here). The best I can imagine is using it to generate the distfiles and checksums sections, but that's basically what go2port already does.

[amake (Aaron Madlon-Kay)]

I should add: In the pre-go-module-system world, there was no danger of dependencies being automatically downloaded, because all projects (as far as I know) either committed their dependencies or used a third-party tool like glide, et al. It's only recently that this has become a problem.

[i0ntempest]

What's the best way of getting the names/urls of all the dependencies used by Go? Once I know that I'll try fixing gotop.

[amake (Aaron Madlon-Kay)]

Replying to i0ntempest:

What's the best way of getting the names/urls of all the dependencies used by Go? Once I know that I'll try fixing gotop.

Currently it's to use ​go2port. When it works well it's good, but there are often little stumbling blocks. See the "fixed" links in the table for details.

[i0ntempest]

So gotop needs to fetch stuff from howett.net/plist which go.vendors doesn't support. Alternative is ​https://github.com/DHowett/go-plist, that fetches it to ${gopath}/src/github.com/DHowett/go-plist, then I'll need to link it to ${gopath}/src/howett.net/plist, only then it will build. Don't know if there's a better way but it works. PR: ​https://github.com/macports/macports-ports/pull/8486

[breun (Nils Breunese)]

I created and maintain the k9s and kustomize ports, because I needed those tools, but I am not very familiar with Go and I’m not really sure what I need to do. If anyone could outline the process that would be helpful. Or maybe someone can even create pull requests for these ports?

[Aaron Madlon-Kay <amake@…>]

In 6640210512cc56fe6ce46af3a0a4710d9d54edba/macports-ports (master):

yq: don't fetch dependencies in build phase

See #61192

[i0ntempest]

Run go2port on your software repo and it will output a portfile. You take the checksum block and the go.vendors block and put them into your portfile (replace the old checksum block), add GOPROXY=off and GO111MODULE=off to build.env, and see if it works. You'll most likely need to do a bit of additional fixing and tweaking.

[Aaron Madlon-Kay <amake@…>]

In 00bbcb613bfaf1aa0d87b2ac85000e441f520197/macports-ports (master):

up: don't fetch dependencies in build phase

See #61192

[i0ntempest]

In cc761aa313fab209e261105a7332d22fc80b91e1/macports-ports (master):

gotop: don't fetch dependencies in build phase

See #61192

[Aaron Madlon-Kay <aaron@…>]

In 439ecf855855cedac020127bb66c3f4ceb8d1c8a/macports-ports (master):

uni: don't fetch dependencies in build phase

See #61192

[Aaron Madlon-Kay <aaron@…>]

In a153862bd9542074156089436102d37273ba8501/macports-ports (master):

staticcheck: don't fetch dependencies in build phase

See #61192

[Aaron Madlon-Kay <aaron@…>]

In 397d9d31d286b03813c6be3f4b30ad9450f457b2/macports-ports (master):

mole: don't fetch dependencies in build phase

See #61192

[Aaron Madlon-Kay <aaron@…>]

In ba97efd7ca98ad4d750042744abcddaa796ee644/macports-ports (master):

newreleases: don't fetch dependencies in build phase

See #61192

[herbygillot (Herby Gillot)]

In 64cac642068e98d6f2ddcbbad398df2f6e664bc1/macports-ports (master):

yaegi: update to 0.9.1

• ensure no Go dependencies are to be fetched from the Internet during

build time with GOPROXY=off

• See: #61192

[herbygillot (Herby Gillot)]

Seeing another issue where things fail when go.vendors contains:

• two modules of the same name even though they live in different repos
• or two or more versions of the exact same module (main project may require vX, but a dependency uses vY)

[amake (Aaron Madlon-Kay)]

Replying to herbygillot:

Seeing another issue where things fail when go.vendors contains:

• two modules of the same name even though they live in different repos
• or two or more versions of the exact same module (main project may require vX, but a dependency uses vY)

I fixed one limitation relating to this recently in ​https://github.com/macports/macports-ports/commit/3ad6f3ce0a0c011466bbd3e076295879f1148975, but there are still limitations relating to how GitHub names tarballs: tarballs fetched from a tag ref have the tag's hash in the name, so to identify the extracted directory you must either

• Specify the hash in addition to the tag (cumbersome), or
• Only use the hash and never a tag (also cumbersome, harder to understand)

The workaround currently used by both golang-1.0 and github-1.0 portgroups is to use globs (${author}-${project}-*). This works for 99% of cases but obviously fails in the case of multiple tarballs from the same repo.

[breun (Nils Breunese)]

I tried using go2port for the k9s port and I got a Portfile with really long list of go.vendors. I then added the other metadata, like description, license, etc. from the current k9s Portfile.

However, when I run portindex I get this error:

Error: go.vendors can't handle dependencies from k8s.io
Failed to parse file sysutils/k9s/Portfile: can't set "go.vendors": invalid command name "unsupported dependency domain"

All entries with github.com URLs in go.vendors look fine, but the ones for k8s.io, sigs.k8s.io, google.golang.org, go.uber.org, helm.sh, go.opencensus.io, rsc.io, vbom.ml all have rmd160, sha256 and size set to 0. It seems that these domains are not supported by go.vendors. Is there a way around this?

[amake (Aaron Madlon-Kay)]

Replying to breun:

All entries with github.com URLs in go.vendors look fine, but the ones for k8s.io, sigs.k8s.io, google.golang.org, go.uber.org, helm.sh, go.opencensus.io, rsc.io, vbom.ml all have rmd160, sha256 and size set to 0. It seems that these domains are not supported by go.vendors. Is there a way around this?

Those domains are indeed not supported by go.vendors. Golang allows arbitrary domains in package identifiers; often those domains simply redirect to well-known hosts like GitHub, but at the moment we don't have a way to figure that out.

The only way to handle this right now would be

1. Manually resolve those package IDs
2. If the actual host is a known one like GitHub, then replace the package ID with the GitHub, etc., equivalent
3. In the post-extract phase, move the extracted directory to its correct place in the GOPATH according to its original package ID

If the package resolves to a host that is not understood by the golang-1.0 portgroup, then you would have to set up the distfile manually the usual way with distfiles and master_sites.

I have an idea for an improvement. I will ping you with more later.

[amake (Aaron Madlon-Kay)]

I've updated both go2port and the golang-1.0 portgroup to support auto-resolving custom domains (thanks to ​@mohd-akram). With this, a lot of the dependencies for k9s will be handled nicely automatically. However some will not:

• Any deps "deeper" than the top level (like github.com/Azure/go-autorest/autorest/adal) don't work right. The root is also a dependency so you can delete those.
• google.golang.org/protobuf is hosted at go.googlesource.com; tarballs from here ​are not deterministic. You can manually replace it with github.com/protocolbuffers/protobuf-go
• The Microsoft dependencies reference github.com/Microsoft, but this has been renamed to github.com/microsoft. This needs to be manually resolved.

Unfortunately after fixing all of that, I got build errors like:

# github.com/deislabs/oras/pkg/content
../../deislabs/oras/pkg/content/file.go:23:2: cannot use &FileStore literal (type *FileStore) as type ProvideIngester in assignment:
	*FileStore does not implement ProvideIngester (wrong type for ReaderAt method)
		have ReaderAt(context.Context, "github.com/opencontainers/image-spec/specs-go/v1".Descriptor) (content.ReaderAt, error)
		want ReaderAt(context.Context, "github.com/containerd/containerd/vendor/github.com/opencontainers/image-spec/specs-go/v1".Descriptor) (content.ReaderAt, error)

The errors seem to have something to do with incorrectly resolving full type names in the GOPATH. I don't know enough about go to fix it.

I'll attach the complete portfile in case it helps.

[amake (Aaron Madlon-Kay)]

k9s portfile with go.vendors

[mohd-akram (Mohamed Akram)]

I'm also having an issue updating ​caddy because of two similarly named packages (qtls and qtls-go1-15). I think fetching from /archive/ instead of /tarball/ on GitHub can fix this issue since it uses more sane naming. Perhaps there could be an option akin to github.tarball_from to not break existing ports. In the meantime, filtering the globbed list via regex could help too.

[herbygillot (Herby Gillot)]

In d7f7ca6dabc2f935ccc179e60fe5ff5200976ffa/macports-ports (master):

aws-vault: update to 6.2.0

• switch deps to go.vendor, disable downloading dependencies at build time
• See: #61192

[herbygillot (Herby Gillot)]

In 236e0c2f119ed5b5fd1d6f2da5b3ec670d36706b/macports-ports (master):

elvish: vendor go dependencies, don't download them at build time

• See: #61192

[herbygillot (Herby Gillot)]

In 76f74db8796dc5f03c9fcaf6b686dfd9ce1addca/macports-ports (master):

gore: vendor Go dependencies, do not download dependencies at build time

• See: #61192

[harens (Haren S)]

In 1d5abbe395918dc567f95b0e65e31646b513d41b/macports-ports (master):

webify: don't fetch dependencies in build phase

[amake (Aaron Madlon-Kay)]

I tried to go.vendorize the kubergrunt port and ran into basically the same issues described ​above. I'll attach the portfile just in case someone figures out how to fix it.

[amake (Aaron Madlon-Kay)]

kubergrunt portfile with go.vendors

[amake (Aaron Madlon-Kay)]

Similar import issue for golangci-lint as with kubergrunt and k9s.

[amake (Aaron Madlon-Kay)]

golangci-lint portfile with go.vendors