Lots of golang ports are downloading dependencies at build time

[amake (Aaron Madlon-Kay)]

With the introduction of the Go modules system, it's very easy to accidentally make a port that downloads dependencies at build time.

I want to add GOPROXY=off GO111MODULE=off to build.env to prevent this, but there are many ports that currently fail in my testing.

Discover ports using the golang-1.0 portgroup (currently 85):

find . -name Portfile | xargs grep -l -E 'PortGroup +golang' | xargs -n 1 dirname | xargs -n 1 basename

Of those, ports that failed to build with GOPROXY=off GO111MODULE=off appended to build.env:

port	maintainer	status
annie	@l2dy,openmaintainer
aws-vault	@herbygillot,openmaintainer
certigo	@herbygillot,openmaintainer
chezmoi	@herbygillot,openmaintainer
cloudmonkey	@herbygillot,openmaintainer
copilot	@herbygillot,openmaintainer
croc	@herbygillot,openmaintainer	​fixed
elvish	@herbygillot,openmaintainer
evans	@herbygillot,openmaintainer
fzf	@cardi,openmaintainer
gitqlite	@herbygillot,openmaintainer
glow	@herbygillot,openmaintainer	​fixed
go-migrate	@herbygillot,openmaintainer
golangci-lint	@herbygillot,openmaintainer
gore	@herbygillot,openmaintainer
gotop	@i0ntempest,openmaintainer	​fixed
grpcurl	@herbygillot,openmaintainer
hugo	@cardi,openmaintainer
ipfs	@sirn,openmaintainer
istioctl	@nickolaev,openmaintainer	due to #61184
jenkins-cli	@harens,openmaintainer
k9s	@breun,openmaintainer
krew	@herbygillot,openmaintainer
kubergrunt	@herbygillot,openmaintainer	#61185
kustomize	@breun,openmaintainer
micro	@herbygillot,openmaintainer
mole	@herbygillot,openmaintainer	​fixed
newreleases	@herbygillot,openmaintainer	​fixed
pulumi	@herbygillot,openmaintainer	Non-trivial: contains numerous dependencies from unsupported domains
rclone	@eborisch,openmaintainer
scw	@dgsb,openmaintainer
staticcheck	@herbygillot,openmaintainer	​fixed
syncthing	@lbschenkel,openmaintainer
tektoncd-cli	@herbygillot,openmaintainer	Non-trivial: contains numerous dependencies from unsupported domains
terragrunt	@mjrc,openmaintainer
trivy	@herbygillot,openmaintainer	Non-trivial: contains numerous dependencies from unsupported domains
uni	@herbygillot,openmaintainer	​fixed
up	@herbygillot,openmaintainer	​fixed
webify	@harens,openmaintainer
wtfutil	@herbygillot,openmaintainer	Non-trivial: contains numerous dependencies from unsupported domains
yq	@herbygillot,openmaintainer	​fixed

I have not yet checked that each of these failures is from being unable to download dependencies, as opposed to e.g. something more esoteric about GO111MODULE.

[ryandesign (Ryan Carsten Schmidt)]

Herby, you may wish to know about this since you've been doing a lot of go-based ports.

[herbygillot (Herby Gillot)]

Thank you @ryandesign, @amake

I will look to transition more of the Go ports over time to using go.vendor. It will be cumbersome, but will give it a go.

[herbygillot (Herby Gillot)]

go2port goes not seem to work very well... the checksums that it generates for go.vendor don't always match what MacPorts sees.... so when attempting to build a Portfile derived from go2port, the build process immediately fails on checksum mismatches.

[amake (Aaron Madlon-Kay)]

go2port is very much a "best effort" tool :(

I just had a go at fixing the glow port ​here and noted the following tricky parts:

1. go2port 20200217 will duplicate some dependencies. I just fixed this and pushed an update to the port, so you should see it soon.
2. There were some checksum mismatches as you mentioned; surprisingly enough it seemed like GitHub sometimes served truncated tarballs, because deleting and re-downloading fixed it. This is quite troubling, but I'm not sure what I can do about it.
3. Simply pasting the generated go.vendors block is not enough; you also have to give the main distfile a filename. Until you do so, the checksum phase will fail in a way that makes it look like issue (2) above.

One issue you might run into is that custom domains are generally not supported by either go2port or the golang portgroup*. So far all the packages with custom domains I've seen have been mere redirects to a supported domain like github.com. If you encounter something else then it might be quite difficult to handle.

*Someone has ​offered a patch to support arbitrary domains in go2port but since the golang portgroup would also need a lot of work I haven't yet properly reviewed it.

[Aaron Madlon-Kay <amake@…>]

In 9876801804d06272278ebfdbb8512532440a0f83/macports-ports (master):

glow: don't fetch dependencies in build phase

See #61192

[herbygillot (Herby Gillot)]

I feel like instead of having go2port reinvent the wheel, we should download deps _exactly_ as Go would... and so perhaps the go mod download command could be of some use here. The only downside is that Go seems to prefer using .zip files, which won't match MacPort's preferred .tar.gz suffix used in most places.

Within a Go project's source tree, go mod download -json will print the manifest that Go would use to download the project's dependencies if that project is using Go modules. This can lead to a much more robust way for MacPorts to vendor the Go deps.

Sample output:

{
        "Path": "cloud.google.com/go",
        "Version": "v0.26.0",
        "Info": "/Users/herby/go/pkg/mod/cache/download/cloud.google.com/go/@v/v0.26.0.info",
        "GoMod": "/Users/herby/go/pkg/mod/cache/download/cloud.google.com/go/@v/v0.26.0.mod",
        "Zip": "/Users/herby/go/pkg/mod/cache/download/cloud.google.com/go/@v/v0.26.0.zip",
        "Dir": "/Users/herby/go/pkg/mod/cloud.google.com/go@v0.26.0",
        "Sum": "h1:e0WKqKTd5BnrG8aKH3J3h+QvEIQtSUcf2n5UZ5ZgLtQ=",
        "GoModSum": "h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw="
}
{
        "Path": "github.com/BurntSushi/toml",
        "Version": "v0.3.1",
        "Info": "/Users/herby/go/pkg/mod/cache/download/github.com/!burnt!sushi/toml/@v/v0.3.1.info",
        "GoMod": "/Users/herby/go/pkg/mod/cache/download/github.com/!burnt!sushi/toml/@v/v0.3.1.mod",
        "Zip": "/Users/herby/go/pkg/mod/cache/download/github.com/!burnt!sushi/toml/@v/v0.3.1.zip",
        "Dir": "/Users/herby/go/pkg/mod/github.com/!burnt!sushi/toml@v0.3.1",
        "Sum": "h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=",
        "GoModSum": "h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU="
}
...

[amake (Aaron Madlon-Kay)]

Some background:

I made go2port before the modern go module system existed, when there were several competing dependency managers (glide, gopkg, glock); it's modeled after other portfile generators like pypi2port and cpan2port. I made the golang-1.0 portgroup largely based on other portgroup idioms and the peco portfile.

I am not a golang guy. I just wanted to make and maintain ports for tools that happened to be in golang. go2port was my first (and last, so far) golang project. So it sounds like you have a much better understanding of the ecosystem than I do.

There is no mandate to use either go2port or the golang-1.0 portgroup. If you have better ideas that better meet MacPorts requirements then by all means we should move on.

The go mod download -json output sounds very useful, but I'm not seeing a way to use that *within* a portfile that lets MacPorts manage the fetching (which is what we're really concerned with here). The best I can imagine is using it to generate the distfiles and checksums sections, but that's basically what go2port already does.

[amake (Aaron Madlon-Kay)]

I should add: In the pre-go-module-system world, there was no danger of dependencies being automatically downloaded, because all projects (as far as I know) either committed their dependencies or used a third-party tool like glide, et al. It's only recently that this has become a problem.

[i0ntempest]

What's the best way of getting the names/urls of all the dependencies used by Go? Once I know that I'll try fixing gotop.

[amake (Aaron Madlon-Kay)]

Replying to i0ntempest:

What's the best way of getting the names/urls of all the dependencies used by Go? Once I know that I'll try fixing gotop.

Currently it's to use ​go2port. When it works well it's good, but there are often little stumbling blocks. See the "fixed" links in the table for details.

[i0ntempest]

So gotop needs to fetch stuff from howett.net/plist which go.vendors doesn't support. Alternative is ​https://github.com/DHowett/go-plist, that fetches it to ${gopath}/src/github.com/DHowett/go-plist, then I'll need to link it to ${gopath}/src/howett.net/plist, only then it will build. Don't know if there's a better way but it works. PR: ​https://github.com/macports/macports-ports/pull/8486

[breun (Nils Breunese)]

I created and maintain the k9s and kustomize ports, because I needed those tools, but I am not very familiar with Go and I’m not really sure what I need to do. If anyone could outline the process that would be helpful. Or maybe someone can even create pull requests for these ports?

[Aaron Madlon-Kay <amake@…>]

In 6640210512cc56fe6ce46af3a0a4710d9d54edba/macports-ports (master):

yq: don't fetch dependencies in build phase

See #61192

[i0ntempest]

Run go2port on your software repo and it will output a portfile. You take the checksum block and the go.vendors block and put them into your portfile (replace the old checksum block), add GOPROXY=off and GO111MODULE=off to build.env, and see if it works. You'll most likely need to do a bit of additional fixing and tweaking.

[Aaron Madlon-Kay <amake@…>]

In 00bbcb613bfaf1aa0d87b2ac85000e441f520197/macports-ports (master):

up: don't fetch dependencies in build phase

See #61192

[i0ntempest]

In cc761aa313fab209e261105a7332d22fc80b91e1/macports-ports (master):

gotop: don't fetch dependencies in build phase

See #61192

[Aaron Madlon-Kay <aaron@…>]

In 439ecf855855cedac020127bb66c3f4ceb8d1c8a/macports-ports (master):

uni: don't fetch dependencies in build phase

See #61192

[Aaron Madlon-Kay <aaron@…>]

In a153862bd9542074156089436102d37273ba8501/macports-ports (master):

staticcheck: don't fetch dependencies in build phase

See #61192

[Aaron Madlon-Kay <aaron@…>]

In 397d9d31d286b03813c6be3f4b30ad9450f457b2/macports-ports (master):

mole: don't fetch dependencies in build phase

See #61192

[Aaron Madlon-Kay <aaron@…>]

In ba97efd7ca98ad4d750042744abcddaa796ee644/macports-ports (master):

newreleases: don't fetch dependencies in build phase

See #61192

[herbygillot (Herby Gillot)]

In 64cac642068e98d6f2ddcbbad398df2f6e664bc1/macports-ports (master):

yaegi: update to 0.9.1

• ensure no Go dependencies are to be fetched from the Internet during

build time with GOPROXY=off

• See: #61192

[herbygillot (Herby Gillot)]

Seeing another issue where things fail when go.vendors contains:

• two modules of the same name even though they live in different repos
• or two or more versions of the exact same module (main project may require vX, but a dependency uses vY)