Opened 4 years ago

Last modified 19 months ago

#61192 new defect

Lots of golang ports are downloading dependencies at build time

Reported by: amake (Aaron Madlon-Kay) Owned by:
Priority: Normal Milestone:
Component: ports Version:
Keywords: Cc: breun (Nils Breunese), cardi (calvin ardi), dgsb (David Bariod), eborisch (Eric A. Borisch), harens (Haren S), herbygillot (Herby Gillot), lbschenkel (Leonardo Brondani Schenkel), mjrc, nickolaev, sirn (Kridsada Thanabulpong), dsedivec, cooljeanius (Eric Gallager), quentinmit (Quentin Smith), jamesog (James O'Gorman), mascguy (Christopher Nielsen)
Port: annie aws-vault certigo chezmoi cloudmonkey copilot croc elvish evans fzf gitqlite glow go-migrate golangci-lint gore gotop grpcurl hugo ipfs istioctl jenkins-cli k9s krew kubergrunt kustomize micro mole newreleases pulumi rclone scw staticcheck syncthing tektoncd-cli terragrunt trivy uni up webify wtfutil yq nebula qri

Description (last modified by amake (Aaron Madlon-Kay))

With the introduction of the Go modules system, it's very easy to accidentally make a port that downloads dependencies at build time.

I want to add GOPROXY=off GO111MODULE=off to build.env to prevent this, but there are many ports that currently fail in my testing.

Discover ports using the golang-1.0 portgroup (currently 85):

find . -name Portfile | xargs grep -l -E 'PortGroup +golang' | xargs -n 1 dirname | xargs -n 1 basename

Of those, ports that failed to build with GOPROXY=off GO111MODULE=off appended to build.env:

port maintainer status
annie@l2dy,openmaintainerfixed
aws-vault@herbygillot,openmaintainerfixed
certigo@herbygillot,openmaintainerfixed
chezmoi@herbygillot,openmaintainerfixed
cloudmonkey@herbygillot,openmaintainerfixed
copilot@herbygillot,openmaintainerNon-trivial: has a complicated asset generating step that relies on multiple golang tools that should be ports of their own: mockgen, packr2
croc@herbygillot,openmaintainerfixed
elvish@herbygillot,openmaintainerfixed
evans@herbygillot,openmaintainerfixed
fzf@cardi,openmaintainerfixed
gitqlite@herbygillot,openmaintainerfixed
glow@herbygillot,openmaintainerfixed
go-migrate@herbygillot,openmaintainerNon-trivial: depends on multiple modules housed in same repo but with different versions
golangci-lint@herbygillot,openmaintainerNon-trivial: import path error
gore@herbygillot,openmaintainerfixed
gotop@i0ntempest,openmaintainerfixed
grpcurl@herbygillot,openmaintainerNon-trivial: depends on multiple modules housed in same repo but with different versions
hugo@cardi,openmaintainerNon-trivial: depends on multiple modules housed in same repo but with different versions
ipfs@sirn,openmaintainerNon-trivial: extracted source file name collisions
istioctl@nickolaev,openmaintainerdue to #61184
jenkins-cli@harens,openmaintainerfixed
k9s@breun,openmaintainerNon-trivial: import path error
krew@herbygillot,openmaintainerfixed
kubergrunt@herbygillot,openmaintainer#61185; Non-trivial: import path error
kustomize@breun,openmaintainerNon-trivial: depends on multiple modules housed in same repo but with different versions
micro@herbygillot,openmaintainerfixed
mole@herbygillot,openmaintainerfixed
newreleases@herbygillot,openmaintainerfixed
pulumi@herbygillot,openmaintainerNon-trivial: contains numerous dependencies from unsupported domains
rclone@eborisch,openmaintainerNon-trivial: depends on multiple modules housed in same repo but with different versions
scw@dgsb,openmaintainerpending
staticcheck@herbygillot,openmaintainerfixed
syncthing@lbschenkel,openmaintainerpending
tektoncd-cli@herbygillot,openmaintainerNon-trivial: contains numerous dependencies from unsupported domains
terragrunt@mjrc,openmaintainerNon-trivial: depends on multiple modules housed in same repo but with different versions
trivy@herbygillot,openmaintainerNon-trivial: contains numerous dependencies from unsupported domains
uni@herbygillot,openmaintainerfixed
up@herbygillot,openmaintainerfixed
webify@harens,openmaintainerfixed
wtfutil@herbygillot,openmaintainerNon-trivial: contains numerous dependencies from unsupported domains
yq@herbygillot,openmaintainerfixed
nebula@herbygillot,openmaintainerfixed
qri@herbygillot,openmaintainerNon-trivial: depends on multiple modules housed in same repo but with different versions

I have not yet checked that each of these failures is from being unable to download dependencies, as opposed to e.g. something more esoteric about GO111MODULE.

Attachments (4)

Portfile (60.5 KB) - added by amake (Aaron Madlon-Kay) 4 years ago.
k9s portfile with go.vendors
Portfile.2 (33.2 KB) - added by amake (Aaron Madlon-Kay) 3 years ago.
kubergrunt portfile with go.vendors
Portfile.3 (38.5 KB) - added by amake (Aaron Madlon-Kay) 3 years ago.
golangci-lint portfile with go.vendors
k9s-with-go.vendors-failure.txt (413.4 KB) - added by breun (Nils Breunese) 3 years ago.

Download all attachments as: .zip

Change History (107)

comment:1 Changed 4 years ago by ryandesign (Ryan Carsten Schmidt)

Cc: herbygillot added

Herby, you may wish to know about this since you've been doing a lot of go-based ports.

comment:2 Changed 4 years ago by herbygillot (Herby Gillot)

Thank you @ryandesign, @amake

I will look to transition more of the Go ports over time to using go.vendor. It will be cumbersome, but will give it a go.

comment:3 Changed 4 years ago by herbygillot (Herby Gillot)

go2port goes not seem to work very well... the checksums that it generates for go.vendor don't always match what MacPorts sees.... so when attempting to build a Portfile derived from go2port, the build process immediately fails on checksum mismatches.

comment:4 Changed 4 years ago by amake (Aaron Madlon-Kay)

go2port is very much a "best effort" tool :(

I just had a go at fixing the glow port here and noted the following tricky parts:

  1. go2port 20200217 will duplicate some dependencies. I just fixed this and pushed an update to the port, so you should see it soon.
  2. There were some checksum mismatches as you mentioned; surprisingly enough it seemed like GitHub sometimes served truncated tarballs, because deleting and re-downloading fixed it. This is quite troubling, but I'm not sure what I can do about it.
  3. Simply pasting the generated go.vendors block is not enough; you also have to give the main distfile a filename. Until you do so, the checksum phase will fail in a way that makes it look like issue (2) above.

One issue you might run into is that custom domains are generally not supported by either go2port or the golang portgroup*. So far all the packages with custom domains I've seen have been mere redirects to a supported domain like github.com. If you encounter something else then it might be quite difficult to handle.

*Someone has offered a patch to support arbitrary domains in go2port but since the golang portgroup would also need a lot of work I haven't yet properly reviewed it.

Last edited 4 years ago by amake (Aaron Madlon-Kay) (previous) (diff)

comment:5 Changed 4 years ago by mf2k (Frank Schima)

Cc: @… @… @… @… @… @… @… @… @… @… @… @… added; herbygillot removed
Description: modified (diff)
Port: annie aws-vault certigo chezmoi cloudmonkey copilot croc elvish evans fzf gitqlite glow go-migrate golangci-lint gore gotop grpcurl hugo ipfs istioctl jenkins-cli k9s krew kubergrunt kustomize micro mole newreleases pulumi rclone scw staticcheck syncthing tektoncd-cli terragrunt trivy uni up webify wtfutil yq added

comment:6 Changed 4 years ago by Aaron Madlon-Kay <amake@…>

In 9876801804d06272278ebfdbb8512532440a0f83/macports-ports (master):

glow: don't fetch dependencies in build phase

See #61192

comment:7 Changed 4 years ago by herbygillot (Herby Gillot)

I feel like instead of having go2port reinvent the wheel, we should download deps _exactly_ as Go would... and so perhaps the go mod download command could be of some use here. The only downside is that Go seems to prefer using .zip files, which won't match MacPort's preferred .tar.gz suffix used in most places.

Within a Go project's source tree, go mod download -json will print the manifest that Go would use to download the project's dependencies if that project is using Go modules. This can lead to a much more robust way for MacPorts to vendor the Go deps.

Sample output:

{
        "Path": "cloud.google.com/go",
        "Version": "v0.26.0",
        "Info": "/Users/herby/go/pkg/mod/cache/download/cloud.google.com/go/@v/v0.26.0.info",
        "GoMod": "/Users/herby/go/pkg/mod/cache/download/cloud.google.com/go/@v/v0.26.0.mod",
        "Zip": "/Users/herby/go/pkg/mod/cache/download/cloud.google.com/go/@v/v0.26.0.zip",
        "Dir": "/Users/herby/go/pkg/mod/cloud.google.com/go@v0.26.0",
        "Sum": "h1:e0WKqKTd5BnrG8aKH3J3h+QvEIQtSUcf2n5UZ5ZgLtQ=",
        "GoModSum": "h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw="
}
{
        "Path": "github.com/BurntSushi/toml",
        "Version": "v0.3.1",
        "Info": "/Users/herby/go/pkg/mod/cache/download/github.com/!burnt!sushi/toml/@v/v0.3.1.info",
        "GoMod": "/Users/herby/go/pkg/mod/cache/download/github.com/!burnt!sushi/toml/@v/v0.3.1.mod",
        "Zip": "/Users/herby/go/pkg/mod/cache/download/github.com/!burnt!sushi/toml/@v/v0.3.1.zip",
        "Dir": "/Users/herby/go/pkg/mod/github.com/!burnt!sushi/toml@v0.3.1",
        "Sum": "h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=",
        "GoModSum": "h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU="
}
...

comment:8 Changed 4 years ago by amake (Aaron Madlon-Kay)

Some background:

I made go2port before the modern go module system existed, when there were several competing dependency managers (glide, gopkg, glock); it's modeled after other portfile generators like pypi2port and cpan2port. I made the golang-1.0 portgroup largely based on other portgroup idioms and the peco portfile.

I am not a golang guy. I just wanted to make and maintain ports for tools that happened to be in golang. go2port was my first (and last, so far) golang project. So it sounds like you have a much better understanding of the ecosystem than I do.

There is no mandate to use either go2port or the golang-1.0 portgroup. If you have better ideas that better meet MacPorts requirements then by all means we should move on.

The go mod download -json output sounds very useful, but I'm not seeing a way to use that *within* a portfile that lets MacPorts manage the fetching (which is what we're really concerned with here). The best I can imagine is using it to generate the distfiles and checksums sections, but that's basically what go2port already does.

comment:9 Changed 4 years ago by amake (Aaron Madlon-Kay)

I should add: In the pre-go-module-system world, there was no danger of dependencies being automatically downloaded, because all projects (as far as I know) either committed their dependencies or used a third-party tool like glide, et al. It's only recently that this has become a problem.

comment:10 Changed 4 years ago by amake (Aaron Madlon-Kay)

Description: modified (diff)

comment:11 Changed 4 years ago by amake (Aaron Madlon-Kay)

Description: modified (diff)

comment:12 Changed 4 years ago by mf2k (Frank Schima)

Cc: breun cardi dgsb eborisch harens herbygillot i0ntempest l2dy lbschenkel mjrc nickolaev sirn added; @… @… @… @… @… @… @… @… @… @… @… @… removed

comment:13 Changed 4 years ago by i0ntempest

What's the best way of getting the names/url of all the dependencies used by Go? Once I know that I'll try fixing gotop.

Version 0, edited 4 years ago by i0ntempest (next)

comment:14 in reply to:  13 Changed 4 years ago by amake (Aaron Madlon-Kay)

Replying to i0ntempest:

What's the best way of getting the names/urls of all the dependencies used by Go? Once I know that I'll try fixing gotop.

Currently it's to use go2port. When it works well it's good, but there are often little stumbling blocks. See the "fixed" links in the table for details.

comment:15 Changed 4 years ago by amake (Aaron Madlon-Kay)

Description: modified (diff)

comment:16 Changed 4 years ago by i0ntempest

So gotop needs to fetch stuff from howett.net/plist which go.vendors doesn't support. Alternative is https://github.com/DHowett/go-plist, that fetches it to ${gopath}/src/github.com/DHowett/go-plist, then I'll need to link it to ${gopath}/src/howett.net/plist, only then it will build. Don't know if there's a better way but it works. PR: https://github.com/macports/macports-ports/pull/8486

Last edited 4 years ago by i0ntempest (previous) (diff)

comment:17 Changed 4 years ago by breun (Nils Breunese)

I created and maintain the k9s and kustomize ports, because I needed those tools, but I am not very familiar with Go and I’m not really sure what I need to do. If anyone could outline the process that would be helpful. Or maybe someone can even create pull requests for these ports?

Last edited 4 years ago by breun (Nils Breunese) (previous) (diff)

comment:18 Changed 4 years ago by Aaron Madlon-Kay <amake@…>

In 6640210512cc56fe6ce46af3a0a4710d9d54edba/macports-ports (master):

yq: don't fetch dependencies in build phase

See #61192

comment:19 Changed 4 years ago by i0ntempest

Run go2port on your software repo and it will output a portfile. You take the checksum block and the go.vendors block and put them into your portfile (replace the old checksum block), add GOPROXY=off and GO111MODULE=off to build.env, and see if it works. You'll most likely need to do a bit of additional fixing and tweaking.

comment:20 Changed 4 years ago by Aaron Madlon-Kay <amake@…>

In 00bbcb613bfaf1aa0d87b2ac85000e441f520197/macports-ports (master):

up: don't fetch dependencies in build phase

See #61192

comment:21 Changed 4 years ago by i0ntempest

In cc761aa313fab209e261105a7332d22fc80b91e1/macports-ports (master):

gotop: don't fetch dependencies in build phase

See #61192

comment:22 Changed 4 years ago by amake (Aaron Madlon-Kay)

Description: modified (diff)

comment:23 Changed 4 years ago by amake (Aaron Madlon-Kay)

Description: modified (diff)

comment:24 Changed 4 years ago by Aaron Madlon-Kay <aaron@…>

In 439ecf855855cedac020127bb66c3f4ceb8d1c8a/macports-ports (master):

uni: don't fetch dependencies in build phase

See #61192

comment:25 Changed 4 years ago by Aaron Madlon-Kay <aaron@…>

In a153862bd9542074156089436102d37273ba8501/macports-ports (master):

staticcheck: don't fetch dependencies in build phase

See #61192

comment:26 Changed 4 years ago by Aaron Madlon-Kay <aaron@…>

In 397d9d31d286b03813c6be3f4b30ad9450f457b2/macports-ports (master):

mole: don't fetch dependencies in build phase

See #61192

comment:27 Changed 4 years ago by Aaron Madlon-Kay <aaron@…>

In ba97efd7ca98ad4d750042744abcddaa796ee644/macports-ports (master):

newreleases: don't fetch dependencies in build phase

See #61192

comment:28 Changed 4 years ago by herbygillot (Herby Gillot)

In 64cac642068e98d6f2ddcbbad398df2f6e664bc1/macports-ports (master):

yaegi: update to 0.9.1

  • ensure no Go dependencies are to be fetched from the Internet during

build time with GOPROXY=off

comment:29 Changed 4 years ago by herbygillot (Herby Gillot)

Seeing another issue where things fail when go.vendors contains:

  • two modules of the same name even though they live in different repos
  • or two or more versions of the exact same module (main project may require vX, but a dependency uses vY)

comment:30 Changed 4 years ago by amake (Aaron Madlon-Kay)

Description: modified (diff)

comment:31 in reply to:  29 Changed 4 years ago by amake (Aaron Madlon-Kay)

Replying to herbygillot:

Seeing another issue where things fail when go.vendors contains:

  • two modules of the same name even though they live in different repos
  • or two or more versions of the exact same module (main project may require vX, but a dependency uses vY)

I fixed one limitation relating to this recently in https://github.com/macports/macports-ports/commit/3ad6f3ce0a0c011466bbd3e076295879f1148975, but there are still limitations relating to how GitHub names tarballs: tarballs fetched from a tag ref have the tag's hash in the name, so to identify the extracted directory you must either

  • Specify the hash in addition to the tag (cumbersome), or
  • Only use the hash and never a tag (also cumbersome, harder to understand)

The workaround currently used by both golang-1.0 and github-1.0 portgroups is to use globs (${author}-${project}-*). This works for 99% of cases but obviously fails in the case of multiple tarballs from the same repo.

comment:32 Changed 4 years ago by breun (Nils Breunese)

I tried using go2port for the k9s port and I got a Portfile with really long list of go.vendors. I then added the other metadata, like description, license, etc. from the current k9s Portfile.

However, when I run portindex I get this error:

Error: go.vendors can't handle dependencies from k8s.io
Failed to parse file sysutils/k9s/Portfile: can't set "go.vendors": invalid command name "unsupported dependency domain"

All entries with github.com URLs in go.vendors look fine, but the ones for k8s.io, sigs.k8s.io, google.golang.org, go.uber.org, helm.sh, go.opencensus.io, rsc.io, vbom.ml all have rmd160, sha256 and size set to 0. It seems that these domains are not supported by go.vendors. Is there a way around this?

comment:33 in reply to:  32 Changed 4 years ago by amake (Aaron Madlon-Kay)

Replying to breun:

All entries with github.com URLs in go.vendors look fine, but the ones for k8s.io, sigs.k8s.io, google.golang.org, go.uber.org, helm.sh, go.opencensus.io, rsc.io, vbom.ml all have rmd160, sha256 and size set to 0. It seems that these domains are not supported by go.vendors. Is there a way around this?

Those domains are indeed not supported by go.vendors. Golang allows arbitrary domains in package identifiers; often those domains simply redirect to well-known hosts like GitHub, but at the moment we don't have a way to figure that out.

The only way to handle this right now would be

  1. Manually resolve those package IDs
  2. If the actual host is a known one like GitHub, then replace the package ID with the GitHub, etc., equivalent
  3. In the post-extract phase, move the extracted directory to its correct place in the GOPATH according to its original package ID

If the package resolves to a host that is not understood by the golang-1.0 portgroup, then you would have to set up the distfile manually the usual way with distfiles and master_sites.

I have an idea for an improvement. I will ping you with more later.

comment:34 Changed 4 years ago by amake (Aaron Madlon-Kay)

I've updated both go2port and the golang-1.0 portgroup to support auto-resolving custom domains (thanks to @mohd-akram). With this, a lot of the dependencies for k9s will be handled nicely automatically. However some will not:

  • Any deps "deeper" than the top level (like github.com/Azure/go-autorest/autorest/adal) don't work right. The root is also a dependency so you can delete those.
  • google.golang.org/protobuf is hosted at go.googlesource.com; tarballs from here are not deterministic. You can manually replace it with github.com/protocolbuffers/protobuf-go
  • The Microsoft dependencies reference github.com/Microsoft, but this has been renamed to github.com/microsoft. This needs to be manually resolved.

Unfortunately after fixing all of that, I got build errors like:

# github.com/deislabs/oras/pkg/content
../../deislabs/oras/pkg/content/file.go:23:2: cannot use &FileStore literal (type *FileStore) as type ProvideIngester in assignment:
	*FileStore does not implement ProvideIngester (wrong type for ReaderAt method)
		have ReaderAt(context.Context, "github.com/opencontainers/image-spec/specs-go/v1".Descriptor) (content.ReaderAt, error)
		want ReaderAt(context.Context, "github.com/containerd/containerd/vendor/github.com/opencontainers/image-spec/specs-go/v1".Descriptor) (content.ReaderAt, error)

The errors seem to have something to do with incorrectly resolving full type names in the GOPATH. I don't know enough about go to fix it.

I'll attach the complete portfile in case it helps.

Changed 4 years ago by amake (Aaron Madlon-Kay)

Attachment: Portfile added

k9s portfile with go.vendors

comment:35 Changed 4 years ago by amake (Aaron Madlon-Kay)

Description: modified (diff)

comment:36 Changed 4 years ago by mohd-akram (Mohamed Akram)

I'm also having an issue updating caddy because of two similarly named packages (qtls and qtls-go1-15). I think fetching from /archive/ instead of /tarball/ on GitHub can fix this issue since it uses more sane naming. Perhaps there could be an option akin to github.tarball_from to not break existing ports. In the meantime, filtering the globbed list via regex could help too.

comment:37 Changed 4 years ago by herbygillot (Herby Gillot)

In d7f7ca6dabc2f935ccc179e60fe5ff5200976ffa/macports-ports (master):

aws-vault: update to 6.2.0

  • switch deps to go.vendor, disable downloading dependencies at build time
  • See: #61192

comment:38 Changed 3 years ago by herbygillot (Herby Gillot)

In 236e0c2f119ed5b5fd1d6f2da5b3ec670d36706b/macports-ports (master):

elvish: vendor go dependencies, don't download them at build time

comment:39 Changed 3 years ago by herbygillot (Herby Gillot)

In 76f74db8796dc5f03c9fcaf6b686dfd9ce1addca/macports-ports (master):

gore: vendor Go dependencies, do not download dependencies at build time

comment:40 Changed 3 years ago by harens (Haren S)

In 1d5abbe395918dc567f95b0e65e31646b513d41b/macports-ports (master):

webify: don't fetch dependencies in build phase

comment:41 Changed 3 years ago by amake (Aaron Madlon-Kay)

Description: modified (diff)

comment:42 Changed 3 years ago by amake (Aaron Madlon-Kay)

I tried to go.vendorize the kubergrunt port and ran into basically the same issues described above. I'll attach the portfile just in case someone figures out how to fix it.

Changed 3 years ago by amake (Aaron Madlon-Kay)

Attachment: Portfile.2 added

kubergrunt portfile with go.vendors

comment:43 Changed 3 years ago by amake (Aaron Madlon-Kay)

Description: modified (diff)

comment:44 Changed 3 years ago by amake (Aaron Madlon-Kay)

Description: modified (diff)

comment:45 Changed 3 years ago by amake (Aaron Madlon-Kay)

Similar import issue for golangci-lint as with kubergrunt and k9s.

Changed 3 years ago by amake (Aaron Madlon-Kay)

Attachment: Portfile.3 added

golangci-lint portfile with go.vendors

comment:46 Changed 3 years ago by amake (Aaron Madlon-Kay)

Description: modified (diff)

comment:47 Changed 3 years ago by i0ntempest

Cc: i0ntempest removed

comment:48 Changed 3 years ago by Aaron Madlon-Kay <amake@…>

In 22fba12fc00e5732fd60ca024c2e2324bc91670e/macports-ports (master):

krew: don't fetch dependencies in build phase

See #61192

comment:49 Changed 3 years ago by Aaron Madlon-Kay <aaron@…>

In 15d58e010e86a746536a2dd7d1d4e4b553267e76/macports-ports (master):

micro: don't fetch dependencies in build phase

See #61192

comment:50 Changed 3 years ago by amake (Aaron Madlon-Kay)

Description: modified (diff)

comment:51 Changed 3 years ago by amake (Aaron Madlon-Kay)

Description: modified (diff)

comment:52 Changed 3 years ago by amake (Aaron Madlon-Kay)

Description: modified (diff)

comment:53 Changed 3 years ago by breun (Nils Breunese)

I should add: In the pre-go-module-system world, there was no danger of dependencies being automatically downloaded, because all projects (as far as I know) either committed their dependencies or used a third-party tool like glide, et al. It's only recently that this has become a problem.

I must admit I do not completely understand what the problem exactly is. I maintain a couple of Go ports, but I'm not very familiar with the Go ecosystem. I am however very familiar with the Java ecosystem and the Maven build tool specifically and I'm used to build tools downloading dependencies at build time (when not previously downloaded and locally cached). Maven uses Maven repositories to download dependencies, which support hashes and signatures, etc. Is the problem that Go builds download dependencies without any checks? If not, what is the problem exactly?

Changed 3 years ago by breun (Nils Breunese)

comment:54 Changed 3 years ago by breun (Nils Breunese)

k9s portfile with go.vendors

Thanks for creating this. However, when I try to build this locally, it takes quite a long time and then fails to build with some Go build error that I don't understand. I attached k9s-with-go.vendors-failure.txt to this ticket. Anyone know what's wrong here?

comment:55 in reply to:  53 ; Changed 3 years ago by amake (Aaron Madlon-Kay)

Replying to breun:

I must admit I do not completely understand what the problem exactly is.

There are two problems:

  1. Downloading dependencies outside of MacPorts' fetch mechanism prevents MacPorts from mirroring distfiles. That's bad for a number of reasons, such as: upstream could disappear, or we send a lot of traffic their way putting a strain on their infrastructure.
  2. Some build-time dependency fetching schemes lack a locking mechanism, so the exact version of the dependency used can differ depending on when exactly the build was performed. This is bad for reproducibility.

I am however very familiar with the Java ecosystem and the Maven build tool specifically and I'm used to build tools downloading dependencies at build time (when not previously downloaded and locally cached).

Sure, that's fine and good, but suboptimal for a MacPorts port.

Maven uses Maven repositories to download dependencies, which support hashes and signatures, etc. Is the problem that Go builds download dependencies without any checks? If not, what is the problem exactly?

No, Maven is also problematic for (1) above. For (2), Maven does allow version ranges but they are not used that often in my experience, so that's good. But in my experience it also doesn't provide a lockfile-like mechanism that would ensure that dependencies can't be silently replaced upstream.

Are there ports building with Maven that are downloading their dependencies at build time? If so, I guess the main reason they have been given a pass in the past is, as I understand it:

  1. Until recently there was no Java port, and Java could not be assumed to be available, so we basically didn't build any Java ports from source; they have mostly been grabbing pre-compiled (JAR) distfiles
  2. No one was paying attention

comment:56 in reply to:  54 ; Changed 3 years ago by amake (Aaron Madlon-Kay)

Replying to breun:

k9s portfile with go.vendors

Thanks for creating this. However, when I try to build this locally, it takes quite a long time and then fails to build with some Go build error that I don't understand. I attached k9s-with-go.vendors-failure.txt to this ticket. Anyone know what's wrong here?

Yes, that's what I reported earlier.

I think the problem is that some of the dependencies have their own vendored dependencies (in a vendor dir at the dependency's root), and when GO111MODULE=off this is unexpected: instead it's expected that all packages should be flattened into the GOPATH.

I'm not sure what to do about it.

comment:57 Changed 3 years ago by Aaron Madlon-Kay <amake@…>

In 3bf12bd32bb5e0c007481c11c9a31dfb687904d5/macports-ports (master):

evans: don't fetch dependencies in build phase

See #61192

comment:58 Changed 3 years ago by amake (Aaron Madlon-Kay)

Description: modified (diff)

comment:59 Changed 3 years ago by dgsb (David Bariod)

I was wondering if it is wise to reimplement something that the go compiler already does. I think the go build problem may not be trivial to replicate in macports.

comment:60 Changed 3 years ago by Aaron Madlon-Kay <amake@…>

In c74c831164a7309e1b6dd142e9d5799910ffe535/macports-ports (master):

chezmoi: don't fetch dependencies in build phase

See #61192

comment:61 Changed 3 years ago by Aaron Madlon-Kay <amake@…>

In bc0da04788973104df02dd4267f675015602dcc7/macports-ports (master):

cloudmonkey: don't fetch dependencies in build phase

See #61192

comment:62 Changed 3 years ago by amake (Aaron Madlon-Kay)

Description: modified (diff)

comment:63 Changed 3 years ago by amake (Aaron Madlon-Kay)

Description: modified (diff)

comment:64 Changed 3 years ago by Aaron Madlon-Kay <aaron@…>

In 9c12bfc242d45fbe70c2c538e46018db0e3f9696/macports-ports (master):

certigo: don't fetch dependencies in build phase

See #61192

comment:65 Changed 3 years ago by Aaron Madlon-Kay <aaron@…>

In 2e02c5335d3bd0da951e330bb7450725c83bad62/macports-ports (master):

gitqlite: don't fetch dependencies in build phase

See #61192

comment:66 Changed 3 years ago by amake (Aaron Madlon-Kay)

Description: modified (diff)

comment:67 Changed 3 years ago by amake (Aaron Madlon-Kay)

Description: modified (diff)

comment:68 Changed 3 years ago by Aaron Madlon-Kay <aaron@…>

In aa1427b26d7ca24dff559d0d51c32bc7fdbb0edf/macports-ports (master):

annie: don't fetch dependencies in build phase

See #61192

comment:69 Changed 3 years ago by Aaron Madlon-Kay <aaron@…>

In 500aa1d2f53673b781d77ebe95dabcd2faf60803/macports-ports (master):

fzf: don't fetch dependencies in build phase

See #61192

comment:70 Changed 3 years ago by Aaron Madlon-Kay <amake@…>

In 25c9ac3c7d7d066eb33dd90c0c26b954e58c1ab2/macports-ports (master):

jenkins-cli: don't fetch dependencies in build phase

See #61192

comment:71 Changed 3 years ago by amake (Aaron Madlon-Kay)

Description: modified (diff)

comment:72 Changed 3 years ago by amake (Aaron Madlon-Kay)

Description: modified (diff)

comment:73 in reply to:  56 Changed 3 years ago by breun (Nils Breunese)

Replying to amake:

I think the problem is that some of the dependencies have their own vendored dependencies (in a vendor dir at the dependency's root), and when GO111MODULE=off this is unexpected: instead it's expected that all packages should be flattened into the GOPATH.

I'm not sure what to do about it.

Neither am I. I maintain these ports because I use these tools myself, but I am not familiar with how Go software gets built. I was already struggling quite a bit to get an initial Portfile for k9s working, and it seems that this go.vendor stuff adds a whole new dimension of complexity. Are there any people familiar enough with Go builds in this thread that could help out?

comment:74 Changed 3 years ago by amake (Aaron Madlon-Kay)

Description: modified (diff)

comment:75 Changed 3 years ago by Aaron Madlon-Kay <amake@…>

In b34a811028a6700b822a2c4aeb691f83ce37acd3/macports-ports (master):

syncthing: don't fetch dependencies in build phase

See #61192

comment:76 Changed 3 years ago by i0ntempest

I'm submitting cloudflared to macports (https://github.com/macports/macports-ports/pull/8662), it has two problematic dependencies. One is known to be unsolvable yet (multiple versions of the same dependency, go-oidc v1 and v2), another one seem to be a new problem (giving error: no files matched glob pattern "/opt/local/var/macports/build/_Users_Admin_Ports_net_cloudflared/cloudflared/work/kshvakov-clickhouse-*") However it builds and works anyway without these two dependencies. Please take a look if interested.

comment:77 Changed 3 years ago by i0ntempest

Cc: i0ntempest added

comment:78 Changed 3 years ago by herbygillot (Herby Gillot)

tektoncd-cli can be marked as fixed since it vendors all its dependencies already. It builds with GOPROXY=off, which has been added as of version 0.13.0:

https://github.com/macports/macports-ports/commit/cc7c646f38bc9729fbf1763df76abdbdff9d3a03

comment:79 Changed 3 years ago by herbygillot (Herby Gillot)

In 93f8502df9e9150e6f055a1a9f6cc06bbb3825c7/macports-ports (master):

ultralist: update to 1.5.1

  • vendor Go dependencies, do not download dependencies at build time

See: #61192

comment:80 Changed 3 years ago by i0ntempest

Cc: i0ntempest removed

comment:81 Changed 3 years ago by l2dy (Zero King)

Cc: l2dy removed

comment:82 Changed 3 years ago by i0ntempest

comment:83 Changed 3 years ago by amake (Aaron Madlon-Kay)

Description: modified (diff)
Port: nebula added

comment:84 Changed 3 years ago by amake (Aaron Madlon-Kay)

Port: qri added

comment:85 Changed 3 years ago by Aaron Madlon-Kay <amake@…>

comment:86 Changed 3 years ago by amake (Aaron Madlon-Kay)

Description: modified (diff)

comment:87 Changed 3 years ago by Aaron Madlon-Kay <amake@…>

In ce448c8a13df8398ee867fffc9f2b6ea71a3c6f2/macports-ports (master):

golang-1.0: disable fetching packages during build, test phases

See #61192

comment:88 Changed 3 years ago by Aaron Madlon-Kay <amake@…>

In a25e7b41049aeea1f57565369e77a2a8c6fab1ad/macports-ports (master):

Various: temporarily allow fetching deps during build

See #61192

comment:89 Changed 3 years ago by Aaron Madlon-Kay <amake@…>

In d79aad1b301412f3ff83438f4e0c23d3c513aeb2/macports-ports (master):

Various: remove superfluous golang envars

These are now set in the golang-1.0 portgroup

See #61192

comment:90 Changed 3 years ago by herbygillot (Herby Gillot)

In 0290dd1c8d056701778cb9a48a051eff77eba981/macports-ports (master):

mkcert: update to 1.4.2

  • use golang portgroup
  • vendor Go dependencies See: #61192

comment:91 Changed 3 years ago by herbygillot (Herby Gillot)

In 8ea3b202f9e30e23b0cf4d5f252ca4e579c5daf0/macports-ports (master):

vgrep: update to 2.5.0

  • vendor Go dependencies See: #61192

comment:92 Changed 3 years ago by herbygillot (Herby Gillot)

In 4270625f2748334ad06ed76f450e48b76b46d1e3/macports-ports (master):

promu: update to 0.7.0

  • use the golang portgroup
  • vendor Go dependencies to prevent downloading them at build time See: #61192

comment:93 Changed 3 years ago by herbygillot (Herby Gillot)

In 62b049b5aa9c1c2e21604fab30e066c8f80643e7/macports-ports (master):

copilot: update to 0.6.0

  • add packr and go-mockgen as build dependencies
  • vendor Go dependencies, do not download them at build time See: #61192

comment:94 Changed 3 years ago by emcrisostomo (Enrico Maria Crisostomo)

I'm trying to update kompose to v. 1.22.0 and I'm hitting this issue: it won't build with GOPROXY=off. Any guidance?

comment:95 Changed 3 years ago by herbygillot (Herby Gillot)

In 0f8835630589663ff4b462f987adae644b056333/macports-ports (master):

kubergrunt: update to 0.6.9

comment:96 Changed 3 years ago by Aaron Madlon-Kay <amake@…>

In 85ad5183de0036aae804c59c7d1a71e2f7d00bf5/macports-ports (master):

scw: don't fetch dependencies in build phase

See #61192

comment:97 in reply to:  55 Changed 3 years ago by gpanders (Gregory Anders)

I was looking at upgrading the caddy port, which is now quite old. Unfortunately, it is no longer possible to generate the caddy binary with version information without using go modules.

However, it looks like it is possible to both 1) download module dependencies before the build step (the issue which this ticket addresses) and 2) still use go's new module system. This can be done using the -mod vendor flag to go build.

The only thing this requires is to create a vendor directory that contains all of the vendored dependencies. Basically, instead of moving downloaded dependencies to gopath/src/, simply move them to vendor/. This also requires a file called vendor/modules.txt that contains all of the module dependencies; however, this can be generated once when the port is updated and kept as a patch alongside the Portfile.

comment:98 Changed 3 years ago by dsedivec

Cc: dsedivec added

comment:99 Changed 2 years ago by cooljeanius (Eric Gallager)

Cc: cooljeanius added

comment:100 in reply to:  55 Changed 22 months ago by quentinmit (Quentin Smith)

GO111MODULES=off is deprecated and scheduled to be removed soon: https://go.dev/blog/go116-module-changes

Replying to amake:

Replying to breun:

I must admit I do not completely understand what the problem exactly is.

There are two problems:

  1. Downloading dependencies outside of MacPorts' fetch mechanism prevents MacPorts from mirroring distfiles. That's bad for a number of reasons, such as: upstream could disappear, or we send a lot of traffic their way putting a strain on their infrastructure.
  2. Some build-time dependency fetching schemes lack a locking mechanism, so the exact version of the dependency used can differ depending on when exactly the build was performed. This is bad for reproducibility.

(2) is a core feature of Go's module system; in fact *every* module dependency is pinned to a specific version by the go.mod and go.sum files in the repo. Turning off GO111MODULE actually causes the latest HEAD of each dependency to be used, so setting GO111MODULE=off does the opposite of what you are hoping for.

I don't think there's an easy answer here; go mod download can be used as part of the fetch step to download all the dependencies, but that's still not going to use MacPorts's distfiles mechanism. Honestly I wouldn't worry about the load on the Go module proxies; they're provisioned at least as well as the distfiles network. But if you are worried, probably the best option would be to have the fetch step download module zip files from the module proxy, and prefill the local module cache in the extract step. Then go build will use the module zip files, and the module zip files can be downloaded as distfiles.

You can also use go mod vendor to download all the dependencies as a source tree, but I don't know how to integrate that with distfiles. You would probably need extra magic to generate a tarball of the output of go mod vendor in the distfiles system.

comment:101 Changed 22 months ago by quentinmit (Quentin Smith)

Cc: quentinmit added

comment:102 Changed 19 months ago by jamesog (James O'Gorman)

Cc: jamesog added

comment:103 Changed 19 months ago by mascguy (Christopher Nielsen)

Cc: mascguy added
Note: See TracTickets for help on using tickets.