Go licenses are not generating with the version 1.11.0 even though cache directory is created and it hangs
Used below command to generate SBOM for GO binaries, Our network is behind proxy, so exported proxy and tried below command but nothing is downloaded in syft-cache directory and licenses are not generated for go binaries.
Hi @anvitha_haviligi, if you run with -vvv are you able to see any activity or get further insight where things seem to get stuck?
I suspect the proxy is the issue here – Syft should be using the standard Go proxy environment variable: HTTP_PROXY. How did you configure this? Just making sure you did not use SYFT_GOLANG_PROXY, which is for specifying any GOPROXY settings, not HTTP proxy configuration.
Hi, Thanks for the reply.
Im doing export HTTP_PROXY="http://[ipv6]:portnumber". Im getting below errors in debug mode.
[0007] DEBUG error reading file://$GOPATH/pkg/mod/github.com/beorn7/perks@v1.0.1/#.: open .: no such file or directory
[0007] DEBUG error reading file://$GOPATH/pkg/mod/github.com/bgentry/speakeasy@v0.1.0/#.: open .: no such file or directory
[0007] DEBUG error reading file://$GOPATH/pkg/mod/github.com/cespare/xxhash/v2@v2.2.0/#.: open .: no such file or directory
[0007] DEBUG error reading file://$GOPATH/pkg/mod/github.com/coreos/go-semver@v0.3.0/#.: open .: no such file or directory
[0007] DEBUG error reading file://$GOPATH/pkg/mod/github.com/coreos/go-systemd/v22@v22.3.2/#.: open .: no such file or directory
[0007] DEBUG error reading file://$GOPATH/pkg/mod/github.com/cpuguy83/go-md2man/v2@v2.0.0/#.: open .: no such file or directory
[0007] DEBUG error reading file://$GOPATH/pkg/mod/github.com/dustin/go-humanize@v1.0.0/#.: open .: no such file or directory
[0007] DEBUG error reading file://$GOPATH/pkg/mod/github.com/gogo/protobuf@v1.3.2/#.: open .: no such file or directory
[0007] DEBUG error reading file://$GOPATH/pkg/mod/github.com/golang-jwt/jwt/v4@v4.4.2/#.: open .: no such file or directory
[0007] DEBUG error reading file://$GOPATH/pkg/mod/github.com/golang/protobuf@v1.5.3/#.: open .: no such file or directory
[0007] DEBUG error reading file://$GOPATH/pkg/mod/github.com/google/btree@v1.0.1/#.: open .: no such file or directory
[0007] DEBUG error reading file://$GOPATH/pkg/mod/github.com/jonboulle/clockwork@v0.2.2/#.: open .: no such file or directory
[0007] DEBUG error reading file://$GOPATH/pkg/mod/github.com/json-iterator/go@v1.1.11/#.: open .: no such file or directory
[0007] DEBUG error reading file://$GOPATH/pkg/mod/github.com/mattn/go-runewidth@v0.0.9/#.: open .: no such file or directory
[0007] DEBUG error reading file://$GOPATH/pkg/mod/github.com/matttproud/golang_protobuf_extensions@v1.0.1/#.: open .: no such file or directory
[0007] DEBUG error reading file://$GOPATH/pkg/mod/github.com/modern-go/concurrent@v0.0.0-20180306012644-bacd9c7ef1dd/#.: open .: no such file or directory
[0007] DEBUG error reading file://$GOPATH/pkg/mod/github.com/modern-go/reflect2@v1.0.1/#.: open .: no such file or directory
[0007] DEBUG error reading file://$GOPATH/pkg/mod/github.com/olekukonko/tablewriter@v0.0.5/#.: open .: no such file or directory
[0007] DEBUG error reading file://$GOPATH/pkg/mod/github.com/prometheus/client_golang@v1.11.1/#.: open .: no such file or directory
[0007] DEBUG error reading file://$GOPATH/pkg/mod/github.com/prometheus/client_model@v0.2.0/#.: open .: no such file or directory
[0007] DEBUG error reading file://$GOPATH/pkg/mod/github.com/prometheus/common@v0.26.0/#.: open .: no such file or directory
[0007] DEBUG error reading file://$GOPATH/pkg/mod/github.com/prometheus/procfs@v0.6.0/#.: open .: no such file or directory
Thanks, those errors are all expected – I’m assuming you don’t have those packages locally in your ~/go/pkg/mod directory (everything which indicates $GOPATH is looking there). Sadly, it looks like I may have omitted some important logging for the network side of things. I’m not sure it will necessarily help, but I fixed that and the latest version of Syft now includes some info logging (which needs at least -v to display). If it seems to hang on something like:
[0010] INFO downloading go module from proxy url=https://proxy.golang.org/github.com/spf13/cast/@v/v1.6.0.zip
We can probably at least tell there is some network issue, which would confirm your original post. The next step might be to see if you can craft a cURL command to download the same URL with appropriate proxy settings and run this in the environment you are running Syft.
Sorry I missed this earlier, but you probably wanted to use use: SYFT_CACHE_DIR=~/syft-cache.
If you’ve run a go build, and go has downloaded the modules already is when you may want to use that mod cache, in which case you would use: SYFT_GOLANG_SEARCH_LOCAL_MOD_CACHE_LICENSES=true, by default this will look in $GOPATH/pkg/mod, where GOPATH defaults to ~/go. You only need to specify SYFT_GOLANG_LOCAL_MOD_CACHE_DIR if you have some configuration where go has downloaded modules to some alternate location and you want to use those to resolve license information.
If you are scanning things where there has not been a go build run, searching the local go mod cache won’t help much, so you could probably just use SYFT_GOLANG_SEARCH_REMOTE_LICENSES=true
While investigating this, one thing I scanned was golang:latest, which has a lot of go binaries with (devel) versions and go library packages like cmd, I see a lot of errors where Syft attempts to blindly resolve these to repositories, e.g.:
{"err":"Get \"https://cmd/cgo/info/refs?service=git-upload-pack\": dial tcp: lookup cmd: no such host -- "}
This means there could be a number of DNS lookups for things like cmd, is this something that might be very slow with your network configuration?