golang github.com private modules no longer work with renovate v38 #30627

michaeldop · 2024-08-06T15:53:46Z

michaeldop
Aug 6, 2024

What would you like help with?

None

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

GitHub

Please tell us more about your question or problem

After upgrading to v38 of renovate I noticed that our private github modules were no longer working. On the latest version, renovate is no longer able to generate the go.sum file.

On 38 I receive this Artifact update problem and the go.sum file isn't generated:

File name: go.sum
Command failed: go get -d -t ./...
go: github.com/xxx/[email protected]: verifying go.mod: github.com/xxx/[email protected]:/go.mod: reading https://sum.golang.org/lookup/github.com/github.com/xxx/[email protected]:: 404 Not Found
	server response:
	not found: github.com/xxx/[email protected]:: invalid version: git ls-remote -q origin in /tmp/gopath/pkg/mod/cache/vcs/defe4c6d8a98d01fb2ec0aa62560e4c0d06f82c4eb6b4b698675c90a79974d53: exit status 128:
		fatal: could not read Username for 'https://github.com': terminal prompts disabled
	Confirm the import path was entered correctly.
	If this is a private repository, see https://golang.org/doc/faq#git_https for additional information.

If I downgrade renovate to 37.440.7 everything is working as expected.

Logs (if relevant)

Logs


DEBUG: Setting current branch to master (repository=XXXXXXXX, branch=renovate/github.com-XXXXXXXX-1.x)
DEBUG: latest commit (repository=XXXXXXXX, branch=renovate/github.com-XXXXXXXX-1.x)
       "branchName": "master",
       "latestCommitDate": "2024-08-06T13:37:17Z"
DEBUG: manager.getUpdatedPackageFiles() reuseExistingBranch=false (repository=XXXXXXXX, branch=renovate/github.com-XXXXXXXX-1.x)
DEBUG: gomod.updateDependency: v1.66.0 (repository=XXXXXXXX, branch=renovate/github.com-XXXXXXXX-1.x)
DEBUG: Updating github.com/XXXXXXXX in go.mod (repository=XXXXXXXX, branch=renovate/github.com-XXXXXXXX-1.x)
DEBUG: updateArtifacts for updatedPackageFiles (repository=XXXXXXXX, branch=renovate/github.com-XXXXXXXX-1.x)
DEBUG: gomod.updateArtifacts(go.mod) (repository=XXXXXXXX, branch=renovate/github.com-XXXXXXXX-1.x)
DEBUG: go mod tidy command included (repository=XXXXXXXX, branch=renovate/github.com-XXXXXXXX-1.x)
DEBUG: go mod tidy command included (repository=XXXXXXXX, branch=renovate/github.com-XXXXXXXX-1.x)
DEBUG: Setting CONTAINERBASE_CACHE_DIR to /tmp/renovate/cache/containerbase (repository=XXXXXXXX, branch=renovate/github.com-XXXXXXXX-1.x)
DEBUG: Using containerbase dynamic installs (repository=XXXXXXXX, branch=renovate/github.com-XXXXXXXX-1.x)
DEBUG: Resolved stable matching version (repository=XXXXXXXX, branch=renovate/github.com-XXXXXXXX-1.x)
       "toolName": "golang",
       "constraint": "^1.22",
       "resolvedVersion": "1.22.5"
DEBUG: Executing command (repository=XXXXXXXX, branch=renovate/github.com-XXXXXXXX-1.x)
       "command": "install-tool golang 1.22.5"
DEBUG: exec completed (repository=XXXXXXXX, branch=renovate/github.com-XXXXXXXX-1.x)
       "durationMs": 257,
       "stdout": "[15:37:08.596] INFO (282): Installing tool [email protected]...\ntool golang v1.22.5 is already installed\ntool is already linked: golang v1.22.5\n[15:37:08.638] INFO (282): Install tool golang succeeded in 42ms.\n",
       "stderr": ""
DEBUG: Executing command (repository=XXXXXXXX, branch=renovate/github.com-XXXXXXXX-1.x)
       "command": "go get -d -t ./..."
DEBUG: rawExec err (repository=XXXXXXXX, branch=renovate/github.com-XXXXXXXX-1.x)
       "err": {
         "cmd": "/bin/sh -c go get -d -t ./...",
         "stderr": "go: github.com/[email protected]: verifying go.mod: github.com/[email protected]/go.mod: reading https://**redacted**@v1.66.0: 404 Not Found\n\tserver response:\n\tnot found: github.com/[email protected]: invalid version: git ls-remote -q origin in /tmp/gopath/pkg/mod/cache/vcs/68bc8c38a0ac47de7b3da393d6a0fed8278b0d8ed1ab824feb5d9f5b3d819ef7: exit status 128:\n\t\tfatal: could not read Username for 'https://github.com': terminal prompts disabled\n\tConfirm the import path was entered correctly.\n\tIf this is a private repository, see https://golang.org/doc/faq#git_https for additional information.\n",
         "stdout": "",
         "options": {
           "cwd": "/tmp/renovate/repos/github/XXXXXXXX",
           "encoding": "utf-8",
           "env": {
             "GOPATH": "/tmp/renovate/cache/others/go",
             "GOFLAGS": "-modcacherw",
             "GIT_CONFIG_KEY_0": "url.https://**redacted**@github.com/.insteadOf",
             "GIT_CONFIG_VALUE_0": "ssh://**redacted**@github.com/",
             "GIT_CONFIG_KEY_1": "url.https://**redacted**@github.com/.insteadOf",
             "GIT_CONFIG_VALUE_1": "[email protected]:",
             "GIT_CONFIG_KEY_2": "url.https://**redacted**@github.com/.insteadOf",
             "GIT_CONFIG_VALUE_2": "https://github.com/",
             "GIT_CONFIG_COUNT": "3",
             "HOME": "/home/ubuntu",
             "PATH": "/home/ubuntu/.local/bin:/home/ubuntu/bin:/home/ubuntu/.local/bin:/home/ubuntu/bin:/home/ubuntu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
             "LC_ALL": "C.UTF-8",
             "LANG": "C.UTF-8",
             "CONTAINERBASE_CACHE_DIR": "/tmp/renovate/cache/containerbase"
           },
           "maxBuffer": 10485760,
           "timeout": 900000
         },
         "exitCode": 1,
         "name": "ExecError",
         "message": "Command failed: go get -d -t ./...\ngo: github.com/[email protected]: verifying go.mod: github.com/[email protected]/go.mod: reading https://**redacted**@v1.66.0: 404 Not Found\n\tserver response:\n\tnot found: github.com/[email protected]: invalid version: git ls-remote -q origin in /tmp/gopath/pkg/mod/cache/vcs/68bc8c38a0ac47de7b3da393d6a0fed8278b0d8ed1ab824feb5d9f5b3d819ef7: exit status 128:\n\t\tfatal: could not read Username for 'https://github.com': terminal prompts disabled\n\tConfirm the import path was entered correctly.\n\tIf this is a private repository, see https://golang.org/doc/faq#git_https for additional information.\n",
         "stack": "ExecError: Command failed: go get -d -t ./...\ngo: github.com/[email protected]: verifying go.mod: github.com/[email protected]/go.mod: reading https://**redacted**@v1.66.0: 404 Not Found\n\tserver response:\n\tnot found: github.com/[email protected]: invalid version: git ls-remote -q origin in /tmp/gopath/pkg/mod/cache/vcs/68bc8c38a0ac47de7b3da393d6a0fed8278b0d8ed1ab824feb5d9f5b3d819ef7: exit status 128:\n\t\tfatal: could not read Username for 'https://github.com': terminal prompts disabled\n\tConfirm the import path was entered correctly.\n\tIf this is a private repository, see https://golang.org/doc/faq#git_https for additional information.\n\n    at ChildProcess.<anonymous> (/usr/local/renovate/lib/util/exec/common.ts:99:11)\n    at ChildProcess.emit (node:events:531:35)\n    at ChildProcess.emit (node:domain:488:12)\n    at Process.ChildProcess._handle.onexit (node:internal/child_process:294:12)"
       },
       "durationMs": 2005
DEBUG: Failed to update go.sum (repository=XXXXXXXX, branch=renovate/github.com-XXXXXXXX-1.x)
       "err": {
         "cmd": "/bin/sh -c go get -d -t ./...",
         "stderr": "go: github.com/[email protected]: verifying go.mod: github.com/[email protected]/go.mod: reading https://**redacted**@v1.66.0: 404 Not Found\n\tserver response:\n\tnot found: github.com/[email protected]: invalid version: git ls-remote -q origin in /tmp/gopath/pkg/mod/cache/vcs/68bc8c38a0ac47de7b3da393d6a0fed8278b0d8ed1ab824feb5d9f5b3d819ef7: exit status 128:\n\t\tfatal: could not read Username for 'https://github.com': terminal prompts disabled\n\tConfirm the import path was entered correctly.\n\tIf this is a private repository, see https://golang.org/doc/faq#git_https for additional information.\n",
         "stdout": "",
         "options": {
           "cwd": "/tmp/renovate/repos/github/XXXXXXXX",
           "encoding": "utf-8",
           "env": {
             "GOPATH": "/tmp/renovate/cache/others/go",
             "GOFLAGS": "-modcacherw",
             "GIT_CONFIG_KEY_0": "url.https://**redacted**@github.com/.insteadOf",
             "GIT_CONFIG_VALUE_0": "ssh://**redacted**@github.com/",
             "GIT_CONFIG_KEY_1": "url.https://**redacted**@github.com/.insteadOf",
             "GIT_CONFIG_VALUE_1": "[email protected]:",
             "GIT_CONFIG_KEY_2": "url.https://**redacted**@github.com/.insteadOf",
             "GIT_CONFIG_VALUE_2": "https://github.com/",
             "GIT_CONFIG_COUNT": "3",
             "HOME": "/home/ubuntu",
             "PATH": "/home/ubuntu/.local/bin:/home/ubuntu/bin:/home/ubuntu/.local/bin:/home/ubuntu/bin:/home/ubuntu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
             "LC_ALL": "C.UTF-8",
             "LANG": "C.UTF-8",
             "CONTAINERBASE_CACHE_DIR": "/tmp/renovate/cache/containerbase"
           },
           "maxBuffer": 10485760,
           "timeout": 900000
         },
         "exitCode": 1,
         "name": "ExecError",
         "message": "Command failed: go get -d -t ./...\ngo: github.com/[email protected]: verifying go.mod: github.com/[email protected]/go.mod: reading https://**redacted**@v1.66.0: 404 Not Found\n\tserver response:\n\tnot found: github.com/[email protected]: invalid version: git ls-remote -q origin in /tmp/gopath/pkg/mod/cache/vcs/68bc8c38a0ac47de7b3da393d6a0fed8278b0d8ed1ab824feb5d9f5b3d819ef7: exit status 128:\n\t\tfatal: could not read Username for 'https://github.com': terminal prompts disabled\n\tConfirm the import path was entered correctly.\n\tIf this is a private repository, see https://golang.org/doc/faq#git_https for additional information.\n",
         "stack": "ExecError: Command failed: go get -d -t ./...\ngo: github.com/[email protected]: verifying go.mod: github.com/[email protected]/go.mod: reading https://**redacted**@v1.66.0: 404 Not Found\n\tserver response:\n\tnot found: github.com/[email protected]: invalid version: git ls-remote -q origin in /tmp/gopath/pkg/mod/cache/vcs/68bc8c38a0ac47de7b3da393d6a0fed8278b0d8ed1ab824feb5d9f5b3d819ef7: exit status 128:\n\t\tfatal: could not read Username for 'https://github.com': terminal prompts disabled\n\tConfirm the import path was entered correctly.\n\tIf this is a private repository, see https://golang.org/doc/faq#git_https for additional information.\n\n    at ChildProcess.<anonymous> (/usr/local/renovate/lib/util/exec/common.ts:99:11)\n    at ChildProcess.emit (node:events:531:35)\n    at ChildProcess.emit (node:domain:488:12)\n    at Process.ChildProcess._handle.onexit (node:internal/child_process:294:12)"
       }

h-no · 2024-08-09T14:17:01Z

h-no
Aug 9, 2024

I run Renovate as a github action (self-hosted config). For me private packages stopped working with v38.0.0. Works with previous version 37.440.7.
I looked at breaking changes for 38.0.0 and the actual diff from previous tag, but it's still unclear to me what might be the issue.

To fix this, I tried setting GOPRIVATE in my config like this:

  customEnvVariables: {
    GOPRIVATE: "github.com/some-org/*",
    GONOSUMDB: "github.com/some-org/*",
    GONOPROXY: "github.com/some-org/*",
  },

Yet I will still see Renovate trying to access my package via proxy:

DEBUG: GET https://proxy.golang.org/github.com/some-org/some-package/@v/list = (code=ERR_NON_2XX_3XX_RESPONSE, statusCode=404 retryCount=0, duration=118) (repository=some-org/user-hannesna-test-renovate)

But if I set GOPRIVATE in my GitHub Action workflow, it will not use proxy as expected. Package is fetched from GitHub ok.

      - uses: renovatebot/github-action@042670e39b8d7335e992c3fa526ecbfbd52ef57b # v40.2.2
        with:
          renovate-version: "38.0.0"
          configurationFile: conf/config.js
          token: ${{ steps.login.outputs.installation_access_token }}
          env-regex: "^(?:RENOVATE_\\w+|LOG_LEVEL|GITHUB_COM_TOKEN|NODE_OPTIONS|GOPRIVATE)$"
        env:
          GOPRIVATE: github.com/some-org

Next problem is that I perform a postUpgradeTasks command including go mod tidy but it always fails on auth in v38.0.0, it succeeds when pinning to previous version 37.440.7.

DEBUG: Executing command (repository=some-org/user-hannesna-test-renovate, branch=renovate/github.com-some-org-some-package-1.x)
       "dep": "github.com/some-org/some-package",
       "command": "go mod tidy -compat=1.17"
DEBUG: rawExec err (repository=some-org/user-hannesna-test-renovate, branch=renovate/github.com-some-org-some-package-1.x)
       "dep": "github.com/some-org/some-package",
       "err": {
         "cmd": "/bin/sh -c go mod tidy -compat=1.17",
         "stderr": "go: downloading github.com/some-org/some-package", v1.12.23\ngo: github.com/some-org/user-hannesna-test-renovate imports\n\tgithub.com/some-org/some-package",/log: reading github.com/some-org/some-package",/go.mod at revision v1.12.23: git ls-remote -q origin in /home/ubuntu/go/pkg/mod/cache/vcs/05b9f4d797987efeca0fe886165df264d4c2917f1de4e709bf34a6a0a794df07: exit status 128:\n\tfatal: could not read Username for 'https://github.com': terminal prompts disabled\nConfirm the import path was entered correctly.\nIf this is a private repository, see https://golang.org/doc/faq#git_https for additional information.\n",
         "stdout": "",
         "options": {
           "cwd": "/tmp/renovate/repos/github/some-org/user-hannesna-test-renovate",
           "encoding": "utf-8",
           "env": {
             "HOME": "/home/ubuntu",
             "PATH": "/home/ubuntu/.local/bin:/home/ubuntu/bin:/home/ubuntu/.local/bin:/home/ubuntu/bin:/home/ubuntu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
             "LC_ALL": "C.UTF-8",
             "LANG": "C.UTF-8",
             "GOPRIVATE": "github.com/some-org/*",
             "GONOSUMDB": "github.com/some-org/*",
             "GONOPROXY": "github.com/some-org/*",
             "CONTAINERBASE_CACHE_DIR": "/tmp/renovate/cache/containerbase"
           },
           "maxBuffer": 10485760,
           "timeout": 900000
         },
         "exitCode": 1,
         "name": "ExecError",
         "message": "Command failed: go mod tidy -compat=1.17\ngo: downloading github.com/some-org/some-package", v1.12.23\ngo: github.com/some-org/user-hannesna-test-renovate imports\n\tgithub.com/some-org/some-package",/log: reading github.com/some-org/some-package",/go.mod at revision v1.12.23: git ls-remote -q origin in /home/ubuntu/go/pkg/mod/cache/vcs/05b9f4d797987efeca0fe886165df264d4c2917f1de4e709bf34a6a0a794df07: exit status 128:\n\tfatal: could not read Username for 'https://github.com': terminal prompts disabled\nConfirm the import path was entered correctly.\nIf this is a private repository, see https://golang.org/doc/faq#git_https for additional information.\n",
         "stack": "ExecError: Command failed: go mod tidy -compat=1.17\ngo: downloading github.com/some-org/some-package", v1.12.23\ngo: github.com/some-org/user-hannesna-test-renovate imports\n\tgithub.com/some-org/some-package",/log: reading github.com/some-org/some-package",/go.mod at revision v1.12.23: git ls-remote -q origin in /home/ubuntu/go/pkg/mod/cache/vcs/05b9f4d797987efeca0fe886165df264d4c2917f1de4e709bf34a6a0a794df07: exit status 128:\n\tfatal: could not read Username for 'https://github.com': terminal prompts disabled\nConfirm the import path was entered correctly.\nIf this is a private repository, see https://golang.org/doc/faq#git_https for additional information.\n\n    at ChildProcess.<anonymous> (/usr/local/renovate/lib/util/exec/common.ts:99:11)\n    at ChildProcess.emit (node:events:531:35)\n    at ChildProcess.emit (node:domain:488:12)\n    at Process.ChildProcess._handle.onexit (node:internal/child_process:294:12)"
       },
       "durationMs": 994

Is the authentication somehow lost/broken when running the postUpgradeTasks command now?

1 reply

rarkins Aug 24, 2024

@michaeldop there was no intentional change to post upgrade tasks auth in v38. I would actually be surprised if the auth ever worked because we only pass through these important GIT_CONFIG_ variables for select managers

gnuletik · 2024-08-07T10:50:52Z

gnuletik
Aug 7, 2024

Same error on our side when using Mend Renovate hosted app on github.com.

Yersteday, "internal" dependencies hosted on our organization's github repo were properly working and Renovate was able to update them with the go.sum update (with "postUpdateOptions": ["gomodTidy"]).

Today, that stopped working: Renovate is now unable to run go mod tidy:

Command failed: go get -d -t ./...
go: github.com/[email protected]: verifying go.mod: github.com/[email protected]/go.mod: reading https://sum.golang.org/lookup/github.com/[email protected]: 404 Not Found
	server response:
	not found: github.com/[email protected]: invalid version: git ls-remote -q origin in /tmp/gopath/pkg/mod/cache/vcs/c1fe54715007438e4b22c32bb4a20510f826ae90f0ffb964f6daf1a3f9110295: exit status 128:
		fatal: could not read Username for 'https://github.com': terminal prompts disabled
	Confirm the import path was entered correctly.
	If this is a private repository, see https://golang.org/doc/faq#git_https for additional information.

After further checks, I think that the variables GOPRIVATE, GONOSUMDB, GONOPROXY are not set.
However, that's not possible to set these variables through customEnvVariables when using the hosted app

https://docs.renovatebot.com/getting-started/private-packages/#example-for-gomod

If that's linked to recent changes on gosumdb, is there a way to disable it?

Thanks!

11 replies

calvis Aug 12, 2024

any information about a workaround for those using the Mend app? we cannot set these environment variables afaict; Renovate is currently failing (as above) when bumping private go packages that were bumping just fine last week

rarkins Aug 12, 2024

@nabeelsaabna can you turn gosumdb off until this is resolved? Or @viceice do you think this is important enough to revert in the CLI?

nabeelsaabna Aug 12, 2024
Maintainer

@nabeelsaabna can you turn gosumdb off until this is resolved? Or @viceice do you think this is important enough to revert in the CLI?

Yes, will do it tomorrow morning

nabeelsaabna Aug 13, 2024
Maintainer

Did you notice any improvement after setting GOSUMDB=off ?

gnuletik Aug 13, 2024

Works like a charm, thanks @nabeelsaabna!

viceice · 2024-08-06T15:59:26Z

viceice
Aug 6, 2024
Maintainer

@rarkins we've enabled gosumdb, maybe that interfere? 🤷‍♂️

@michaeldop how do you configure go for your private repos?

2 replies

rarkins Aug 6, 2024

Part of the error: go: github.com/[email protected]: verifying go.mod: github.com/[email protected]/go.mod: reading https://**redacted**@v1.66.0: 404 Not Found\n\tserver response:\n\tnot found: github.com/[email protected]

It seems go fails to verify a private github module. Question: was it able to verify it before, or was it skipping verification?

gnuletik Aug 7, 2024

It seems that it previously succeeded to verify private Github modules.
You can check the following jobs ID on Mend:

Job 01912457-c622-7612-bff5-a7d336a47795 (yersteday): downloading dependencies and running go mod tidy run without errors.

Job id 1eb4e6a7-e1bc-41c3-aa9d-801c1a5112d7 (today): same error as above.

golang github.com private modules no longer work with renovate v38 #30627

Uh oh!

What would you like help with?

How are you running Renovate?

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

Please tell us more about your question or problem

Logs (if relevant)

Replies: 3 comments · 14 replies

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nabeelsaabna Aug 12, 2024 Maintainer

Uh oh!

nabeelsaabna Aug 13, 2024 Maintainer

Uh oh!

Uh oh!

viceice Aug 6, 2024 Maintainer

Uh oh!

Uh oh!

Replies: 3 comments 14 replies

nabeelsaabna Aug 12, 2024
Maintainer

nabeelsaabna Aug 13, 2024
Maintainer

viceice
Aug 6, 2024
Maintainer