chore(ci): disable buildvcs stamping into binaries#19095
chore(ci): disable buildvcs stamping into binaries#19095
Conversation
Go 1.18+ (adopted Sep 2022) silently enabled buildvcs=auto, which stamps vcs.revision/vcs.time/vcs.modified into every binary via the linker. This forces a full re-link on every commit even when no source files changed, costing ~5-6s per binary. No code in the repository reads this VCS info (checked via debug.ReadBuildInfo). Version stamping is done independently via -X ldflags from status.sh. The buildvcs data was never intentionally enabled and has been unused for nearly 4 years. Local measurement: buildvcs=auto: 8.0s per binary (re-links every build) buildvcs=false: 1.9s per binary (fully cached) For 7 CLI cross-compilation targets, this saves ~40s per build. Partially generated by AI. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
Skipping CI for Draft Pull Request. |
|
Images are ready for the commit at b23a670. To use with deploy scripts, first |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #19095 +/- ##
==========================================
- Coverage 49.52% 49.52% -0.01%
==========================================
Files 2672 2672
Lines 201665 201671 +6
==========================================
- Hits 99879 99871 -8
- Misses 94327 94341 +14
Partials 7459 7459
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Allow cache saves on davdhacs/gha-disable-buildvcs so we can measure the buildvcs=false improvement on a warm branch cache. First run saves, second run should show the link cache speedup. Partially generated by AI. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Hey - I've found 2 issues, and left some high level feedback:
- The updated cache save/restore conditions are getting quite complex and are duplicated four times; consider introducing a single reusable expression (e.g., a
should_save_cacheoutput or job-levelenv) to centralize the logic and make it easier to reason about and change safely. - The
contains(github.event.pull_request.labels.*.name, 'ci-save-cache')checks are not guarded by an event-type predicate; to avoid issues whengithub.event.pull_requestis undefined (e.g., onpushevents) and to make intent clearer, wrap the label check in an explicitgithub.event_name == 'pull_request' && ...clause.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- The updated cache save/restore conditions are getting quite complex and are duplicated four times; consider introducing a single reusable expression (e.g., a `should_save_cache` output or job-level `env`) to centralize the logic and make it easier to reason about and change safely.
- The `contains(github.event.pull_request.labels.*.name, 'ci-save-cache')` checks are not guarded by an event-type predicate; to avoid issues when `github.event.pull_request` is undefined (e.g., on `push` events) and to make intent clearer, wrap the label check in an explicit `github.event_name == 'pull_request' && ...` clause.
## Individual Comments
### Comment 1
<location> `.github/actions/cache-go-dependencies/action.yaml:22-23` </location>
<code_context>
- # All other events (PRs, etc.) restore only.
+ # Save caches on pushes to the default branch, or on PRs with the
+ # ci-save-cache label (for testing cache-affecting changes on branches).
- name: Cache Go Dependencies (save)
- if: inputs.save == 'true' && (github.event_name == 'push' && github.ref_name == github.event.repository.default_branch)
+ if: inputs.save == 'true' && (github.event_name == 'push' && github.ref_name == github.event.repository.default_branch || contains(github.event.pull_request.labels.*.name, 'ci-save-cache'))
uses: actions/cache@v5
with:
</code_context>
<issue_to_address>
**issue (bug_risk):** Referencing `github.event.pull_request` on non-PR events can break the workflow expression.
On push events, `github.event.pull_request` is undefined, and accessing its properties can cause the workflow condition to error. To avoid failures on pushes, wrap the label check in an event guard, e.g.:
```yaml
if: inputs.save == 'true' && (
(github.event_name == 'push' && github.ref_name == github.event.repository.default_branch) ||
(github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'ci-save-cache'))
)
```
This preserves the intended behavior without touching `github.event.pull_request` when it doesn’t exist.
</issue_to_address>
### Comment 2
<location> `.github/actions/cache-go-dependencies/action.yaml:30-31` </location>
<code_context>
restore-keys: go-mod-v1-
- name: Cache Go Dependencies (restore)
- if: ${{ !(inputs.save == 'true' && (github.event_name == 'push' && github.ref_name == github.event.repository.default_branch)) }}
+ if: ${{ !(inputs.save == 'true' && (github.event_name == 'push' && github.ref_name == github.event.repository.default_branch || contains(github.event.pull_request.labels.*.name, 'ci-save-cache'))) }}
uses: actions/cache/restore@v5
with:
</code_context>
<issue_to_address>
**suggestion (bug_risk):** The restore condition has the same `pull_request` access risk and is hard to reason about with the negation.
To avoid undefined access and keep the logic easier to validate, consider reusing the guarded save condition and inverting it for restore, e.g.:
```yaml
if: ${{ !(inputs.save == 'true' && (
(github.event_name == 'push' && github.ref_name == github.event.repository.default_branch) ||
(github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'ci-save-cache'))
)) }}
```
This keeps restore as the exact inverse of the guarded save condition while only evaluating `github.event.pull_request` for PR events.
Suggested implementation:
```
- name: Cache Go Dependencies (save)
if: inputs.save == 'true' && ((github.event_name == 'push' && github.ref_name == github.event.repository.default_branch) || (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'ci-save-cache')))
uses: actions/cache@v5
```
```
- name: Cache Go Dependencies (restore)
if: ${{ !(inputs.save == 'true' && ((github.event_name == 'push' && github.ref_name == github.event.repository.default_branch) || (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'ci-save-cache')))) }}
uses: actions/cache/restore@v5
```
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
| - name: Cache Go Dependencies (save) | ||
| if: inputs.save == 'true' && (github.event_name == 'push' && github.ref_name == github.event.repository.default_branch) | ||
| if: inputs.save == 'true' && (github.event_name == 'push' && github.ref_name == github.event.repository.default_branch || contains(github.event.pull_request.labels.*.name, 'ci-save-cache')) |
There was a problem hiding this comment.
issue (bug_risk): Referencing github.event.pull_request on non-PR events can break the workflow expression.
On push events, github.event.pull_request is undefined, and accessing its properties can cause the workflow condition to error. To avoid failures on pushes, wrap the label check in an event guard, e.g.:
if: inputs.save == 'true' && (
(github.event_name == 'push' && github.ref_name == github.event.repository.default_branch) ||
(github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'ci-save-cache'))
)This preserves the intended behavior without touching github.event.pull_request when it doesn’t exist.
| - name: Cache Go Dependencies (restore) | ||
| if: ${{ !(inputs.save == 'true' && (github.event_name == 'push' && github.ref_name == github.event.repository.default_branch)) }} | ||
| if: ${{ !(inputs.save == 'true' && (github.event_name == 'push' && github.ref_name == github.event.repository.default_branch || contains(github.event.pull_request.labels.*.name, 'ci-save-cache'))) }} |
There was a problem hiding this comment.
suggestion (bug_risk): The restore condition has the same pull_request access risk and is hard to reason about with the negation.
To avoid undefined access and keep the logic easier to validate, consider reusing the guarded save condition and inverting it for restore, e.g.:
if: ${{ !(inputs.save == 'true' && (
(github.event_name == 'push' && github.ref_name == github.event.repository.default_branch) ||
(github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'ci-save-cache'))
)) }}This keeps restore as the exact inverse of the guarded save condition while only evaluating github.event.pull_request for PR events.
Suggested implementation:
- name: Cache Go Dependencies (save)
if: inputs.save == 'true' && ((github.event_name == 'push' && github.ref_name == github.event.repository.default_branch) || (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'ci-save-cache')))
uses: actions/cache@v5
- name: Cache Go Dependencies (restore)
if: ${{ !(inputs.save == 'true' && ((github.event_name == 'push' && github.ref_name == github.event.repository.default_branch) || (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'ci-save-cache')))) }}
uses: actions/cache/restore@v5
Instead of hardcoding a branch name, enable cache saves when the ci-save-cache label is applied to a PR. Appends a -seeded suffix to the cache key so it doesn't match master's existing cache, forcing a fresh build that saves a branch-scoped entry. Subsequent runs on the labeled PR restore from that entry and demonstrate the improvement (e.g. buildvcs=false speedup). The label can be reused for any future cache experiments. Partially generated by AI. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
18c3db7 to
8eb5eba
Compare
GitShortSha was injected via -X ldflags on every build, changing the linker inputs on every commit and preventing the Go link cache from being reused. The SHA was only used for diagnostic output (roxctl version --json, Central debug endpoint) and is redundant since MainVersion already embeds it (e.g. "4.11.x-143-g4982da58fd"). Replace internal.GitShortSha with getGitCommitFromMainVersion() which extracts the SHA suffix from MainVersion at runtime. This eliminates the per-commit ldflags change, allowing the link cache to be reused across commits when no source files change. Also removes STABLE_GIT_SHORT_SHA from status.sh (no longer needed) and the //XDef:STABLE_GIT_SHORT_SHA annotation that triggered the ldflags injection. Partially generated by AI. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds a go patch to use git index for file change checks in `cmd/go`, replacing per-file content read+hashes with lookups from `.git/index`. The workflow_dispatch workflow builds a patched Go toolchain, then benchmarks roxctl warm rebuilds in three modes (stock, enabled, disabled) to measure the impact on CI runner storage. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
workflow_dispatch requires the workflow to exist on the default branch. Add a push trigger scoped to this branch so it runs on push. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Use restored GOCACHE from cache action (real CI scenario) - Build with -v to show which packages are recompiled (cache misses) - Two builds per mode: first with restored cache, second truly warm - Add GODEBUG=gocachehash=1 step to trace cache hash computations - Add modindex instrumentation output Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Duplicates both jobs with a patched Go variant that uses the git index optimization. Each job does a build + rebuild so we get a known-warm cache measurement. Adds cache-job-name input to cache-go-dependencies action so the patched jobs can use their own GOCACHE keys (stock and patched produce different ActionIDs due to different hash formats). Structure: - pre-build-cli: stock Go, build + rebuild (warm baseline) - pre-build-cli-patched: patched Go, build + rebuild (warm comparison) - pre-build-go-binaries: stock Go, amd64, build + rebuild - pre-build-go-binaries-patched: patched Go, amd64, build + rebuild Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove cache-job-name from cache-go-dependencies action - Cache patched Go toolchain by patch file hash (skip rebuild on hit) - Split patched build into cold→warm jobs sharing GOCACHE - Fix GONOBITINDEX=1 time syntax (use env: block) - Add compiler SHA256 + -v flag for cache miss diagnostics Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Prevent non-essential step failures from killing the job. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- GOCACHE key: per-run unique save, prefix-match restore from previous runs - Add GOTAGS, CGO, CC to environment diagnostics - Show git diff after build-prep to detect generated file changes - Show package compile count with -v flag - Drop cache-go-dependencies from patched jobs (manual GOMODCACHE restore) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- GONOBITINDEX=1 now disables ALL three optimizations: git blob hash in fileHash, git file hash in dirHash, and vendor modroot fix - Baseline job uses same patched Go binary with GONOBITINDEX=1 and fresh per-run GOCACHE (separate baseline-gocache-* key prefix) - Removes stale weekly cache dependency for fair A/B comparison Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
6801a5b to
098af8d
Compare
Description
Go 1.18+ (adopted Sep 2022) silently enabled
buildvcs=auto, stampingvcs.revision/vcs.time/vcs.modifiedinto every binary via the linker.No code in the repository reads this VCS info (
debug.ReadBuildInfo()is never called). Version stamping is done independently via
-Xldflagsfrom
status.sh. The buildvcs data was never intentionally enabled — itwas an unnoticed side effect of the Go 1.18 upgrade.
Fix: add
-buildvcs=falsetoinvoke_go_buildinscripts/go-tool.sh.What this does and does not fix
Does fix: Removes ~5-6s of unnecessary VCS stamping per binary in the
post-build step. With ~15 binaries across all jobs, this saves roughly 75-90s
of aggregate CI time and eliminates unused metadata from all built binaries.
Does NOT fix: The link cache is still invalidated on every commit because
status.shinjects a per-commit SHA via-X pkg/version/internal.GitShortSha.This causes ~40s of re-linking per job regardless of
-buildvcs=false.Fixing the ldflags (replacing
-X GitShortShawith runtime VCS info viadebug.ReadBuildInfo()) is a separate effort tracked in thedavdhacs/test-build-on-runnerprototype. That change would reduce warm builds from ~40s to ~2s per binary.
Measurements
Local measurement (same binary, no source changes):
buildvcs=auto: 8.0s (re-stamps VCS info via linker)buildvcs=false: 1.9s (link cache fully reused)CI measurement (different commits,
GitShortShachanges via ldflags):buildvcs=auto: ~42s (re-links due to VCS stamp + ldflags change)buildvcs=false: ~42s (link cache still invalidated by changed ldflags)CI improvement is only the removal of the VCS stamping overhead, not the
full link cache benefit. The full benefit requires the ldflags fix.
Research
debug.ReadBuildInfo()is never called for VCS info-buildvcs=false(PR ci(scanner): remove vcs information from binary #11935, Jul 2024) was areactive fix when VCS stamping broke a scanner CI workflow
.dockerignoreexcludes/.git/)Partially generated by AI.