Skip to content

chore(ci): disable buildvcs stamping into binaries#19095

Closed
davdhacs wants to merge 35 commits intomasterfrom
davdhacs/gha-disable-buildvcs
Closed

chore(ci): disable buildvcs stamping into binaries#19095
davdhacs wants to merge 35 commits intomasterfrom
davdhacs/gha-disable-buildvcs

Conversation

@davdhacs
Copy link
Contributor

@davdhacs davdhacs commented Feb 18, 2026

Description

Go 1.18+ (adopted Sep 2022) silently enabled buildvcs=auto, stamping
vcs.revision/vcs.time/vcs.modified into every binary via the linker.
No code in the repository reads this VCS info (debug.ReadBuildInfo()
is never called). Version stamping is done independently via -X ldflags
from status.sh. The buildvcs data was never intentionally enabled — it
was an unnoticed side effect of the Go 1.18 upgrade.

Fix: add -buildvcs=false to invoke_go_build in scripts/go-tool.sh.

What this does and does not fix

Does fix: Removes ~5-6s of unnecessary VCS stamping per binary in the
post-build step. With ~15 binaries across all jobs, this saves roughly 75-90s
of aggregate CI time and eliminates unused metadata from all built binaries.

Does NOT fix: The link cache is still invalidated on every commit because
status.sh injects a per-commit SHA via -X pkg/version/internal.GitShortSha.
This causes ~40s of re-linking per job regardless of -buildvcs=false.

Fixing the ldflags (replacing -X GitShortSha with runtime VCS info via
debug.ReadBuildInfo()) is a separate effort tracked in the davdhacs/test-build-on-runner
prototype. That change would reduce warm builds from ~40s to ~2s per binary.

Measurements

Local measurement (same binary, no source changes):

  • buildvcs=auto: 8.0s (re-stamps VCS info via linker)
  • buildvcs=false: 1.9s (link cache fully reused)

CI measurement (different commits, GitShortSha changes via ldflags):

  • buildvcs=auto: ~42s (re-links due to VCS stamp + ldflags change)
  • buildvcs=false: ~42s (link cache still invalidated by changed ldflags)

CI improvement is only the removal of the VCS stamping overhead, not the
full link cache benefit. The full benefit requires the ldflags fix.

Research

  • No commit on master ever intentionally enabled buildvcs
  • debug.ReadBuildInfo() is never called for VCS info
  • The only prior instance of -buildvcs=false (PR ci(scanner): remove vcs information from binary #11935, Jul 2024) was a
    reactive fix when VCS stamping broke a scanner CI workflow
  • Dockerized builds already had no VCS stamping (.dockerignore excludes /.git/)

Partially generated by AI.

Go 1.18+ (adopted Sep 2022) silently enabled buildvcs=auto, which
stamps vcs.revision/vcs.time/vcs.modified into every binary via the
linker. This forces a full re-link on every commit even when no
source files changed, costing ~5-6s per binary.

No code in the repository reads this VCS info (checked via
debug.ReadBuildInfo). Version stamping is done independently via
-X ldflags from status.sh. The buildvcs data was never intentionally
enabled and has been unused for nearly 4 years.

Local measurement:
  buildvcs=auto:  8.0s per binary (re-links every build)
  buildvcs=false: 1.9s per binary (fully cached)

For 7 CLI cross-compilation targets, this saves ~40s per build.

Partially generated by AI.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@openshift-ci
Copy link

openshift-ci bot commented Feb 18, 2026

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@rhacs-bot
Copy link
Contributor

rhacs-bot commented Feb 18, 2026

Images are ready for the commit at b23a670.

To use with deploy scripts, first export MAIN_IMAGE_TAG=4.11.x-188-gb23a67005a.

@codecov
Copy link

codecov bot commented Feb 19, 2026

Codecov Report

❌ Patch coverage is 0% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 49.52%. Comparing base (d3feadf) to head (b23a670).
⚠️ Report is 29 commits behind head on master.

Files with missing lines Patch % Lines
pkg/version/version.go 0.00% 7 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master   #19095      +/-   ##
==========================================
- Coverage   49.52%   49.52%   -0.01%     
==========================================
  Files        2672     2672              
  Lines      201665   201671       +6     
==========================================
- Hits        99879    99871       -8     
- Misses      94327    94341      +14     
  Partials     7459     7459              
Flag Coverage Δ
go-unit-tests 49.52% <0.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

davdhacs and others added 5 commits February 19, 2026 07:46
Allow cache saves on davdhacs/gha-disable-buildvcs so we can
measure the buildvcs=false improvement on a warm branch cache.
First run saves, second run should show the link cache speedup.

Partially generated by AI.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@davdhacs davdhacs added the ci-save-cache Enable Go cache saves on this PR branch for testing label Feb 19, 2026
Copy link
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 2 issues, and left some high level feedback:

  • The updated cache save/restore conditions are getting quite complex and are duplicated four times; consider introducing a single reusable expression (e.g., a should_save_cache output or job-level env) to centralize the logic and make it easier to reason about and change safely.
  • The contains(github.event.pull_request.labels.*.name, 'ci-save-cache') checks are not guarded by an event-type predicate; to avoid issues when github.event.pull_request is undefined (e.g., on push events) and to make intent clearer, wrap the label check in an explicit github.event_name == 'pull_request' && ... clause.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- The updated cache save/restore conditions are getting quite complex and are duplicated four times; consider introducing a single reusable expression (e.g., a `should_save_cache` output or job-level `env`) to centralize the logic and make it easier to reason about and change safely.
- The `contains(github.event.pull_request.labels.*.name, 'ci-save-cache')` checks are not guarded by an event-type predicate; to avoid issues when `github.event.pull_request` is undefined (e.g., on `push` events) and to make intent clearer, wrap the label check in an explicit `github.event_name == 'pull_request' && ...` clause.

## Individual Comments

### Comment 1
<location> `.github/actions/cache-go-dependencies/action.yaml:22-23` </location>
<code_context>
-    # All other events (PRs, etc.) restore only.
+    # Save caches on pushes to the default branch, or on PRs with the
+    # ci-save-cache label (for testing cache-affecting changes on branches).
     - name: Cache Go Dependencies (save)
-      if: inputs.save == 'true' && (github.event_name == 'push' && github.ref_name == github.event.repository.default_branch)
+      if: inputs.save == 'true' && (github.event_name == 'push' && github.ref_name == github.event.repository.default_branch || contains(github.event.pull_request.labels.*.name, 'ci-save-cache'))
       uses: actions/cache@v5
       with:
</code_context>

<issue_to_address>
**issue (bug_risk):** Referencing `github.event.pull_request` on non-PR events can break the workflow expression.

On push events, `github.event.pull_request` is undefined, and accessing its properties can cause the workflow condition to error. To avoid failures on pushes, wrap the label check in an event guard, e.g.:

```yaml
if: inputs.save == 'true' && (
  (github.event_name == 'push' && github.ref_name == github.event.repository.default_branch) ||
  (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'ci-save-cache'))
)
```

This preserves the intended behavior without touching `github.event.pull_request` when it doesn’t exist.
</issue_to_address>

### Comment 2
<location> `.github/actions/cache-go-dependencies/action.yaml:30-31` </location>
<code_context>
         restore-keys: go-mod-v1-

     - name: Cache Go Dependencies (restore)
-      if: ${{ !(inputs.save == 'true' && (github.event_name == 'push' && github.ref_name == github.event.repository.default_branch)) }}
+      if: ${{ !(inputs.save == 'true' && (github.event_name == 'push' && github.ref_name == github.event.repository.default_branch || contains(github.event.pull_request.labels.*.name, 'ci-save-cache'))) }}
       uses: actions/cache/restore@v5
       with:
</code_context>

<issue_to_address>
**suggestion (bug_risk):** The restore condition has the same `pull_request` access risk and is hard to reason about with the negation.

To avoid undefined access and keep the logic easier to validate, consider reusing the guarded save condition and inverting it for restore, e.g.: 

```yaml
if: ${{ !(inputs.save == 'true' && (
  (github.event_name == 'push' && github.ref_name == github.event.repository.default_branch) ||
  (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'ci-save-cache'))
)) }}
```

This keeps restore as the exact inverse of the guarded save condition while only evaluating `github.event.pull_request` for PR events.

Suggested implementation:

```
    - name: Cache Go Dependencies (save)
      if: inputs.save == 'true' && ((github.event_name == 'push' && github.ref_name == github.event.repository.default_branch) || (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'ci-save-cache')))
      uses: actions/cache@v5

```

```
    - name: Cache Go Dependencies (restore)
      if: ${{ !(inputs.save == 'true' && ((github.event_name == 'push' && github.ref_name == github.event.repository.default_branch) || (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'ci-save-cache')))) }}
      uses: actions/cache/restore@v5

```
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Comment on lines 22 to +23
- name: Cache Go Dependencies (save)
if: inputs.save == 'true' && (github.event_name == 'push' && github.ref_name == github.event.repository.default_branch)
if: inputs.save == 'true' && (github.event_name == 'push' && github.ref_name == github.event.repository.default_branch || contains(github.event.pull_request.labels.*.name, 'ci-save-cache'))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (bug_risk): Referencing github.event.pull_request on non-PR events can break the workflow expression.

On push events, github.event.pull_request is undefined, and accessing its properties can cause the workflow condition to error. To avoid failures on pushes, wrap the label check in an event guard, e.g.:

if: inputs.save == 'true' && (
  (github.event_name == 'push' && github.ref_name == github.event.repository.default_branch) ||
  (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'ci-save-cache'))
)

This preserves the intended behavior without touching github.event.pull_request when it doesn’t exist.

Comment on lines 30 to +31
- name: Cache Go Dependencies (restore)
if: ${{ !(inputs.save == 'true' && (github.event_name == 'push' && github.ref_name == github.event.repository.default_branch)) }}
if: ${{ !(inputs.save == 'true' && (github.event_name == 'push' && github.ref_name == github.event.repository.default_branch || contains(github.event.pull_request.labels.*.name, 'ci-save-cache'))) }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion (bug_risk): The restore condition has the same pull_request access risk and is hard to reason about with the negation.

To avoid undefined access and keep the logic easier to validate, consider reusing the guarded save condition and inverting it for restore, e.g.:

if: ${{ !(inputs.save == 'true' && (
  (github.event_name == 'push' && github.ref_name == github.event.repository.default_branch) ||
  (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'ci-save-cache'))
)) }}

This keeps restore as the exact inverse of the guarded save condition while only evaluating github.event.pull_request for PR events.

Suggested implementation:

    - name: Cache Go Dependencies (save)
      if: inputs.save == 'true' && ((github.event_name == 'push' && github.ref_name == github.event.repository.default_branch) || (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'ci-save-cache')))
      uses: actions/cache@v5

    - name: Cache Go Dependencies (restore)
      if: ${{ !(inputs.save == 'true' && ((github.event_name == 'push' && github.ref_name == github.event.repository.default_branch) || (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'ci-save-cache')))) }}
      uses: actions/cache/restore@v5

Instead of hardcoding a branch name, enable cache saves when
the ci-save-cache label is applied to a PR. Appends a -seeded
suffix to the cache key so it doesn't match master's existing
cache, forcing a fresh build that saves a branch-scoped entry.
Subsequent runs on the labeled PR restore from that entry and
demonstrate the improvement (e.g. buildvcs=false speedup).

The label can be reused for any future cache experiments.

Partially generated by AI.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@davdhacs davdhacs force-pushed the davdhacs/gha-disable-buildvcs branch from 18c3db7 to 8eb5eba Compare February 19, 2026 16:42
davdhacs and others added 13 commits February 19, 2026 09:56
GitShortSha was injected via -X ldflags on every build, changing the
linker inputs on every commit and preventing the Go link cache from
being reused. The SHA was only used for diagnostic output (roxctl
version --json, Central debug endpoint) and is redundant since
MainVersion already embeds it (e.g. "4.11.x-143-g4982da58fd").

Replace internal.GitShortSha with getGitCommitFromMainVersion() which
extracts the SHA suffix from MainVersion at runtime. This eliminates
the per-commit ldflags change, allowing the link cache to be reused
across commits when no source files change.

Also removes STABLE_GIT_SHORT_SHA from status.sh (no longer needed)
and the //XDef:STABLE_GIT_SHORT_SHA annotation that triggered the
ldflags injection.

Partially generated by AI.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds a go patch to use git index for file change checks in `cmd/go`,
replacing per-file content read+hashes with lookups from `.git/index`.

The workflow_dispatch workflow builds a patched Go toolchain, then
benchmarks roxctl warm rebuilds in three modes (stock, enabled, disabled)
to measure the impact on CI runner storage.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
workflow_dispatch requires the workflow to exist on the default branch.
Add a push trigger scoped to this branch so it runs on push.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
davdhacs and others added 12 commits February 20, 2026 21:45
- Use restored GOCACHE from cache action (real CI scenario)
- Build with -v to show which packages are recompiled (cache misses)
- Two builds per mode: first with restored cache, second truly warm
- Add GODEBUG=gocachehash=1 step to trace cache hash computations
- Add modindex instrumentation output

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Duplicates both jobs with a patched Go variant that uses the git index
optimization. Each job does a build + rebuild so we get a known-warm
cache measurement.

Adds cache-job-name input to cache-go-dependencies action so the
patched jobs can use their own GOCACHE keys (stock and patched produce
different ActionIDs due to different hash formats).

Structure:
- pre-build-cli: stock Go, build + rebuild (warm baseline)
- pre-build-cli-patched: patched Go, build + rebuild (warm comparison)
- pre-build-go-binaries: stock Go, amd64, build + rebuild
- pre-build-go-binaries-patched: patched Go, amd64, build + rebuild

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove cache-job-name from cache-go-dependencies action
- Cache patched Go toolchain by patch file hash (skip rebuild on hit)
- Split patched build into cold→warm jobs sharing GOCACHE
- Fix GONOBITINDEX=1 time syntax (use env: block)
- Add compiler SHA256 + -v flag for cache miss diagnostics

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Prevent non-essential step failures from killing the job.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- GOCACHE key: per-run unique save, prefix-match restore from previous runs
- Add GOTAGS, CGO, CC to environment diagnostics
- Show git diff after build-prep to detect generated file changes
- Show package compile count with -v flag
- Drop cache-go-dependencies from patched jobs (manual GOMODCACHE restore)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- GONOBITINDEX=1 now disables ALL three optimizations: git blob hash
  in fileHash, git file hash in dirHash, and vendor modroot fix
- Baseline job uses same patched Go binary with GONOBITINDEX=1 and
  fresh per-run GOCACHE (separate baseline-gocache-* key prefix)
- Removes stale weekly cache dependency for fair A/B comparison

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@davdhacs davdhacs force-pushed the davdhacs/gha-disable-buildvcs branch from 6801a5b to 098af8d Compare February 21, 2026 16:13
@davdhacs davdhacs closed this Feb 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-review area/ci ci-save-cache Enable Go cache saves on this PR branch for testing do-not-merge/work-in-progress

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants