Skip to content

fix(ci): GHA cache keys unique vs shared#19580

Open
davdhacs wants to merge 6 commits intomasterfrom
davdhacs/fix-cache-key-separation
Open

fix(ci): GHA cache keys unique vs shared#19580
davdhacs wants to merge 6 commits intomasterfrom
davdhacs/fix-cache-key-separation

Conversation

@davdhacs
Copy link
Contributor

@davdhacs davdhacs commented Mar 24, 2026

Fix key issues for GHA cache of GOCACHE:

unit-tests.yaml: add key-suffix for GOTAGS matrix

go and go-postgres have GOTAGS matrix ("" / release) sharing one cache key. Each variant's trim deletes ~3GB of the other's entries every run. Fix: add key-suffix: ${{ matrix.gotags }}.

scanner-build.yaml: set GOARCH on cache step

pre-build-scanner-go-binary cross-compiles for 4 architectures but doesn't set GOARCH on the cache step. go env GOARCH returns the runner's native amd64, so all arches share one cache key with incompatible entries. Fix: add env: GOARCH: ${{ matrix.goarch }}.

build.yaml: only separate race-condition-debug

pre-build-go-binaries: default and prerelease share 93% of GOCACHE entries (only 6 packages use release build tags). Only separate race-condition-debug (-race/CGO_ENABLED=1, 39% unique).

Verification

Local GOCACHE entry overlap measurements:

Variant New entries vs default Unique % Action
prerelease (GOTAGS=release) +691 7% share
race (-race, CGO_ENABLED=1) +6300 39% separate
GOTAGS="" vs release (tests) +691 7% separate (trim causes 3GB/run thrashing)

All CI passed (Build, Unit Tests, Style, Scanner).

Partially generated by AI.

… entries

pre-build-go-binaries: default and prerelease (GOTAGS=release) share 93%
of cache entries (only 6 packages use release build tags). Separating them
doubles cache storage and causes the trim to delete shared entries. Keep
key-suffix only for race-condition-debug which uses -race/CGO_ENABLED=1
and has 39% unique entries.

build-and-push-operator: branding (RHACS vs STACKROX) is a runtime env
var, not a build tag. Compiled output is identical. Remove key-suffix
entirely.

Verified locally:
  default:    8911 entries (baseline)
  prerelease: +691 new (7% unique, 93% shared)
  race:       +6300 new (39% unique, 61% shared)
  branding:   +0 new (0% unique, 100% shared)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@openshift-ci
Copy link

openshift-ci bot commented Mar 24, 2026

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

Copy link
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 1 issue, and left some high level feedback:

  • Using matrix.name == 'race-condition-debug' && 'race' || '' relies on GitHub’s truthy/falsey behavior and is a bit opaque; consider moving the condition into the matrix definition (e.g., a race: true flag) or a small reusable expression to make the cache key logic more readable and less brittle to matrix-name changes.
  • For the pre-build-go-binaries job, double-check that passing an empty string as key-suffix is treated the same by .github/actions/cache-go-dependencies as omitting the input entirely (as done in build-and-push-operator), or adjust the action so unset vs empty suffix behaves consistently.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- Using `matrix.name == 'race-condition-debug' && 'race' || ''` relies on GitHub’s truthy/falsey behavior and is a bit opaque; consider moving the condition into the matrix definition (e.g., a `race: true` flag) or a small reusable expression to make the cache key logic more readable and less brittle to matrix-name changes.
- For the `pre-build-go-binaries` job, double-check that passing an empty string as `key-suffix` is treated the same by `.github/actions/cache-go-dependencies` as omitting the input entirely (as done in `build-and-push-operator`), or adjust the action so unset vs empty suffix behaves consistently.

## Individual Comments

### Comment 1
<location path=".github/workflows/build.yaml" line_range="213" />
<code_context>
         uses: ./.github/actions/cache-go-dependencies
         with:
-          key-suffix: ${{ matrix.name }}
+          key-suffix: ${{ matrix.name == 'race-condition-debug' && 'race' || '' }}

       - uses: ./.github/actions/handle-tagged-build
</code_context>
<issue_to_address>
**suggestion:** Consider making this conditional expression more explicit to avoid ambiguity in evaluation and future edits.

GitHub Actions’ `&&`/`||` precedence and truthiness can be hard to parse. Adding parentheses like `${{ (matrix.name == 'race-condition-debug' && 'race') || '' }}` makes the grouping clearer. If the action distinguishes `null` from `''`, you could instead use `${{ matrix.name == 'race-condition-debug' && 'race' || null }}` to signal that the suffix is intentionally omitted.

```suggestion
          key-suffix: ${{ (matrix.name == 'race-condition-debug' && 'race') || '' }}
```
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@rhacs-bot
Copy link
Contributor

rhacs-bot commented Mar 24, 2026

Images are ready for the commit at 1015a87.

To use with deploy scripts, first export MAIN_IMAGE_TAG=4.11.x-441-g1015a87dc8.

@codecov
Copy link

codecov bot commented Mar 24, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 49.29%. Comparing base (be2f16f) to head (1015a87).

Additional details and impacted files
@@           Coverage Diff           @@
##           master   #19580   +/-   ##
=======================================
  Coverage   49.28%   49.29%           
=======================================
  Files        2735     2735           
  Lines      206170   206170           
=======================================
+ Hits       101617   101625    +8     
+ Misses      97012    97006    -6     
+ Partials     7541     7539    -2     
Flag Coverage Δ
go-unit-tests 49.29% <ø> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

davdhacs and others added 2 commits March 24, 2026 16:46
unit-tests.yaml: go and go-postgres have GOTAGS matrix (""/ release)
sharing one cache key. With the GOCACHE trim, each variant deletes
the other's entries on every run, causing 3GB of churn. Add key-suffix
to give each variant its own cache.

scanner-build.yaml: pre-build-scanner-go-binary cross-compiles for
multiple architectures but doesn't set GOARCH on the cache step,
so all arches share one amd64-labeled cache key. Add env GOARCH.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Address sourcery-ai review: make GHA expression grouping explicit.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@davdhacs
Copy link
Contributor Author

@sourcery-ai review

Copy link
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've left some high level feedback:

  • The GitHub Actions expression ${{ (matrix.name == 'race-condition-debug' && 'race') || '' }} is a bit terse; consider using a more explicit pattern (e.g., the if function or a short comment) so future readers immediately understand that only the race build gets a distinct cache key.
  • In scanner-build.yaml, you set GOARCH only on the cache step; if any later Go steps rely on go env GOARCH, it may be safer to set GOARCH at the job level (or at least consistently on all Go-related steps) to avoid subtle cross-arch issues.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- The GitHub Actions expression `${{ (matrix.name == 'race-condition-debug' && 'race') || '' }}` is a bit terse; consider using a more explicit pattern (e.g., the `if` function or a short comment) so future readers immediately understand that only the race build gets a distinct cache key.
- In `scanner-build.yaml`, you set `GOARCH` only on the cache step; if any later Go steps rely on `go env GOARCH`, it may be safer to set `GOARCH` at the job level (or at least consistently on all Go-related steps) to avoid subtle cross-arch issues.

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@davdhacs davdhacs changed the title perf(ci): stop over-separating GOCACHE for build matrix variants fix(ci): GHA cache keys unique vs shared Mar 25, 2026
davdhacs and others added 3 commits March 24, 2026 21:10
The operator branding key-suffix is unnecessary for GOCACHE (0% unique
entries), but PR #19417 will unify operator builds entirely. Removing
the key-suffix here would create a merge conflict with that PR.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Address sourcery-ai feedback: clarify why only race-condition-debug
gets a separate cache key (39% unique entries from -race/CGO_ENABLED=1)
while default and prerelease share (93% overlap).

GOARCH is intentionally set only on the cache step, not job-level,
to avoid changing the default arch for other steps.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@davdhacs davdhacs marked this pull request as ready for review March 25, 2026 04:09
@davdhacs davdhacs requested a review from a team as a code owner March 25, 2026 04:09
Copy link
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've left some high level feedback:

  • The conditional expression for key-suffix in build.yaml (${{ (matrix.name == 'race-condition-debug' && 'race') || '' }}) is a bit opaque and tightly coupled to the matrix name; consider either using a more explicit mapping or a dedicated matrix field (e.g. cacheKeySuffix) to make future variant changes less error-prone.
  • In unit-tests.yaml, using matrix.gotags directly as the key-suffix assumes only '' and release; if more complex tag sets are introduced later it may be safer to normalize or hash them to avoid unexpected characters in cache keys or collisions between equivalent tag permutations.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- The conditional expression for `key-suffix` in `build.yaml` (`${{ (matrix.name == 'race-condition-debug' && 'race') || '' }}`) is a bit opaque and tightly coupled to the matrix name; consider either using a more explicit mapping or a dedicated matrix field (e.g. `cacheKeySuffix`) to make future variant changes less error-prone.
- In `unit-tests.yaml`, using `matrix.gotags` directly as the `key-suffix` assumes only `''` and `release`; if more complex tag sets are introduced later it may be safer to normalize or hash them to avoid unexpected characters in cache keys or collisions between equivalent tag permutations.

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@openshift-ci
Copy link

openshift-ci bot commented Mar 25, 2026

@davdhacs: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/gke-nongroovy-e2e-tests 1015a87 link true /test gke-nongroovy-e2e-tests

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants