Skip to content

fix(ci): increase operator build free-disk-space to 40GB#19396

Merged
janisz merged 1 commit intomasterfrom
davdhacs/fix-op-build-disk-space
Mar 14, 2026
Merged

fix(ci): increase operator build free-disk-space to 40GB#19396
janisz merged 1 commit intomasterfrom
davdhacs/fix-op-build-disk-space

Conversation

@davdhacs
Copy link
Contributor

@davdhacs davdhacs commented Mar 12, 2026 •
edited
Loading

Description

Our operator build used to fit in 30GB and now it uses 31GB.

Azure-based GitHub Actions runners can start with only 77GB root disk (~19GB free). The job-preamble cleanup stops as soon as it reaches the threshold, so the previous 30GB target left the operator Docker build without enough headroom. The multi-stage build (UBI go-toolset base image + full monorepo go mod download + source copy + compile) needs more than 30GB.

Bumping to 40GB is safe per the preamble's own docs ("ubuntu-24.04 runner starts with 20GB free and we can delete and reach 40GB") and consistent with other jobs (emailsender uses 40GB, scanner-versioned-definitions uses 50GB).

See also #19397 for a background disk usage monitor added to the job-preamble to track disk consumption over time.

Disk usage measurements (RHACS amd64, 145G runner)

Phase Used Avail Delta
After preamble (no cleanup needed) 57G 88G —
After go mod tidy + cache 79G 66G -22G
After bundle build 80G 65G -1G
After unit tests 80G 65G 0
After operator image build (peak) 88G 57G -8G

Peak build consumption: ~31G. On the failing 77G runner where cleanup only freed 30G, this build would have exhausted disk space. The 40GB threshold provides ~9G of headroom.

User-facing documentation

Testing and quality

  • the change is production ready: the change is GA, or otherwise the functionality is gated by a feature flag
  • CI results are inspected

Automated testing

  • added unit tests
  • added e2e tests
  • added regression tests
  • added compatibility tests
  • modified existing tests

How I validated my change

  • All 4 operator build matrix jobs (RHACS/STACKROX x amd64/arm64) passed with the 40GB threshold
  • Disk usage was measured at each build phase via temporary df instrumentation (see table above)
  • Peak consumption of ~31G confirms 30G was insufficient and 40G provides adequate headroom

Generated with Claude Code

@openshift-ci
Copy link

openshift-ci bot commented Mar 12, 2026

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

sourcery-ai[bot]
sourcery-ai bot reviewed Mar 12, 2026
View reviewed changes
Copy link
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've left some high level feedback:

  • Since the new df -h / steps are being used for diagnostics, consider adding if: always() to the ones after long-running build/test steps so they still run and collect disk data even when a preceding step fails.
Prompt for AI Agents 
Please address the comments from this code review:

## Overall Comments
- Since the new `df -h /` steps are being used for diagnostics, consider adding `if: always()` to the ones after long-running build/test steps so they still run and collect disk data even when a preceding step fails.
Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@davdhacs @claude
fix(ci): increase operator build free-disk-space to 40GB
9ec0403 
Azure-based GitHub runners can start with only 77GB root disk, leaving
~19GB free. The preamble cleanup stops at the threshold, so 30GB left
the operator Docker build (go mod download + UBI base image + source
copy + compile) without enough headroom. Bumping to 40GB is safe per
preamble docs and consistent with other jobs (emailsender uses 40GB,
scanner uses 50GB).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@rhacs-bot
Copy link
Contributor

rhacs-bot commented Mar 12, 2026 •
edited
Loading

Images are ready for the commit at 9ec0403.

To use with deploy scripts, first export MAIN_IMAGE_TAG=4.11.x-310-g9ec0403a2e.

@davdhacs davdhacs force-pushed the davdhacs/fix-op-build-disk-space branch from 7f07721 to 9ec0403 Compare March 12, 2026 16:08
@davdhacs davdhacs marked this pull request as ready for review March 12, 2026 16:10
@davdhacs davdhacs requested a review from a team as a code owner March 12, 2026 16:10
sourcery-ai[bot]
sourcery-ai bot reviewed Mar 12, 2026
View reviewed changes
Copy link
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've reviewed your changes and they look great!

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@davdhacs davdhacs requested review from a team and porridge March 12, 2026 16:11
@codecov
Copy link

codecov bot commented Mar 12, 2026 •
edited
Loading

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 49.68%. Comparing base (52688db) to head (9ec0403).
⚠️ Report is 11 commits behind head on master.

Additional details and impacted files 
@@           Coverage Diff           @@
##           master   #19396   +/-   ##
=======================================
  Coverage   49.68%   49.68%           
=======================================
  Files        2700     2700           
  Lines      203278   203278           
=======================================
  Hits       100999   100999           
- Misses      94753    94754    +1     
+ Partials     7526     7525    -1
Flag Coverage Δ
go-unit-tests 49.68% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@janisz
Copy link
Contributor

janisz commented Mar 13, 2026

I think I was just hit by this issue https://github.com/stackrox/stackrox/actions/runs/23043750790/job/66927778020?pr=19379

@davdhacs
Copy link
Contributor Author

davdhacs commented Mar 13, 2026

may be not necessary when we apply #19417 to reduce the disk space used by the operator build 🥳

@davdhacs
Copy link
Contributor Author

davdhacs commented Mar 13, 2026 •
edited
Loading

I think I was just hit by this issue stackrox/stackrox/actions/runs/23043750790/job/66927778020?pr=19379

@janisz Can I get your +1 for this, and then #19417 will follow-up to reduce the disk usage? Or do you want to keep the 30gb setting because the changes will keep it under 30GB (and go in soon so it doesn't hit/block other PRs)?

@janisz janisz merged commit ac93c38 into master Mar 14, 2026
100 checks passed
@janisz janisz deleted the davdhacs/fix-op-build-disk-space branch March 14, 2026 05:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sourcery-ai sourcery-ai[bot] sourcery-ai[bot] left review comments

@janisz janisz janisz approved these changes

@porridge porridge Awaiting requested review from porridge

Assignees

No one assigned

Labels

ai-assisted ai-review area/ci

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@davdhacs @rhacs-bot @janisz