Skip to content

fix(ci): resolve disk space exhaustion in operator build #19417

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
janisz wants to merge 9 commits into
base: master
Choose a base branch
from
+40 −22
Draft

fix(ci): resolve disk space exhaustion in operator build #19417

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
73138a1
fix(ci): resolve disk space exhaustion in operator build
janisz Mar 13, 2026
9f0b3da
refactor(ci): use GitHub Actions cache backend instead of local cache
janisz Mar 13, 2026
8acf0d9
fix(ci): install Go 1.25 for operator builds
janisz Mar 13, 2026
893c0e3
refactor(ci): use setup-go built-in cache instead of custom action
janisz Mar 13, 2026
2fdda35
fix(ci): make BuildKit cache paths conditional
janisz Mar 13, 2026
8aa4abe
fix(ci): install Python 3.9 for operator bundle build
janisz Mar 13, 2026
d0a6bb9
fix(ci): restore cache-go-dependencies for cross-platform builds
janisz Mar 13, 2026
9a97411
perf(ci): skip proto generation in all operator build steps
janisz Mar 13, 2026
d682788
perf(ci): remove duplicate operator unit tests from build job
janisz Mar 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 25 additions & 20 deletions .github/workflows/build.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -598,22 +598,16 @@ jobs:
runs-on: ubuntu-latest
needs:
- define-job-matrix
container:
image: quay.io/stackrox-io/apollo-ci:stackrox-test-0.5.3
volumes:
- /usr:/mnt/usr
- /opt:/mnt/opt
env:
QUAY_RHACS_ENG_RW_USERNAME: ${{ secrets.QUAY_RHACS_ENG_RW_USERNAME }}
QUAY_RHACS_ENG_RW_PASSWORD: ${{ secrets.QUAY_RHACS_ENG_RW_PASSWORD }}
QUAY_RHACS_ENG_BEARER_TOKEN: ${{ secrets.QUAY_RHACS_ENG_BEARER_TOKEN }}
QUAY_STACKROX_IO_RW_USERNAME: ${{ secrets.QUAY_STACKROX_IO_RW_USERNAME }}
QUAY_STACKROX_IO_RW_PASSWORD: ${{ secrets.QUAY_STACKROX_IO_RW_PASSWORD }}
strategy:
fail-fast: false
matrix: ${{ fromJson(needs.define-job-matrix.outputs.matrix).build_and_push_operator }}
env:
ROX_PRODUCT_BRANDING: ${{ matrix.name }}
QUAY_RHACS_ENG_RW_USERNAME: ${{ secrets.QUAY_RHACS_ENG_RW_USERNAME }}
QUAY_RHACS_ENG_RW_PASSWORD: ${{ secrets.QUAY_RHACS_ENG_RW_PASSWORD }}
QUAY_RHACS_ENG_BEARER_TOKEN: ${{ secrets.QUAY_RHACS_ENG_BEARER_TOKEN }}
QUAY_STACKROX_IO_RW_USERNAME: ${{ secrets.QUAY_STACKROX_IO_RW_USERNAME }}
QUAY_STACKROX_IO_RW_PASSWORD: ${{ secrets.QUAY_STACKROX_IO_RW_PASSWORD }}
steps:
- name: Checkout
uses: actions/checkout@v6
Expand All @@ -626,6 +620,12 @@ jobs:
free-disk-space: 30
gcp-account: ${{ secrets.GCP_SERVICE_ACCOUNT_STACKROX_CI }}

- name: Set up Go
uses: actions/setup-go@v5
with:
go-version-file: go.mod
cache: false

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

Expand Down Expand Up @@ -661,16 +661,16 @@ jobs:
run: |
./scripts/ci/lib.sh registry_rw_login "quay.io/${QUAY_ORG}"

- name: Set up Python for bundle build
uses: actions/setup-python@v5
with:
python-version-file: operator/bundle_helpers/.python-version

- name: Build Operator Bundle image
if: |
matrix.name != 'STACKROX_BRANDING'
run: |
make -C operator/ bundle bundle-build

- name: Operator unit tests
if: matrix.arch == 'amd64'
run: |
scripts/lib.sh retry 2 true make -C operator/ test
ROX_OPERATOR_SKIP_PROTO_GENERATED_SRCS=true make -C operator/ bundle bundle-build

- name: Build Operator image
run: |
Expand All @@ -680,10 +680,15 @@ jobs:
# or in a separate intermediate target.
# + protoc-gen-go
# go: cannot install cross-compiled binaries when GOBIN is set
CGO_ENABLED=0 scripts/lib.sh retry 6 true make -C operator/ build
# Skip proto generation in CI - sources are already committed.
CGO_ENABLED=0 ROX_OPERATOR_SKIP_PROTO_GENERATED_SRCS=true scripts/lib.sh retry 6 true make -C operator/ build
# Once the native architecture dependencies are installed by the above command, the following one
# simply builds the actual operator binary for the correct target architecture.
CGO_ENABLED=0 GOARCH=${{ matrix.arch }} scripts/lib.sh retry 6 true make -C operator/ build docker-build
# simply builds the actual operator binary for the correct target architecture. Skip proto generation.
# Use GitHub Actions cache backend (type=gha) which handles cache management automatically.
CGO_ENABLED=0 GOARCH=${{ matrix.arch }} ROX_OPERATOR_SKIP_PROTO_GENERATED_SRCS=true \
BUILDKIT_CACHE_FROM="type=gha,scope=operator-${{ matrix.arch }}" \
BUILDKIT_CACHE_TO="type=gha,scope=operator-${{ matrix.arch }},mode=max" \
scripts/lib.sh retry 6 true make -C operator/ docker-build

- name: Login to docker.io to mitigate rate limiting on downloading images
uses: docker/login-action@v4
Expand Down
11 changes: 10 additions & 1 deletion operator/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
# syntax=docker/dockerfile:1

# We have to emulate directory layout as in the repo so that imports in go files work fine.
ARG roxpath=/workspace/src/github.com/stackrox/rox
ARG TARGET_ARCH=amd64
Expand All @@ -7,6 +9,9 @@ FROM --platform=$BUILDPLATFORM registry.access.redhat.com/ubi9/go-toolset:1.25 A
# Build the manager binary
ARG TARGET_ARCH
ARG roxpath
# Cache mount paths - can be overridden to match local go env
ARG GOMODCACHE_PATH=/workspace/pkg/mod
ARG GOCACHE_PATH=/root/.cache/go-build
Copy link
Contributor

@davdhacs davdhacs Mar 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Great! So this allows the go build within the docker build to use the gocache saved into github actions cache and shared across CI runs?

Copy link
Contributor Author

@janisz janisz Mar 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's the plan also it will use the local cache when run locally


WORKDIR ${roxpath}/
ENV GOPATH=/workspace
Expand All @@ -29,7 +34,11 @@ COPY go.sum go.sum
# image).
# Retry as the proxy can be unavailable at times.
ENV GOPROXY=https://proxy.golang.org|https://goproxy.io|direct
RUN go mod download || go mod download || go mod download
ARG GOMODCACHE_PATH
ARG GOCACHE_PATH
RUN --mount=type=cache,target=${GOMODCACHE_PATH},uid=1001,gid=0 \
--mount=type=cache,target=${GOCACHE_PATH},uid=1001,gid=0 \
go mod download || go mod download || go mod download

# Copy operator source
COPY operator/ operator/
Expand Down
6 changes: 5 additions & 1 deletion operator/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -357,9 +357,13 @@ build/Dockerfile.gen: Dockerfile

.PHONY: docker-build
docker-build: build/Dockerfile.gen smuggled-status-sh ## Build docker image with the operator.
BUILDKIT_PROGRESS=plain ../scripts/docker-build.sh \
DOCKER_BUILDKIT=1 BUILDKIT_PROGRESS=plain ../scripts/docker-build.sh \
-t ${IMG} \
$(if $(GOARCH),--build-arg TARGET_ARCH=$(GOARCH)) \
$(if $(GOMODCACHE_PATH),--build-arg GOMODCACHE_PATH=$(GOMODCACHE_PATH)) \
$(if $(GOCACHE_PATH),--build-arg GOCACHE_PATH=$(GOCACHE_PATH)) \
$(if $(BUILDKIT_CACHE_FROM),--cache-from $(BUILDKIT_CACHE_FROM)) \
$(if $(BUILDKIT_CACHE_TO),--cache-to $(BUILDKIT_CACHE_TO)) \
-f $< \
..

Expand Down
Loading