Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
c55e335
perf(ci): add Docker buildx layer caching for image builds
davdhacs Apr 2, 2026
3b67987
ci: trigger warm buildx cache run
davdhacs Apr 2, 2026
c917dc8
perf(ci): restructure Dockerfile for better layer caching
davdhacs Apr 2, 2026
ad814a0
fix: copy save-dir-contents into base-with-packages stage
davdhacs Apr 3, 2026
e1dca2b
ci: trigger warm run with Dockerfile cache
davdhacs Apr 3, 2026
c48a86d
perf(ci): use docker/build-push-action for main image with GHA cache
davdhacs Apr 3, 2026
6e39717
ci: trigger warm run
davdhacs Apr 3, 2026
81346d5
perf(ci): use COPY --link for independent layer caching
davdhacs Apr 3, 2026
6ecf0a3
ci: warm run with COPY --link
davdhacs Apr 3, 2026
23a6713
perf(ci): push main image directly from buildx, skip --load
davdhacs Apr 3, 2026
b10b256
fix(ci): use secrets directly for registry login
davdhacs Apr 3, 2026
9d051d2
fix(ci): add arch-suffixed tags for manifest list creation
davdhacs Apr 3, 2026
16b7296
fix(ci): use registry_rw_login for quay.io push credentials
davdhacs Apr 3, 2026
d3454d6
fix(ci): push main to rhacs-eng only via buildx, use existing push fo…
davdhacs Apr 3, 2026
8172824
fix(ci): handle missing local main image after buildx push
davdhacs Apr 3, 2026
5468d96
fix(ci): revert to load:true with existing push pipeline
davdhacs Apr 3, 2026
b5b96c1
perf(ci): push main image directly via buildx with provenance:false
davdhacs Apr 3, 2026
079fd0f
fix(ci): use docker/login-action for buildx push, skopeo for cross-or…
davdhacs Apr 3, 2026
9096228
fix(ci): use explicit skopeo credentials for cross-org image copy
davdhacs Apr 3, 2026
da69f5f
perf(ci): switch Docker layer cache from GHA to GHCR registry
davdhacs Apr 4, 2026
97c431f
perf(ci): use docker buildx driver to skip ~40s container boot
davdhacs Apr 4, 2026
2af6fca
ci: warm run with GHCR cache
davdhacs Apr 4, 2026
8a15c85
perf(ci): add inline cache with docker driver
davdhacs Apr 4, 2026
e6f6f05
ci: warm run with docker driver + inline cache
davdhacs Apr 4, 2026
16db856
experiment: stable build ldflags for Docker layer caching
davdhacs Apr 4, 2026
5752a50
ci: warm run - test layer cache with stable binaries
davdhacs Apr 4, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
111 changes: 84 additions & 27 deletions .github/workflows/build.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -181,8 +181,8 @@ jobs:
- /usr:/mnt/usr
- /opt:/mnt/opt
env:
BUILD_TAG: ${{ needs.define-job-matrix.outputs.build-tag }}
SHORTCOMMIT: ${{ needs.define-job-matrix.outputs.short-commit }}
BUILD_TAG: 0.0.0
SHORTCOMMIT: "0000000"
Comment on lines +184 to +185
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Don't bake 0.0.0/0000000 into the published binaries.

These overrides apply to the same workflow that runs on push to master and *-nightly-*, so the main/roxctl/operator artifacts published from those builds will also self-report the placeholder version. If this trade-off is only acceptable for CI/PR testing, gate it away from publishing contexts.

Possible guard
-      BUILD_TAG: 0.0.0
-      SHORTCOMMIT: "0000000"
+      BUILD_TAG: ${{ github.event_name == 'pull_request' && '0.0.0' || needs.define-job-matrix.outputs.build-tag }}
+      SHORTCOMMIT: ${{ github.event_name == 'pull_request' && '0000000' || needs.define-job-matrix.outputs.short-commit }}

Apply the same pattern in both pre-build-cli and pre-build-go-binaries.

Also applies to: 226-230

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/build.yaml around lines 184 - 185, The workflow currently
hardcodes BUILD_TAG and SHORTCOMMIT to "0.0.0" and "0000000" (seen in the
variables block) which causes published artifacts to self-report placeholder
versions; change the logic in the pre-build-cli and pre-build-go-binaries steps
so these placeholder overrides are only applied for CI/PR/test runs (e.g., when
event is pull_request or a non-publishing branch) and are skipped for publishing
contexts (push-to-master, nightly tag builds, or when a publish flag is set);
update both occurrences (also the similar block at lines 226-230) to
conditionally set BUILD_TAG/SHORTCOMMIT based on the workflow context or a
publish-specific input/env var rather than unconditionally overriding them.

GOTAGS: ${{ needs.define-job-matrix.outputs.gotags }}
steps:
- name: Checkout
Expand Down Expand Up @@ -223,8 +223,11 @@ jobs:
- /usr:/mnt/usr
- /opt:/mnt/opt
env:
BUILD_TAG: ${{ needs.define-job-matrix.outputs.build-tag }}
SHORTCOMMIT: ${{ needs.define-job-matrix.outputs.short-commit }}
# Stable version for Go binaries — produces byte-identical output
# across commits that don't change Go source. Enables Docker
# COPY --link layer caching: unchanged binary = cached layer = skip push.
BUILD_TAG: 0.0.0
SHORTCOMMIT: "0000000"
GOTAGS: ${{ needs.define-job-matrix.outputs.gotags }}
steps:
- name: Checkout
Expand Down Expand Up @@ -417,6 +420,8 @@ jobs:

build-and-push-main:
runs-on: ubuntu-latest
permissions:
packages: write # for GHCR buildx layer cache
needs:
- define-job-matrix
- pre-build-cli
Expand Down Expand Up @@ -500,6 +505,9 @@ jobs:

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
with:
# Use docker driver (built-in buildkit) — skips ~40s container boot.
driver: docker

- name: Checkout submodules
run: |
Expand Down Expand Up @@ -544,30 +552,59 @@ jobs:
if: matrix.name == 'race-condition-debug'
run: echo "BUILD_TAG=$(make --quiet --no-print-directory tag)-rcd" >> "$GITHUB_ENV"

- name: Build main images
- name: Prepare main image build context
run: |
GOOS=linux GOARCH=${{ matrix.arch }} scripts/lib.sh retry 6 true make docker-build-main-image
GOOS=linux GOARCH=${{ matrix.arch }} make copy-binaries-to-image-dir central-db-image

# docker/login-action injects credentials into the buildx builder,
# unlike plain `docker login` which only writes to host config.
- name: Login to quay.io for buildx push
if: |
github.event_name == 'push' || !github.event.pull_request.head.repo.fork
uses: docker/login-action@v4
with:
registry: quay.io
username: ${{ secrets.QUAY_RHACS_ENG_RW_USERNAME }}
password: ${{ secrets.QUAY_RHACS_ENG_RW_PASSWORD }}

- name: Check debugger presence in the main image
run: make check-debugger
- name: Build and push main image
uses: docker/build-push-action@v6
with:
context: image/rhel
file: image/rhel/Dockerfile
platforms: linux/${{ matrix.arch }}
# Push directly to registry — avoids slow --load export (~90s).
# provenance: false produces a plain image (not a manifest list),
# compatible with the downstream push-main-manifests job.
push: ${{ github.event_name == 'push' || !github.event.pull_request.head.repo.fork }}
load: ${{ !(github.event_name == 'push' || !github.event.pull_request.head.repo.fork) }}
provenance: false
# Push to rhacs-eng via buildx (fast, cached layers).
# stackrox-io is handled by the Push remaining step below.
tags: |
quay.io/rhacs-eng/main:${{ env.BUILD_TAG }}
quay.io/rhacs-eng/main:${{ env.BUILD_TAG }}-${{ matrix.arch }}
Comment on lines +585 to +586
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Keep the replacement push path brand-aware and per-arch.

This matrix still builds distinct RHACS_BRANDING/STACKROX_BRANDING artifacts, but these tags are now shared across both variants and then mirrored/pushed to both registries. That lets one branding overwrite the other, and it also publishes the shared ${BUILD_TAG} tag before push-main-manifests assembles the final manifest list. The old push_main_image_set flow kept registry selection brand-aware, published per-arch tags from the build jobs, and added latest-${arch} on merge-to-master.

Also applies to: 620-629, 634-639

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/build.yaml around lines 585 - 586, The current push uses
shared tags (quay.io/rhacs-eng/main:${{ env.BUILD_TAG }} and
quay.io/rhacs-eng/main:${{ env.BUILD_TAG }}-${{ matrix.arch }}) which lets one
branding overwrite the other; update the push steps so registry paths remain
brand-aware and per-arch by including the branding identifier in the image name
(e.g., incorporate the branding env var into the registry/repo component) for
all occurrences of those tags and the equivalent blocks mentioned (also adjust
the analogous blocks at the other ranges), stop publishing the shared
${BUILD_TAG} tag from build jobs (only publish per-arch tags and per-arch
latest-${{ matrix.arch }} on merge-to-master) and only create/push the shared
manifest tag during the push-main-manifests / push_main_image_set manifest
assembly step.

quay.io/rhacs-eng/main:cache-${{ matrix.arch }}
build-args: |
DEBUG_BUILD=${{ env.DEBUG_BUILD || 'no' }}
ROX_PRODUCT_BRANDING=${{ env.ROX_PRODUCT_BRANDING }}
TARGET_ARCH=${{ matrix.arch }}
ROX_IMAGE_FLAVOR=development_build
LABEL_VERSION=${{ env.BUILD_TAG }}
LABEL_RELEASE=${{ env.BUILD_TAG }}
# Pull the previous build as cache source. With COPY --link,
Comment on lines +588 to +595
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

QUAY_TAG_EXPIRATION was dropped from the new build invocation.

image/rhel/Dockerfile still populates quay.expires-after from this arg, and the old docker-build-main-image target in Makefile passed it through. Without it, the pushed main images lose their expiry label, so ephemeral PR/nightly tags can linger indefinitely.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/build.yaml around lines 588 - 595, The build workflow
dropped the QUAY_TAG_EXPIRATION build-arg, causing image/rhel/Dockerfile's
quay.expires-after label to be empty; restore the missing argument by adding
QUAY_TAG_EXPIRATION=${{ env.QUAY_TAG_EXPIRATION || '<appropriate-default>' }}
back into the build-args block so the published main images receive the expiry
label (matching how the old docker-build-main-image Makefile target propagated
QUAY_TAG_EXPIRATION).

# unchanged layers (packages, UI, static data) are reused from
# the cache image — only changed binary layers are rebuilt+pushed.
cache-from: type=registry,ref=quay.io/rhacs-eng/main:cache-${{ matrix.arch }}
cache-to: type=inline

- name: Build roxctl image
run: |
GOOS=linux GOARCH=${{ matrix.arch }} scripts/lib.sh retry 6 true make docker-build-roxctl-image

# needed for docs ensure_image.sh initial pull with RHACS_BRANDING
- name: Docker login
# Skip for external contributions.
if: |
github.event_name == 'push' || !github.event.pull_request.head.repo.fork
env:
QUAY_RHACS_ENG_RO_USERNAME: ${{ secrets.QUAY_RHACS_ENG_RO_USERNAME }}
QUAY_RHACS_ENG_RO_PASSWORD: ${{ secrets.QUAY_RHACS_ENG_RO_PASSWORD }}
run: |
./scripts/ci/lib.sh registry_ro_login "quay.io/rhacs-eng"

- name: Push images
# Skip for external contributions.
- name: Push remaining images
# Main image already pushed to rhacs-eng by build-push-action.
# Push roxctl/central-db and copy main to stackrox-io.
if: |
github.event_name == 'push' || !github.event.pull_request.head.repo.fork
env:
Expand All @@ -577,12 +614,30 @@ jobs:
QUAY_STACKROX_IO_RW_PASSWORD: ${{ secrets.QUAY_STACKROX_IO_RW_PASSWORD }}
run: |
source ./scripts/ci/lib.sh
echo "Will determine context from: ${{ github.event_name }} & ${{ github.ref_name }}"
push_context=""
if [[ "${{ github.event_name }}" == "push" && "${{ github.ref_name }}" == "master" ]]; then
push_context="merge-to-master"
fi
push_main_image_set "$push_context" "${{ env.ROX_PRODUCT_BRANDING }}" "${{ matrix.arch }}"

# Push roxctl and central-db (built locally)
for image in roxctl central-db; do
docker tag "stackrox/${image}:${BUILD_TAG}" "quay.io/rhacs-eng/${image}:${BUILD_TAG}"
docker tag "stackrox/${image}:${BUILD_TAG}" "quay.io/rhacs-eng/${image}:${BUILD_TAG}-${{ matrix.arch }}"
docker tag "stackrox/${image}:${BUILD_TAG}" "quay.io/stackrox-io/${image}:${BUILD_TAG}"
docker tag "stackrox/${image}:${BUILD_TAG}" "quay.io/stackrox-io/${image}:${BUILD_TAG}-${{ matrix.arch }}"
registry_rw_login "quay.io/rhacs-eng"
retry 5 true docker push "quay.io/rhacs-eng/${image}:${BUILD_TAG}" | cat
retry 5 true docker push "quay.io/rhacs-eng/${image}:${BUILD_TAG}-${{ matrix.arch }}" | cat
registry_rw_login "quay.io/stackrox-io"
retry 5 true docker push "quay.io/stackrox-io/${image}:${BUILD_TAG}" | cat
Comment on lines +619 to +628
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion (performance): Repeated registry logins inside the image loop add avoidable overhead and latency.

registry_rw_login is invoked for quay.io/rhacs-eng and quay.io/stackrox-io on each loop iteration even though both credentials and registries are constant. Consider moving these two login calls outside the for image in ... loop so they run once per registry, and keep only the docker push operations inside the loop to avoid repeated API calls.

retry 5 true docker push "quay.io/stackrox-io/${image}:${BUILD_TAG}-${{ matrix.arch }}" | cat
done

# Copy main from rhacs-eng to stackrox-io (blobs shared on quay.io, lightweight).
# Use explicit credentials since docker login can only hold one quay.io login.
for tag in "${BUILD_TAG}" "${BUILD_TAG}-${{ matrix.arch }}"; do
retry 5 true skopeo copy \
--src-creds "${QUAY_RHACS_ENG_RW_USERNAME}:${QUAY_RHACS_ENG_RW_PASSWORD}" \
--dest-creds "${QUAY_STACKROX_IO_RW_USERNAME}:${QUAY_STACKROX_IO_RW_PASSWORD}" \
"docker://quay.io/rhacs-eng/main:${tag}" \
"docker://quay.io/stackrox-io/main:${tag}"
done

push-matching-collector-scanner:
runs-on: ubuntu-latest
Expand Down Expand Up @@ -809,6 +864,8 @@ jobs:
./scripts/ci/lib.sh registry_rw_login "quay.io/${QUAY_ORG}"

- name: Build Operator image
env:
DOCKER_BUILDX_CACHE: operator-${{ matrix.arch }}
run: |
GOARCH=${{ matrix.arch }} scripts/lib.sh retry 6 true make -C operator/ docker-build-prebuilt

Expand Down
92 changes: 52 additions & 40 deletions image/rhel/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,39 @@ WORKDIR /
COPY fetch-stackrox-data.sh .
RUN /fetch-stackrox-data.sh /stackrox-data

FROM ${BASE_REGISTRY}/${BASE_IMAGE}:${BASE_TAG}
# Install OS packages in a separate stage so Docker can cache this layer
# independently of binary changes. Package installs rarely change, but
# binaries change every commit — this ordering avoids rebuilding packages
# when only binaries change.
FROM ${BASE_REGISTRY}/${BASE_IMAGE}:${BASE_TAG} AS base-with-packages

COPY signatures/RPM-GPG-KEY-CentOS-Official /
COPY static-bin/save-dir-contents /stackrox/save-dir-contents
COPY static-bin/restore-all-dir-contents /stackrox/restore-all-dir-contents
ENV PATH="/stackrox:$PATH"
COPY --from=downloads /output/rpms/ /tmp/

RUN rpm --import RPM-GPG-KEY-CentOS-Official && \
microdnf -y upgrade --nobest && \
rpm -i --nodeps /tmp/postgres-libs.rpm && \
rpm -i --nodeps /tmp/postgres.rpm && \
microdnf install --setopt=install_weak_deps=0 --nodocs -y util-linux && \
microdnf clean all -y && \
rm /tmp/postgres.rpm /tmp/postgres-libs.rpm RPM-GPG-KEY-CentOS-Official && \
# (Optional) Remove line below to keep package management utilities
rpm -e --nodeps $(rpm -qa curl '*rpm*' '*dnf*' '*libsolv*' '*hawkey*' 'yum*') && \
rm -rf /var/cache/dnf /var/cache/yum && \
# The contents of paths mounted as emptyDir volumes in Kubernetes are saved
# by the script `save-dir-contents` during the image build. The directory
# contents are then restored by the script `restore-all-dir-contents`
# during the container start.
chown -R 4000:4000 /etc/pki/ca-trust && save-dir-contents /etc/pki/ca-trust/source && \
mkdir -p /var/lib/stackrox && chown -R 4000:4000 /var/lib/stackrox && \
mkdir -p /var/log/stackrox && chown -R 4000:4000 /var/log/stackrox && \
mkdir -p /var/cache/stackrox && chown -R 4000:4000 /var/cache/stackrox && \
chown -R 4000:4000 /tmp

FROM base-with-packages

ARG LABEL_VERSION
ARG LABEL_RELEASE
Expand All @@ -45,48 +77,28 @@ ENV PATH="/stackrox:$PATH" \
ROX_IMAGE_FLAVOR=${ROX_IMAGE_FLAVOR} \
ROX_PRODUCT_BRANDING=${ROX_PRODUCT_BRANDING}

COPY signatures/RPM-GPG-KEY-CentOS-Official /
COPY static-bin /stackrox/
# Use --link so each COPY is an independent overlay layer.
# Changing one binary doesn't invalidate other layers.
COPY --link static-bin /stackrox/

COPY --from=downloads /output/rpms/ /tmp/
COPY --from=downloads /output/go/ /go/
COPY --link --from=downloads /output/go/ /go/

RUN rpm --import RPM-GPG-KEY-CentOS-Official && \
microdnf -y upgrade --nobest && \
rpm -i --nodeps /tmp/postgres-libs.rpm && \
rpm -i --nodeps /tmp/postgres.rpm && \
microdnf install --setopt=install_weak_deps=0 --nodocs -y util-linux && \
microdnf clean all -y && \
rm /tmp/postgres.rpm /tmp/postgres-libs.rpm RPM-GPG-KEY-CentOS-Official && \
# (Optional) Remove line below to keep package management utilities
rpm -e --nodeps $(rpm -qa curl '*rpm*' '*dnf*' '*libsolv*' '*hawkey*' 'yum*') && \
rm -rf /var/cache/dnf /var/cache/yum && \
# The contents of paths mounted as emptyDir volumes in Kubernetes are saved
# by the script `save-dir-contents` during the image build. The directory
# contents are then restored by the script `restore-all-dir-contents`
# during the container start.
chown -R 4000:4000 /etc/pki/ca-trust && save-dir-contents /etc/pki/ca-trust/source && \
mkdir -p /var/lib/stackrox && chown -R 4000:4000 /var/lib/stackrox && \
mkdir -p /var/log/stackrox && chown -R 4000:4000 /var/log/stackrox && \
mkdir -p /var/cache/stackrox && chown -R 4000:4000 /var/cache/stackrox && \
chown -R 4000:4000 /tmp
COPY --link --from=stackrox_data /stackrox-data /stackrox/static-data
COPY --link ./docs/api/v1/swagger.json /stackrox/static-data/docs/api/v1/swagger.json
COPY --link ./docs/api/v2/swagger.json /stackrox/static-data/docs/api/v2/swagger.json
COPY --link THIRD_PARTY_NOTICES /THIRD_PARTY_NOTICES/

COPY --link ui /ui
Comment on lines +82 to +91
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (bug_risk): Using COPY --link requires BuildKit and may break non-buildx/local docker builds using this Dockerfile.

In CI this is fine because we use buildx, but any remaining docker build usage (local dev, older daemons, or CI jobs without BuildKit) will fail on this Dockerfile. If those paths must keep working, either make BuildKit a documented, enforced requirement for all builds that use this file (ideally with a clear failure if it’s not enabled), or drop --link and rely on the existing caching strategy.


COPY --from=stackrox_data /stackrox-data /stackrox/static-data
COPY ./docs/api/v1/swagger.json /stackrox/static-data/docs/api/v1/swagger.json
COPY ./docs/api/v2/swagger.json /stackrox/static-data/docs/api/v2/swagger.json
COPY THIRD_PARTY_NOTICES /THIRD_PARTY_NOTICES/

COPY ui /ui

COPY bin/central /stackrox/central
COPY bin/migrator /stackrox/bin/migrator
COPY bin/compliance /stackrox/bin/compliance
COPY bin/kubernetes-sensor /stackrox/bin/kubernetes-sensor
COPY bin/sensor-upgrader /stackrox/bin/sensor-upgrader
COPY bin/admission-control /stackrox/bin/admission-control
COPY bin/config-controller /stackrox/bin/config-controller
COPY bin/roxagent /stackrox/bin/roxagent
COPY bin/roxctl* /assets/downloads/cli/
COPY --link bin/central /stackrox/central
COPY --link bin/migrator /stackrox/bin/migrator
COPY --link bin/compliance /stackrox/bin/compliance
COPY --link bin/kubernetes-sensor /stackrox/bin/kubernetes-sensor
COPY --link bin/sensor-upgrader /stackrox/bin/sensor-upgrader
COPY --link bin/admission-control /stackrox/bin/admission-control
COPY --link bin/config-controller /stackrox/bin/config-controller
COPY --link bin/roxagent /stackrox/bin/roxagent
COPY --link bin/roxctl* /assets/downloads/cli/

RUN ln -s /assets/downloads/cli/roxctl-linux-${TARGET_ARCH} /stackrox/roxctl && \
ln -s /assets/downloads/cli/roxctl-linux-amd64 /assets/downloads/cli/roxctl-linux
Expand Down
12 changes: 11 additions & 1 deletion scripts/docker-build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,19 @@

set -e

cache_args=()
if [[ -n "${DOCKER_BUILDX_CACHE}" ]]; then
# GHA buildx cache: reuse Docker layers (base image pulls, package installs)
# across CI runs. Scope per-arch to avoid cache collisions.
cache_args+=(
--cache-from "type=gha,scope=${DOCKER_BUILDX_CACHE}"
--cache-to "type=gha,mode=max,scope=${DOCKER_BUILDX_CACHE}"
)
fi

echo "Building with platform linux/${GOARCH}"
if docker info | grep buildx; then
docker buildx build --platform "linux/${GOARCH}" --load "$@"
docker buildx build --platform "linux/${GOARCH}" "${cache_args[@]}" --load "$@"
else
docker build --platform "linux/${GOARCH}" "$@"
fi
Loading