-
Notifications
You must be signed in to change notification settings - Fork 174
experiment: stable build ldflags for Docker layer caching #19829
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
c55e335
3b67987
c917dc8
ad814a0
e1dca2b
c48a86d
6e39717
81346d5
6ecf0a3
23a6713
b10b256
9d051d2
16b7296
d3454d6
8172824
5468d96
b5b96c1
079fd0f
9096228
da69f5f
97c431f
2af6fca
8a15c85
e6f6f05
16db856
5752a50
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -181,8 +181,8 @@ jobs: | |
| - /usr:/mnt/usr | ||
| - /opt:/mnt/opt | ||
| env: | ||
| BUILD_TAG: ${{ needs.define-job-matrix.outputs.build-tag }} | ||
| SHORTCOMMIT: ${{ needs.define-job-matrix.outputs.short-commit }} | ||
| BUILD_TAG: 0.0.0 | ||
| SHORTCOMMIT: "0000000" | ||
| GOTAGS: ${{ needs.define-job-matrix.outputs.gotags }} | ||
| steps: | ||
| - name: Checkout | ||
|
|
@@ -223,8 +223,11 @@ jobs: | |
| - /usr:/mnt/usr | ||
| - /opt:/mnt/opt | ||
| env: | ||
| BUILD_TAG: ${{ needs.define-job-matrix.outputs.build-tag }} | ||
| SHORTCOMMIT: ${{ needs.define-job-matrix.outputs.short-commit }} | ||
| # Stable version for Go binaries — produces byte-identical output | ||
| # across commits that don't change Go source. Enables Docker | ||
| # COPY --link layer caching: unchanged binary = cached layer = skip push. | ||
| BUILD_TAG: 0.0.0 | ||
| SHORTCOMMIT: "0000000" | ||
| GOTAGS: ${{ needs.define-job-matrix.outputs.gotags }} | ||
| steps: | ||
| - name: Checkout | ||
|
|
@@ -417,6 +420,8 @@ jobs: | |
|
|
||
| build-and-push-main: | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| packages: write # for GHCR buildx layer cache | ||
| needs: | ||
| - define-job-matrix | ||
| - pre-build-cli | ||
|
|
@@ -500,6 +505,9 @@ jobs: | |
|
|
||
| - name: Set up Docker Buildx | ||
| uses: docker/setup-buildx-action@v4 | ||
| with: | ||
| # Use docker driver (built-in buildkit) — skips ~40s container boot. | ||
| driver: docker | ||
|
|
||
| - name: Checkout submodules | ||
| run: | | ||
|
|
@@ -544,30 +552,59 @@ jobs: | |
| if: matrix.name == 'race-condition-debug' | ||
| run: echo "BUILD_TAG=$(make --quiet --no-print-directory tag)-rcd" >> "$GITHUB_ENV" | ||
|
|
||
| - name: Build main images | ||
| - name: Prepare main image build context | ||
| run: | | ||
| GOOS=linux GOARCH=${{ matrix.arch }} scripts/lib.sh retry 6 true make docker-build-main-image | ||
| GOOS=linux GOARCH=${{ matrix.arch }} make copy-binaries-to-image-dir central-db-image | ||
|
|
||
| # docker/login-action injects credentials into the buildx builder, | ||
| # unlike plain `docker login` which only writes to host config. | ||
| - name: Login to quay.io for buildx push | ||
| if: | | ||
| github.event_name == 'push' || !github.event.pull_request.head.repo.fork | ||
| uses: docker/login-action@v4 | ||
| with: | ||
| registry: quay.io | ||
| username: ${{ secrets.QUAY_RHACS_ENG_RW_USERNAME }} | ||
| password: ${{ secrets.QUAY_RHACS_ENG_RW_PASSWORD }} | ||
|
|
||
| - name: Check debugger presence in the main image | ||
| run: make check-debugger | ||
| - name: Build and push main image | ||
| uses: docker/build-push-action@v6 | ||
| with: | ||
| context: image/rhel | ||
| file: image/rhel/Dockerfile | ||
| platforms: linux/${{ matrix.arch }} | ||
| # Push directly to registry — avoids slow --load export (~90s). | ||
| # provenance: false produces a plain image (not a manifest list), | ||
| # compatible with the downstream push-main-manifests job. | ||
| push: ${{ github.event_name == 'push' || !github.event.pull_request.head.repo.fork }} | ||
| load: ${{ !(github.event_name == 'push' || !github.event.pull_request.head.repo.fork) }} | ||
| provenance: false | ||
| # Push to rhacs-eng via buildx (fast, cached layers). | ||
| # stackrox-io is handled by the Push remaining step below. | ||
| tags: | | ||
| quay.io/rhacs-eng/main:${{ env.BUILD_TAG }} | ||
| quay.io/rhacs-eng/main:${{ env.BUILD_TAG }}-${{ matrix.arch }} | ||
|
Comment on lines
+585
to
+586
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Keep the replacement push path brand-aware and per-arch. This matrix still builds distinct Also applies to: 620-629, 634-639 🤖 Prompt for AI Agents |
||
| quay.io/rhacs-eng/main:cache-${{ matrix.arch }} | ||
| build-args: | | ||
| DEBUG_BUILD=${{ env.DEBUG_BUILD || 'no' }} | ||
| ROX_PRODUCT_BRANDING=${{ env.ROX_PRODUCT_BRANDING }} | ||
| TARGET_ARCH=${{ matrix.arch }} | ||
| ROX_IMAGE_FLAVOR=development_build | ||
| LABEL_VERSION=${{ env.BUILD_TAG }} | ||
| LABEL_RELEASE=${{ env.BUILD_TAG }} | ||
| # Pull the previous build as cache source. With COPY --link, | ||
|
Comment on lines
+588
to
+595
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
🤖 Prompt for AI Agents |
||
| # unchanged layers (packages, UI, static data) are reused from | ||
| # the cache image — only changed binary layers are rebuilt+pushed. | ||
| cache-from: type=registry,ref=quay.io/rhacs-eng/main:cache-${{ matrix.arch }} | ||
| cache-to: type=inline | ||
|
|
||
| - name: Build roxctl image | ||
| run: | | ||
| GOOS=linux GOARCH=${{ matrix.arch }} scripts/lib.sh retry 6 true make docker-build-roxctl-image | ||
|
|
||
| # needed for docs ensure_image.sh initial pull with RHACS_BRANDING | ||
| - name: Docker login | ||
| # Skip for external contributions. | ||
| if: | | ||
| github.event_name == 'push' || !github.event.pull_request.head.repo.fork | ||
| env: | ||
| QUAY_RHACS_ENG_RO_USERNAME: ${{ secrets.QUAY_RHACS_ENG_RO_USERNAME }} | ||
| QUAY_RHACS_ENG_RO_PASSWORD: ${{ secrets.QUAY_RHACS_ENG_RO_PASSWORD }} | ||
| run: | | ||
| ./scripts/ci/lib.sh registry_ro_login "quay.io/rhacs-eng" | ||
|
|
||
| - name: Push images | ||
| # Skip for external contributions. | ||
| - name: Push remaining images | ||
| # Main image already pushed to rhacs-eng by build-push-action. | ||
| # Push roxctl/central-db and copy main to stackrox-io. | ||
| if: | | ||
| github.event_name == 'push' || !github.event.pull_request.head.repo.fork | ||
| env: | ||
|
|
@@ -577,12 +614,30 @@ jobs: | |
| QUAY_STACKROX_IO_RW_PASSWORD: ${{ secrets.QUAY_STACKROX_IO_RW_PASSWORD }} | ||
| run: | | ||
| source ./scripts/ci/lib.sh | ||
| echo "Will determine context from: ${{ github.event_name }} & ${{ github.ref_name }}" | ||
| push_context="" | ||
| if [[ "${{ github.event_name }}" == "push" && "${{ github.ref_name }}" == "master" ]]; then | ||
| push_context="merge-to-master" | ||
| fi | ||
| push_main_image_set "$push_context" "${{ env.ROX_PRODUCT_BRANDING }}" "${{ matrix.arch }}" | ||
|
|
||
| # Push roxctl and central-db (built locally) | ||
| for image in roxctl central-db; do | ||
| docker tag "stackrox/${image}:${BUILD_TAG}" "quay.io/rhacs-eng/${image}:${BUILD_TAG}" | ||
| docker tag "stackrox/${image}:${BUILD_TAG}" "quay.io/rhacs-eng/${image}:${BUILD_TAG}-${{ matrix.arch }}" | ||
| docker tag "stackrox/${image}:${BUILD_TAG}" "quay.io/stackrox-io/${image}:${BUILD_TAG}" | ||
| docker tag "stackrox/${image}:${BUILD_TAG}" "quay.io/stackrox-io/${image}:${BUILD_TAG}-${{ matrix.arch }}" | ||
| registry_rw_login "quay.io/rhacs-eng" | ||
| retry 5 true docker push "quay.io/rhacs-eng/${image}:${BUILD_TAG}" | cat | ||
| retry 5 true docker push "quay.io/rhacs-eng/${image}:${BUILD_TAG}-${{ matrix.arch }}" | cat | ||
| registry_rw_login "quay.io/stackrox-io" | ||
| retry 5 true docker push "quay.io/stackrox-io/${image}:${BUILD_TAG}" | cat | ||
|
Comment on lines
+619
to
+628
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. suggestion (performance): Repeated registry logins inside the image loop add avoidable overhead and latency.
|
||
| retry 5 true docker push "quay.io/stackrox-io/${image}:${BUILD_TAG}-${{ matrix.arch }}" | cat | ||
| done | ||
|
|
||
| # Copy main from rhacs-eng to stackrox-io (blobs shared on quay.io, lightweight). | ||
| # Use explicit credentials since docker login can only hold one quay.io login. | ||
| for tag in "${BUILD_TAG}" "${BUILD_TAG}-${{ matrix.arch }}"; do | ||
| retry 5 true skopeo copy \ | ||
| --src-creds "${QUAY_RHACS_ENG_RW_USERNAME}:${QUAY_RHACS_ENG_RW_PASSWORD}" \ | ||
| --dest-creds "${QUAY_STACKROX_IO_RW_USERNAME}:${QUAY_STACKROX_IO_RW_PASSWORD}" \ | ||
| "docker://quay.io/rhacs-eng/main:${tag}" \ | ||
| "docker://quay.io/stackrox-io/main:${tag}" | ||
| done | ||
|
|
||
| push-matching-collector-scanner: | ||
| runs-on: ubuntu-latest | ||
|
|
@@ -809,6 +864,8 @@ jobs: | |
| ./scripts/ci/lib.sh registry_rw_login "quay.io/${QUAY_ORG}" | ||
|
|
||
| - name: Build Operator image | ||
| env: | ||
| DOCKER_BUILDX_CACHE: operator-${{ matrix.arch }} | ||
| run: | | ||
| GOARCH=${{ matrix.arch }} scripts/lib.sh retry 6 true make -C operator/ docker-build-prebuilt | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -22,7 +22,39 @@ WORKDIR / | |
| COPY fetch-stackrox-data.sh . | ||
| RUN /fetch-stackrox-data.sh /stackrox-data | ||
|
|
||
| FROM ${BASE_REGISTRY}/${BASE_IMAGE}:${BASE_TAG} | ||
| # Install OS packages in a separate stage so Docker can cache this layer | ||
| # independently of binary changes. Package installs rarely change, but | ||
| # binaries change every commit — this ordering avoids rebuilding packages | ||
| # when only binaries change. | ||
| FROM ${BASE_REGISTRY}/${BASE_IMAGE}:${BASE_TAG} AS base-with-packages | ||
|
|
||
| COPY signatures/RPM-GPG-KEY-CentOS-Official / | ||
| COPY static-bin/save-dir-contents /stackrox/save-dir-contents | ||
| COPY static-bin/restore-all-dir-contents /stackrox/restore-all-dir-contents | ||
| ENV PATH="/stackrox:$PATH" | ||
| COPY --from=downloads /output/rpms/ /tmp/ | ||
|
|
||
| RUN rpm --import RPM-GPG-KEY-CentOS-Official && \ | ||
| microdnf -y upgrade --nobest && \ | ||
| rpm -i --nodeps /tmp/postgres-libs.rpm && \ | ||
| rpm -i --nodeps /tmp/postgres.rpm && \ | ||
| microdnf install --setopt=install_weak_deps=0 --nodocs -y util-linux && \ | ||
| microdnf clean all -y && \ | ||
| rm /tmp/postgres.rpm /tmp/postgres-libs.rpm RPM-GPG-KEY-CentOS-Official && \ | ||
| # (Optional) Remove line below to keep package management utilities | ||
| rpm -e --nodeps $(rpm -qa curl '*rpm*' '*dnf*' '*libsolv*' '*hawkey*' 'yum*') && \ | ||
| rm -rf /var/cache/dnf /var/cache/yum && \ | ||
| # The contents of paths mounted as emptyDir volumes in Kubernetes are saved | ||
| # by the script `save-dir-contents` during the image build. The directory | ||
| # contents are then restored by the script `restore-all-dir-contents` | ||
| # during the container start. | ||
| chown -R 4000:4000 /etc/pki/ca-trust && save-dir-contents /etc/pki/ca-trust/source && \ | ||
| mkdir -p /var/lib/stackrox && chown -R 4000:4000 /var/lib/stackrox && \ | ||
| mkdir -p /var/log/stackrox && chown -R 4000:4000 /var/log/stackrox && \ | ||
| mkdir -p /var/cache/stackrox && chown -R 4000:4000 /var/cache/stackrox && \ | ||
| chown -R 4000:4000 /tmp | ||
|
|
||
| FROM base-with-packages | ||
|
|
||
| ARG LABEL_VERSION | ||
| ARG LABEL_RELEASE | ||
|
|
@@ -45,48 +77,28 @@ ENV PATH="/stackrox:$PATH" \ | |
| ROX_IMAGE_FLAVOR=${ROX_IMAGE_FLAVOR} \ | ||
| ROX_PRODUCT_BRANDING=${ROX_PRODUCT_BRANDING} | ||
|
|
||
| COPY signatures/RPM-GPG-KEY-CentOS-Official / | ||
| COPY static-bin /stackrox/ | ||
| # Use --link so each COPY is an independent overlay layer. | ||
| # Changing one binary doesn't invalidate other layers. | ||
| COPY --link static-bin /stackrox/ | ||
|
|
||
| COPY --from=downloads /output/rpms/ /tmp/ | ||
| COPY --from=downloads /output/go/ /go/ | ||
| COPY --link --from=downloads /output/go/ /go/ | ||
|
|
||
| RUN rpm --import RPM-GPG-KEY-CentOS-Official && \ | ||
| microdnf -y upgrade --nobest && \ | ||
| rpm -i --nodeps /tmp/postgres-libs.rpm && \ | ||
| rpm -i --nodeps /tmp/postgres.rpm && \ | ||
| microdnf install --setopt=install_weak_deps=0 --nodocs -y util-linux && \ | ||
| microdnf clean all -y && \ | ||
| rm /tmp/postgres.rpm /tmp/postgres-libs.rpm RPM-GPG-KEY-CentOS-Official && \ | ||
| # (Optional) Remove line below to keep package management utilities | ||
| rpm -e --nodeps $(rpm -qa curl '*rpm*' '*dnf*' '*libsolv*' '*hawkey*' 'yum*') && \ | ||
| rm -rf /var/cache/dnf /var/cache/yum && \ | ||
| # The contents of paths mounted as emptyDir volumes in Kubernetes are saved | ||
| # by the script `save-dir-contents` during the image build. The directory | ||
| # contents are then restored by the script `restore-all-dir-contents` | ||
| # during the container start. | ||
| chown -R 4000:4000 /etc/pki/ca-trust && save-dir-contents /etc/pki/ca-trust/source && \ | ||
| mkdir -p /var/lib/stackrox && chown -R 4000:4000 /var/lib/stackrox && \ | ||
| mkdir -p /var/log/stackrox && chown -R 4000:4000 /var/log/stackrox && \ | ||
| mkdir -p /var/cache/stackrox && chown -R 4000:4000 /var/cache/stackrox && \ | ||
| chown -R 4000:4000 /tmp | ||
| COPY --link --from=stackrox_data /stackrox-data /stackrox/static-data | ||
| COPY --link ./docs/api/v1/swagger.json /stackrox/static-data/docs/api/v1/swagger.json | ||
| COPY --link ./docs/api/v2/swagger.json /stackrox/static-data/docs/api/v2/swagger.json | ||
| COPY --link THIRD_PARTY_NOTICES /THIRD_PARTY_NOTICES/ | ||
|
|
||
| COPY --link ui /ui | ||
|
Comment on lines
+82
to
+91
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. issue (bug_risk): Using COPY --link requires BuildKit and may break non-buildx/local docker builds using this Dockerfile. In CI this is fine because we use buildx, but any remaining |
||
|
|
||
| COPY --from=stackrox_data /stackrox-data /stackrox/static-data | ||
| COPY ./docs/api/v1/swagger.json /stackrox/static-data/docs/api/v1/swagger.json | ||
| COPY ./docs/api/v2/swagger.json /stackrox/static-data/docs/api/v2/swagger.json | ||
| COPY THIRD_PARTY_NOTICES /THIRD_PARTY_NOTICES/ | ||
|
|
||
| COPY ui /ui | ||
|
|
||
| COPY bin/central /stackrox/central | ||
| COPY bin/migrator /stackrox/bin/migrator | ||
| COPY bin/compliance /stackrox/bin/compliance | ||
| COPY bin/kubernetes-sensor /stackrox/bin/kubernetes-sensor | ||
| COPY bin/sensor-upgrader /stackrox/bin/sensor-upgrader | ||
| COPY bin/admission-control /stackrox/bin/admission-control | ||
| COPY bin/config-controller /stackrox/bin/config-controller | ||
| COPY bin/roxagent /stackrox/bin/roxagent | ||
| COPY bin/roxctl* /assets/downloads/cli/ | ||
| COPY --link bin/central /stackrox/central | ||
| COPY --link bin/migrator /stackrox/bin/migrator | ||
| COPY --link bin/compliance /stackrox/bin/compliance | ||
| COPY --link bin/kubernetes-sensor /stackrox/bin/kubernetes-sensor | ||
| COPY --link bin/sensor-upgrader /stackrox/bin/sensor-upgrader | ||
| COPY --link bin/admission-control /stackrox/bin/admission-control | ||
| COPY --link bin/config-controller /stackrox/bin/config-controller | ||
| COPY --link bin/roxagent /stackrox/bin/roxagent | ||
| COPY --link bin/roxctl* /assets/downloads/cli/ | ||
|
|
||
| RUN ln -s /assets/downloads/cli/roxctl-linux-${TARGET_ARCH} /stackrox/roxctl && \ | ||
| ln -s /assets/downloads/cli/roxctl-linux-amd64 /assets/downloads/cli/roxctl-linux | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't bake
0.0.0/0000000into the published binaries.These overrides apply to the same workflow that runs on
pushtomasterand*-nightly-*, so the main/roxctl/operator artifacts published from those builds will also self-report the placeholder version. If this trade-off is only acceptable for CI/PR testing, gate it away from publishing contexts.Possible guard
Apply the same pattern in both
pre-build-cliandpre-build-go-binaries.Also applies to: 226-230
🤖 Prompt for AI Agents