Skip to content

ROX-25572: prefetch dependencies for operator-bundle on Konflux#13209

Merged
tommartensen merged 11 commits intomasterfrom
tm/konflux-prefetch-operator-bundle
Nov 6, 2024
Merged

ROX-25572: prefetch dependencies for operator-bundle on Konflux#13209
tommartensen merged 11 commits intomasterfrom
tm/konflux-prefetch-operator-bundle

Conversation

@tommartensen
Copy link
Contributor

@tommartensen tommartensen commented Nov 4, 2024

Description

Prefetch the Python dependencies required to run patch-csv.py bundle helpers script.
This will fill the CSV template with information about related images, ...

We can switch to the Python image as builder, because we don't use Go anymore to construct, patch and test the CSV.

User-facing documentation

Testing and quality

  • the change is production ready: the change is GA or otherwise the functionality is gated by a feature flag
  • CI results are inspected

Automated testing

  • existing tests suffice

How I validated my change

  • Create a intermediate Docker image that has all RPM dependencies prefetched, like
FROM registry.access.redhat.com/ubi9:latest AS ubi-repo-donor

FROM brew.registry.redhat.io/rh-osbs/openshift-golang-builder:rhel_9_1.22 AS builder-runner

# For some reason, openshift-golang-builder 9 comes without any RPM repos in /etc/yum.repos.d/
# We, however, need to install some packages and so we need to configure RPM repos. The ones for UBI are sufficient.
COPY --from=ubi-repo-donor /etc/yum.repos.d/ubi.repo /etc/yum.repos.d/ubi.repo
RUN dnf -y upgrade --nobest && dnf -y install --nodocs --noplugins jq python3-pip

COPY ./operator/bundle_helpers/requirements.txt /tmp/requirements.txt
RUN pip3 install --no-cache-dir --only-binary=:all: -r /tmp/requirements.txt && rm /tmp/requirements.txt
  • Use this as first stage in the konflux.bundle.Dockerfile
  • Set hermetic: true in bundle build pipeline
  • (remove waiting for other images, just to speed things up)

Validation build: https://console.redhat.com/application-pipeline/workspaces/rh-acs/applications/acs/pipelineruns/operator-bundle-on-push-hcgr2 (Commit: f6c5ac0)

Then reverted those changes and ran a full validation pipeline: https://console.redhat.com/application-pipeline/workspaces/rh-acs/applications/acs/pipelineruns/operator-bundle-on-push-cxsf8

[1/2] STEP 4/44: RUN . /cachi2/cachi2.env &&     pip3 install --no-cache-dir --only-binary=:all: -r /tmp/requirements.txt
Looking in links: /cachi2/output/deps/pip
Ignoring atomicwrites: markers 'python_version >= "3.6" and python_version < "4.0" and sys_platform == "win32"' don't match your environment
Ignoring colorama: markers 'python_version >= "3.6" and python_version < "4.0" and sys_platform == "win32"' don't match your environment
Ignoring importlib-metadata: markers 'python_version >= "3.6" and python_version < "3.8"' don't match your environment
Ignoring typing-extensions: markers 'python_version >= "3.6" and python_version < "3.8"' don't match your environment
Ignoring zipp: markers 'python_version >= "3.6" and python_version < "3.8"' don't match your environment
Processing /cachi2/output/deps/pip/attrs-22.2.0-py3-none-any.whl (from -r /tmp/requirements.txt (line 3))
Processing /cachi2/output/deps/pip/iniconfig-1.1.1-py2.py3-none-any.whl (from -r /tmp/requirements.txt (line 12))
Processing /cachi2/output/deps/pip/packaging-21.3-py3-none-any.whl (from -r /tmp/requirements.txt (line 15))
Processing /cachi2/output/deps/pip/pluggy-1.0.0-py2.py3-none-any.whl (from -r /tmp/requirements.txt (line 18))
Processing /cachi2/output/deps/pip/py-1.11.0-py2.py3-none-any.whl (from -r /tmp/requirements.txt (line 21))
Processing /cachi2/output/deps/pip/pyparsing-3.0.7-py3-none-any.whl (from -r /tmp/requirements.txt (line 24))
Processing /cachi2/output/deps/pip/pytest-7.0.1-py3-none-any.whl (from -r /tmp/requirements.txt (line 27))
Processing /cachi2/output/deps/pip/PyYAML-6.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (from -r /tmp/requirements.txt (line 30))
Processing /cachi2/output/deps/pip/tomli-1.2.3-py3-none-any.whl (from -r /tmp/requirements.txt (line 71))
Installing collected packages: iniconfig, tomli, pyyaml, pyparsing, py, pluggy, attrs, packaging, pytest
Successfully installed attrs-22.2.0 iniconfig-1.1.1 packaging-21.3 pluggy-1.0.0 py-1.11.0 pyparsing-3.0.7 pytest-7.0.1 pyyaml-6.0 tomli-1.2.3

We see that prefetch-dependencies fetched our required dependencies and artifacts were used during the build.

@openshift-ci
Copy link

openshift-ci bot commented Nov 4, 2024

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@rhacs-bot
Copy link
Contributor

rhacs-bot commented Nov 4, 2024

Images are ready for the commit at 5bc5967.

To use with deploy scripts, first export MAIN_IMAGE_TAG=4.7.x-42-g5bc596758d.

@codecov
Copy link

codecov bot commented Nov 4, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 48.50%. Comparing base (0b65ee2) to head (5bc5967).
Report is 11 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #13209      +/-   ##
==========================================
- Coverage   48.50%   48.50%   -0.01%     
==========================================
  Files        2468     2468              
  Lines      177995   177999       +4     
==========================================
- Hits        86334    86332       -2     
- Misses      84728    84733       +5     
- Partials     6933     6934       +1     
Flag Coverage Δ
go-unit-tests 48.50% <ø> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@tommartensen tommartensen changed the title another attempt ROX-25572: prefetch dependencies for operator-bundle on Konflux (xth attempt) Nov 5, 2024
@tommartensen tommartensen changed the title ROX-25572: prefetch dependencies for operator-bundle on Konflux (xth attempt) ROX-25572: prefetch dependencies for operator-bundle on Konflux Nov 5, 2024
@tommartensen
Copy link
Contributor Author

/retest roxctl-on-push

@openshift-ci
Copy link

openshift-ci bot commented Nov 5, 2024

@tommartensen: The /retest command does not accept any targets.
The following commands are available to trigger required jobs:

  • /test gke-nongroovy-e2e-tests
  • /test shell-unit-tests

The following commands are available to trigger optional jobs:

  • /test aks-qa-e2e-tests
  • /test aro-qa-e2e-tests
  • /test eks-qa-e2e-tests
  • /test gke-nongroovy-compatibility-tests
  • /test gke-operator-e2e-tests
  • /test gke-perf-scale-tests
  • /test gke-qa-e2e-tests
  • /test gke-race-condition-qa-e2e-tests
  • /test gke-scale-tests
  • /test gke-scanner-v4-tests
  • /test gke-sensor-integration-tests
  • /test gke-ui-e2e-tests
  • /test gke-upgrade-tests
  • /test gke-version-compatibility-tests
  • /test ibmcloudz-4-14-qa-e2e-tests
  • /test ibmcloudz-4-15-qa-e2e-tests
  • /test ibmcloudz-4-16-qa-e2e-tests
  • /test ocp-4-12-compliance-e2e-tests
  • /test ocp-4-12-ebpf-qa-e2e-tests
  • /test ocp-4-12-nongroovy-e2e-tests
  • /test ocp-4-12-operator-e2e-tests
  • /test ocp-4-12-qa-e2e-tests
  • /test ocp-4-12-scanner-v4-tests
  • /test ocp-4-12-sensor-integration-tests
  • /test ocp-4-12-ui-e2e-tests
  • /test ocp-4-17-compliance-e2e-tests
  • /test ocp-4-17-crun-qa-e2e-tests
  • /test ocp-4-17-ebpf-qa-e2e-tests
  • /test ocp-4-17-fips-qa-e2e-tests
  • /test ocp-4-17-nongroovy-e2e-tests
  • /test ocp-4-17-operator-e2e-tests
  • /test ocp-4-17-qa-e2e-tests
  • /test ocp-4-17-scanner-v4-tests
  • /test ocp-4-17-sensor-integration-tests
  • /test ocp-4-17-ui-e2e-tests
  • /test ocp-dev-preview-compliance-e2e-tests
  • /test ocp-dev-preview-ebpf-qa-e2e-tests
  • /test ocp-dev-preview-fips-qa-e2e-tests
  • /test ocp-dev-preview-nongroovy-e2e-tests
  • /test ocp-dev-preview-operator-e2e-tests
  • /test ocp-dev-preview-qa-e2e-tests
  • /test ocp-dev-preview-scanner-v4-tests
  • /test ocp-dev-preview-sensor-integration-tests
  • /test ocp-dev-preview-ui-e2e-tests
  • /test ocp-next-candidate-compliance-e2e-tests
  • /test ocp-next-candidate-ebpf-qa-e2e-tests
  • /test ocp-next-candidate-fips-qa-e2e-tests
  • /test ocp-next-candidate-nongroovy-e2e-tests
  • /test ocp-next-candidate-operator-e2e-tests
  • /test ocp-next-candidate-qa-e2e-tests
  • /test ocp-next-candidate-scanner-v4-tests
  • /test ocp-next-candidate-sensor-integration-tests
  • /test ocp-next-candidate-ui-e2e-tests
  • /test ocp-stable-scanner-v4-compliance-e2e-tests
  • /test ocp-stable-scanner-v4-ebpf-qa-e2e-tests
  • /test ocp-stable-scanner-v4-nongroovy-e2e-tests
  • /test ocp-stable-scanner-v4-operator-e2e-tests
  • /test ocp-stable-scanner-v4-perf-scale-tests
  • /test ocp-stable-scanner-v4-qa-e2e-tests
  • /test ocp-stable-scanner-v4-scanner-v4-tests
  • /test ocp-stable-scanner-v4-sensor-integration-tests
  • /test ocp-stable-scanner-v4-ui-e2e-tests
  • /test osd-aws-qa-e2e-tests
  • /test osd-gcp-qa-e2e-tests
  • /test powervs-4-14-qa-corebpf-e2e-tests
  • /test powervs-4-15-qa-corebpf-e2e-tests
  • /test powervs-4-16-qa-corebpf-e2e-tests
  • /test powervs-4-17-qa-corebpf-e2e-tests
  • /test rosa-hcp-qa-e2e-tests
  • /test rosa-qa-e2e-tests
  • /test ui-component-tests

Use /test all to run the following jobs that were automatically triggered:

  • pull-ci-stackrox-stackrox-master-gke-nongroovy-e2e-tests
  • pull-ci-stackrox-stackrox-master-gke-operator-e2e-tests
  • pull-ci-stackrox-stackrox-master-gke-qa-e2e-tests
  • pull-ci-stackrox-stackrox-master-gke-sensor-integration-tests
  • pull-ci-stackrox-stackrox-master-gke-upgrade-tests
  • pull-ci-stackrox-stackrox-master-ocp-4-12-nongroovy-e2e-tests
  • pull-ci-stackrox-stackrox-master-ocp-4-12-operator-e2e-tests
  • pull-ci-stackrox-stackrox-master-ocp-4-12-qa-e2e-tests
  • pull-ci-stackrox-stackrox-master-ocp-4-17-nongroovy-e2e-tests
  • pull-ci-stackrox-stackrox-master-ocp-4-17-operator-e2e-tests
  • pull-ci-stackrox-stackrox-master-ocp-4-17-qa-e2e-tests
Details

In response to this:

/retest roxctl-on-push

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@tommartensen
Copy link
Contributor Author

/retest roxctl-on-push

@openshift-ci
Copy link

openshift-ci bot commented Nov 5, 2024

@tommartensen: The /retest command does not accept any targets.
The following commands are available to trigger required jobs:

  • /test gke-nongroovy-e2e-tests
  • /test shell-unit-tests

The following commands are available to trigger optional jobs:

  • /test aks-qa-e2e-tests
  • /test aro-qa-e2e-tests
  • /test eks-qa-e2e-tests
  • /test gke-nongroovy-compatibility-tests
  • /test gke-operator-e2e-tests
  • /test gke-perf-scale-tests
  • /test gke-qa-e2e-tests
  • /test gke-race-condition-qa-e2e-tests
  • /test gke-scale-tests
  • /test gke-scanner-v4-tests
  • /test gke-sensor-integration-tests
  • /test gke-ui-e2e-tests
  • /test gke-upgrade-tests
  • /test gke-version-compatibility-tests
  • /test ibmcloudz-4-14-qa-e2e-tests
  • /test ibmcloudz-4-15-qa-e2e-tests
  • /test ibmcloudz-4-16-qa-e2e-tests
  • /test ocp-4-12-compliance-e2e-tests
  • /test ocp-4-12-ebpf-qa-e2e-tests
  • /test ocp-4-12-nongroovy-e2e-tests
  • /test ocp-4-12-operator-e2e-tests
  • /test ocp-4-12-qa-e2e-tests
  • /test ocp-4-12-scanner-v4-tests
  • /test ocp-4-12-sensor-integration-tests
  • /test ocp-4-12-ui-e2e-tests
  • /test ocp-4-17-compliance-e2e-tests
  • /test ocp-4-17-crun-qa-e2e-tests
  • /test ocp-4-17-ebpf-qa-e2e-tests
  • /test ocp-4-17-fips-qa-e2e-tests
  • /test ocp-4-17-nongroovy-e2e-tests
  • /test ocp-4-17-operator-e2e-tests
  • /test ocp-4-17-qa-e2e-tests
  • /test ocp-4-17-scanner-v4-tests
  • /test ocp-4-17-sensor-integration-tests
  • /test ocp-4-17-ui-e2e-tests
  • /test ocp-dev-preview-compliance-e2e-tests
  • /test ocp-dev-preview-ebpf-qa-e2e-tests
  • /test ocp-dev-preview-fips-qa-e2e-tests
  • /test ocp-dev-preview-nongroovy-e2e-tests
  • /test ocp-dev-preview-operator-e2e-tests
  • /test ocp-dev-preview-qa-e2e-tests
  • /test ocp-dev-preview-scanner-v4-tests
  • /test ocp-dev-preview-sensor-integration-tests
  • /test ocp-dev-preview-ui-e2e-tests
  • /test ocp-next-candidate-compliance-e2e-tests
  • /test ocp-next-candidate-ebpf-qa-e2e-tests
  • /test ocp-next-candidate-fips-qa-e2e-tests
  • /test ocp-next-candidate-nongroovy-e2e-tests
  • /test ocp-next-candidate-operator-e2e-tests
  • /test ocp-next-candidate-qa-e2e-tests
  • /test ocp-next-candidate-scanner-v4-tests
  • /test ocp-next-candidate-sensor-integration-tests
  • /test ocp-next-candidate-ui-e2e-tests
  • /test ocp-stable-scanner-v4-compliance-e2e-tests
  • /test ocp-stable-scanner-v4-ebpf-qa-e2e-tests
  • /test ocp-stable-scanner-v4-nongroovy-e2e-tests
  • /test ocp-stable-scanner-v4-operator-e2e-tests
  • /test ocp-stable-scanner-v4-perf-scale-tests
  • /test ocp-stable-scanner-v4-qa-e2e-tests
  • /test ocp-stable-scanner-v4-scanner-v4-tests
  • /test ocp-stable-scanner-v4-sensor-integration-tests
  • /test ocp-stable-scanner-v4-ui-e2e-tests
  • /test osd-aws-qa-e2e-tests
  • /test osd-gcp-qa-e2e-tests
  • /test powervs-4-14-qa-corebpf-e2e-tests
  • /test powervs-4-15-qa-corebpf-e2e-tests
  • /test powervs-4-16-qa-corebpf-e2e-tests
  • /test powervs-4-17-qa-corebpf-e2e-tests
  • /test rosa-hcp-qa-e2e-tests
  • /test rosa-qa-e2e-tests
  • /test ui-component-tests

Use /test all to run the following jobs that were automatically triggered:

  • pull-ci-stackrox-stackrox-master-gke-nongroovy-e2e-tests
  • pull-ci-stackrox-stackrox-master-gke-operator-e2e-tests
  • pull-ci-stackrox-stackrox-master-gke-qa-e2e-tests
  • pull-ci-stackrox-stackrox-master-gke-sensor-integration-tests
  • pull-ci-stackrox-stackrox-master-gke-upgrade-tests
  • pull-ci-stackrox-stackrox-master-ocp-4-12-nongroovy-e2e-tests
  • pull-ci-stackrox-stackrox-master-ocp-4-12-operator-e2e-tests
  • pull-ci-stackrox-stackrox-master-ocp-4-12-qa-e2e-tests
  • pull-ci-stackrox-stackrox-master-ocp-4-17-nongroovy-e2e-tests
  • pull-ci-stackrox-stackrox-master-ocp-4-17-operator-e2e-tests
  • pull-ci-stackrox-stackrox-master-ocp-4-17-qa-e2e-tests
Details

In response to this:

/retest roxctl-on-push

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@tommartensen tommartensen marked this pull request as ready for review November 5, 2024 12:26
@tommartensen tommartensen requested review from a team as code owners November 5, 2024 12:26
@tommartensen tommartensen requested review from vladbologa and removed request for a team November 5, 2024 12:26
@tommartensen tommartensen requested a review from a team November 5, 2024 12:26
@msugakov msugakov requested a review from porridge November 5, 2024 13:10
Copy link
Contributor

@msugakov msugakov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Please enable hermetic and I'll gladly approve

@tommartensen tommartensen requested a review from msugakov November 5, 2024 13:37
@tommartensen
Copy link
Contributor Author

/test gke-sensor-integration-tests

Copy link
Contributor

@msugakov msugakov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address the new comments before merging.

@openshift-ci
Copy link

openshift-ci bot commented Nov 5, 2024

@tommartensen: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/gke-qa-e2e-tests ab8529e link false /test gke-qa-e2e-tests

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Co-authored-by: Misha Sugakov <537715+msugakov@users.noreply.github.com>
@tommartensen tommartensen enabled auto-merge (squash) November 6, 2024 07:50
@tommartensen tommartensen added the backport-for-4.6-konflux-release https://redhat-internal.slack.com/archives/C05TS9N0S7L/p1730134914487439 label Nov 6, 2024
@tommartensen tommartensen merged commit 86785c6 into master Nov 6, 2024
@tommartensen tommartensen deleted the tm/konflux-prefetch-operator-bundle branch November 6, 2024 09:35
aaa5kameric pushed a commit that referenced this pull request Nov 14, 2024
Co-authored-by: Misha Sugakov <537715+msugakov@users.noreply.github.com>
ajheflin pushed a commit that referenced this pull request Jun 24, 2025
Co-authored-by: Misha Sugakov <537715+msugakov@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/operator backport-for-4.6-konflux-release https://redhat-internal.slack.com/archives/C05TS9N0S7L/p1730134914487439

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants