ROX-30961: simplify to single thread by dashrews78 · Pull Request #17049 · stackrox/stackrox

dashrews78 · 2025-09-29T14:37:34Z

Description

Competitor to #16935 which does the migration sequentially instead of concurrently. On the other one encountering some race conditions in pgx. Need to go through all process indicators which could be large. Grabbing and batching them by cluster seemed like a reasonable choice. This is single threaded, the real thought behind getting them by cluster was to run some of the updates concurrently.

Ran a cluster with 11M indicators across 7 clusters. The migration took 15 minutes.

User-facing documentation

CHANGELOG.md is updated OR update is not needed
documentation PR is created and is linked above OR is not needed

Testing and quality

the change is production ready: the change is GA, or otherwise the functionality is gated by a feature flag
CI results are inspected

Automated testing

How I validated my change

Unit test, upgrade test, and local test with around 7 clusters and 11M process indicators.

dashrews78 · 2025-09-29T14:37:35Z

This change is part of the following stack:

ROX-30961: simplify to single thread #17049 ◀

_{Change managed by git-spice.}

rhacs-bot · 2025-09-29T15:02:22Z

Images are ready for the commit at b2f3c17.

To use with deploy scripts, first export MAIN_IMAGE_TAG=4.9.x-918-gb2f3c1718a.

codecov · 2025-09-29T15:19:00Z

Codecov Report

❌ Patch coverage is 65.78947% with 52 lines in your changes missing coverage. Please review.
✅ Project coverage is 48.81%. Comparing base (2b2a970) to head (b2f3c17).
⚠️ Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
...dicators/test/schema/convert_process_indicators.go	0.00%	28 Missing ⚠️
...ainer_start_column_to_indicators/migration_impl.go	72.72%	8 Missing and 4 partials ⚠️
...to_indicators/schema/convert_process_indicators.go	79.31%	4 Missing and 2 partials ⚠️
...lumn_to_indicators/test/schema/convert_clusters.go	71.42%	4 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #17049      +/-   ##
==========================================
+ Coverage   48.80%   48.81%   +0.01%     
==========================================
  Files        2706     2716      +10     
  Lines      202090   202346     +256     
==========================================
+ Hits        98631    98778     +147     
- Misses      95695    95789      +94     
- Partials     7764     7779      +15

Flag	Coverage Δ
go-unit-tests	`48.81% <65.78%> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

dashrews78 · 2025-09-29T18:05:45Z

/retest

red-hat-konflux · 2025-09-29T18:06:03Z

Caution

There are some errors in your PipelineRun template.

PipelineRun	Error
central-db-on-push	`CEL expression evaluation error: expression "(\n event == \"push\" && target_branch.matches(\"^(master\|release-.\|refs/tags/.)$\")\n) \|\| (\n event == \"pull_request\" && (\n target_branch.startsWith(\"release-\") \|\|\n source_branch.matches(\"(konflux\|renovate\|appstudio\|rhtap)\") \|\|\n (has(body.pull_request.labels) && body.pull_request.labels.exists(l, l.name == \"konflux-build\"))\n ) && body.action != \"ready_for_review\"\n)\n" failed to evaluate: no such key: pull_request`
main-on-push	`CEL expression evaluation error: expression "(\n event == \"push\" && target_branch.matches(\"^(master\|release-.\|refs/tags/.)$\")\n) \|\| (\n event == \"pull_request\" && (\n target_branch.startsWith(\"release-\") \|\|\n source_branch.matches(\"(konflux\|renovate\|appstudio\|rhtap)\") \|\|\n (has(body.pull_request.labels) && body.pull_request.labels.exists(l, l.name == \"konflux-build\"))\n ) && body.action != \"ready_for_review\"\n)\n" failed to evaluate: no such key: pull_request`
operator-on-push	`CEL expression evaluation error: expression "(\n event == \"push\" && target_branch.matches(\"^(master\|release-.\|refs/tags/.)$\")\n) \|\| (\n event == \"pull_request\" && (\n target_branch.startsWith(\"release-\") \|\|\n source_branch.matches(\"(konflux\|renovate\|appstudio\|rhtap)\") \|\|\n (has(body.pull_request.labels) && body.pull_request.labels.exists(l, l.name == \"konflux-build\"))\n ) && body.action != \"ready_for_review\"\n)\n" failed to evaluate: no such key: pull_request`
operator-bundle-on-push	`CEL expression evaluation error: expression "(\n event == \"push\" && target_branch.matches(\"^(master\|release-.\|refs/tags/.)$\")\n) \|\| (\n event == \"pull_request\" && (\n target_branch.startsWith(\"release-\") \|\|\n source_branch.matches(\"(konflux\|renovate\|appstudio\|rhtap)\") \|\|\n (has(body.pull_request.labels) && body.pull_request.labels.exists(l, l.name == \"konflux-build\"))\n ) && body.action != \"ready_for_review\"\n)\n" failed to evaluate: no such key: pull_request`
retag-collector	`CEL expression evaluation error: expression "(\n event == \"push\" && target_branch.matches(\"^(master\|release-.\|refs/tags/.)$\")\n) \|\| (\n event == \"pull_request\" && (\n target_branch.startsWith(\"release-\") \|\|\n source_branch.matches(\"(konflux\|renovate\|appstudio\|rhtap)\") \|\|\n (has(body.pull_request.labels) && body.pull_request.labels.exists(l, l.name == \"konflux-build\"))\n ) && body.action != \"ready_for_review\"\n)\n" failed to evaluate: no such key: pull_request`
retag-scanner-db-slim	`CEL expression evaluation error: expression "(\n event == \"push\" && target_branch.matches(\"^(master\|release-.\|refs/tags/.)$\")\n) \|\| (\n event == \"pull_request\" && (\n target_branch.startsWith(\"release-\") \|\|\n source_branch.matches(\"(konflux\|renovate\|appstudio\|rhtap)\") \|\|\n (has(body.pull_request.labels) && body.pull_request.labels.exists(l, l.name == \"konflux-build\"))\n ) && body.action != \"ready_for_review\"\n)\n" failed to evaluate: no such key: pull_request`
retag-scanner-db	`CEL expression evaluation error: expression "(\n event == \"push\" && target_branch.matches(\"^(master\|release-.\|refs/tags/.)$\")\n) \|\| (\n event == \"pull_request\" && (\n target_branch.startsWith(\"release-\") \|\|\n source_branch.matches(\"(konflux\|renovate\|appstudio\|rhtap)\") \|\|\n (has(body.pull_request.labels) && body.pull_request.labels.exists(l, l.name == \"konflux-build\"))\n ) && body.action != \"ready_for_review\"\n)\n" failed to evaluate: no such key: pull_request`
retag-scanner-slim	`CEL expression evaluation error: expression "(\n event == \"push\" && target_branch.matches(\"^(master\|release-.\|refs/tags/.)$\")\n) \|\| (\n event == \"pull_request\" && (\n target_branch.startsWith(\"release-\") \|\|\n source_branch.matches(\"(konflux\|renovate\|appstudio\|rhtap)\") \|\|\n (has(body.pull_request.labels) && body.pull_request.labels.exists(l, l.name == \"konflux-build\"))\n ) && body.action != \"ready_for_review\"\n)\n" failed to evaluate: no such key: pull_request`
retag-scanner	`CEL expression evaluation error: expression "(\n event == \"push\" && target_branch.matches(\"^(master\|release-.\|refs/tags/.)$\")\n) \|\| (\n event == \"pull_request\" && (\n target_branch.startsWith(\"release-\") \|\|\n source_branch.matches(\"(konflux\|renovate\|appstudio\|rhtap)\") \|\|\n (has(body.pull_request.labels) && body.pull_request.labels.exists(l, l.name == \"konflux-build\"))\n ) && body.action != \"ready_for_review\"\n)\n" failed to evaluate: no such key: pull_request`
roxctl-on-push	`CEL expression evaluation error: expression "(\n event == \"push\" && target_branch.matches(\"^(master\|release-.\|refs/tags/.)$\")\n) \|\| (\n event == \"pull_request\" && (\n target_branch.startsWith(\"release-\") \|\|\n source_branch.matches(\"(konflux\|renovate\|appstudio\|rhtap)\") \|\|\n (has(body.pull_request.labels) && body.pull_request.labels.exists(l, l.name == \"konflux-build\"))\n ) && body.action != \"ready_for_review\"\n)\n" failed to evaluate: no such key: pull_request`
scanner-v4-on-push	`CEL expression evaluation error: expression "(\n event == \"push\" && target_branch.matches(\"^(master\|release-.\|refs/tags/.)$\")\n) \|\| (\n event == \"pull_request\" && (\n target_branch.startsWith(\"release-\") \|\|\n source_branch.matches(\"(konflux\|renovate\|appstudio\|rhtap)\") \|\|\n (has(body.pull_request.labels) && body.pull_request.labels.exists(l, l.name == \"konflux-build\"))\n ) && body.action != \"ready_for_review\"\n)\n" failed to evaluate: no such key: pull_request`
scanner-v4-db-on-push	`CEL expression evaluation error: expression "(\n event == \"push\" && target_branch.matches(\"^(master\|release-.\|refs/tags/.)$\")\n) \|\| (\n event == \"pull_request\" && (\n target_branch.startsWith(\"release-\") \|\|\n source_branch.matches(\"(konflux\|renovate\|appstudio\|rhtap)\") \|\|\n (has(body.pull_request.labels) && body.pull_request.labels.exists(l, l.name == \"konflux-build\"))\n ) && body.action != \"ready_for_review\"\n)\n" failed to evaluate: no such key: pull_request`

openshift-ci · 2025-09-29T20:18:38Z

@dashrews78: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/gke-operator-e2e-tests	`b2f3c17`	link	false	`/test gke-operator-e2e-tests`
ci/prow/gke-qa-e2e-tests	`b2f3c17`	link	false	`/test gke-qa-e2e-tests`
ci/prow/ocp-4-19-operator-e2e-tests	`b2f3c17`	link	false	`/test ocp-4-19-operator-e2e-tests`
ci/prow/ocp-4-12-operator-e2e-tests	`b2f3c17`	link	false	`/test ocp-4-12-operator-e2e-tests`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

simplify to single thread

7aa3776

dashrews78 requested a review from a team as a code owner September 29, 2025 14:37

forgot to commit

b2f3c17

dashrews78 mentioned this pull request Sep 29, 2025

ROX-30963: Only query indicator fields needed for evalution #16965

Merged

9 tasks

charmik-redhat approved these changes Sep 29, 2025

View reviewed changes

dashrews78 merged commit 939f0fc into master Sep 30, 2025
92 of 96 checks passed

dashrews78 deleted the dashrews/container-start-migration-30961-sequential branch September 30, 2025 09:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ROX-30961: simplify to single thread#17049

ROX-30961: simplify to single thread#17049
dashrews78 merged 2 commits intomasterfrom
dashrews/container-start-migration-30961-sequential

dashrews78 commented Sep 29, 2025 •

edited

Loading

Uh oh!

dashrews78 commented Sep 29, 2025 •

edited

Loading

Uh oh!

rhacs-bot commented Sep 29, 2025

Uh oh!

codecov bot commented Sep 29, 2025 •

edited

Loading

Uh oh!

dashrews78 commented Sep 29, 2025

Uh oh!

red-hat-konflux bot commented Sep 29, 2025

Uh oh!

openshift-ci bot commented Sep 29, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

dashrews78 commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

User-facing documentation

Testing and quality

Automated testing

How I validated my change

Uh oh!

dashrews78 commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rhacs-bot commented Sep 29, 2025

Uh oh!

codecov bot commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

dashrews78 commented Sep 29, 2025

Uh oh!

red-hat-konflux bot commented Sep 29, 2025

Uh oh!

openshift-ci bot commented Sep 29, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dashrews78 commented Sep 29, 2025 •

edited

Loading

dashrews78 commented Sep 29, 2025 •

edited

Loading

codecov bot commented Sep 29, 2025 •

edited

Loading