Skip to content

ROX-31176: improve reprocess deployment risk#17126

Merged
janisz merged 2 commits intomasterfrom
optimize_reprocess_deployment_risk_v2
Oct 6, 2025
Merged

ROX-31176: improve reprocess deployment risk#17126
janisz merged 2 commits intomasterfrom
optimize_reprocess_deployment_risk_v2

Conversation

@janisz
Copy link
Contributor

@janisz janisz commented Oct 3, 2025 •
edited
Loading

The current GetProcessIndicatorsRiskView calls are inefficient for our use case, since we need to loop through the returned process indicators. Switching to a callback-based approach allows us to process items one by one, avoiding the need to load all objects into a slice. This reduces memory usage and lowers GC pressure.

go test -tags sql_integration -run=NO -bench=BenchmarkEvaluateBaselinesAndPersistResult  ./central/processbaseline/evaluator/. -count 6 -benchmem -benchtime 3s 
                                                                              │ master-2.txt │               new.txt               │
                                                                              │    sec/op    │    sec/op     vs base               │
EvaluateBaselinesAndPersistResult/100_processes_2_containers-8                  1.781m ± 32%   1.720m ± 55%        ~ (p=0.240 n=6)
EvaluateBaselinesAndPersistResult/500_processes_3_containers-8                  4.968m ± 28%   4.865m ± 43%        ~ (p=0.485 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers-8                 9.076m ± 46%   8.471m ± 32%        ~ (p=0.180 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers_large_args-8      11.86m ± 12%   11.64m ±  9%        ~ (p=0.485 n=6)
EvaluateBaselinesAndPersistResult/2000_processes_10_containers-8                15.86m ± 29%   15.19m ± 29%        ~ (p=0.240 n=6)
EvaluateBaselinesAndPersistResult/10000_processes_20_containers-8               81.46m ± 44%   74.28m ± 17%        ~ (p=0.132 n=6)
EvaluateBaselinesAndPersistResult/25000_processes_50_containers-8               228.4m ± 14%   182.5m ± 12%  -20.12% (p=0.026 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers-8              483.2m ± 17%   346.9m ± 33%  -28.21% (p=0.009 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers_large_args-8   633.3m ± 24%   542.8m ± 13%  -14.30% (p=0.004 n=6)
geomean                                                                         35.34m         31.66m        -10.41%

                                                                              │  master-2.txt  │                new.txt                │
                                                                              │ max_heap_bytes │ max_heap_bytes  vs base               │
EvaluateBaselinesAndPersistResult/100_processes_2_containers-8                    11.23M ±  9%     11.23M ± 17%        ~ (p=0.818 n=6)
EvaluateBaselinesAndPersistResult/500_processes_3_containers-8                    13.56M ± 18%     12.79M ± 15%        ~ (p=0.132 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers-8                   15.10M ± 13%     13.50M ± 28%        ~ (p=0.240 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers_large_args-8        20.17M ± 35%     18.61M ± 33%   -7.73% (p=0.041 n=6)
EvaluateBaselinesAndPersistResult/2000_processes_10_containers-8                  16.49M ±  7%     15.79M ±  2%   -4.28% (p=0.004 n=6)
EvaluateBaselinesAndPersistResult/10000_processes_20_containers-8                 38.74M ±  9%     32.44M ±  8%  -16.27% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/25000_processes_50_containers-8                 76.33M ±  5%     39.86M ± 18%  -47.78% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers-8               137.26M ±  3%     41.16M ± 19%  -70.02% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers_large_args-8    362.78M ±  6%     86.84M ± 35%  -76.06% (p=0.002 n=6)
geomean                                                                           36.42M           23.99M        -34.12%

                                                                              │   master-2.txt   │                 new.txt                 │
                                                                              │ max_heap_objects │ max_heap_objects  vs base               │
EvaluateBaselinesAndPersistResult/100_processes_2_containers-8                      11.23M ±  9%       11.23M ± 17%        ~ (p=0.818 n=6)
EvaluateBaselinesAndPersistResult/500_processes_3_containers-8                      13.56M ± 18%       12.79M ± 15%        ~ (p=0.132 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers-8                     15.10M ± 13%       13.50M ± 28%        ~ (p=0.240 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers_large_args-8          20.17M ± 35%       18.61M ± 33%   -7.73% (p=0.041 n=6)
EvaluateBaselinesAndPersistResult/2000_processes_10_containers-8                    16.49M ±  7%       15.79M ±  2%   -4.28% (p=0.004 n=6)
EvaluateBaselinesAndPersistResult/10000_processes_20_containers-8                   38.74M ±  9%       32.44M ±  8%  -16.27% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/25000_processes_50_containers-8                   76.33M ±  5%       39.86M ± 18%  -47.78% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers-8                 137.26M ±  3%       41.16M ± 19%  -70.02% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers_large_args-8      362.78M ±  6%       86.84M ± 35%  -76.06% (p=0.002 n=6)
geomean                                                                             36.42M             23.99M        -34.12%

                                                                              │ master-2.txt │               new.txt               │
                                                                              │     B/op     │     B/op      vs base               │
EvaluateBaselinesAndPersistResult/100_processes_2_containers-8                  678.9Ki ± 0%   660.3Ki ± 0%   -2.74% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/500_processes_3_containers-8                  2.116Mi ± 0%   2.029Mi ± 0%   -4.11% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers-8                 3.774Mi ± 0%   3.609Mi ± 0%   -4.39% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers_large_args-8      8.955Mi ± 0%   8.790Mi ± 0%   -1.85% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/2000_processes_10_containers-8                7.132Mi ± 0%   6.673Mi ± 0%   -6.43% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/10000_processes_20_containers-8               34.66Mi ± 0%   31.69Mi ± 0%   -8.55% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/25000_processes_50_containers-8               87.49Mi ± 0%   77.93Mi ± 0%  -10.93% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers-8              175.7Mi ± 1%   155.0Mi ± 0%  -11.80% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers_large_args-8   463.0Mi ± 0%   442.3Mi ± 0%   -4.47% (p=0.002 n=6)
geomean                                                                         16.35Mi        15.33Mi        -6.20%

                                                                              │ master-2.txt │              new.txt              │
                                                                              │  allocs/op   │  allocs/op   vs base              │
EvaluateBaselinesAndPersistResult/100_processes_2_containers-8                   15.59k ± 0%   15.45k ± 0%  -0.90% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/500_processes_3_containers-8                   57.99k ± 0%   57.81k ± 0%  -0.31% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers-8                  108.7k ± 0%   108.5k ± 0%  -0.18% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers_large_args-8       108.7k ± 0%   108.5k ± 0%  -0.18% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/2000_processes_10_containers-8                 208.7k ± 0%   208.4k ± 0%  -0.11% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/10000_processes_20_containers-8                1.015M ± 0%   1.015M ± 0%  -0.03% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/25000_processes_50_containers-8                2.515M ± 0%   2.515M ± 0%  -0.02% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers-8               5.015M ± 0%   5.015M ± 0%  -0.01% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers_large_args-8    5.015M ± 0%   5.015M ± 0%  -0.01% (p=0.002 n=6)
geomean                                                                          374.0k        373.2k       -0.20%

refs:

janisz added 2 commits October 3, 2025 12:53
@janisz
fix(risk): improve reprocess deployment risk
0b4536b 
Signed-off-by: Tomasz Janiszewski <tomek@redhat.com>
@janisz
wip
1857c3b 
Signed-off-by: Tomasz Janiszewski <tomek@redhat.com>
@janisz janisz requested a review from a team as a code owner October 3, 2025 11:07
@janisz janisz changed the title Optimize reprocess deployment risk v2 fix(risk): improve reprocess deployment risk Oct 3, 2025
@janisz janisz requested a review from dashrews78 October 3, 2025 11:09
dashrews78
dashrews78 approved these changes Oct 3, 2025
View reviewed changes
Copy link
Contributor

@dashrews78 dashrews78 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome. Please update the description to refer to the correct method though as this doesn't apply to SearchRawProcessIndicators.

@rhacs-bot
Copy link
Contributor

rhacs-bot commented Oct 3, 2025

Images are ready for the commit at 1857c3b.

To use with deploy scripts, first export MAIN_IMAGE_TAG=4.9.x-963-g1857c3bdf1.

@clickboo
Copy link
Contributor

clickboo commented Oct 3, 2025

Indeed awesome!

@janisz janisz requested review from mtodor and rhybrillou October 3, 2025 11:37
@codecov
Copy link

codecov bot commented Oct 3, 2025 •
edited
Loading

Codecov Report

❌ Patch coverage is 57.14286% with 21 lines in your changes missing coverage. Please review.
✅ Project coverage is 48.76%. Comparing base (f830aaf) to head (1857c3b).
⚠️ Report is 19 commits behind head on master.

Files with missing lines Patch % Lines
...entral/processbaseline/evaluator/evaluator_impl.go 47.36% 7 Missing and 3 partials ⚠️
...ntral/processindicator/datastore/datastore_impl.go 0.00% 6 Missing ⚠️
pkg/search/postgres/select.go 79.16% 4 Missing and 1 partial ⚠️
Additional details and impacted files 
@@           Coverage Diff           @@
##           master   #17126   +/-   ##
=======================================
  Coverage   48.76%   48.76%           
=======================================
  Files        2712     2712           
  Lines      202430   202443   +13     
=======================================
+ Hits        98720    98729    +9     
- Misses      95931    95935    +4     
  Partials     7779     7779
Flag Coverage Δ
go-unit-tests 48.76% <57.14%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@janisz
Copy link
Contributor Author

janisz commented Oct 6, 2025

/retest

@red-hat-konflux
Copy link
Contributor

red-hat-konflux bot commented Oct 6, 2025

Caution

There are some errors in your PipelineRun template.

PipelineRun Error
central-db-on-push CEL expression evaluation error: expression "(\n event == \"push\" && target_branch.matches(\"^(master|release-.*|refs/tags/.*)$\")\n) || (\n event == \"pull_request\" && (\n target_branch.startsWith(\"release-\") ||\n source_branch.matches(\"(konflux|renovate|appstudio|rhtap)\") ||\n (has(body.pull_request.labels) && body.pull_request.labels.exists(l, l.name == \"konflux-build\"))\n ) && body.action != \"ready_for_review\"\n)\n" failed to evaluate: no such key: pull_request
main-on-push CEL expression evaluation error: expression "(\n event == \"push\" && target_branch.matches(\"^(master|release-.*|refs/tags/.*)$\")\n) || (\n event == \"pull_request\" && (\n target_branch.startsWith(\"release-\") ||\n source_branch.matches(\"(konflux|renovate|appstudio|rhtap)\") ||\n (has(body.pull_request.labels) && body.pull_request.labels.exists(l, l.name == \"konflux-build\"))\n ) && body.action != \"ready_for_review\"\n)\n" failed to evaluate: no such key: pull_request
operator-on-push CEL expression evaluation error: expression "(\n event == \"push\" && target_branch.matches(\"^(master|release-.*|refs/tags/.*)$\")\n) || (\n event == \"pull_request\" && (\n target_branch.startsWith(\"release-\") ||\n source_branch.matches(\"(konflux|renovate|appstudio|rhtap)\") ||\n (has(body.pull_request.labels) && body.pull_request.labels.exists(l, l.name == \"konflux-build\"))\n ) && body.action != \"ready_for_review\"\n)\n" failed to evaluate: no such key: pull_request
operator-bundle-on-push CEL expression evaluation error: expression "(\n event == \"push\" && target_branch.matches(\"^(master|release-.*|refs/tags/.*)$\")\n) || (\n event == \"pull_request\" && (\n target_branch.startsWith(\"release-\") ||\n source_branch.matches(\"(konflux|renovate|appstudio|rhtap)\") ||\n (has(body.pull_request.labels) && body.pull_request.labels.exists(l, l.name == \"konflux-build\"))\n ) && body.action != \"ready_for_review\"\n)\n" failed to evaluate: no such key: pull_request
retag-collector CEL expression evaluation error: expression "(\n event == \"push\" && target_branch.matches(\"^(master|release-.*|refs/tags/.*)$\")\n) || (\n event == \"pull_request\" && (\n target_branch.startsWith(\"release-\") ||\n source_branch.matches(\"(konflux|renovate|appstudio|rhtap)\") ||\n (has(body.pull_request.labels) && body.pull_request.labels.exists(l, l.name == \"konflux-build\"))\n ) && body.action != \"ready_for_review\"\n)\n" failed to evaluate: no such key: pull_request
retag-scanner-db-slim CEL expression evaluation error: expression "(\n event == \"push\" && target_branch.matches(\"^(master|release-.*|refs/tags/.*)$\")\n) || (\n event == \"pull_request\" && (\n target_branch.startsWith(\"release-\") ||\n source_branch.matches(\"(konflux|renovate|appstudio|rhtap)\") ||\n (has(body.pull_request.labels) && body.pull_request.labels.exists(l, l.name == \"konflux-build\"))\n ) && body.action != \"ready_for_review\"\n)\n" failed to evaluate: no such key: pull_request
retag-scanner-db CEL expression evaluation error: expression "(\n event == \"push\" && target_branch.matches(\"^(master|release-.*|refs/tags/.*)$\")\n) || (\n event == \"pull_request\" && (\n target_branch.startsWith(\"release-\") ||\n source_branch.matches(\"(konflux|renovate|appstudio|rhtap)\") ||\n (has(body.pull_request.labels) && body.pull_request.labels.exists(l, l.name == \"konflux-build\"))\n ) && body.action != \"ready_for_review\"\n)\n" failed to evaluate: no such key: pull_request
retag-scanner-slim CEL expression evaluation error: expression "(\n event == \"push\" && target_branch.matches(\"^(master|release-.*|refs/tags/.*)$\")\n) || (\n event == \"pull_request\" && (\n target_branch.startsWith(\"release-\") ||\n source_branch.matches(\"(konflux|renovate|appstudio|rhtap)\") ||\n (has(body.pull_request.labels) && body.pull_request.labels.exists(l, l.name == \"konflux-build\"))\n ) && body.action != \"ready_for_review\"\n)\n" failed to evaluate: no such key: pull_request
retag-scanner CEL expression evaluation error: expression "(\n event == \"push\" && target_branch.matches(\"^(master|release-.*|refs/tags/.*)$\")\n) || (\n event == \"pull_request\" && (\n target_branch.startsWith(\"release-\") ||\n source_branch.matches(\"(konflux|renovate|appstudio|rhtap)\") ||\n (has(body.pull_request.labels) && body.pull_request.labels.exists(l, l.name == \"konflux-build\"))\n ) && body.action != \"ready_for_review\"\n)\n" failed to evaluate: no such key: pull_request
roxctl-on-push CEL expression evaluation error: expression "(\n event == \"push\" && target_branch.matches(\"^(master|release-.*|refs/tags/.*)$\")\n) || (\n event == \"pull_request\" && (\n target_branch.startsWith(\"release-\") ||\n source_branch.matches(\"(konflux|renovate|appstudio|rhtap)\") ||\n (has(body.pull_request.labels) && body.pull_request.labels.exists(l, l.name == \"konflux-build\"))\n ) && body.action != \"ready_for_review\"\n)\n" failed to evaluate: no such key: pull_request
scanner-v4-on-push CEL expression evaluation error: expression "(\n event == \"push\" && target_branch.matches(\"^(master|release-.*|refs/tags/.*)$\")\n) || (\n event == \"pull_request\" && (\n target_branch.startsWith(\"release-\") ||\n source_branch.matches(\"(konflux|renovate|appstudio|rhtap)\") ||\n (has(body.pull_request.labels) && body.pull_request.labels.exists(l, l.name == \"konflux-build\"))\n ) && body.action != \"ready_for_review\"\n)\n" failed to evaluate: no such key: pull_request
scanner-v4-db-on-push CEL expression evaluation error: expression "(\n event == \"push\" && target_branch.matches(\"^(master|release-.*|refs/tags/.*)$\")\n) || (\n event == \"pull_request\" && (\n target_branch.startsWith(\"release-\") ||\n source_branch.matches(\"(konflux|renovate|appstudio|rhtap)\") ||\n (has(body.pull_request.labels) && body.pull_request.labels.exists(l, l.name == \"konflux-build\"))\n ) && body.action != \"ready_for_review\"\n)\n" failed to evaluate: no such key: pull_request

@janisz janisz added the auto-retest PRs with this label will be automatically retested if prow checks fails label Oct 6, 2025
@janisz janisz mentioned this pull request Oct 6, 2025
Open
@janisz janisz merged commit 0a5f411 into master Oct 6, 2025
99 of 100 checks passed
@janisz janisz deleted the optimize_reprocess_deployment_risk_v2 branch October 6, 2025 11:47
@janisz janisz mentioned this pull request Oct 6, 2025
Closed
9 tasks
@janisz janisz changed the title fix(risk): improve reprocess deployment risk ROX-31176: improve reprocess deployment risk Oct 6, 2025
@janisz janisz mentioned this pull request Oct 7, 2025
Open
@dashrews78 dashrews78 mentioned this pull request Feb 3, 2026
Merged
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dashrews78 dashrews78 dashrews78 approved these changes

@mtodor mtodor Awaiting requested review from mtodor

@rhybrillou rhybrillou Awaiting requested review from rhybrillou

Assignees

No one assigned

Labels

area/central area/postgres auto-retest PRs with this label will be automatically retested if prow checks fails

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@janisz @rhacs-bot @clickboo @dashrews78