test(bench): mesure max heap bytes and objects by janisz · Pull Request #17078 · stackrox/stackrox

janisz · 2025-09-30T16:00:35Z

Description

This PR adds a custom metric to measure max heap bytes and objects. This is important as allocations could be misleading. If we use the same amount of memory there will be no or minimal difference on allocations but could be a significant difference on peak of used memory.

Below is an example of the results processed by benchstat

                                                                              │   master.txt   │                new.txt                │
                                                                              │ max_heap_bytes │ max_heap_bytes  vs base               │
EvaluateBaselinesAndPersistResult/100_processes_2_containers-8                     10.76M ± 1%     10.79M ±  0%        ~ (p=0.180 n=6)
EvaluateBaselinesAndPersistResult/500_processes_3_containers-8                     14.75M ± 0%     14.72M ±  0%   -0.18% (p=0.015 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers-8                    19.54M ± 1%     19.26M ±  1%   -1.42% (p=0.026 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers_large_args-8         25.30M ± 0%     19.75M ±  3%  -21.91% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/2000_processes_10_containers-8                   29.68M ± 0%     22.67M ± 14%  -23.62% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/10000_processes_20_containers-8                 109.78M ± 1%     41.24M ± 56%  -62.43% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/25000_processes_50_containers-8                 260.41M ± 1%     50.99M ± 44%  -80.42% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers-8                509.47M ± 1%     80.57M ± 44%  -84.19% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers_large_args-8      832.9M ± 0%     100.1M ± 69%  -87.98% (p=0.002 n=6)
geomean                                                                            67.28M          30.56M        -54.58%

                                                                              │    master.txt    │                 new.txt                 │
                                                                              │ max_heap_objects │ max_heap_objects  vs base               │
EvaluateBaselinesAndPersistResult/100_processes_2_containers-8                       68.75k ± 1%       68.91k ±  0%        ~ (p=0.065 n=6)
EvaluateBaselinesAndPersistResult/500_processes_3_containers-8                       96.87k ± 0%       96.99k ±  0%        ~ (p=0.240 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers-8                      130.3k ± 1%       128.5k ±  1%        ~ (p=0.065 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers_large_args-8           130.2k ± 0%       106.1k ±  1%  -18.53% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/2000_processes_10_containers-8                     200.1k ± 0%       153.1k ± 15%  -23.46% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/10000_processes_20_containers-8                    758.0k ± 0%       285.9k ± 50%  -62.29% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/25000_processes_50_containers-8                   1807.4k ± 0%       355.3k ± 31%  -80.34% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers-8                  3557.0k ± 0%       546.7k ± 42%  -84.63% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers_large_args-8       3557.1k ± 0%       436.0k ± 54%  -87.74% (p=0.002 n=6)
geomean                                                                              418.3k            191.0k        -54.34%

And here is an example line of the result

BenchmarkEvaluateBaselinesAndPersistResult/100_processes_2_containers-8         	     146	  22047236 ns/op	  10697832 max_heap_bytes	     68389 max_heap_objects	 1198255 B/op	   10385 allocs/op

User-facing documentation

CHANGELOG.md is updated OR update is not needed
documentation PR is created and is linked above OR is not needed

Testing and quality

the change is production ready: the change is GA, or otherwise the functionality is gated by a feature flag
CI results are inspected

Automated testing

How I validated my change

go test -tags sql_integration -run=NO -bench=BenchmarkEvaluateBaselinesAndPersistResult  ./central/processbaseline/evaluator/. -count 6 -benchmem -benchtime 3s

codecov · 2025-09-30T16:33:47Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 48.76%. Comparing base (f830aaf) to head (23120b1).

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #17078   +/-   ##
=======================================
  Coverage   48.76%   48.76%           
=======================================
  Files        2712     2712           
  Lines      202430   202430           
=======================================
+ Hits        98720    98725    +5     
+ Misses      95931    95929    -2     
+ Partials     7779     7776    -3

Flag	Coverage Δ
go-unit-tests	`48.76% <ø> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Signed-off-by: Tomasz Janiszewski <tomek@redhat.com>

rhacs-bot · 2025-10-03T09:00:13Z

Images are ready for the commit at 23120b1.

To use with deploy scripts, first export MAIN_IMAGE_TAG=4.9.x-963-g23120b1331.

janisz · 2025-10-06T09:46:53Z

/retest

red-hat-konflux · 2025-10-06T09:47:00Z

Caution

There are some errors in your PipelineRun template.

PipelineRun	Error
central-db-on-push	`CEL expression evaluation error: expression "(\n event == \"push\" && target_branch.matches(\"^(master\|release-.\|refs/tags/.)$\")\n) \|\| (\n event == \"pull_request\" && (\n target_branch.startsWith(\"release-\") \|\|\n source_branch.matches(\"(konflux\|renovate\|appstudio\|rhtap)\") \|\|\n (has(body.pull_request.labels) && body.pull_request.labels.exists(l, l.name == \"konflux-build\"))\n ) && body.action != \"ready_for_review\"\n)\n" failed to evaluate: no such key: pull_request`
main-on-push	`CEL expression evaluation error: expression "(\n event == \"push\" && target_branch.matches(\"^(master\|release-.\|refs/tags/.)$\")\n) \|\| (\n event == \"pull_request\" && (\n target_branch.startsWith(\"release-\") \|\|\n source_branch.matches(\"(konflux\|renovate\|appstudio\|rhtap)\") \|\|\n (has(body.pull_request.labels) && body.pull_request.labels.exists(l, l.name == \"konflux-build\"))\n ) && body.action != \"ready_for_review\"\n)\n" failed to evaluate: no such key: pull_request`
operator-on-push	`CEL expression evaluation error: expression "(\n event == \"push\" && target_branch.matches(\"^(master\|release-.\|refs/tags/.)$\")\n) \|\| (\n event == \"pull_request\" && (\n target_branch.startsWith(\"release-\") \|\|\n source_branch.matches(\"(konflux\|renovate\|appstudio\|rhtap)\") \|\|\n (has(body.pull_request.labels) && body.pull_request.labels.exists(l, l.name == \"konflux-build\"))\n ) && body.action != \"ready_for_review\"\n)\n" failed to evaluate: no such key: pull_request`
operator-bundle-on-push	`CEL expression evaluation error: expression "(\n event == \"push\" && target_branch.matches(\"^(master\|release-.\|refs/tags/.)$\")\n) \|\| (\n event == \"pull_request\" && (\n target_branch.startsWith(\"release-\") \|\|\n source_branch.matches(\"(konflux\|renovate\|appstudio\|rhtap)\") \|\|\n (has(body.pull_request.labels) && body.pull_request.labels.exists(l, l.name == \"konflux-build\"))\n ) && body.action != \"ready_for_review\"\n)\n" failed to evaluate: no such key: pull_request`
retag-collector	`CEL expression evaluation error: expression "(\n event == \"push\" && target_branch.matches(\"^(master\|release-.\|refs/tags/.)$\")\n) \|\| (\n event == \"pull_request\" && (\n target_branch.startsWith(\"release-\") \|\|\n source_branch.matches(\"(konflux\|renovate\|appstudio\|rhtap)\") \|\|\n (has(body.pull_request.labels) && body.pull_request.labels.exists(l, l.name == \"konflux-build\"))\n ) && body.action != \"ready_for_review\"\n)\n" failed to evaluate: no such key: pull_request`
retag-scanner-db-slim	`CEL expression evaluation error: expression "(\n event == \"push\" && target_branch.matches(\"^(master\|release-.\|refs/tags/.)$\")\n) \|\| (\n event == \"pull_request\" && (\n target_branch.startsWith(\"release-\") \|\|\n source_branch.matches(\"(konflux\|renovate\|appstudio\|rhtap)\") \|\|\n (has(body.pull_request.labels) && body.pull_request.labels.exists(l, l.name == \"konflux-build\"))\n ) && body.action != \"ready_for_review\"\n)\n" failed to evaluate: no such key: pull_request`
retag-scanner-db	`CEL expression evaluation error: expression "(\n event == \"push\" && target_branch.matches(\"^(master\|release-.\|refs/tags/.)$\")\n) \|\| (\n event == \"pull_request\" && (\n target_branch.startsWith(\"release-\") \|\|\n source_branch.matches(\"(konflux\|renovate\|appstudio\|rhtap)\") \|\|\n (has(body.pull_request.labels) && body.pull_request.labels.exists(l, l.name == \"konflux-build\"))\n ) && body.action != \"ready_for_review\"\n)\n" failed to evaluate: no such key: pull_request`
retag-scanner-slim	`CEL expression evaluation error: expression "(\n event == \"push\" && target_branch.matches(\"^(master\|release-.\|refs/tags/.)$\")\n) \|\| (\n event == \"pull_request\" && (\n target_branch.startsWith(\"release-\") \|\|\n source_branch.matches(\"(konflux\|renovate\|appstudio\|rhtap)\") \|\|\n (has(body.pull_request.labels) && body.pull_request.labels.exists(l, l.name == \"konflux-build\"))\n ) && body.action != \"ready_for_review\"\n)\n" failed to evaluate: no such key: pull_request`
retag-scanner	`CEL expression evaluation error: expression "(\n event == \"push\" && target_branch.matches(\"^(master\|release-.\|refs/tags/.)$\")\n) \|\| (\n event == \"pull_request\" && (\n target_branch.startsWith(\"release-\") \|\|\n source_branch.matches(\"(konflux\|renovate\|appstudio\|rhtap)\") \|\|\n (has(body.pull_request.labels) && body.pull_request.labels.exists(l, l.name == \"konflux-build\"))\n ) && body.action != \"ready_for_review\"\n)\n" failed to evaluate: no such key: pull_request`
roxctl-on-push	`CEL expression evaluation error: expression "(\n event == \"push\" && target_branch.matches(\"^(master\|release-.\|refs/tags/.)$\")\n) \|\| (\n event == \"pull_request\" && (\n target_branch.startsWith(\"release-\") \|\|\n source_branch.matches(\"(konflux\|renovate\|appstudio\|rhtap)\") \|\|\n (has(body.pull_request.labels) && body.pull_request.labels.exists(l, l.name == \"konflux-build\"))\n ) && body.action != \"ready_for_review\"\n)\n" failed to evaluate: no such key: pull_request`
scanner-v4-on-push	`CEL expression evaluation error: expression "(\n event == \"push\" && target_branch.matches(\"^(master\|release-.\|refs/tags/.)$\")\n) \|\| (\n event == \"pull_request\" && (\n target_branch.startsWith(\"release-\") \|\|\n source_branch.matches(\"(konflux\|renovate\|appstudio\|rhtap)\") \|\|\n (has(body.pull_request.labels) && body.pull_request.labels.exists(l, l.name == \"konflux-build\"))\n ) && body.action != \"ready_for_review\"\n)\n" failed to evaluate: no such key: pull_request`
scanner-v4-db-on-push	`CEL expression evaluation error: expression "(\n event == \"push\" && target_branch.matches(\"^(master\|release-.\|refs/tags/.)$\")\n) \|\| (\n event == \"pull_request\" && (\n target_branch.startsWith(\"release-\") \|\|\n source_branch.matches(\"(konflux\|renovate\|appstudio\|rhtap)\") \|\|\n (has(body.pull_request.labels) && body.pull_request.labels.exists(l, l.name == \"konflux-build\"))\n ) && body.action != \"ready_for_review\"\n)\n" failed to evaluate: no such key: pull_request`

openshift-ci · 2025-10-06T10:35:26Z

@janisz: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/gke-qa-e2e-tests	`23120b1`	link	false	`/test gke-qa-e2e-tests`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

janisz · 2025-10-06T11:55:18Z

Merged in #17126

janisz requested a review from dashrews78 September 30, 2025 16:00

github-actions bot added the area/central label Sep 30, 2025

janisz requested review from mtodor and rhybrillou September 30, 2025 16:00

This was referenced Sep 30, 2025

fix(risk): improve reprocess deployment risk #16987

Closed

ROX-30963: Only query indicator fields needed for evalution #16965

Merged

janisz added 2 commits October 3, 2025 10:10

test(bench): mesure max heap bytes and objects

1b31abe

Signed-off-by: Tomasz Janiszewski <tomek@redhat.com>

cleanup

23120b1

Signed-off-by: Tomasz Janiszewski <tomek@redhat.com>

janisz force-pushed the monitor_max_heap_bytes_in_benchmark branch from ee7cbee to 23120b1 Compare October 3, 2025 08:14

janisz mentioned this pull request Oct 3, 2025

ROX-31176: improve reprocess deployment risk #17126

Merged

janisz closed this Oct 6, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test(bench): mesure max heap bytes and objects#17078

test(bench): mesure max heap bytes and objects#17078
janisz wants to merge 2 commits intomasterfrom
monitor_max_heap_bytes_in_benchmark

janisz commented Sep 30, 2025

Uh oh!

codecov bot commented Sep 30, 2025 •

edited

Loading

Uh oh!

rhacs-bot commented Oct 3, 2025

Uh oh!

janisz commented Oct 6, 2025

Uh oh!

red-hat-konflux bot commented Oct 6, 2025

Uh oh!

openshift-ci bot commented Oct 6, 2025

Uh oh!

janisz commented Oct 6, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

janisz commented Sep 30, 2025

Description

User-facing documentation

Testing and quality

Automated testing

How I validated my change

Uh oh!

codecov bot commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

rhacs-bot commented Oct 3, 2025

Uh oh!

janisz commented Oct 6, 2025

Uh oh!

red-hat-konflux bot commented Oct 6, 2025

Uh oh!

openshift-ci bot commented Oct 6, 2025

Uh oh!

janisz commented Oct 6, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov bot commented Sep 30, 2025 •

edited

Loading