ROX-32873: Metrics for process arguments by JoukoVirtanen · Pull Request #18794 · stackrox/stackrox

JoukoVirtanen · 2026-02-01T17:54:24Z

Description

We would like to know how much memory is taken up by process arguments. The reason we would like to know this is that it would better help us plan how to handle process arguments. It would help us better understand the pros and cons of truncating process baselines in the future.

See the related PR for process lineage info #19406

User-facing documentation

CHANGELOG.md is updated OR update is not needed
documentation PR is created and is linked above OR is not needed

Testing and quality

the change is production ready: the change is GA, or otherwise the functionality is gated by a feature flag
CI results are inspected

Automated testing

How I validated my change

Deployed ACS using deploy/deploy-local.sh.

Created a port forward

kubectl -n stackrox port-forward deploy/central 9090:9090 > /dev/null 2>&1 &

curl http://localhost:9090/metrics

# HELP rox_central_process_upserted_args_size Distribution of process argument sizes in characters for upserted indicators
# TYPE rox_central_process_upserted_args_size histogram
rox_central_process_upserted_args_size_bucket{le="0"} 29
rox_central_process_upserted_args_size_bucket{le="128"} 439
rox_central_process_upserted_args_size_bucket{le="256"} 483
rox_central_process_upserted_args_size_bucket{le="512"} 495
rox_central_process_upserted_args_size_bucket{le="1024"} 499
rox_central_process_upserted_args_size_bucket{le="2048"} 500
rox_central_process_upserted_args_size_bucket{le="4096"} 501
rox_central_process_upserted_args_size_bucket{le="8192"} 501
rox_central_process_upserted_args_size_bucket{le="16384"} 501
rox_central_process_upserted_args_size_bucket{le="32768"} 501
rox_central_process_upserted_args_size_bucket{le="65536"} 501
rox_central_process_upserted_args_size_bucket{le="+Inf"} 501
rox_central_process_upserted_args_size_sum 31352
rox_central_process_upserted_args_size_count 501
# HELP rox_central_process_upserted_args_size_total Total process argument sizes in characters by cluster and namespace
# TYPE rox_central_process_upserted_args_size_total counter
rox_central_process_upserted_args_size_total{cluster="05d9c020-2a1d-439a-8c9c-2cbdf5413673",namespace="default"} 8
rox_central_process_upserted_args_size_total{cluster="05d9c020-2a1d-439a-8c9c-2cbdf5413673",namespace="gke-managed-cim"} 1747
rox_central_process_upserted_args_size_total{cluster="05d9c020-2a1d-439a-8c9c-2cbdf5413673",namespace="gmp-system"} 6193
rox_central_process_upserted_args_size_total{cluster="05d9c020-2a1d-439a-8c9c-2cbdf5413673",namespace="kube-system"} 19811
rox_central_process_upserted_args_size_total{cluster="05d9c020-2a1d-439a-8c9c-2cbdf5413673",namespace="stackrox"} 3593

openshift-ci · 2026-02-01T17:54:27Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

rhacs-bot · 2026-02-01T18:28:24Z

Images are ready for the commit at ccc26a8.

To use with deploy scripts, first export MAIN_IMAGE_TAG=4.11.x-310-gccc26a81f9.

codecov · 2026-02-01T18:32:25Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 49.71%. Comparing base (82657e5) to head (ccc26a8).
⚠️ Report is 20 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #18794      +/-   ##
==========================================
+ Coverage   49.64%   49.71%   +0.07%     
==========================================
  Files        2698     2701       +3     
  Lines      203075   203471     +396     
==========================================
+ Hits       100817   101160     +343     
- Misses      94737    94784      +47     
- Partials     7521     7527       +6

Flag	Coverage Δ
go-unit-tests	`49.71% <100.00%> (+0.07%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

JoukoVirtanen · 2026-02-05T19:42:55Z

/test gke-ui-e2e-tests

erthalion

Thanks, few commentaries below.

erthalion · 2026-02-10T10:22:45Z

central/processindicator/datastore/metrics.go

+}
+
+// recordProcessIndicatorAdded records metrics for a single process indicator added to DB.
+func recordProcessIndicatorAdded(argsSize int) {


Does this have to be a separate function?

No, it does not have to be a separate function. I have removed it.

erthalion · 2026-02-10T10:23:46Z

central/processindicator/datastore/metrics.go

+		Buckets:   []float64{0, 128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536},
+	})
+
+	processIndicatorsAddedCounter = prometheus.NewCounter(prometheus.CounterOpts{


I think it makes sense to rename it into processIndicatorsLiveCounter, then increment it as in the PR and decrement it whenever a process indicator is removed (either in the pruner or for some other reasons).

I considered that, but I don't think it is practical. When the process indicators are pruned, they are not returned. In most cases the IDs of the pruned process indicators are returned, but not the process indicators themselves. To decrement the counter we would need to know the process arguments that have been pruned. It could theoretically be done, but it would degrade performance to get the process arguments, and it would add to the maintenance burden. I don't think it would be worth it.

What does this counter have to do with arguments? There are two metrics, one is a histogram of argument sizes, another one is number of added process indicators. The latter one could be added and decremented, since we're adding and removing process indicators. Correct?

I misunderstood. I should have read more closely. I thought that you wanted the histogram to represent the distribution of process arguments in the database. Decrementing the count of process indicators makes sense.

There is already a count of the number of rows in the process_indicators table so I think processIndicatorsLiveCounter would be redundant.

Not quite, process_indicators is a metric taken form the database statistics and is only an estimated number of records. It may drift from the real number in under certain circumstances.

Another important point is that currently there are not easily consumable metrics regarding process pruning, and adding those is the main point of improving runtime data metrics. Since you're busy with that anyway, let's cover it as well.

I have a PR to keep track of pruned process indicators here #19130

It is still in draft form.

JoukoVirtanen · 2026-02-17T18:25:08Z

/test gke-ui-e2e-tests

JoukoVirtanen · 2026-02-23T07:34:31Z

/test gke-qa-e2e-tests

…nts stored, and number of process indicators inserted

…roken down by cluster and namespace

openshift-ci bot added the do-not-merge/work-in-progress label Feb 1, 2026

github-actions bot added the area/central label Feb 1, 2026

JoukoVirtanen marked this pull request as ready for review February 4, 2026 23:03

openshift-ci bot removed the do-not-merge/work-in-progress label Feb 4, 2026

JoukoVirtanen force-pushed the jv-ROX-32873-metrics-for-process-arguments branch from bc54504 to d8e2a4e Compare February 5, 2026 16:23

JoukoVirtanen requested a review from erthalion February 5, 2026 22:40

erthalion reviewed Feb 10, 2026

View reviewed changes

JoukoVirtanen force-pushed the jv-ROX-32873-metrics-for-process-arguments branch from 3000c06 to 73aec63 Compare February 12, 2026 03:54

JoukoVirtanen requested a review from erthalion February 12, 2026 07:40

JoukoVirtanen force-pushed the jv-ROX-32873-metrics-for-process-arguments branch from 73aec63 to 49ebcfa Compare February 23, 2026 02:04

JoukoVirtanen force-pushed the jv-ROX-32873-metrics-for-process-arguments branch from 49ebcfa to 066620c Compare March 2, 2026 23:49

JoukoVirtanen added 9 commits March 11, 2026 08:18

X-Smart-Branch-Parent: master

2a4c7c2

Added metrics for histogram of argument length, total bytes of argume…

70007d6

…nts stored, and number of process indicators inserted

Removed a redundant metric

5f98b79

Removed recordProcessIndicatorAdded

84026e7

Added a metric for the number of characters in arguments

9a04b18

Removed processIndicatorsAddedCounter

ea58b3e

Removed process_args_size_bytes metric

b33b185

Renamed process_args_size_chars to process_upserted_args_size

1d3e13b

The metric takes in the cluster id as a parameter

3ce4c0a

JoukoVirtanen force-pushed the jv-ROX-32873-metrics-for-process-arguments branch from 142c53e to 3ce4c0a Compare March 11, 2026 15:18

Process arguments lengths are now specified by cluster and namespace

0758d89

JoukoVirtanen mentioned this pull request Mar 13, 2026

ROX-33352: Add histogram of lineage size #19406

Open

9 tasks

There is just one histogram for central. Argument lengths are still b…

ccc26a8

…roken down by cluster and namespace

Conversation

JoukoVirtanen commented Feb 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

User-facing documentation

Testing and quality

Automated testing

How I validated my change

Uh oh!

openshift-ci bot commented Feb 1, 2026

Uh oh!

rhacs-bot commented Feb 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Feb 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

JoukoVirtanen commented Feb 5, 2026

Uh oh!

erthalion left a comment

Choose a reason for hiding this comment

Uh oh!

erthalion Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

JoukoVirtanen Feb 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

erthalion Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

JoukoVirtanen Feb 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

erthalion Feb 18, 2026

Choose a reason for hiding this comment

Uh oh!

JoukoVirtanen Feb 18, 2026

Choose a reason for hiding this comment

Uh oh!

JoukoVirtanen Feb 19, 2026

Choose a reason for hiding this comment

Uh oh!

erthalion Feb 19, 2026

Choose a reason for hiding this comment

Uh oh!

JoukoVirtanen Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

JoukoVirtanen commented Feb 17, 2026

Uh oh!

JoukoVirtanen commented Feb 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

JoukoVirtanen commented Feb 1, 2026 •

edited

Loading

rhacs-bot commented Feb 1, 2026 •

edited

Loading

codecov bot commented Feb 1, 2026 •

edited

Loading

JoukoVirtanen Feb 10, 2026 •

edited

Loading

JoukoVirtanen Feb 10, 2026 •

edited

Loading