Skip to content

ROX-19064: Scanner V4 Bundle Filtering#19835

Open
dcaravel wants to merge 1 commit intomasterfrom
dc/scan4-filter-import
Open

ROX-19064: Scanner V4 Bundle Filtering#19835
dcaravel wants to merge 1 commit intomasterfrom
dc/scan4-filter-import

Conversation

@dcaravel
Copy link
Copy Markdown
Contributor

@dcaravel dcaravel commented Apr 6, 2026

Description

Adds the ability to set an allow list of bundles for which to import into Scanner V4 DB.

While enabling Scanner V4, CI jobs were timing out due to taking ~40m to load vulns occasionally.

CI can be sped up by not loading data that isn't utilized in tests, such as SUSE which was taking ~18m to load in GKE tests.

=== Per-Bundle Load Time Breakdown ===

BUNDLE                          RUNS       AVG       MAX       MIN
------------------------------------------------------------------
alpine                            16       19s       36s       12s
aws                               16       27s       38s       20s
debian                            16       25s       32s       20s
epss                              16       45s     1m06s       29s
manual                            16        0s        1s        0s
nvd                               16     1m19s     2m20s       48s
oracle                            16     2m00s     2m51s     1m23s
osv                               16     1m56s     2m25s     1m25s
photon                            16       34s       45s       22s
rhel-vex                          16     7m52s    10m18s     5m55s
stackrox-rhel-csaf                16       12s       15s       10s
suse                              16    14m22s    17m59s    10m44s
ubuntu                            16     5m46s     7m22s     4m36s

User-facing documentation

Testing and quality

  • the change is production ready: the change is GA, or otherwise the functionality is gated by a feature flag
  • CI results are inspected

Automated testing

  • added unit tests

How I validated my change

Manually via modifying the scanner-v4-matcher-config configmap, adding key:

vuln_bundle_allowlist: rhel-vex, manual, nvd, epss, stackrox-rhel-csaf

And then observing via logs, database, and comparing before/after scans the only the expected data was loaded:

{"level":"warn","host":"scanner-v4-matcher-645d9f58c5-l5ld7","component":"matcher/updater/vuln/Updater.Start","bundles":"epss, stackrox-rhel-csaf, rhel-vex, manual, nvd","time":"2026-04-06T00:48:19Z","message":"vulnerability bundle allowlist is active: only listed bundles will be imported; do not use in production"}
{"level":"info","host":"scanner-v4-matcher-645d9f58c5-l5ld7","component":"matcher/updater/vuln/Updater.runMultiBundleUpdate","timestamp":"0001-01-01T00:00:00Z","time":"2026-04-06T00:48:19Z","message":"previous vuln update"}
{"level":"info","host":"scanner-v4-matcher-645d9f58c5-l5ld7","component":"matcher/updater/vuln/Updater.fetch","url":"https://central.stackrox.svc/api/extensions/scannerdefinitions?version=dev","attempt":1,"time":"2026-04-06T00:48:19Z","message":"fetching vuln update"}
{"level":"info","host":"scanner-v4-matcher-645d9f58c5-l5ld7","bundle":"bundles/alpine.json.zst","component":"matcher/updater/vuln/Updater.runMultiBundleUpdate","time":"2026-04-06T00:48:19Z","message":"skipping bundle (not in allowlist)"}
{"level":"info","host":"scanner-v4-matcher-645d9f58c5-l5ld7","component":"matcher/updater/vuln/Updater.runMultiBundleUpdate","bundle":"bundles/aws.json.zst","time":"2026-04-06T00:48:19Z","message":"skipping bundle (not in allowlist)"}
{"level":"info","host":"scanner-v4-matcher-645d9f58c5-l5ld7","component":"matcher/updater/vuln/Updater.runMultiBundleUpdate","bundle":"bundles/debian.json.zst","time":"2026-04-06T00:48:19Z","message":"skipping bundle (not in allowlist)"}
{"level":"info","host":"scanner-v4-matcher-645d9f58c5-l5ld7","component":"matcher/updater/vuln/Updater.runMultiBundleUpdate","bundle":"bundles/epss.json.zst","time":"2026-04-06T00:48:19Z","message":"starting bundle update"}
{"level":"info","host":"scanner-v4-matcher-645d9f58c5-l5ld7","bundle":"bundles/epss.json.zst","component":"matcher/updater/vuln/Updater.Import","updater":"clair.epss","kind":"enrichment","time":"2026-04-06T00:48:19Z","message":"importing update"}
{"level":"info","host":"scanner-v4-matcher-645d9f58c5-l5ld7","bundle":"bundles/epss.json.zst","component":"matcher/updater/vuln/Updater.Import","updater":"clair.epss","kind":"enrichment","ref":"14508ec4-f9ba-4294-a73f-77fb672dfef9","count":323647,"time":"2026-04-06T00:48:57Z","message":"update imported"}
{"level":"info","host":"scanner-v4-matcher-645d9f58c5-l5ld7","component":"matcher/updater/vuln/Updater.runMultiBundleUpdate","bundle":"bundles/epss.json.zst","time":"2026-04-06T00:48:57Z","message":"completed bundle update"}
{"level":"info","host":"scanner-v4-matcher-645d9f58c5-l5ld7","bundle":"bundles/manual.json.zst","component":"matcher/updater/vuln/Updater.runMultiBundleUpdate","time":"2026-04-06T00:48:57Z","message":"starting bundle update"}
{"level":"info","host":"scanner-v4-matcher-645d9f58c5-l5ld7","bundle":"bundles/manual.json.zst","component":"matcher/updater/vuln/Updater.Import","updater":"stackrox-manual","kind":"vulnerability","time":"2026-04-06T00:48:57Z","message":"importing update"}
{"level":"info","host":"scanner-v4-matcher-645d9f58c5-l5ld7","bundle":"bundles/manual.json.zst","component":"matcher/updater/vuln/Updater.Import","updater":"stackrox-manual","kind":"vulnerability","ref":"182d1f5c-625f-4243-a268-aa716de04ee4","count":24,"time":"2026-04-06T00:48:57Z","message":"update imported"}
{"level":"info","host":"scanner-v4-matcher-645d9f58c5-l5ld7","bundle":"bundles/manual.json.zst","component":"matcher/updater/vuln/Updater.runMultiBundleUpdate","time":"2026-04-06T00:48:57Z","message":"completed bundle update"}
{"level":"info","host":"scanner-v4-matcher-645d9f58c5-l5ld7","component":"matcher/updater/vuln/Updater.runMultiBundleUpdate","bundle":"bundles/nvd.json.zst","time":"2026-04-06T00:48:57Z","message":"starting bundle update"}
{"level":"info","host":"scanner-v4-matcher-645d9f58c5-l5ld7","bundle":"bundles/nvd.json.zst","component":"matcher/updater/vuln/Updater.Import","updater":"nvd","kind":"enrichment","time":"2026-04-06T00:48:57Z","message":"importing update"}
{"level":"info","host":"scanner-v4-matcher-645d9f58c5-l5ld7","component":"matcher/updater/vuln/Updater.Import","bundle":"bundles/nvd.json.zst","updater":"nvd","kind":"enrichment","ref":"1a0753e0-6ebd-402d-95b9-bfd27f831264","count":323688,"time":"2026-04-06T00:50:00Z","message":"update imported"}
{"level":"info","host":"scanner-v4-matcher-645d9f58c5-l5ld7","component":"matcher/updater/vuln/Updater.runMultiBundleUpdate","bundle":"bundles/nvd.json.zst","time":"2026-04-06T00:50:00Z","message":"completed bundle update"}
{"level":"info","host":"scanner-v4-matcher-645d9f58c5-l5ld7","component":"matcher/updater/vuln/Updater.runMultiBundleUpdate","bundle":"bundles/oracle.json.zst","time":"2026-04-06T00:50:00Z","message":"skipping bundle (not in allowlist)"}
{"level":"info","host":"scanner-v4-matcher-645d9f58c5-l5ld7","component":"matcher/updater/vuln/Updater.runMultiBundleUpdate","bundle":"bundles/osv.json.zst","time":"2026-04-06T00:50:00Z","message":"skipping bundle (not in allowlist)"}
{"level":"info","host":"scanner-v4-matcher-645d9f58c5-l5ld7","component":"matcher/updater/vuln/Updater.runMultiBundleUpdate","bundle":"bundles/photon.json.zst","time":"2026-04-06T00:50:00Z","message":"skipping bundle (not in allowlist)"}
{"level":"info","host":"scanner-v4-matcher-645d9f58c5-l5ld7","bundle":"bundles/rhel-vex.json.zst","component":"matcher/updater/vuln/Updater.runMultiBundleUpdate","time":"2026-04-06T00:50:00Z","message":"starting bundle update"}
{"level":"info","host":"scanner-v4-matcher-645d9f58c5-l5ld7","bundle":"bundles/rhel-vex.json.zst","component":"matcher/updater/vuln/Updater.Import","updater":"rhel-vex","kind":"vulnerability","time":"2026-04-06T00:50:00Z","message":"importing update"}
{"level":"info","host":"scanner-v4-matcher-645d9f58c5-l5ld7","bundle":"bundles/rhel-vex.json.zst","component":"matcher/updater/vuln/Updater.Import","updater":"rhel-vex","kind":"vulnerability","ref":"2e702a83-1c7b-4b80-91a9-538e0ca70b14","count":2299189,"time":"2026-04-06T00:56:42Z","message":"update imported"}
{"level":"info","host":"scanner-v4-matcher-645d9f58c5-l5ld7","component":"matcher/updater/vuln/Updater.runMultiBundleUpdate","bundle":"bundles/rhel-vex.json.zst","time":"2026-04-06T00:56:42Z","message":"completed bundle update"}
{"level":"info","host":"scanner-v4-matcher-645d9f58c5-l5ld7","component":"matcher/updater/vuln/Updater.runMultiBundleUpdate","bundle":"bundles/stackrox-rhel-csaf.json.zst","time":"2026-04-06T00:56:42Z","message":"starting bundle update"}
{"level":"info","host":"scanner-v4-matcher-645d9f58c5-l5ld7","bundle":"bundles/stackrox-rhel-csaf.json.zst","component":"matcher/updater/vuln/Updater.Import","updater":"stackrox.rhel-csaf","kind":"enrichment","time":"2026-04-06T00:56:42Z","message":"importing update"}
{"level":"info","host":"scanner-v4-matcher-645d9f58c5-l5ld7","bundle":"bundles/stackrox-rhel-csaf.json.zst","component":"matcher/updater/vuln/Updater.Import","updater":"stackrox.rhel-csaf","kind":"enrichment","ref":"31ee1692-cf26-4d25-b8b1-5ee835e58794","count":19819,"time":"2026-04-06T00:56:53Z","message":"update imported"}
{"level":"info","host":"scanner-v4-matcher-645d9f58c5-l5ld7","component":"matcher/updater/vuln/Updater.runMultiBundleUpdate","bundle":"bundles/stackrox-rhel-csaf.json.zst","time":"2026-04-06T00:56:53Z","message":"completed bundle update"}
{"level":"info","host":"scanner-v4-matcher-645d9f58c5-l5ld7","component":"matcher/updater/vuln/Updater.runMultiBundleUpdate","bundle":"bundles/suse.json.zst","time":"2026-04-06T00:56:53Z","message":"skipping bundle (not in allowlist)"}
{"level":"info","host":"scanner-v4-matcher-645d9f58c5-l5ld7","component":"matcher/updater/vuln/Updater.runMultiBundleUpdate","bundle":"bundles/ubuntu.json.zst","time":"2026-04-06T00:56:53Z","message":"skipping bundle (not in allowlist)"}
{"level":"info","host":"scanner-v4-matcher-645d9f58c5-l5ld7","component":"updater/vuln/dists.update","time":"2026-04-06T00:56:53Z","message":"updating vuln distributions"}
{"level":"info","host":"scanner-v4-matcher-645d9f58c5-l5ld7","component":"updater/vuln/dists.update","updated":true,"time":"2026-04-06T00:57:07Z","message":"done updating vuln distributions"}
{"level":"info","host":"scanner-v4-matcher-645d9f58c5-l5ld7","component":"matcher/updater/vuln/Updater.Initialized","time":"2026-04-06T00:57:07Z","message":"all vulnerability bundles were updated at least once: setting to initialized"}
postgres=# select * from last_vuln_update;
                 key                 | timestamp |  update_timestamp   
-------------------------------------+-----------+---------------------
 bundles/epss.json.zst               |           | 2026-03-30 18:42:59
 bundles/manual.json.zst             |           | 2026-03-30 18:42:59
 bundles/nvd.json.zst                |           | 2026-03-30 18:42:59
 bundles/rhel-vex.json.zst           |           | 2026-03-30 18:42:59
 bundles/stackrox-rhel-csaf.json.zst |           | 2026-03-30 18:42:59

Scanned quay.io/rhacs-eng/main:4.9.0 before and after - differences were as expected (ie: OSV vulns were missing)

Scanned nginx:1.23.0 before and after - no vulns in the latter which is expected - image is debian based and no debian vulns were loaded.

@openshift-ci
Copy link
Copy Markdown

openshift-ci bot commented Apr 6, 2026

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@codecov
Copy link
Copy Markdown

codecov bot commented Apr 6, 2026

Codecov Report

❌ Patch coverage is 57.14286% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 49.60%. Comparing base (2d5d7a2) to head (c4e6e53).
⚠️ Report is 7 commits behind head on master.

Files with missing lines Patch % Lines
scanner/matcher/updater/vuln/updater.go 57.14% 9 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##           master   #19835   +/-   ##
=======================================
  Coverage   49.60%   49.60%           
=======================================
  Files        2763     2763           
  Lines      208339   208359   +20     
=======================================
+ Hits       103341   103353   +12     
- Misses      97331    97339    +8     
  Partials     7667     7667           
Flag Coverage Δ
go-unit-tests 49.60% <57.14%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 6, 2026

🚀 Build Images Ready

Images are ready for commit c4e6e53. To use with deploy scripts:

export MAIN_IMAGE_TAG=4.11.x-561-gc4e6e53703

@dcaravel
Copy link
Copy Markdown
Contributor Author

dcaravel commented Apr 6, 2026

/test all

@dcaravel dcaravel marked this pull request as ready for review April 6, 2026 13:50
@dcaravel dcaravel requested a review from a team as a code owner April 6, 2026 13:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant