Skip to content

[PROTOTYPE / DO NOT MERGE] Feature/CVE normalized table#19543

Draft
stehessel wants to merge 18 commits intomasterfrom
feature/cve-normalized-table
Draft

[PROTOTYPE / DO NOT MERGE] Feature/CVE normalized table#19543
stehessel wants to merge 18 commits intomasterfrom
feature/cve-normalized-table

Conversation

@stehessel
Copy link
Collaborator

Description

change me!

User-facing documentation

Testing and quality

  • the change is production ready: the change is GA, or otherwise the functionality is gated by a feature flag
  • CI results are inspected

Automated testing

  • added unit tests
  • added e2e tests
  • added regression tests
  • added compatibility tests
  • modified existing tests

How I validated my change

change me!

stehessel and others added 10 commits March 23, 2026 11:46
…ces.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Introduces schema Go files for the new normalized cves table and
component_cve_edges edge table. Removes the image_cves_v2 schema file.

Generated with AI assistance.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…for normalized CVE table.

This is part of the CVE data model rework that introduces an append-only
cves table and a component_cve_edges edge table to replace the denormalized
image_cves_v2 table.

Generated with AI assistance.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…lication.

Implements null-byte-delimited SHA256 hash over (cve_name, source, severity,
cvss_v3, summary). Adds helper functions SourceToString and SeverityToString.
Includes unit tests for hash determinism and boundary collision prevention.

Generated with AI assistance.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Creates new cves and component_cve_edges tables, drops image_cves_v2
and image_cve_infos tables. No data migration needed (rescan repopulates).

Generated with AI assistance.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…tions

Replaces auto-generated ImageCVEV2 store with a custom store operating on
the new cves and component_cve_edges tables. Implements two-phase CVE upsert,
edge upsert with preserved first_system_occurrence, stale-edge deletion via
!= ALL semantics, and orphaned CVE batch deletion for GC.

Key changes:
- CVERow and EdgeRow data types for normalized table rows
- UpsertCVE: two-phase insert+fetch to obtain UUID for new/existing CVEs
- UpsertEdge: upserts edges while preserving first_system_occurrence
- DeleteStaleEdges: removes edges not in keepCVEIDs using != ALL(array)
- GetCVEsForImage: joins cves → edges → components for image queries
- GetAllReferencedCVEs: fetches all CVEs with at least one edge
- DeleteOrphanedCVEsBatch: GC support for unreferenced CVEs

Generated with AI assistance.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Implements a batched NOT-EXISTS deletion loop (up to 100k CVEs per run)
registered as a daily cron job. Processes at most 100 batches of 1000
rows per run to avoid long transactions.

Context: Step 7 of CVE data model rework implementing background garbage
collection for the new append-only cves table. Orphaned CVEs (those not
referenced by any component_cve_edges) are periodically cleaned up to
prevent unbounded growth.

Generated with AI assistance.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
- Restore image_cves_v2.go schema file so the search framework (globalindex,
  views/imagecve, views/imagecveflat) continues to compile. The table is
  dropped at runtime by migration m_222_to_m_223; the Go definitions are
  kept as stubs.
- Add Search, Count, SearchImageCVEs, SearchRawImageCVEs, GetBatch stub
  methods to the new DataStore interface so existing callers (GraphQL
  resolvers, report generator, search.Searcher registration) continue to
  compile. All stubs return empty results, consistent with the dropped table.
- Stub out the DataKind-based loader in loaders/image_cves_v2.go.

Generated with AI assistance.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Adds GetNormalizedCVEsForCSV and GetCVEsForImageCSV functions to the
image CVE CSV handler that query the new cves + component_cve_edges tables
directly. Adds GetAllNormalizedCVEs to the imagecveflat view.

These new functions provide CSV export capabilities using the normalized
CVE data model, alongside the existing GraphQL-based export functions.

Generated with AI assistance.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…CVE upsert.

Scan ingestion now calls UpsertCVE (two-phase), UpsertEdge, and
DeleteStaleEdges on the normalized cves/component_cve_edges tables instead
of bulk-inserting into image_cves_v2. The old copyFromImageComponentV2Cves
bulk copy is removed.

This change applies to both the imagev2 store
(central/imagev2/datastore/store/postgres/store.go) and the non-v2 image
store (central/image/datastore/store/v2/postgres/store.go).

The new implementation:
- Iterates over ImageCVEV2 objects from scan results
- Determines the primary source from CVSS metrics (NVD > RED_HAT > OSV)
- Computes content hash for CVE deduplication
- Upserts each CVE row into the cves table
- Upserts corresponding edge into component_cve_edges table
- Deletes stale edges for components whose CVEs have changed

Generated with AI assistance.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@openshift-ci
Copy link

openshift-ci bot commented Mar 23, 2026

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

…avoid conflict.

relations.proto already defines a deprecated ComponentCVEEdge message in the
storage package. Rename the new edge message to NormalizedComponentCVEEdge to
eliminate the duplicate-symbol error from protoc. Regenerate proto-generated
sources (make proto-generated-srcs).

Generated with AI assistance.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@rhacs-bot
Copy link
Contributor

rhacs-bot commented Mar 23, 2026

Images are ready for the commit at d785652.

To use with deploy scripts, first export MAIN_IMAGE_TAG=4.11.x-424-gd785652bc4.

stehessel and others added 7 commits March 23, 2026 14:32
…irective.

The 'regenerate mocks' commit ran make go-generated-srcs which re-executed
gen.go's pg-table-bindings-wrapper directive, silently overwriting the custom
store implementation with the old auto-generated ImageCVEV2 store.

Fixes:
- Restore store.go to the hand-written normalized CVE implementation
  (UpsertCVE, UpsertEdge, DeleteStaleEdges, DeleteOrphanedCVEsBatch, etc.)
- Delete gen.go so that make go-generated-srcs can never overwrite this
  package again
- Delete store_test.go (was auto-generated for the old Store interface;
  it referenced postgres.Store which no longer exists in this package)

Generated with AI assistance.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Three related issues:

1. Import cycle: imagev2/image store.go called cvev2ds.Singleton() which
   imports the full datastore package. The datastore test graph imports image
   stores back, creating a cycle. Fix: replace Singleton() with
   cvev2pgstore.New(s.db) — the raw postgres layer has no upstream deps.

2. Missing Count on store.Store: test files called cvePgStore.Count() but
   store.Store lacked the method. Add Count(ctx, *v1.Query) (int, error)
   to store.Store; implement as SELECT COUNT(*) FROM cves (query param
   accepted for interface compatibility, ignored for now). Delegate in
   datastore_impl.go instead of returning hardcoded 0.

3. Undefined cveStore.Store in test files: test files imported store/postgres
   for the Store type, but the postgres package no longer exports that
   interface (New returns store.Store from the parent package). Fix: use
   cveStore "...store" for the interface type and cvev2pgstore "...store/postgres"
   for the constructor call.

Generated with AI assistance.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
roxvet errors:
- datastore_sac_test.go used DataStore.Exists() — add Exists(ctx, id) to
  both store.Store (SELECT EXISTS FROM cves) and DataStore (delegates).
- store_test.go used cvePgStore.GetIDs() — add GetIDs(ctx) to store.Store
  (SELECT id FROM cves) and the GC testStore stub.

gofmt errors in files written by subagents:
- store/store.go: trailing spaces in struct field comments
- image store.go (both paths): map literal alignment in priority map
- migration_impl.go: const block alignment
- schema/cves.go, schema/component_cve_edges.go: struct field alignment

Generated with AI assistance.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants