[PROTOTYPE / DO NOT MERGE] Feature/CVE normalized table#19543
Draft
[PROTOTYPE / DO NOT MERGE] Feature/CVE normalized table#19543
Conversation
…ces. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Introduces schema Go files for the new normalized cves table and component_cve_edges edge table. Removes the image_cves_v2 schema file. Generated with AI assistance. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…for normalized CVE table. This is part of the CVE data model rework that introduces an append-only cves table and a component_cve_edges edge table to replace the denormalized image_cves_v2 table. Generated with AI assistance. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…lication. Implements null-byte-delimited SHA256 hash over (cve_name, source, severity, cvss_v3, summary). Adds helper functions SourceToString and SeverityToString. Includes unit tests for hash determinism and boundary collision prevention. Generated with AI assistance. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Creates new cves and component_cve_edges tables, drops image_cves_v2 and image_cve_infos tables. No data migration needed (rescan repopulates). Generated with AI assistance. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…tions Replaces auto-generated ImageCVEV2 store with a custom store operating on the new cves and component_cve_edges tables. Implements two-phase CVE upsert, edge upsert with preserved first_system_occurrence, stale-edge deletion via != ALL semantics, and orphaned CVE batch deletion for GC. Key changes: - CVERow and EdgeRow data types for normalized table rows - UpsertCVE: two-phase insert+fetch to obtain UUID for new/existing CVEs - UpsertEdge: upserts edges while preserving first_system_occurrence - DeleteStaleEdges: removes edges not in keepCVEIDs using != ALL(array) - GetCVEsForImage: joins cves → edges → components for image queries - GetAllReferencedCVEs: fetches all CVEs with at least one edge - DeleteOrphanedCVEsBatch: GC support for unreferenced CVEs Generated with AI assistance. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Implements a batched NOT-EXISTS deletion loop (up to 100k CVEs per run) registered as a daily cron job. Processes at most 100 batches of 1000 rows per run to avoid long transactions. Context: Step 7 of CVE data model rework implementing background garbage collection for the new append-only cves table. Orphaned CVEs (those not referenced by any component_cve_edges) are periodically cleaned up to prevent unbounded growth. Generated with AI assistance. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
- Restore image_cves_v2.go schema file so the search framework (globalindex, views/imagecve, views/imagecveflat) continues to compile. The table is dropped at runtime by migration m_222_to_m_223; the Go definitions are kept as stubs. - Add Search, Count, SearchImageCVEs, SearchRawImageCVEs, GetBatch stub methods to the new DataStore interface so existing callers (GraphQL resolvers, report generator, search.Searcher registration) continue to compile. All stubs return empty results, consistent with the dropped table. - Stub out the DataKind-based loader in loaders/image_cves_v2.go. Generated with AI assistance. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Adds GetNormalizedCVEsForCSV and GetCVEsForImageCSV functions to the image CVE CSV handler that query the new cves + component_cve_edges tables directly. Adds GetAllNormalizedCVEs to the imagecveflat view. These new functions provide CSV export capabilities using the normalized CVE data model, alongside the existing GraphQL-based export functions. Generated with AI assistance. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…CVE upsert. Scan ingestion now calls UpsertCVE (two-phase), UpsertEdge, and DeleteStaleEdges on the normalized cves/component_cve_edges tables instead of bulk-inserting into image_cves_v2. The old copyFromImageComponentV2Cves bulk copy is removed. This change applies to both the imagev2 store (central/imagev2/datastore/store/postgres/store.go) and the non-v2 image store (central/image/datastore/store/v2/postgres/store.go). The new implementation: - Iterates over ImageCVEV2 objects from scan results - Determines the primary source from CVSS metrics (NVD > RED_HAT > OSV) - Computes content hash for CVE deduplication - Upserts each CVE row into the cves table - Upserts corresponding edge into component_cve_edges table - Deletes stale edges for components whose CVEs have changed Generated with AI assistance. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
|
Skipping CI for Draft Pull Request. |
…avoid conflict. relations.proto already defines a deprecated ComponentCVEEdge message in the storage package. Rename the new edge message to NormalizedComponentCVEEdge to eliminate the duplicate-symbol error from protoc. Regenerate proto-generated sources (make proto-generated-srcs). Generated with AI assistance. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Contributor
|
Images are ready for the commit at d785652. To use with deploy scripts, first |
…irective. The 'regenerate mocks' commit ran make go-generated-srcs which re-executed gen.go's pg-table-bindings-wrapper directive, silently overwriting the custom store implementation with the old auto-generated ImageCVEV2 store. Fixes: - Restore store.go to the hand-written normalized CVE implementation (UpsertCVE, UpsertEdge, DeleteStaleEdges, DeleteOrphanedCVEsBatch, etc.) - Delete gen.go so that make go-generated-srcs can never overwrite this package again - Delete store_test.go (was auto-generated for the old Store interface; it referenced postgres.Store which no longer exists in this package) Generated with AI assistance. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Three related issues: 1. Import cycle: imagev2/image store.go called cvev2ds.Singleton() which imports the full datastore package. The datastore test graph imports image stores back, creating a cycle. Fix: replace Singleton() with cvev2pgstore.New(s.db) — the raw postgres layer has no upstream deps. 2. Missing Count on store.Store: test files called cvePgStore.Count() but store.Store lacked the method. Add Count(ctx, *v1.Query) (int, error) to store.Store; implement as SELECT COUNT(*) FROM cves (query param accepted for interface compatibility, ignored for now). Delegate in datastore_impl.go instead of returning hardcoded 0. 3. Undefined cveStore.Store in test files: test files imported store/postgres for the Store type, but the postgres package no longer exports that interface (New returns store.Store from the parent package). Fix: use cveStore "...store" for the interface type and cvev2pgstore "...store/postgres" for the constructor call. Generated with AI assistance. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
roxvet errors: - datastore_sac_test.go used DataStore.Exists() — add Exists(ctx, id) to both store.Store (SELECT EXISTS FROM cves) and DataStore (delegates). - store_test.go used cvePgStore.GetIDs() — add GetIDs(ctx) to store.Store (SELECT id FROM cves) and the GC testStore stub. gofmt errors in files written by subagents: - store/store.go: trailing spaces in struct field comments - image store.go (both paths): map literal alignment in priority map - migration_impl.go: const block alignment - schema/cves.go, schema/component_cve_edges.go: struct field alignment Generated with AI assistance. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
change me!
User-facing documentation
Testing and quality
Automated testing
How I validated my change
change me!