Skip to content

ROX-31175: Prefer RunSelectRequestForSchemaFn#17141

Open
janisz wants to merge 7 commits intomasterfrom
optimizew_interating_over_results
Open

ROX-31175: Prefer RunSelectRequestForSchemaFn#17141
janisz wants to merge 7 commits intomasterfrom
optimizew_interating_over_results

Conversation

@janisz
Copy link
Contributor

@janisz janisz commented Oct 6, 2025

This PR improves memory efficiency by replacing calls to RunSelectRequestForSchema with RunSelectRequestForSchemaFn. It aligns with the approach used in the following PRs:

The primary objective is not to enhance processing speed, but to reduce overall memory consumption by minimizing the number and size of heap allocations.

🤖 Generated with Claude Code

Co-Authored-By: Claude noreply@anthropic.com

@janisz janisz requested a review from a team as a code owner October 6, 2025 09:58
@janisz janisz removed the request for review from a team October 6, 2025 09:58
@rhacs-bot
Copy link
Contributor

rhacs-bot commented Oct 6, 2025

Images are ready for the commit at daf8539.

To use with deploy scripts, first export MAIN_IMAGE_TAG=4.11.x-413-gdaf8539f9b.

@codecov
Copy link

codecov bot commented Oct 6, 2025

Codecov Report

❌ Patch coverage is 58.36576% with 107 lines in your changes missing coverage. Please review.
✅ Project coverage is 49.26%. Comparing base (5b741e9) to head (daf8539).
⚠️ Report is 9 commits behind head on master.

Files with missing lines Patch % Lines
pkg/search/postgres/select.go 0.00% 45 Missing ⚠️
central/views/imagecve/view_impl.go 55.55% 16 Missing and 4 partials ⚠️
central/views/imagecveflat/view_impl.go 38.46% 6 Missing and 2 partials ⚠️
...ts/scheduler/v2/reportgenerator/report_gen_impl.go 73.91% 6 Missing ⚠️
central/image/datastore/store/v2/postgres/store.go 0.00% 5 Missing ⚠️
central/imagev2/datastore/store/postgres/store.go 0.00% 5 Missing ⚠️
central/views/imagecomponentflat/view_impl.go 0.00% 5 Missing ⚠️
central/views/nodecve/view_impl.go 73.68% 1 Missing and 4 partials ⚠️
...erator/v2/checkresults/datastore/datastore_impl.go 87.50% 0 Missing and 4 partials ⚠️
central/views/platformcve/view_impl.go 81.81% 0 Missing and 4 partials ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##           master   #17141   +/-   ##
=======================================
  Coverage   49.25%   49.26%           
=======================================
  Files        2727     2727           
  Lines      205788   205815   +27     
=======================================
+ Hits       101368   101390   +22     
- Misses      96885    96899   +14     
+ Partials     7535     7526    -9     
Flag Coverage Δ
go-unit-tests 49.26% <58.36%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@janisz janisz force-pushed the optimizew_interating_over_results branch from 4a55d62 to b6fd77a Compare October 6, 2025 14:59
@janisz janisz changed the title WIP: fix: Optimize slice preallocation for database query results ROX-31175: Prefer RunSelectRequestForSchemaFn Oct 7, 2025
@janisz janisz requested a review from dashrews78 October 7, 2025 15:22
Copy link
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey there - I've reviewed your changes and they look great!

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location> `pkg/search/postgres/select_query_test.go:1032` </location>
<code_context>
 	var err error
 	switch tc.resultStruct.(type) {
 	case Struct1:
-		results, err = pgSearch.RunSelectRequestForSchema[Struct1](ctx, testDB.DB, schema.TestStructsSchema, tc.q)
+		var structs []*Struct1
+		err = pgSearch.RunSelectRequestForSchemaFn[Struct1](ctx, testDB.DB, schema.TestStructsSchema, tc.q, func(r *Struct1) error {
</code_context>

<issue_to_address>
**suggestion (testing):** Tests have been updated to use RunSelectRequestForSchemaFn, but edge cases for the callback function are not covered.

Add tests where the callback returns a non-nil error to confirm that RunSelectRequestForSchemaFn properly handles and propagates callback errors.

Suggested implementation:

```golang
	var err error
	switch tc.resultStruct.(type) {
	case Struct1:
		var structs []*Struct1
		// Normal callback test
		err = pgSearch.RunSelectRequestForSchemaFn[Struct1](ctx, testDB.DB, schema.TestStructsSchema, tc.q, func(r *Struct1) error {
			structs = append(structs, r)
			return nil
		})
		results = structs

		// Edge case: callback returns error
		expectedErr := errors.New("callback error")
		errCallback := pgSearch.RunSelectRequestForSchemaFn[Struct1](ctx, testDB.DB, schema.TestStructsSchema, tc.q, func(r *Struct1) error {
			return expectedErr
		})
		if errCallback == nil || !errors.Is(errCallback, expectedErr) {
			t.Errorf("expected callback error to be propagated, got %v", errCallback)
		}

	case Struct2:
		var structs []*Struct2
		// Normal callback test
		err = pgSearch.RunSelectRequestForSchemaFn[Struct2](ctx, testDB.DB, schema.TestStructsSchema, tc.q, func(r *Struct2) error {
			structs = append(structs, r)
			return nil
		})
		results = structs

		// Edge case: callback returns error
		expectedErr := errors.New("callback error")
		errCallback := pgSearch.RunSelectRequestForSchemaFn[Struct2](ctx, testDB.DB, schema.TestStructsSchema, tc.q, func(r *Struct2) error {
			return expectedErr
		})
		if errCallback == nil || !errors.Is(errCallback, expectedErr) {
			t.Errorf("expected callback error to be propagated, got %v", errCallback)
		}

```

- Ensure that `errors` and `testing` packages are imported at the top of the file if not already present:
  ```go
  import (
      "errors"
      "testing"
      // other imports...
  )
  ```
- If this code is inside a loop or table-driven test, you may want to use unique error messages or subtests for clarity.
</issue_to_address>

### Comment 2
<location> `pkg/search/postgres/select.go:171` </location>
<code_context>
 	return parsedQuery, nil
 }

+func retryableRunSelectOneForSchema[T any](ctx context.Context, db postgres.DB, query *query) (*T, error) {
+	if len(query.SelectedFields) == 0 {
+		return nil, errors.New("select fields required for select query")
</code_context>

<issue_to_address>
**issue (complexity):** Consider introducing a shared helper function to unify repeated query execution and error handling logic.

```suggestion
Introduce a shared helper to collapse the repeated “check-fields → Retry2 → tracedQuery → scan” boilerplate. For example:

```go
// runSelectWithRetry wraps panic recovery, retry logic, tracedQuery and scanning.
func runSelectWithRetry[T any](
    ctx context.Context,
    db postgres.DB,
    query *query,
    scanFn func(rows pgx.Rows) error,
) error {
    if len(query.SelectedFields) == 0 {
        return errors.New("select fields required for select query")
    }

    // convert panics to errors:
    var panicErr error
    defer func() {
        if r := recover(); r != nil {
            panicErr = fmt.Errorf("unexpected panic: %v", r)
        }
    }()

    return pgutils.Retry2(ctx, func() error {
        if panicErr != nil {
            return panicErr
        }
        sql := query.AsSQL()
        rows, err := tracedQuery(ctx, db, sql, query.Data...)
        if err != nil {
            return errors.Wrapf(err, "executing %s", sql)
        }
        defer rows.Close()
        return scanFn(rows)
    })
}
```

Then collapse your two helpers into thin wrappers:

```go
func retryableRunSelectOneForSchema[T any](ctx context.Context, db postgres.DB, query *query) (*T, error) {
    var result T
    err := runSelectWithRetry(ctx, db, query, func(rows pgx.Rows) error {
        return scanAPI.ScanOne(&result, rows)
    })
    return &result, pgutils.ErrNilIfNoRows(err)
}

func retryableRunSelectRequestForSchemaFn[T any](
    ctx context.Context,
    db postgres.DB,
    query *query,
    fn func(*T) error,
) error {
    return runSelectWithRetry(ctx, db, query, func(rows pgx.Rows) error {
        for rows.Next() {
            var elem T
            if err := scanAPI.Scan(&elem, rows); err != nil {
                return err
            }
            if err := fn(&elem); err != nil {
                return err
            }
        }
        return nil
    })
}
```

This removes duplicated checks, panic-to-error, retry logic, query execution and defer-Close calls, while preserving all existing behavior.
</issue_to_address>

### Comment 3
<location> `central/reports/scheduler/v2/reportgenerator/report_gen_impl.go:261` </location>
<code_context>
-			deployedImagesQueryParts.Schema, query)
+		err = pgSearch.RunSelectRequestForSchemaFn[ImageCVEQueryResponse](reportGenCtx, rg.db,
+			deployedImagesQueryParts.Schema, query, func(r *ImageCVEQueryResponse) error {
+				cveResponses = append(cveResponses, r)
+				numDeployedImageResults++
+				return nil
</code_context>

<issue_to_address>
**issue (complexity):** Consider refactoring to use a shared helper function for streaming query results and measuring lengths before and after appending, rather than maintaining per-row counters and duplicate closures.

Here’s a small refactor that preserves the streaming callback but drops your per‐row counters and duplicate closures. Instead you measure lengths before/after the callback and compute “new” rows in one go:

```go
// helper to stream into a slice
func streamImageCVEs(
    ctx context.Context,
    db DB,
    schema string,
    query search.Query,
) ([]*ImageCVEQueryResponse, error) {
    var out []*ImageCVEQueryResponse
    err := pgSearch.RunSelectRequestForSchemaFn[ImageCVEQueryResponse](
        ctx, db, schema, query,
        func(r *ImageCVEQueryResponse) error {
            out = append(out, r)
            return nil
        },
    )
    return out, err
}
```

Then in your main logic:

```go
// Deployed
start := len(cveResponses)
if filterOnImageType(..., DEPLOYED) {
    deployed, err := streamImageCVEs(reportGenCtx, rg.db,
        deployedImagesQueryParts.Schema, query)
    if err != nil {
        return nil, errors.Wrap(err, "…deployed images")
    }
    numDeployedImageResults = len(deployed)
    cveResponses = append(cveResponses, deployed...)
}

// Watched
start = len(cveResponses)
if filterOnImageType(..., WATCHED) && len(watchedImages) > 0 {
    watched, err := streamImageCVEs(reportGenCtx, rg.db,
        watchedImagesQueryParts.Schema, watchedQuery)
    if err != nil {
        return nil, errors.Wrap(err, "…watched images")
    }
    numWatchedImageResults = len(watched)
    cveResponses = append(cveResponses, watched...)
}
```

Advantages:

  * No per‐row counter in the closure
  * Shared helper for both queries
  * Clear “before/after” length measurement rather than manual increments
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Copy link
Contributor

@dashrews78 dashrews78 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. (Sorry I thought I approved this days ago. )

janisz and others added 7 commits March 23, 2026 13:55
- Replace deprecated RunSelectRequestForSchema with RunSelectOneForSchema for count queries
- Updated single-result count patterns in:
  - central/views/platformcve/view_impl.go (3 methods)
  - central/views/nodecve/view_impl.go (1 method)
  - central/views/imagecve/view_impl.go (1 method)
  - central/views/imagecveflat/view_impl.go (1 method)
  - central/complianceoperator/v2/checkresults/datastore/datastore_impl.go (4 methods)
- Simplified error handling by removing length checks since RunSelectOneForSchema returns nil for no results
- Removed unused imports (errors, utils packages)

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
…emaFn

- Update multiple-result patterns in view implementations:
  - central/views/platformcve/view_impl.go: Get() and GetClusterIDs() methods
  - central/views/nodecve/view_impl.go: Get() and GetNodes() methods
  - central/views/imagecve/view_impl.go: Get() method and fixed CountBySeverity()
- Convert from results slice iteration to callback-based processing
- Memory efficiency improvements by avoiding intermediate slice allocation
- Remove unused imports (errors, utils packages)

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
…sing patterns

- Replace RunSelectRequestForSchema with RunSelectRequestForSchemaFn across 24 files
- Move result processing directly into callbacks to reduce memory allocations
- Eliminate intermediate slice collections when data is immediately processed
- Fix variable redeclaration errors in compliance operator and test files
- Optimize view implementations to avoid unnecessary GC pressure

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Tomasz Janiszewski <tomek@redhat.com>
…calls

Optimize memory allocation by preallocating slice capacity using pagination limits:
- Added paginated import to view_impl.go files
- Updated slice allocations to use paginated.GetLimit() with default capacity of 100
- Applied optimization to 9 files across view implementations and datastore layers

Benefits:
- Reduces memory allocations and GC pressure
- Uses query pagination limits when available
- Provides sensible default for unlimited queries

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Tomasz Janiszewski <tomek@redhat.com>
Signed-off-by: Tomasz Janiszewski <tomek@redhat.com>
Fixed variable shadowing issues where 'err' was incorrectly declared
in some compliance operator datastore functions after the rebase.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@janisz janisz force-pushed the optimizew_interating_over_results branch from bd5aba8 to daf8539 Compare March 23, 2026 13:16
@openshift-ci
Copy link

openshift-ci bot commented Mar 23, 2026

@janisz: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/ocp-4-12-qa-e2e-tests daf8539 link false /test ocp-4-12-qa-e2e-tests
ci/prow/ocp-4-21-qa-e2e-tests daf8539 link false /test ocp-4-21-qa-e2e-tests
ci/prow/ocp-4-20-qa-e2e-tests daf8539 link false /test ocp-4-20-qa-e2e-tests

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@janisz
Copy link
Contributor Author

janisz commented Mar 23, 2026

/retest

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants