Description

This PR is adding the GetByQueryFn function to storage. This way of handling rows returned by DB helps with:

• faster processing because callbacks will be called as rows are received from DB over the network layer
• reduced memory usage because intermediary objects have a shorter lifetime, and GC can clear them faster

During the evaluation of improvements, I have done the following testing:

1. I have used default benchmark tests, but they didn't show any significant improvement (~4%) in memory allocations.
2. More specific testing of impact on GC and ability to free memory

For 2. I did the following:

• Parallel query execution The test spawns multiple goroutines to simulate memory-intensive workloads. This workload runs queries in a loop.
• GC triggering: A separate goroutine forces garbage collection every second to observe GC effects.
• Memory tracking: Another goroutine monitors and records the peak memory usage using runtime.ReadMemStats().

This test shows the following results (5 executions on each version):
OLD

    bench_test.go:151: Maximum memory usage: 598.23 MB
--- PASS: TestMemoryUsage (51.60s)

    bench_test.go:151: Maximum memory usage: 595.63 MB
--- PASS: TestMemoryUsage (52.03s)

    bench_test.go:151: Maximum memory usage: 578.43 MB
--- PASS: TestMemoryUsage (52.46s)

    bench_test.go:151: Maximum memory usage: 584.55 MB
--- PASS: TestMemoryUsage (53.32s)

    bench_test.go:151: Maximum memory usage: 587.40 MB
--- PASS: TestMemoryUsage (50.92s)

NEW

    bench_test.go:151: Maximum memory usage: 406.72 MB
--- PASS: TestMemoryUsage (51.33s)

    bench_test.go:151: Maximum memory usage: 307.58 MB
--- PASS: TestMemoryUsage (51.02s)

    bench_test.go:151: Maximum memory usage: 390.79 MB
--- PASS: TestMemoryUsage (50.57s)

    bench_test.go:151: Maximum memory usage: 329.87 MB
--- PASS: TestMemoryUsage (50.78s)

    bench_test.go:151: Maximum memory usage: 311.44 MB
--- PASS: TestMemoryUsage (51.74s)

Memory levels are way better with callbacks. The memory level was 70% of the initial code.

User-facing documentation

• CHANGELOG is updated OR update is not needed
• documentation PR is created and is linked above OR is not needed

Testing and quality

• the change is production ready: the change is GA or otherwise the functionality is gated by a feature flag
• CI results are inspected

Automated testing

• no additional tests

How I validated my change

CI pipeline and local bench testing