Skip to content

ROX-33188: Wire Sensor label providers for label-based policy scoping#19359

Open
AlexVulaj wants to merge 1 commit intomasterfrom
AlexVulaj/ROX-33188-sensor-label-providers
Open

ROX-33188: Wire Sensor label providers for label-based policy scoping#19359
AlexVulaj wants to merge 1 commit intomasterfrom
AlexVulaj/ROX-33188-sensor-label-providers

Conversation

@AlexVulaj
Copy link
Contributor

@AlexVulaj AlexVulaj commented Mar 10, 2026

Description

Note, this PR's end-to-end functionality also relies on PR #19530 . They were written and tested together, but split to simplify the review process.

Wire Sensor's in-memory stores as label providers for policy evaluation, enabling full label-based policy scoping (cluster + namespace labels) in both runtime detection and admission control.

Policies with cluster_label and namespace_label scopes now work in Sensor's runtime detection and admission control.

User-facing documentation

Testing and quality

  • the change is production ready: the change is GA, or otherwise the functionality is gated by a feature flag
  • CI results are inspected

Automated testing

  • added unit tests
  • added e2e tests
  • added regression tests
  • added compatibility tests
  • modified existing tests

How I validated my change

Added comprehensive unit tests.

Executed comprehensive manual test plan covering three workflows:

Workflow 1: Admission control deploy-time enforcement (4/4 passed)

  • Cluster label scoping (env=prod) blocks privileged containers
  • Namespace label scoping (team=backend) blocks privileged containers
  • Combined cluster + namespace label scoping works correctly
  • Dynamic cluster label changes (prod→dev) and namespace filtering verified

Workflow 2: Runtime detection (Sensor) (3/3 passed)

  • Cluster-scoped runtime policy triggers on bash execution in prod cluster
  • Namespace-scoped runtime policy triggers only in matching namespace (backend)
  • Namespace filtering works correctly (frontend namespace doesn't trigger backend policy)

Workflow 3: Verification checks (2/2 passed)

  • Cluster labels flow from SecuredCluster CR → ConfigMap → Admission Control
  • No errors in Sensor or admission-control logs related to label providers or scoping

Test setup:

  • ACS deployed via roxie with feature flag ROX_LABEL_BASED_POLICY_SCOPING=true
  • Image: 4.11.x-348-g89eb433197 (includes namespace sync and enrichment fixes)
  • Cluster labels set via SecuredCluster CR (env=prod/dev, region=us-east-1)
  • Namespace labels set via kubectl (team=backend/frontend, tier=app)
  • 5 test policies created via API (3 deploy-time with SCALE_TO_ZERO_ENFORCEMENT, 2 runtime)
  • Tested both admission control (deploy-time) and Sensor (runtime) label-based scoping

@openshift-ci
Copy link

openshift-ci bot commented Mar 10, 2026

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

Copy link
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've left some high level feedback:

  • Both namespace label lookup methods (NamespaceStore.LookupNamespaceLabelsByID and namespaceStore.LookupNamespaceLabelsByID) return the underlying label map directly; consider returning a cloned map to avoid accidental mutation of shared state by callers.
  • In settingsToConfigMap, you always marshal/compress cluster labels even when the map is nil or empty; consider skipping ClusterLabelsGZDataKey in that case to reduce configmap size and align with the len(data) == 0 fast-path in decompressAndUnmarshalClusterLabels.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- Both namespace label lookup methods (`NamespaceStore.LookupNamespaceLabelsByID` and `namespaceStore.LookupNamespaceLabelsByID`) return the underlying label map directly; consider returning a cloned map to avoid accidental mutation of shared state by callers.
- In `settingsToConfigMap`, you always marshal/compress cluster labels even when the map is nil or empty; consider skipping `ClusterLabelsGZDataKey` in that case to reduce configmap size and align with the `len(data) == 0` fast-path in `decompressAndUnmarshalClusterLabels`.

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@rhacs-bot
Copy link
Contributor

rhacs-bot commented Mar 10, 2026

Images are ready for the commit at 68171b8.

To use with deploy scripts, first export MAIN_IMAGE_TAG=4.11.x-395-g68171b8f5a.

@codecov
Copy link

codecov bot commented Mar 11, 2026

Codecov Report

❌ Patch coverage is 21.56863% with 40 lines in your changes missing coverage. Please review.
✅ Project coverage is 49.24%. Comparing base (c2045aa) to head (68171b8).
⚠️ Report is 7 commits behind head on master.

Files with missing lines Patch % Lines
...ommon/admissioncontroller/settings_manager_impl.go 0.00% 18 Missing ⚠️
...r/kubernetes/listener/resources/namespace_store.go 52.38% 10 Missing ⚠️
...or/kubernetes/listener/resources/store_provider.go 0.00% 10 Missing ⚠️
sensor/common/detector/detector.go 0.00% 2 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##           master   #19359   +/-   ##
=======================================
  Coverage   49.23%   49.24%           
=======================================
  Files        2727     2727           
  Lines      205764   205825   +61     
=======================================
+ Hits       101304   101354   +50     
- Misses      96921    96934   +13     
+ Partials     7539     7537    -2     
Flag Coverage Δ
go-unit-tests 49.24% <21.56%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@AlexVulaj AlexVulaj force-pushed the AlexVulaj/ROX-33188-sensor-label-providers branch 7 times, most recently from 97c1c1c to 89eb433 Compare March 17, 2026 14:58
@AlexVulaj AlexVulaj self-assigned this Mar 17, 2026
@AlexVulaj AlexVulaj marked this pull request as ready for review March 17, 2026 18:44
@AlexVulaj AlexVulaj requested a review from a team as a code owner March 17, 2026 18:44
Copy link
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 3 issues, and left some high level feedback:

  • In enrichDeploymentWithNamespaceID, logging a warning for every deployment that lacks a namespace ID could be very noisy in misconfigured clusters; consider downgrading this to debug level or adding some form of rate limiting so logs remain usable.
  • Both ClusterLabelProviderAdapter.GetClusterLabels and the namespace label providers return the underlying label maps directly; if there is any chance that callers might mutate these maps, it would be safer to return shallow copies to avoid data races or unintended shared state between the stores and the policy engine.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- In `enrichDeploymentWithNamespaceID`, logging a warning for every deployment that lacks a namespace ID could be very noisy in misconfigured clusters; consider downgrading this to debug level or adding some form of rate limiting so logs remain usable.
- Both `ClusterLabelProviderAdapter.GetClusterLabels` and the namespace label providers return the underlying label maps directly; if there is any chance that callers might mutate these maps, it would be safer to return shallow copies to avoid data races or unintended shared state between the stores and the policy engine.

## Individual Comments

### Comment 1
<location path="sensor/common/admissioncontroller/settings_manager_impl.go" line_range="163-160" />
<code_context>
 		})
 	}
+
+	for _, ns := range p.namespaces.GetAll() {
+		ret = append(ret, &sensor.AdmCtrlUpdateResourceRequest{
+			Action: central.ResourceAction_CREATE_RESOURCE,
+			Resource: &sensor.AdmCtrlUpdateResourceRequest_Namespace{
+				Namespace: ns,
+			},
+		})
+	}
 	return ret
</code_context>
<issue_to_address>
**issue (bug_risk):** Guard against nil NamespaceStore to avoid panics in GetResourcesForSync

This adds a dependency on `p.namespaces` being non-nil. While `NewSettingsManager` wires this correctly, tests or other constructors could still pass a nil `NamespaceStore`, causing `p.namespaces.GetAll()` to panic. Please either guard this loop with a nil check or enforce a non-nil `NamespaceStore` invariant in the constructor.
</issue_to_address>

### Comment 2
<location path="sensor/admission-control/manager/manager_impl.go" line_range="45" />
<code_context>
 	)
 )

+// clusterLabelProviderAdapter adapts admission control's static cluster labels to the ClusterLabelProvider interface.
+type clusterLabelProviderAdapter struct {
+	labels map[string]string
</code_context>
<issue_to_address>
**issue (complexity):** Consider reusing the shared label provider adapters from the common package instead of defining local adapter types to keep the abstraction centralized and avoid duplication.

You can drop the local adapter types and reuse the shared adapters from `sensor/common/labelproviders` to avoid duplicate concepts and keep the wiring local.

```go
// Remove these local types entirely:
// - clusterLabelProviderAdapter
// - namespaceLabelProviderAdapter
```

Then, in the wiring code, construct the providers via the shared helpers:

```go
import (
    // ...
    "github.com/stackrox/rox/sensor/common/labelproviders"
)

// ...

// Wire label providers for admission control policies.
clusterLabelProvider := labelproviders.NewClusterLabelProviderAdapter(
    newSettings.GetClusterLabels(),
)

namespaceLabelProvider := labelproviders.NewNamespaceLabelProviderAdapter(
    // lookupFunc: namespaceID -> (labels, found)
    m.namespaces.LookupNamespaceLabelsByID,
)

allK8sEventPolicies := detection.NewPolicySet(clusterLabelProvider, namespaceLabelProvider)
deployFieldK8sEventPolicies, k8sEventOnlyPolicies :=
    detection.NewPolicySet(clusterLabelProvider, namespaceLabelProvider),
    detection.NewPolicySet(clusterLabelProvider, namespaceLabelProvider)

// ...

// Wire label providers for deploy-time policies.
specOnlyPolicies := detection.NewPolicySet(clusterLabelProvider, namespaceLabelProvider)
enrichmentRequiredPolicies := detection.NewPolicySet(clusterLabelProvider, namespaceLabelProvider)

compiled, err := detection.CompilePolicy(policy, clusterLabelProvider, namespaceLabelProvider)
```

This keeps all functionality intact while eliminating redundant adapter implementations and centralizing the label-provider abstraction in one package.
</issue_to_address>

### Comment 3
<location path="sensor/common/labelproviders/providers.go" line_range="10" />
<code_context>
+)
+
+// ClusterLabelProviderAdapter adapts Sensor's cluster labels store to the ClusterLabelProvider interface.
+type ClusterLabelProviderAdapter struct {
+	store *clusterlabels.Store
+}
</code_context>
<issue_to_address>
**issue (complexity):** Consider refactoring these label adapter structs into shared function-based adapters so that all callers use a single, unified abstraction for cluster and namespace labels.

You can reduce the “fragmented abstraction surface” by making this package the *only* place that defines these adapters and by simplifying the adapters to be reusable function adapters. That lets `sensor/admission-control/manager/manager_impl.go` stop defining its own nearly identical types.

### 1. Replace struct adapters with function adapters

Instead of a dedicated `NamespaceLabelProviderAdapter` struct that wraps a `lookupFunc`, make the function type itself implement the interface. This removes one level of indirection and makes it trivial for callers (like the manager) to use existing functions directly.

```go
package labelproviders

import (
    "context"

    "github.com/stackrox/rox/sensor/common/clusterlabels"
)

// --- Cluster labels ---

// ClusterLabelProviderFunc adapts a function to the ClusterLabelProvider interface.
type ClusterLabelProviderFunc func(ctx context.Context, clusterID string) (map[string]string, error)

func (f ClusterLabelProviderFunc) GetClusterLabels(ctx context.Context, clusterID string) (map[string]string, error) {
    return f(ctx, clusterID)
}

// NewClusterLabelProviderAdapter adapts clusterlabels.Store to ClusterLabelProvider.
func NewClusterLabelProviderAdapter(store *clusterlabels.Store) ClusterLabelProviderFunc {
    return func(_ context.Context, _ string) (map[string]string, error) {
        return store.Get(), nil
    }
}

// --- Namespace labels ---

// NamespaceLabelProviderFunc adapts a function to the NamespaceLabelProvider interface.
type NamespaceLabelProviderFunc func(ctx context.Context, namespaceID string) (map[string]string, error)

func (f NamespaceLabelProviderFunc) GetNamespaceLabels(ctx context.Context, namespaceID string) (map[string]string, error) {
    return f(ctx, namespaceID)
}

// NewNamespaceLabelProviderAdapter adapts a lookup func to NamespaceLabelProvider.
func NewNamespaceLabelProviderAdapter(lookupFunc func(string) (map[string]string, bool)) NamespaceLabelProviderFunc {
    return func(_ context.Context, namespaceID string) (map[string]string, error) {
        labels, found := lookupFunc(namespaceID)
        if !found {
            return nil, nil
        }
        return labels, nil
    }
}
```

### 2. Remove local adapter types in the manager

In `sensor/admission-control/manager/manager_impl.go`, replace local adapter structs with the function types above and/or directly use the constructors:

```go
// Before: local adapter type in manager_impl.go
type namespaceLabelProviderAdapter struct {
    lookupFunc func(string) (map[string]string, bool)
}

func (a *namespaceLabelProviderAdapter) GetNamespaceLabels(ctx context.Context, nsID string) (map[string]string, error) {
    labels, found := a.lookupFunc(nsID)
    if !found {
        return nil, nil
    }
    return labels, nil
}
```

```go
// After: use shared adapter
import "github.com/stackrox/rox/sensor/admission-control/labelproviders"

// wherever the provider is created:
nsProvider := labelproviders.NewNamespaceLabelProviderAdapter(manager.lookupNamespaceLabels)
```

Do the same for cluster labels:

```go
// Before: local cluster adapter type
type clusterLabelProviderAdapter struct {
    store *clusterlabels.Store
}

func (a *clusterLabelProviderAdapter) GetClusterLabels(ctx context.Context, _ string) (map[string]string, error) {
    return a.store.Get(), nil
}
```

```go
// After: use shared adapter
import "github.com/stackrox/rox/sensor/admission-control/labelproviders"

clusterProvider := labelproviders.NewClusterLabelProviderAdapter(clusterLabelStore)
```

### 3. Outcome

- All adapter logic lives in `labelproviders`, eliminating parallel, nearly identical types.
- Callers (including the manager) just pass their existing lookup functions/stores into the shared constructors.
- The abstraction surface is unified: one way to adapt label sources to the interfaces, no duplicated adapter types.
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@AlexVulaj AlexVulaj force-pushed the AlexVulaj/ROX-33188-sensor-label-providers branch 9 times, most recently from 3a3ffc4 to 26bd430 Compare March 19, 2026 15:52
@AlexVulaj
Copy link
Contributor Author

/retest

Copy link
Contributor

@vikin91 vikin91 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed mainly the sensor part skipping all hardcore admission controller (maybe I left a comment here or there) - thus not giving ✅ .

The Sensor part looks good! I am a bit concerned about the number of copies that we create to the namespaces objects - I left detailed comments. Other than that, the sensor part looks good!

Comment on lines +34 to +37
copy := make(map[string]string, len(labels))
for k, v := range labels {
copy[k] = v
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe https://pkg.go.dev/maps#Clone would fit here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this change required? I remember providing feedback about this during the addition of these stores. I think these can be reverted.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll revert this, it's unnecessary overhead for internal code.

Comment on lines +12 to +13
tests := []struct {
name string
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Observation: We often do tests := map[string]struct{ so that the key holds the name. Maybe you want to join the club?

Comment on lines +25 to +30
map<string, string> cluster_labels = 9;
}

// ClusterLabels is a simple wrapper for cluster labels map to enable serialization.
message ClusterLabels {
map<string, string> labels = 1;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am scratching my head... shouldn't it be then:

Suggested change
map<string, string> cluster_labels = 9;
}
// ClusterLabels is a simple wrapper for cluster labels map to enable serialization.
message ClusterLabels {
map<string, string> labels = 1;
ClusterLabels cluster_labels = 9;
}
// ClusterLabels is a simple wrapper for cluster labels map to enable serialization.
message ClusterLabels {
map<string, string> labels = 1;

?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had originally used map<string, string> in the Settings message, and then as I went forward with testing realized it was becoming a pain to serialize/deserialize a map<string, string>, so I made the wrapper ClusterLabels to use in Go code. That said, you're right that it looks weird to have that wrapper and not use it in the proto too. I'll make this change (and the appropriate code changes as well).

Comment on lines 75 to +78
func New(clusterID clusterIDPeekWaiter, enforcer enforcer.Enforcer, admCtrlSettingsMgr admissioncontroller.SettingsManager,
deploymentStore store.DeploymentStore, serviceAccountStore store.ServiceAccountStore, cache cache.Image, auditLogEvents chan *sensor.AuditEvents,
auditLogUpdater updater.Component, networkPolicyStore store.NetworkPolicyStore, registryStore *registry.Store, localScan *scan.LocalScan, nodeStore store.NodeStore) Detector {
auditLogUpdater updater.Component, networkPolicyStore store.NetworkPolicyStore, registryStore *registry.Store, localScan *scan.LocalScan, nodeStore store.NodeStore,
clusterLabelProvider scopecomp.ClusterLabelProvider, namespaceLabelProvider scopecomp.NamespaceLabelProvider) Detector {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

15 args, wow :)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ikr :-( maybe a builder pattern id due in a separate tight refactor.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah definitely - I didn't love adding more params to this already giant signature, but I agree that a builder pattern refactor should be saved for a future cleanup.

Comment on lines +12 to +13
namespaces map[string]*storage.NamespaceMetadata
namespacesByID map[string]*storage.NamespaceMetadata
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
namespaces map[string]*storage.NamespaceMetadata
namespacesByID map[string]*storage.NamespaceMetadata
namespacesByName map[string]*storage.NamespaceMetadata
namespacesByID map[string]*storage.NamespaceMetadata

Maybe to save some memory (memory is gold for Sensor), we should rather do:

map name -> ID
map ID -> object

(depending on how big are the objects)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both maps currently store pointers to the same cloned object (line 36 below), so there's no duplication of NamespaceMetadata itself - just two pointers. That is, unless I'm missing something about map bucket overhead?


var ret []*storage.NamespaceMetadata
for _, ns := range n.namespaces {
ret = append(ret, ns.CloneVT())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see we are cloning twice, once to store the namespace in the store and the second time here to prevent the caller of this func to modify the object.

Do we really need to do this? I know that there are not so many namespaces in a cluster, but I would be for avoiding such things in Sensor unless really necessary.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah this is probably overly defensive. I'll remove both clones to match the pattern we're using in clusterlabels store.

@@ -0,0 +1,52 @@
package labelproviders
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you evaluated the admission controller manager to embed clusterlabelprovider and namespacelabelprovider interfaces? If I am seeing this correctly, the adapters are not needed and there is a lot of opportunity for simplification.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Honestly great suggestion, and not too hard to change. I'll go forward with this.

@AlexVulaj AlexVulaj force-pushed the AlexVulaj/ROX-33188-sensor-label-providers branch from 9e4ca7f to 0260fd9 Compare March 20, 2026 20:10
@AlexVulaj AlexVulaj force-pushed the AlexVulaj/ROX-33188-sensor-label-providers branch 2 times, most recently from 5ec645a to ae0a45a Compare March 20, 2026 20:32
@AlexVulaj AlexVulaj force-pushed the AlexVulaj/ROX-33188-sensor-label-providers branch from ae0a45a to 68171b8 Compare March 20, 2026 20:55
@openshift-ci
Copy link

openshift-ci bot commented Mar 20, 2026

@AlexVulaj: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/gke-operator-e2e-tests 0260fd9 link false /test gke-operator-e2e-tests
ci/prow/gke-scanner-v4-install-tests 0260fd9 link false /test gke-scanner-v4-install-tests
ci/prow/gke-qa-e2e-tests 0260fd9 link false /test gke-qa-e2e-tests
ci/prow/ocp-4-12-scanner-v4-install-tests 0260fd9 link false /test ocp-4-12-scanner-v4-install-tests
ci/prow/ocp-4-12-qa-e2e-tests 0260fd9 link false /test ocp-4-12-qa-e2e-tests
ci/prow/ocp-4-12-nongroovy-e2e-tests 0260fd9 link false /test ocp-4-12-nongroovy-e2e-tests
ci/prow/ocp-4-21-operator-e2e-tests 0260fd9 link false /test ocp-4-21-operator-e2e-tests
ci/prow/ocp-4-21-nongroovy-e2e-tests 0260fd9 link false /test ocp-4-21-nongroovy-e2e-tests
ci/prow/ocp-4-20-nongroovy-e2e-tests 0260fd9 link false /test ocp-4-20-nongroovy-e2e-tests
ci/prow/ocp-4-21-scanner-v4-install-tests 0260fd9 link false /test ocp-4-21-scanner-v4-install-tests
ci/prow/ocp-4-20-qa-e2e-tests 0260fd9 link false /test ocp-4-20-qa-e2e-tests
ci/prow/ocp-4-21-qa-e2e-tests 0260fd9 link false /test ocp-4-21-qa-e2e-tests
ci/prow/ocp-4-20-scanner-v4-install-tests 0260fd9 link false /test ocp-4-20-scanner-v4-install-tests
ci/prow/ocp-4-20-operator-e2e-tests 0260fd9 link false /test ocp-4-20-operator-e2e-tests
ci/prow/ocp-4-12-operator-e2e-tests 0260fd9 link false /test ocp-4-12-operator-e2e-tests

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants