ROX-33188: Wire Sensor label providers for label-based policy scoping#19359
ROX-33188: Wire Sensor label providers for label-based policy scoping#19359
Conversation
|
Skipping CI for Draft Pull Request. |
There was a problem hiding this comment.
Hey - I've left some high level feedback:
- Both namespace label lookup methods (
NamespaceStore.LookupNamespaceLabelsByIDandnamespaceStore.LookupNamespaceLabelsByID) return the underlying label map directly; consider returning a cloned map to avoid accidental mutation of shared state by callers. - In
settingsToConfigMap, you always marshal/compress cluster labels even when the map is nil or empty; consider skippingClusterLabelsGZDataKeyin that case to reduce configmap size and align with thelen(data) == 0fast-path indecompressAndUnmarshalClusterLabels.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- Both namespace label lookup methods (`NamespaceStore.LookupNamespaceLabelsByID` and `namespaceStore.LookupNamespaceLabelsByID`) return the underlying label map directly; consider returning a cloned map to avoid accidental mutation of shared state by callers.
- In `settingsToConfigMap`, you always marshal/compress cluster labels even when the map is nil or empty; consider skipping `ClusterLabelsGZDataKey` in that case to reduce configmap size and align with the `len(data) == 0` fast-path in `decompressAndUnmarshalClusterLabels`.Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
|
Images are ready for the commit at 68171b8. To use with deploy scripts, first |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #19359 +/- ##
=======================================
Coverage 49.23% 49.24%
=======================================
Files 2727 2727
Lines 205764 205825 +61
=======================================
+ Hits 101304 101354 +50
- Misses 96921 96934 +13
+ Partials 7539 7537 -2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
97c1c1c to
89eb433
Compare
There was a problem hiding this comment.
Hey - I've found 3 issues, and left some high level feedback:
- In
enrichDeploymentWithNamespaceID, logging a warning for every deployment that lacks a namespace ID could be very noisy in misconfigured clusters; consider downgrading this to debug level or adding some form of rate limiting so logs remain usable. - Both
ClusterLabelProviderAdapter.GetClusterLabelsand the namespace label providers return the underlying label maps directly; if there is any chance that callers might mutate these maps, it would be safer to return shallow copies to avoid data races or unintended shared state between the stores and the policy engine.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- In `enrichDeploymentWithNamespaceID`, logging a warning for every deployment that lacks a namespace ID could be very noisy in misconfigured clusters; consider downgrading this to debug level or adding some form of rate limiting so logs remain usable.
- Both `ClusterLabelProviderAdapter.GetClusterLabels` and the namespace label providers return the underlying label maps directly; if there is any chance that callers might mutate these maps, it would be safer to return shallow copies to avoid data races or unintended shared state between the stores and the policy engine.
## Individual Comments
### Comment 1
<location path="sensor/common/admissioncontroller/settings_manager_impl.go" line_range="163-160" />
<code_context>
})
}
+
+ for _, ns := range p.namespaces.GetAll() {
+ ret = append(ret, &sensor.AdmCtrlUpdateResourceRequest{
+ Action: central.ResourceAction_CREATE_RESOURCE,
+ Resource: &sensor.AdmCtrlUpdateResourceRequest_Namespace{
+ Namespace: ns,
+ },
+ })
+ }
return ret
</code_context>
<issue_to_address>
**issue (bug_risk):** Guard against nil NamespaceStore to avoid panics in GetResourcesForSync
This adds a dependency on `p.namespaces` being non-nil. While `NewSettingsManager` wires this correctly, tests or other constructors could still pass a nil `NamespaceStore`, causing `p.namespaces.GetAll()` to panic. Please either guard this loop with a nil check or enforce a non-nil `NamespaceStore` invariant in the constructor.
</issue_to_address>
### Comment 2
<location path="sensor/admission-control/manager/manager_impl.go" line_range="45" />
<code_context>
)
)
+// clusterLabelProviderAdapter adapts admission control's static cluster labels to the ClusterLabelProvider interface.
+type clusterLabelProviderAdapter struct {
+ labels map[string]string
</code_context>
<issue_to_address>
**issue (complexity):** Consider reusing the shared label provider adapters from the common package instead of defining local adapter types to keep the abstraction centralized and avoid duplication.
You can drop the local adapter types and reuse the shared adapters from `sensor/common/labelproviders` to avoid duplicate concepts and keep the wiring local.
```go
// Remove these local types entirely:
// - clusterLabelProviderAdapter
// - namespaceLabelProviderAdapter
```
Then, in the wiring code, construct the providers via the shared helpers:
```go
import (
// ...
"github.com/stackrox/rox/sensor/common/labelproviders"
)
// ...
// Wire label providers for admission control policies.
clusterLabelProvider := labelproviders.NewClusterLabelProviderAdapter(
newSettings.GetClusterLabels(),
)
namespaceLabelProvider := labelproviders.NewNamespaceLabelProviderAdapter(
// lookupFunc: namespaceID -> (labels, found)
m.namespaces.LookupNamespaceLabelsByID,
)
allK8sEventPolicies := detection.NewPolicySet(clusterLabelProvider, namespaceLabelProvider)
deployFieldK8sEventPolicies, k8sEventOnlyPolicies :=
detection.NewPolicySet(clusterLabelProvider, namespaceLabelProvider),
detection.NewPolicySet(clusterLabelProvider, namespaceLabelProvider)
// ...
// Wire label providers for deploy-time policies.
specOnlyPolicies := detection.NewPolicySet(clusterLabelProvider, namespaceLabelProvider)
enrichmentRequiredPolicies := detection.NewPolicySet(clusterLabelProvider, namespaceLabelProvider)
compiled, err := detection.CompilePolicy(policy, clusterLabelProvider, namespaceLabelProvider)
```
This keeps all functionality intact while eliminating redundant adapter implementations and centralizing the label-provider abstraction in one package.
</issue_to_address>
### Comment 3
<location path="sensor/common/labelproviders/providers.go" line_range="10" />
<code_context>
+)
+
+// ClusterLabelProviderAdapter adapts Sensor's cluster labels store to the ClusterLabelProvider interface.
+type ClusterLabelProviderAdapter struct {
+ store *clusterlabels.Store
+}
</code_context>
<issue_to_address>
**issue (complexity):** Consider refactoring these label adapter structs into shared function-based adapters so that all callers use a single, unified abstraction for cluster and namespace labels.
You can reduce the “fragmented abstraction surface” by making this package the *only* place that defines these adapters and by simplifying the adapters to be reusable function adapters. That lets `sensor/admission-control/manager/manager_impl.go` stop defining its own nearly identical types.
### 1. Replace struct adapters with function adapters
Instead of a dedicated `NamespaceLabelProviderAdapter` struct that wraps a `lookupFunc`, make the function type itself implement the interface. This removes one level of indirection and makes it trivial for callers (like the manager) to use existing functions directly.
```go
package labelproviders
import (
"context"
"github.com/stackrox/rox/sensor/common/clusterlabels"
)
// --- Cluster labels ---
// ClusterLabelProviderFunc adapts a function to the ClusterLabelProvider interface.
type ClusterLabelProviderFunc func(ctx context.Context, clusterID string) (map[string]string, error)
func (f ClusterLabelProviderFunc) GetClusterLabels(ctx context.Context, clusterID string) (map[string]string, error) {
return f(ctx, clusterID)
}
// NewClusterLabelProviderAdapter adapts clusterlabels.Store to ClusterLabelProvider.
func NewClusterLabelProviderAdapter(store *clusterlabels.Store) ClusterLabelProviderFunc {
return func(_ context.Context, _ string) (map[string]string, error) {
return store.Get(), nil
}
}
// --- Namespace labels ---
// NamespaceLabelProviderFunc adapts a function to the NamespaceLabelProvider interface.
type NamespaceLabelProviderFunc func(ctx context.Context, namespaceID string) (map[string]string, error)
func (f NamespaceLabelProviderFunc) GetNamespaceLabels(ctx context.Context, namespaceID string) (map[string]string, error) {
return f(ctx, namespaceID)
}
// NewNamespaceLabelProviderAdapter adapts a lookup func to NamespaceLabelProvider.
func NewNamespaceLabelProviderAdapter(lookupFunc func(string) (map[string]string, bool)) NamespaceLabelProviderFunc {
return func(_ context.Context, namespaceID string) (map[string]string, error) {
labels, found := lookupFunc(namespaceID)
if !found {
return nil, nil
}
return labels, nil
}
}
```
### 2. Remove local adapter types in the manager
In `sensor/admission-control/manager/manager_impl.go`, replace local adapter structs with the function types above and/or directly use the constructors:
```go
// Before: local adapter type in manager_impl.go
type namespaceLabelProviderAdapter struct {
lookupFunc func(string) (map[string]string, bool)
}
func (a *namespaceLabelProviderAdapter) GetNamespaceLabels(ctx context.Context, nsID string) (map[string]string, error) {
labels, found := a.lookupFunc(nsID)
if !found {
return nil, nil
}
return labels, nil
}
```
```go
// After: use shared adapter
import "github.com/stackrox/rox/sensor/admission-control/labelproviders"
// wherever the provider is created:
nsProvider := labelproviders.NewNamespaceLabelProviderAdapter(manager.lookupNamespaceLabels)
```
Do the same for cluster labels:
```go
// Before: local cluster adapter type
type clusterLabelProviderAdapter struct {
store *clusterlabels.Store
}
func (a *clusterLabelProviderAdapter) GetClusterLabels(ctx context.Context, _ string) (map[string]string, error) {
return a.store.Get(), nil
}
```
```go
// After: use shared adapter
import "github.com/stackrox/rox/sensor/admission-control/labelproviders"
clusterProvider := labelproviders.NewClusterLabelProviderAdapter(clusterLabelStore)
```
### 3. Outcome
- All adapter logic lives in `labelproviders`, eliminating parallel, nearly identical types.
- Callers (including the manager) just pass their existing lookup functions/stores into the shared constructors.
- The abstraction surface is unified: one way to adapt label sources to the interfaces, no duplicated adapter types.
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
3a3ffc4 to
26bd430
Compare
|
/retest |
vikin91
left a comment
There was a problem hiding this comment.
I reviewed mainly the sensor part skipping all hardcore admission controller (maybe I left a comment here or there) - thus not giving ✅ .
The Sensor part looks good! I am a bit concerned about the number of copies that we create to the namespaces objects - I left detailed comments. Other than that, the sensor part looks good!
pkg/clusterlabels/clusterlabels.go
Outdated
| copy := make(map[string]string, len(labels)) | ||
| for k, v := range labels { | ||
| copy[k] = v | ||
| } |
There was a problem hiding this comment.
Maybe https://pkg.go.dev/maps#Clone would fit here?
There was a problem hiding this comment.
Why is this change required? I remember providing feedback about this during the addition of these stores. I think these can be reverted.
There was a problem hiding this comment.
I'll revert this, it's unnecessary overhead for internal code.
pkg/labelproviders/providers_test.go
Outdated
| tests := []struct { | ||
| name string |
There was a problem hiding this comment.
Observation: We often do tests := map[string]struct{ so that the key holds the name. Maybe you want to join the club?
| map<string, string> cluster_labels = 9; | ||
| } | ||
|
|
||
| // ClusterLabels is a simple wrapper for cluster labels map to enable serialization. | ||
| message ClusterLabels { | ||
| map<string, string> labels = 1; |
There was a problem hiding this comment.
I am scratching my head... shouldn't it be then:
| map<string, string> cluster_labels = 9; | |
| } | |
| // ClusterLabels is a simple wrapper for cluster labels map to enable serialization. | |
| message ClusterLabels { | |
| map<string, string> labels = 1; | |
| ClusterLabels cluster_labels = 9; | |
| } | |
| // ClusterLabels is a simple wrapper for cluster labels map to enable serialization. | |
| message ClusterLabels { | |
| map<string, string> labels = 1; |
?
There was a problem hiding this comment.
I had originally used map<string, string> in the Settings message, and then as I went forward with testing realized it was becoming a pain to serialize/deserialize a map<string, string>, so I made the wrapper ClusterLabels to use in Go code. That said, you're right that it looks weird to have that wrapper and not use it in the proto too. I'll make this change (and the appropriate code changes as well).
| func New(clusterID clusterIDPeekWaiter, enforcer enforcer.Enforcer, admCtrlSettingsMgr admissioncontroller.SettingsManager, | ||
| deploymentStore store.DeploymentStore, serviceAccountStore store.ServiceAccountStore, cache cache.Image, auditLogEvents chan *sensor.AuditEvents, | ||
| auditLogUpdater updater.Component, networkPolicyStore store.NetworkPolicyStore, registryStore *registry.Store, localScan *scan.LocalScan, nodeStore store.NodeStore) Detector { | ||
| auditLogUpdater updater.Component, networkPolicyStore store.NetworkPolicyStore, registryStore *registry.Store, localScan *scan.LocalScan, nodeStore store.NodeStore, | ||
| clusterLabelProvider scopecomp.ClusterLabelProvider, namespaceLabelProvider scopecomp.NamespaceLabelProvider) Detector { |
There was a problem hiding this comment.
Ikr :-( maybe a builder pattern id due in a separate tight refactor.
There was a problem hiding this comment.
Yeah definitely - I didn't love adding more params to this already giant signature, but I agree that a builder pattern refactor should be saved for a future cleanup.
| namespaces map[string]*storage.NamespaceMetadata | ||
| namespacesByID map[string]*storage.NamespaceMetadata |
There was a problem hiding this comment.
| namespaces map[string]*storage.NamespaceMetadata | |
| namespacesByID map[string]*storage.NamespaceMetadata | |
| namespacesByName map[string]*storage.NamespaceMetadata | |
| namespacesByID map[string]*storage.NamespaceMetadata |
Maybe to save some memory (memory is gold for Sensor), we should rather do:
map name -> ID
map ID -> object
(depending on how big are the objects)?
There was a problem hiding this comment.
Both maps currently store pointers to the same cloned object (line 36 below), so there's no duplication of NamespaceMetadata itself - just two pointers. That is, unless I'm missing something about map bucket overhead?
|
|
||
| var ret []*storage.NamespaceMetadata | ||
| for _, ns := range n.namespaces { | ||
| ret = append(ret, ns.CloneVT()) |
There was a problem hiding this comment.
I see we are cloning twice, once to store the namespace in the store and the second time here to prevent the caller of this func to modify the object.
Do we really need to do this? I know that there are not so many namespaces in a cluster, but I would be for avoiding such things in Sensor unless really necessary.
There was a problem hiding this comment.
Yeah this is probably overly defensive. I'll remove both clones to match the pattern we're using in clusterlabels store.
pkg/labelproviders/providers.go
Outdated
| @@ -0,0 +1,52 @@ | |||
| package labelproviders | |||
There was a problem hiding this comment.
Have you evaluated the admission controller manager to embed clusterlabelprovider and namespacelabelprovider interfaces? If I am seeing this correctly, the adapters are not needed and there is a lot of opportunity for simplification.
There was a problem hiding this comment.
Honestly great suggestion, and not too hard to change. I'll go forward with this.
9e4ca7f to
0260fd9
Compare
5ec645a to
ae0a45a
Compare
ae0a45a to
68171b8
Compare
|
@AlexVulaj: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
Description
Note, this PR's end-to-end functionality also relies on PR #19530 . They were written and tested together, but split to simplify the review process.
Wire Sensor's in-memory stores as label providers for policy evaluation, enabling full label-based policy scoping (cluster + namespace labels) in both runtime detection and admission control.
Policies with
cluster_labelandnamespace_labelscopes now work in Sensor's runtime detection and admission control.User-facing documentation
Testing and quality
Automated testing
How I validated my change
Added comprehensive unit tests.
Executed comprehensive manual test plan covering three workflows:
Workflow 1: Admission control deploy-time enforcement (4/4 passed)
Workflow 2: Runtime detection (Sensor) (3/3 passed)
Workflow 3: Verification checks (2/2 passed)
Test setup:
ROX_LABEL_BASED_POLICY_SCOPING=true4.11.x-348-g89eb433197(includes namespace sync and enrichment fixes)env=prod/dev,region=us-east-1)team=backend/frontend,tier=app)