ROX-31229: Add compliance E2E tests for tailored profiles#19880
ROX-31229: Add compliance E2E tests for tailored profiles#19880
Conversation
|
Skipping CI for Draft Pull Request. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #19880 +/- ##
=======================================
Coverage 49.56% 49.56%
=======================================
Files 2764 2764
Lines 208357 208357
=======================================
Hits 103269 103269
Misses 97436 97436
Partials 7652 7652
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
🚀 Build Images ReadyImages are ready for commit bcf6d30. To use with deploy scripts: export MAIN_IMAGE_TAG=4.11.x-639-gbcf6d30ae8 |
🚀 Build Images ReadyImages are ready for commit 5dd8504. To use with deploy scripts: export MAIN_IMAGE_TAG=4.11.x-570-g5dd85047c9 |
|
/test ocp-4-21-compliance-e2e-tests |
Problem: The install_e2e_compliance_resources() function was creating a TailoredProfile immediately after creating the CustomRule it references, without waiting for the CustomRule to be processed by the compliance operator. This caused a race condition where the TailoredProfile could fail to reach READY state if the CustomRule wasn't validated yet. Solution: Add a wait loop after creating the CustomRule to ensure it exists and give the compliance operator time to process it (10 seconds) before creating the TailoredProfile. Also improved error reporting to show the TailoredProfile's error message and current state during the wait loop. The e2e-tailored-profile.yaml references check-cm-marker CustomRule in its enableRules, so the CustomRule must be ready before the TailoredProfile can be validated successfully. Fixes: #19880 CI failure in ocp-4-21-compliance-e2e-tests Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
|
/test ocp-4-21-compliance-e2e-tests |
There was a problem hiding this comment.
Hey - I've found 1 issue, and left some high level feedback:
- There’s quite a bit of inline TailoredProfile and CustomRule construction (e.g., in TestComplianceV2TailoredProfileVariants) that mirrors the helper-style logic in createTailoredProfile/createCustomRule; consider extracting common patterns into reusable helpers to keep the tests shorter and reduce duplication of ready-wait loops and cleanup logic.
- Some calls to ListComplianceScanConfigurations now deliberately ignore errors (e.g.,
scanConfigs, _ := ...), which could hide real problems during debugging; it would be safer to assert or require on the error and only proceed when the call succeeds.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- There’s quite a bit of inline TailoredProfile and CustomRule construction (e.g., in TestComplianceV2TailoredProfileVariants) that mirrors the helper-style logic in createTailoredProfile/createCustomRule; consider extracting common patterns into reusable helpers to keep the tests shorter and reduce duplication of ready-wait loops and cleanup logic.
- Some calls to ListComplianceScanConfigurations now deliberately ignore errors (e.g., `scanConfigs, _ := ...`), which could hide real problems during debugging; it would be safer to assert or require on the error and only proceed when the call succeeds.
## Individual Comments
### Comment 1
<location path="tests/compliance_operator_v2_test.go" line_range="1206-1209" />
<code_context>
+ }
+
+ // --- Verify from-scratch TP with custom rule: custom rule present ---
+ fromScratchCRDetail, err := profileClient.GetComplianceProfile(ctx, &v2.ResourceByID{Id: fromScratchCRInACS.GetId()})
+ require.NoError(t, err)
+ foundCustomRule := false
+ for _, r := range fromScratchCRDetail.GetRules() {
+ if strings.Contains(r.GetName(), crName) {
+ foundCustomRule = true
</code_context>
<issue_to_address>
**issue (testing):** Make the custom-rule presence assertion more robust in TailoredProfileVariants
This assertion may be brittle: `strings.Contains(r.GetName(), crName)` assumes the ACS rule name matches the `CustomRule` resource name, but ACS may expose the XCCDF ID (e.g. `xccdf_org.example_rule_<name_with_underscores>`). Since `createCustomRule` builds the ID via `strings.ReplaceAll(name, "-", "_")`, `crName` with hyphens might never appear verbatim in `r.GetName()`. Instead, derive the expected rule ID using the same transformation as `createCustomRule` and compare for equality, or match on a prefix like `"xccdf_org.example_rule_"` plus the transformed name to avoid false negatives.
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
📝 WalkthroughWalkthroughAdded shared test helpers for creating CustomRules and TailoredProfiles with ingestion polling. Introduced Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes 🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
tests/compliance_operator_v2_test.go (2)
296-304: Consider shorter poll intervals for faster feedback.The 10-second poll interval for TailoredProfile readiness may delay test failure detection. A 5-second interval would provide faster feedback without significantly increasing load on the cluster.
Proposed change
- }, 3*time.Minute, 10*time.Second) + }, 3*time.Minute, 5*time.Second)🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@tests/compliance_operator_v2_test.go` around lines 296 - 304, The test is using a 10-second poll interval in require.EventuallyWithT which slows feedback; update the call in tests/compliance_operator_v2_test.go (the require.EventuallyWithT block that fetches complianceoperatorv1.TailoredProfile using client.Get with NamespacedName{Name: name, Namespace: coNamespaceV2}) to use a 5-second poll interval instead of 10 seconds to get faster failure detection while keeping the same overall timeout.
1205-1216: Consider documenting whystrings.Containsis used for custom rule matching.Using
strings.Containsinstead of exact matching suggests the Compliance Operator may transform rule names. A brief comment explaining this would improve maintainability.Suggested comment
// --- Verify from-scratch TP with custom rule: custom rule present --- fromScratchCRDetail, err := profileClient.GetComplianceProfile(ctx, &v2.ResourceByID{Id: fromScratchCRInACS.GetId()}) require.NoError(t, err) foundCustomRule := false for _, r := range fromScratchCRDetail.GetRules() { + // Use Contains because CO may prefix/transform rule names during processing. if strings.Contains(r.GetName(), crName) { foundCustomRule = true break } }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@tests/compliance_operator_v2_test.go` around lines 1205 - 1216, Tests use strings.Contains when checking rule names on the fromScratchCRDetail.GetRules() list because rule names can be transformed (prefixes/suffixes/IDs) by the Compliance Operator; add a concise inline comment above the loop that references fromScratchCRDetail, the loop over GetRules(), r.GetName(), and crName explaining that non-exact matching is intentional due to operator-driven name normalization and therefore we use substring matching (strings.Contains) rather than exact equality.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@tests/compliance_operator_v2_test.go`:
- Around line 707-733: The test calls err.Error() after
CreateComplianceScanConfiguration for both duplicateProfileReq and
duplicateTPReq which can nil-panic if the call unexpectedly succeeds; update the
checks to first assert that an error was returned (e.g., use require.Error(t,
err) or assert.Error(t, err)) before asserting on the message with err.Error(),
referencing the CreateComplianceScanConfiguration calls, scanConfigService,
duplicateProfileReq and duplicateTPReq to locate the two failing assertions.
---
Nitpick comments:
In `@tests/compliance_operator_v2_test.go`:
- Around line 296-304: The test is using a 10-second poll interval in
require.EventuallyWithT which slows feedback; update the call in
tests/compliance_operator_v2_test.go (the require.EventuallyWithT block that
fetches complianceoperatorv1.TailoredProfile using client.Get with
NamespacedName{Name: name, Namespace: coNamespaceV2}) to use a 5-second poll
interval instead of 10 seconds to get faster failure detection while keeping the
same overall timeout.
- Around line 1205-1216: Tests use strings.Contains when checking rule names on
the fromScratchCRDetail.GetRules() list because rule names can be transformed
(prefixes/suffixes/IDs) by the Compliance Operator; add a concise inline comment
above the loop that references fromScratchCRDetail, the loop over GetRules(),
r.GetName(), and crName explaining that non-exact matching is intentional due to
operator-driven name normalization and therefore we use substring matching
(strings.Contains) rather than exact equality.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Central YAML (base), Organization UI (inherited)
Review profile: CHILL
Plan: Pro
Run ID: a234cc97-b475-4304-af67-a3ddc55a7efb
📒 Files selected for processing (1)
tests/compliance_operator_v2_test.go
Problem: The install_e2e_compliance_resources() function was creating a TailoredProfile immediately after creating the CustomRule it references, without waiting for the CustomRule to be processed by the compliance operator. This caused a race condition where the TailoredProfile could fail to reach READY state if the CustomRule wasn't validated yet. Solution: Add a wait loop after creating the CustomRule to ensure it exists and give the compliance operator time to process it (10 seconds) before creating the TailoredProfile. Also improved error reporting to show the TailoredProfile's error message and current state during the wait loop. The e2e-tailored-profile.yaml references check-cm-marker CustomRule in its enableRules, so the CustomRule must be ready before the TailoredProfile can be validated successfully. Fixes: #19880 CI failure in ocp-4-21-compliance-e2e-tests Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
|
This change is part of the following stack: Change managed by git-spice. |
36ad77c to
ef67153
Compare
Install a CustomRule + from-scratch TailoredProfile (e2e-tailored-profile)
during compliance operator setup so they are available to all compliance e2e
tests.
Test changes:
- Add profileRef{name, operatorKind} helper type so assertScanSettingBinding
can verify the Kind field on each SSB profile entry (Profile vs
TailoredProfile), not just the name
- TestComplianceV2ProfileGet/GetSummaries: assert operator_kind on both
the e2e TP and a regular profile
- TestComplianceV2CreateGetScanConfigurations: use mixed profiles (regular +
TP); assert SSB kinds; test TP duplicate rejection
- TestComplianceV2UpdateScanConfigurations: update from single Profile to
mixed Profile+TP; assert SSB kinds for both
- TestComplianceV2DeleteComplianceScanConfigurations: use mixed profiles
- TestComplianceV2CentralSendsScanConfiguration: validate startup sync path
preserves profile_refs with correct kinds across Sensor reconnect
- TestComplianceV2ScheduleRescan: include TP alongside regular profile
- TestComplianceV2TailoredProfileVariants (new): creates an extends-base TP
dynamically, verifies operator_kind == TAILORED_PROFILE for both from-scratch
and extends-base variants, and verifies the disabled rule is excluded from
the effective rules list
Note: CO does not allow mixing CustomRules and regular Rules in an
extends-based TP, so the variant test uses only DisableRules.
All tests verified on cluster (rox-26032-cluster-1).
AI-assisted implementation.
Problem: The install_e2e_compliance_resources() function was creating a TailoredProfile immediately after creating the CustomRule it references, without waiting for the CustomRule to be processed by the compliance operator. This caused a race condition where the TailoredProfile could fail to reach READY state if the CustomRule wasn't validated yet. Solution: Add a wait loop after creating the CustomRule to ensure it exists and give the compliance operator time to process it (10 seconds) before creating the TailoredProfile. Also improved error reporting to show the TailoredProfile's error message and current state during the wait loop. The e2e-tailored-profile.yaml references check-cm-marker CustomRule in its enableRules, so the CustomRule must be ready before the TailoredProfile can be validated successfully. Fixes: #19880 CI failure in ocp-4-21-compliance-e2e-tests Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Tests now create their own TP/CR resources per-test with unique names and proper cleanup, so the shared install_e2e_compliance_resources() and its yaml manifests are no longer needed. Partially generated by AI. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
createTailoredProfile now creates a simple extends-based TP (ocp4-e8) instead of requiring a custom rule. Tests that just need "any TP" use this helper; variant-specific logic moved to the variants test. TestComplianceV2TailoredProfileVariants now covers all TP types inline: - extends-base with disabled rules - from-scratch with custom rules - from-scratch with regular rules only Partially generated by AI. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove customRuleNamespace constant, reuse coNamespaceV2 for ConfigMap - Simplify CustomRule ID to match name (per CO team recommendation) - Truncate createTailoredProfile comment, include name in Title - Remove -tp suffix from TP names (separate UUIDs where needed) - Reduce timeouts from 3min/2min to 1min (compromise vs 10s) - Fix "platform Profile" → "Profile" in comment - Specify (Profile/TailoredProfile) in profileRef comment Partially generated by AI. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Per review feedback — empirically these complete well within 10s. Partially generated by AI. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Restore error checking on ListComplianceScanConfigurations calls that were silently ignoring errors (could hide real failures) - Tighten custom rule assertion from strings.Contains to exact match (safe now that ID=name per CO team recommendation) - Remove unused strings import Partially generated by AI. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
If CreateComplianceScanConfiguration unexpectedly succeeds for a duplicate profile, err is nil and err.Error() panics. Add require.Error before accessing the error message. Partially generated by AI. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The old name used "ingestion" which is an internal concept that isn't obvious to readers. The new name makes the intent explicit: we're waiting for the tailored profile to appear in Central's database. Also adds a name query filter to ListComplianceProfiles so we don't fetch all profiles on every poll iteration, and restores the deleted "UUID-based name for test isolation" comment. Generated with AI assistance. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rename variable to match the naming convention used elsewhere in the test (regularProfile). Also removes the duplicate operator_kind assertion after GetComplianceProfile — it was already verified on the ListComplianceProfiles result two lines above. Generated with AI assistance. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Drop "inherited from base profile" from rules assertion message since TP rules are not necessarily inherited (from-scratch TPs have their own rules). - Remove redundant operator_kind check on regular profile via GetComplianceProfile — already asserted on the List result above. Generated with AI assistance. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ScanSettingBinding profile kind assertions are already covered in TestComplianceV2CentralSendsScanConfiguration. This test focuses on the scan config CRUD API, not the SSB reconciliation. Generated with AI assistance. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The same schedule (daily at 15:00, days 1-6) was inlined 7 times across different test functions. Extract it into a package-level variable to reduce duplication and noise. Generated with AI assistance. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Use tpName := testID instead of generating separate UUIDs with "-tp" suffixes. The TP name just needs to be unique, and testID already is. - Move TP creation in UpdateScanConfigurations right before the update step where it's actually needed, not at the top of the test. - Remove "per-test" from comments — it adds no value. Generated with AI assistance. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add EnableRules to extends-base TP variant to test enable+disable combination (different rules, since CO doesn't allow both on the same rule). Verify enabled rule is present in the stored profile. - Rename variant 2 from fromScratchCR to customRulesTP since custom rules can only be from-scratch, making "fromScratch" redundant. - Use separate fromScratchRule const in variant 3 instead of reusing disabledRule in an EnableRules context, which was confusing. - Rename stored profile vars to storedExtendsTP, storedCustomRulesTP, storedFromScratchTP for clarity. Generated with AI assistance. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Each variant's spec, expected rules, and rejected rules are now co-located in a map entry. The shared creation/wait/assertion logic runs once per subtest, making it easier to follow what each variant tests and to add new variants. Generated with AI assistance. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…comment - Remove redundant expectRules/rejectRules from variants table; derive expected/rejected rules directly from the spec's EnableRules/DisableRules. - Replace map[string]bool with []string + assert.Contains/NotContains. - Add CO recommendation comment above CustomRule ID field. - Increase waitUntilTPInCentralDB timeout to 1min (10s was flaky under load). - Use a different enabled rule (alwaysadmit) to avoid overlap with the disabled rule's profile family. Partially generated by AI. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1dd1603 to
bcf6d30
Compare
There was a problem hiding this comment.
Hey - I've left some high level feedback:
- The TailoredProfile creation/wait logic in
TestComplianceV2TailoredProfileVariantsduplicates thecreateTailoredProfilehelper; consider reusing the helper there to keep the readiness checks and cleanup behavior consistent across tests. - In
createCustomRule, the sharede2e-cr-configConfigMap is created but never cleaned up; if you expect these tests to run repeatedly against longer-lived clusters, consider either cleaning it up int.Cleanupor explicitly documenting that it is intentionally left behind as a shared fixture.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- The TailoredProfile creation/wait logic in `TestComplianceV2TailoredProfileVariants` duplicates the `createTailoredProfile` helper; consider reusing the helper there to keep the readiness checks and cleanup behavior consistent across tests.
- In `createCustomRule`, the shared `e2e-cr-config` ConfigMap is created but never cleaned up; if you expect these tests to run repeatedly against longer-lived clusters, consider either cleaning it up in `t.Cleanup` or explicitly documenting that it is intentionally left behind as a shared fixture.Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
Description
Every existing compliance e2e test now creates its own TP dynamically (instead of relying on a globally-installed one), and uses mixed profiles (Profile + TailoredProfile) wherever scan configs are created. This validates that ACS stores
operator_kindcorrectly and thatScanSettingBindingresources carry the right k8sKindper profile entry.A new test (
TestComplianceV2TailoredProfileVariants) has been added to ensure that different TP flavors are handled correctly:CustomRule)User-facing documentation
Testing and quality
Automated testing
How I validated my change
All 10 compliance v2 e2e tests pass on a real OCP cluster with Compliance Operator v1.8.2: