ROX-31229: add e2e tests for tailored profile tracking#19880
Draft
ROX-31229: add e2e tests for tailored profile tracking#19880
Conversation
Install a CustomRule + from-scratch TailoredProfile (e2e-tailored-profile)
during compliance operator setup so they are available to all compliance e2e
tests.
Test changes:
- Add profileRef{name, operatorKind} helper type so assertScanSettingBinding
can verify the Kind field on each SSB profile entry (Profile vs
TailoredProfile), not just the name
- TestComplianceV2ProfileGet/GetSummaries: assert operator_kind on both
the e2e TP and a regular profile
- TestComplianceV2CreateGetScanConfigurations: use mixed profiles (regular +
TP); assert SSB kinds; test TP duplicate rejection
- TestComplianceV2UpdateScanConfigurations: update from single Profile to
mixed Profile+TP; assert SSB kinds for both
- TestComplianceV2DeleteComplianceScanConfigurations: use mixed profiles
- TestComplianceV2CentralSendsScanConfiguration: validate startup sync path
preserves profile_refs with correct kinds across Sensor reconnect
- TestComplianceV2ScheduleRescan: include TP alongside regular profile
- TestComplianceV2TailoredProfileVariants (new): creates an extends-base TP
dynamically, verifies operator_kind == TAILORED_PROFILE for both from-scratch
and extends-base variants, and verifies the disabled rule is excluded from
the effective rules list
Note: CO does not allow mixing CustomRules and regular Rules in an
extends-based TP, so the variant test uses only DisableRules.
All tests verified on cluster (rox-26032-cluster-1).
AI-assisted implementation.
|
Skipping CI for Draft Pull Request. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #19880 +/- ##
==========================================
- Coverage 49.60% 49.57% -0.03%
==========================================
Files 2763 2766 +3
Lines 208339 208540 +201
==========================================
+ Hits 103342 103390 +48
- Misses 97332 97474 +142
- Partials 7665 7676 +11
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Contributor
🚀 Build Images ReadyImages are ready for commit 5834b14. To use with deploy scripts: export MAIN_IMAGE_TAG=4.11.x-573-g5834b144dc |
Contributor
🚀 Build Images ReadyImages are ready for commit 5dd8504. To use with deploy scripts: export MAIN_IMAGE_TAG=4.11.x-570-g5dd85047c9 |
Contributor
Author
|
/test ocp-4-21-compliance-e2e-tests |
|
@guzalv: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
Problem: The install_e2e_compliance_resources() function was creating a TailoredProfile immediately after creating the CustomRule it references, without waiting for the CustomRule to be processed by the compliance operator. This caused a race condition where the TailoredProfile could fail to reach READY state if the CustomRule wasn't validated yet. Solution: Add a wait loop after creating the CustomRule to ensure it exists and give the compliance operator time to process it (10 seconds) before creating the TailoredProfile. Also improved error reporting to show the TailoredProfile's error message and current state during the wait loop. The e2e-tailored-profile.yaml references check-cm-marker CustomRule in its enableRules, so the CustomRule must be ready before the TailoredProfile can be validated successfully. Fixes: #19880 CI failure in ocp-4-21-compliance-e2e-tests Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Tests now create their own TP/CR resources per-test with unique names and proper cleanup, so the shared install_e2e_compliance_resources() and its yaml manifests are no longer needed. Partially generated by AI. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
createTailoredProfile now creates a simple extends-based TP (ocp4-e8) instead of requiring a custom rule. Tests that just need "any TP" use this helper; variant-specific logic moved to the variants test. TestComplianceV2TailoredProfileVariants now covers all TP types inline: - extends-base with disabled rules - from-scratch with custom rules - from-scratch with regular rules only Partially generated by AI. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Add e2e test coverage for tailored profile tracking (ROX-31229). Installs a
CustomRuleand a from-scratchTailoredProfile(e2e-tailored-profile) during compliance operator setup, then verifies that ACS correctly stores and exposes them with the rightoperator_kind(TAILORED_PROFILEvsPROFILE), and thatScanSettingBindingresources are created with the correct k8sKindon each profile entry.User-facing documentation
Testing and quality
Automated testing
How I validated my change
All 10 compliance v2 e2e tests were run on a real OCP cluster (
rox-26032-cluster-1) with a live Compliance Operator installation:TestComplianceV2ProfileGet— assertsTAILORED_PROFILEone2e-tailored-profile,PROFILEon a regular profile, andcheck-cm-markerin TP rulesTestComplianceV2ProfileGetSummaries— assertsTAILORED_PROFILEin summariesTestComplianceV2CreateGetScanConfigurations— mixed profile scan config; verifies SSBKindfields; tests TP duplicate rejectionTestComplianceV2UpdateScanConfigurations— updates from Profile-only to mixed Profile+TP; verifies SSB kindsTestComplianceV2DeleteComplianceScanConfigurations— mixed profilesTestComplianceV2CentralSendsScanConfiguration— validates startup sync preservesprofile_refskinds across Sensor reconnectTestComplianceV2ScheduleRescan— TP alongside regular profileTestComplianceV2TailoredProfileVariants(new) — creates an extends-base TP dynamically, verifiesTAILORED_PROFILEfor both variants, verifies disabled rule is excluded from effective rulesTestComplianceV2Integration,TestComplianceV2ComplianceObjectMetadata— all passingKey discovery: CO does not allow mixing
CustomRulesand regularRulesin an extends-based TP. The variant test therefore uses onlyDisableRules.