ROX-19064: Scanner V4 Automated Tests (Common Test Image + Compliance)#19641
ROX-19064: Scanner V4 Automated Tests (Common Test Image + Compliance)#19641
Conversation
Add null guard to found checks so that it actually retries when image not found yet vs. failing with NullPointerException. This was causing CI flakes during testing.
…y flag - Creates a ScannerV4Integration type for tests to use (will be part of future commits) - The Scanner V4 integration cannot be deleted in Central by design, the new isDeletable() method allows tests to act accordingly. - Makes isTestable() for StackRoxScannerIntegration conditional on Scanner V4 being enabled. When Scanner V4 is enabled StackRox Scanner is not reachable for image scans (exception: delegated scanning, but that is not tested in groovy tests)
Scanner V4 does not support debian 10 or less images, updating this test image should allow both V2 and V4 to scan succesfully
The "Verify checks based on Integrations" tests relied on scanner integrations being deleted, however the scanner V4 integration cannot be deleted by design, therefore the test is now skipped when scanner v4 enabled. The "Verify controls that checks for fixable CVEs" tests relied on the test image having 100 or more fixable vulnerabilities, scanner v4 reports less vulnerabilities and therefore the regex had to be updated.
|
Skipping CI for Draft Pull Request. |
|
/test all |
|
@dcaravel: No presubmit jobs available for stackrox/stackrox@dc/scanv4-base-groovy DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/test all |
|
Images are ready for the commit at be0e2ae. To use with deploy scripts, first |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #19641 +/- ##
==========================================
- Coverage 49.37% 49.36% -0.01%
==========================================
Files 2742 2742
Lines 206921 206921
==========================================
- Hits 102165 102154 -11
- Misses 97172 97181 +9
- Partials 7584 7586 +2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
@dcaravel: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
Closed, relies on the image prefetch changes that are part of #19642 |
Description
Part of a series of changes that modify groovy tests to work with both
StackRox ScannerandScanner V4.Scanner V4 is not enabled yet, this is prep work so that it can be enabled in a future PR.
Updated the shared
TEST_IMAGEto one that both scanners will report vulnerabilities for (the old image was based ondebian:9which is not supported by Scanner V4)The
ComplianceTesthad to be updated to work with the new image and both scanners as a result.Stacked on top of
User-facing documentation
Testing and quality
Automated testing
How I validated my change
Against StackROX Scanner these changes will be tested by CI as part of this PR
Against Scanner V4 these changes were validated in #19236