Description

Test failures

Components list Page and its entity detail page, (related entities) sub list validations with VM updates ON should display all the columns expected in node components list page

cypress/integration/vulnmanagement/componentsListPages.test.js

1. 1567400461261082624 from branch build for ROX-12343: unauthenticated email integration #2984 on 2022-09-07

   Timed out retrying after 5000ms: cy.wait() timed out waiting 5000ms for the 1st request to the route: auth/status. No request ever occurred.

2. 1567566650423644160 from branch build for Clone values to prevent infinite loop and fix edge case in QuayIntegrationForm #2996 on 2022-09-07

   Timed out retrying after 5000ms: cy.wait() timed out waiting 5000ms for the 1st request to the route: credentialexpiry_SCANNER. No request ever occurred.

The trend continues: during past 12 hours on 2022-09-13, 6 more failures for master builds compared to only 3 successes.

Recent test failures are similar to an earlier test failure on 2021-11-30

Timed out retrying after 5000ms: cy.wait() timed out waiting 5000ms for the 1st request to the route: getImages. No request ever occurred.

Analysis

The test files for entities have a big test, for example: 'Clusters list Page and its single entity detail page, and sub list validations'

• Much opportunity for request timeouts. Not enough information to wait on requests.
• In hypothetical case that test fails because of a problem other than timeout, where is the problem: this, that, or the other?
• If test criteria are skipped conditionally because of data (that is, because first row of table does not have a link) then unreliable criteria can remain dormant, and then suddenly cause tests to fail at unconvenient times.

componentsListPages has 2 big tests back-to-back, and the second fails often: double trouble!

Solution

Sadly, the solution is difficult to break down into smaller steps.

1. Separate big test into smaller tests, each of which has a short description.
2. Rewrite complicated helper functions to explicitly identify steps through a workflow.
3. Add constants for all entities so helper functions can wait on relevant requests.
4. Verify the first link in the column instead of verify only if the first row has a link in the column.

Changed files

1. Edit cypress/constants/apiEndpoints.js

   • Rename objects and add missing properties (see Residue point 2).

2. Edit cypress/helpers/vulnmanagement/entities.js

   • Factor out requestConfigForDashboard for functions which visit dashboard.
   • Import opname objects and add keyForEntity object.
   • Add verifyLinksForSecondaryEntities to replace allChecksForEntities and allCVECheck function calls.
   • Add verifyFilteredSecondaryEntitiesLink to replace allFixableCheck function calls.
   • Add verifyFixableCVEsLinkAndRiskAcceptanceTabs to replace allFixableCheck function call plus inline assertions for images.

3. Edit 7 test files:

   • clustersListPages
   • componentsListPages: superseded by new tests, therefore to be delete in future clean up
   • cvesListPages superseded by new tests, therefore to be delete in future clean up
   • deploymentsListPages
   • imagesListPages
   • namespacesListPages
   • policiesListPages

4. Add 5 test files:

   • clusterCvesListPages
   • imageComponentsListPages
   • imageCvesListPages: has describe.skip see Residue point 5
   • nodeComponentsListPages
   • nodeCvesListPages

Residue

1. Fix incorrect counts and then remove special cases:
   • zero count: clusterCvesListPages
   • null count: nodeComponentsListPages
2. Move endpoint constants from apiEndpoints.js to entities.js simplify future maintenance. Rewrite references to helper methods in test files:
   • dashboard.test.js
   • entitypages.test.js
3. Move hasExpectedHeaderColumns from vmWorkflowUtils.js to entities.js as prerequisite to delete vmWorkflowUtils when we delete 'ROX_FRONTEND_VM_UPDATES' feature flag. Decide whether to explicitly include checkbox in expected columns.
4. Add tests to sort lists by Risk Priority column. Probably replace CVSS Score with Severity as column for tests for sort lists of vulnerabilities.
5. Decide whether describe.skip block in imageCvesListPages is still relevant.

Checklist

• Investigated and inspected CI test results
• Edited integration tests

Testing Performed

1. Ran individual tests in local deployment
2. Used staging-db demo to write tests for scenarios not in local deployment.