Description

13 tests in 5 files under configmanagement fail intermittently since cypress 10.6.0 upgrade, for example:

• 1561658257045458944 from master build for Upgrade cypress 10.6.0 devDependencies in ui #2743 on 2022-08-22
• 1561807085534973952 from master build for Bump k8s.io/apiserver from 0.23.9 to 0.23.10 #2781 on 2022-08-23
• 1562047103482466304 from master build for CI: Bump -rcd wait timeout #2745 on 2022-08-23

Failing tests

1. configmanagement/entitiespage.test.js
   • should not modify the URL when clicking the Overview tab when in the Overview section: Config Management Entity Page should not modify the URL when clicking the Overview tab when in the Overview section
2. configmanagement/dashboard.test.js
   • clicking the "CIS Standard Across Clusters" widget's "passing controls" link should take you to the controls list and filter by passing controls: Config Management Dashboard Page
   • clicking the "CIS Standard Across Clusters" widget's "failing controls" link should take you to the controls list and filter by failing controls: Config Management Dashboard Page
3. configmanagement/clusters.test.js
   • should click on the controls link in the clusters list and open the side panel with the controls list: Config Management Entities (Clusters) should click on the controls link in the clusters list and open the side panel with the controls list
4. configmanagement/ciscontrols.test.js
   • should render the controls list and open the side panel when a row is clicked: Config Management Entities (CIS controls) should render the controls list and open the side panel when a row is clicked
   • should take you to a control single when the "navigate away" button is clicked: Config Management Entities (CIS controls) should take you to a control single when the "navigate away" button is clicked
   • should have the correct count widgets for a single entity view: Config Management Entities (CIS controls) should have the correct count widgets for a single entity view
   • should click on the nodes count widget in the entity page and show the nodes tab: Config Management Entities (CIS controls) should click on the nodes count widget in the entity page and show the nodes tab
   • should have the correct tabs for a single entity view: Config Management Entities (CIS controls) should have the correct tabs for a single entity view
   • should have the same number of Nodes in the count widget as in the Nodes table: Config Management Entities (CIS controls) should have the same number of Nodes in the count widget as in the Nodes table
   • should show no failing nodes in the control findings section of a passing control: Config Management Entities (CIS controls) should show no failing nodes in the control findings section of a passing control
   • should show failing nodes in the control findings section of a failing control: Config Management Entities (CIS controls) should show failing nodes in the control findings section of a failing control
5. configmanagement/nodes.test.js
   • should click on the controls count widget in the entity page and show the controls tab: Config Management Entities (Nodes) should click on the controls count widget in the entity page and show the controls tab

Analysis

The failing tests assume that scan results are available.

• compliance/complianceDashboard.test.js bends the guideline that tests not depend on other tests, because subsequent tests in the same file depend on side effect 'should scan for compliance data from the Dashboard page' from the first test So far, cypress does run tests in alphabetical order.
• configmanagement tests break the guideline, because cypress 10 apparently no longer runs tests files in a determinate order.

The following lines are filtered and stream edited from build-log.txt files.

cypress 9

• 1560369927628525568 from master build for Fix scope lookups in image and cluster Vulnerabilities sub-resolvers #2715 on 2022-08-18 which has unrelated failure

  access.test.js                                                                 (1 of 79)
  accessControl/accessControlAccessScopes.test.js                                (2 of 79)
  accessControl/accessControlAuthProviders.test.js                               (3 of 79)
  accessControl/accessControlPermissionSets.test.js                              (4 of 79)
  accessControl/accessControlRoles.test.js                                       (5 of 79)
  auth.test.js                                                                   (6 of 79)
  certexpiration.test.js                                                         (7 of 79)
  clusters/clusterDeletion.test.js                                               (8 of 79)
  clusters/clusters.test.js                                                      (9 of 79)
  compliance/complianceDashboard.test.js                                        (10 of 79)
  compliance/complianceList.test.js                                             (11 of 79)
  configmanagement/ciscontrols.test.js                                          (12 of 79)
  configmanagement/clusters.test.js                                             (13 of 79)
  configmanagement/dashboard.test.js                                            (14 of 79)
  configmanagement/deployments.test.js                                          (15 of 79)
  configmanagement/entitiespage.test.js                                         (16 of 79)
  configmanagement/images.test.js                                               (17 of 79)
  configmanagement/namespaces.test.js                                           (18 of 79)
  configmanagement/nodes.test.js                                                (19 of 79)
  configmanagement/policies.test.js                                             (20 of 79)
  configmanagement/roles.test.js                                                (21 of 79)
  configmanagement/secrets.test.js                                              (22 of 79)
  configmanagement/serviceaccount.test.js                                       (23 of 79)
  configmanagement/usersandgroups.test.js                                       (24 of 79)
  

cypress 10

• 1561658257045458944 from master build for Upgrade cypress 10.6.0 devDependencies in ui #2743 on 2022-08-22

  access.test.js                                                                 (1 of 78)
  auth.test.js                                                                   (2 of 78)
  certexpiration.test.js                                                         (3 of 78)
  dashboard.test.js                                                              (4 of 78)
  docs.test.js                                                                   (5 of 78)
  general.test.js                                                                (6 of 78)
  network.test.js                                                                (7 of 78)
  productBranding.test.js                                                        (8 of 78)
  search.test.js                                                                 (9 of 78)
  systemconfig.test.js                                                          (10 of 78)
  userinfo.test.js                                                              (11 of 78)
  accessControlAccessScopes.test.js                                             (12 of 78)
  accessControlAuthProviders.test.js                                            (13 of 78)
  accessControlPermissionSets.test.js                                           (14 of 78)
  accessControlRoles.test.js                                                    (15 of 78)
  clusterDeletion.test.js                                                       (16 of 78)
  clusters.test.js                                                              (17 of 78)
  ciscontrols.test.js                                                           (18 of 78)
  clusters.test.js                                                              (19 of 78)
  dashboard.test.js                                                             (20 of 78)
  deployments.test.js                                                           (21 of 78)
  entitiespage.test.js                                                          (22 of 78)
  images.test.js                                                                (23 of 78)
  namespaces.test.js                                                            (24 of 78)
  nodes.test.js                                                                 (25 of 78)
  policies.test.js                                                              (26 of 78)
  roles.test.js                                                                 (27 of 78)
  secrets.test.js                                                               (28 of 78)
  serviceaccount.test.js                                                        (29 of 78)
  usersandgroups.test.js                                                        (30 of 78)
  
  complianceDashboard.test.js                                                   (58 of 78)
  complianceList.test.js                                                        (59 of 78)
  

• 1561807085534973952 from master build for Bump k8s.io/apiserver from 0.23.9 to 0.23.10 #2781 on 2022-08-23

  ciscontrols.test.js                                                           (18 of 78)
  clusters.test.js                                                              (19 of 78)
  dashboard.test.js                                                             (20 of 78)
  deployments.test.js                                                           (21 of 78)
  entitiespage.test.js                                                          (22 of 78)
  images.test.js                                                                (23 of 78)
  namespaces.test.js                                                            (24 of 78)
  nodes.test.js                                                                 (25 of 78)
  policies.test.js                                                              (26 of 78)
  roles.test.js                                                                 (27 of 78)
  secrets.test.js                                                               (28 of 78)
  serviceaccount.test.js                                                        (29 of 78)
  usersandgroups.test.js                                                        (30 of 78)
  complianceDashboard.test.js                                                   (31 of 78)
  complianceList.test.js                                                        (32 of 78)
  

• from master build for ROX-9766: Add severity column to Image CVEs and Node CVEs tables #2768 on 2022-08-22 which has unrelated failures

  complianceDashboard.test.js                                                   (18 of 78)
  complianceList.test.js                                                        (19 of 78)
  ciscontrols.test.js                                                           (20 of 78)
  clusters.test.js                                                              (21 of 78)
  dashboard.test.js                                                             (22 of 78)
  deployments.test.js                                                           (23 of 78)
  entitiespage.test.js                                                          (24 of 78)
  images.test.js                                                                (25 of 78)
  namespaces.test.js                                                            (26 of 78)
  nodes.test.js                                                                 (27 of 78)
  policies.test.js                                                              (28 of 78)
  roles.test.js                                                                 (29 of 78)
  secrets.test.js                                                               (30 of 78)
  serviceaccount.test.js                                                        (31 of 78)
  usersandgroups.test.js                                                        (32 of 78)
  

Changed files

1. Export triggerScan function which makes triggerScan request and waits on response.

   • helpers/compliance.js

2. Call triggerScan function in first test which assumes that scan results are available.

   • integration/configmanagement/ciscontrols.test.js
   • integration/configmanagement/clusters.test.js
   • integration/configmanagement/dashboard.test.js
   • integration/configmanagement/entitiespage.test.js

Checklist

• Investigated and inspected CI test results
• Edited integration tests

Testing Performed

Run test files locally (with teardown and deploy-local between each run to clear the scan results).

See that tests fail before changes and pass after changes:

• ciscontrols.test.js: 8 tests failed before and 0 tests failed after
• clusters.test.js: 1 tests failed before and 0 tests failed after
• dashboard.test.js: 3 tests failed before and 1 test still failed after: 'should show the same number of low severity policies in the "Policy Violations By Severity" widget as it does in the Policies list'
• entitiespage.test.js: 1 test failed before and 0 tests failed after
• nodes.test.js: 1 test failed before and 0 tests failed after