Description

Overview

1. Because a minority of vulnmanagement tests have 'ROX_POSTGRES_DATASTORE' feature flag disabled, we have waited for evidence of timing problems.
2. Because previous true positives for non-postgress regressions were mixed with simultaneous false positives, apply the solution to replace confusion with clarity in the case of future regressions.
3. Prepare now for future removal of non-postgress tests and conditional skips in postgress tests. Move implicit head knowledge into explicit test organization, so it becomes a no-brainer.

Test failure

1. 1592676501193494528 from master build for Add migration for groups with invalid values #3789 on 2022-11-15

Vulnerability Management Deployments with VM updates OFF should display all the columns and links expected in deployments list page

Timed out retrying after 4000ms: Expected to find element: [data-testid="allCvesLink"], but never found it.

at validateAllCVELinks cypress/helpers/vmWorkflowUtils.js:126:7
at allCVECheck cypress/helpers/vmWorkflowUtils.js:233:4
at Context.eval cypress/integration/vulnmanagement/deploymentsListPages.test.js:60:32

Analysis

The test files for entities have a big test:

• Much opportunity for request timeouts. Not enough information to wait on requests.
• In hypothetical case that test fails because of a problem other than timeout, where is the problem: this, that, or the other?
• If test criteria are skipped conditionally because of data (that is, because first row of table does not have a link) then unreliable criteria can remain dormant, and then suddenly cause tests to fail at unconvenient times.

Solution

Apply pattern for tests with 'ROX_POSTGRES_DATASTORE' feature flag enabled in #3068

Organization

1. Rename and separate entity tests so camelCase corresponds to hyphen-case path segments for entities.
   • Remove ListPages suffix for postgress test files to leave plural camelCase of entity.
     For example, rename clusterCvesListPages.test.js as clusterCves.test.js
     • Two-part entities like clusterCves are postgress only.
     • Objective of test file is to test entities, even if many of its tests start from list page.
   • Replace ListPages suffix with NonPostgress for non-postgress test files.
     For example, rename cvesListPages.test.js as cvesNonPostgress.test.js
   • Separate test files which have combined tests.
     • deployments.test.js and deploymentsNonPostgress.test.js
     • namespaces.test.js and namespacesNonPostgress.test.js
2. Replace test patterns which import from vmWorkflowUtils with entities helpers file.
3. Do not separate tests in files which do not skip tests, but instead have ternary expressions:
   • dashboard.test.js
   • entitypages.test.js

Residue

Because hasTableColumnHeadings function explicitly asserts existence of checkbox, hidden, and invisible column headings:

• Increment columnIndex arguments of verify function calls to be consistent with arrays of column heading strings in the function call.
• Replace columnIndex + 1 with columnIndex in nth-child pseudo-selector.

Future removal of non-postgress tests

1. Delete whateverNonPostgress.test.js files

2. Delete skip from test files:

   before(function beforeHook() {
       if (!hasFeatureFlag('ROX_POSTGRES_DATASTORE')) {
           this.skip();
       }
   });

3. Delete skip from tests:

   if (!hasFeatureFlag('ROX_POSTGRES_DATASTORE')) {
       this.skip();
   }

4. Replace conditional ternary expressions with postgress-only value, for example:

   • replace const entitiesKey = hasFeatureFlag('ROX_POSTGRES_DATASTORE') ? 'image-cves' : 'cves';
   • with const entitiesKey = 'image-cves';

Checklist

• Investigated and inspected CI test results
• Edited integration tests

Testing Performed