/test gke-qa-e2e-tests

Images are ready for the commit at e2ebc0c.

To use with deploy scripts, first export MAIN_IMAGE_TAG=4.8.x-869-ge2ebc0c2a7.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 49.26%. Comparing base (14e8fe1) to head (e2ebc0c).
Report is 329 commits behind head on master.

@@           Coverage Diff           @@
##           master   #15337   +/-   ##
=======================================
  Coverage   49.25%   49.26%           
=======================================
  Files        2581     2581           
  Lines      189350   189350           
=======================================
+ Hits        93261    93277   +16     
+ Misses      88746    88732   -14     
+ Partials     7343     7341    -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

/retest

looking at one of the tests, https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/stackrox_stackrox/15337/pull-ci-stackrox-stackrox-master-ocp-4-18-qa-e2e-tests/1923844275313840128

central-6db6cdfc98-jzk4k 0/1 CrashLoopBackOff 6 (4m33s ago)

$ head -5 ./stackrox/pods/central-6db6cdfc98-jzk4k-central-previous.log
cp: cannot remove '/etc/pki/ca-trust/extracted/pem/directory-hash/002c0b4f.0': Permission denied
cp: cannot remove '/etc/pki/ca-trust/extracted/pem/directory-hash/01419da9.0': Permission denied
cp: cannot remove '/etc/pki/ca-trust/extracted/pem/directory-hash/0179095f.0': Permission denied
cp: cannot remove '/etc/pki/ca-trust/extracted/pem/directory-hash/02265526.0': Permission denied
cp: cannot remove '/etc/pki/ca-trust/extracted/pem/directory-hash/04f60c28.0': Permission denied

so, it looks like something with updating ca's is being blocked. Maybe the owner changed in the base image (not sure if we're using a different uid) or there is an additional step required on rhel9 for an update

I don't pretend to understand why we're messing around with /etc/pki in the first place. Maybe this is a good opportunity to find out whether it's still a good idea?

I don't pretend to understand why we're messing around with /etc/pki in the first place. Maybe this is a good opportunity to find out whether it's still a good idea?

Maybe it is a system service that is running or triggered somehow; I do not know if we're running anything outside our entrypoint

We are replacing these files in the https://github.com/stackrox/stackrox/blob/master/image/rhel/static-bin/central-entrypoint.sh#L25
calling https://github.com/stackrox/stackrox/blob/master/image/rhel/static-bin/import-additional-cas
which runs the update-ca-trust utility script that creates the /etc/pki/ca-trust/extracted files (the man page explains the dirs+files and to only use the script to modify these files. it looks like we're following the guidelines).

/test gke-qa-e2e-tests

/test gke-qa-e2e-tests

I was wrong. The file perms error is earlier (https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/stackrox_stackrox/15337/pull-ci-stackrox-stackrox-master-gke-qa-e2e-tests/1924872987819905024/artifacts/gke-qa-e2e-tests/stackrox-stackrox-e2e-test/artifacts/central-74d7c79ff5-5kkm4-central-previous.log):

++ id -u
+ '[' 4000 == 0 ']'
+ restore-all-dir-contents
cp: cannot remove '/etc/pki/ca-trust/extracted/pem/directory-hash/002c0b4f.0': Permission denied
cp: cannot remove '/etc/pki/ca-trust/extracted/pem/directory-hash/01419da9.0': Permission denied
cp: cannot remove '/etc/pki/ca-trust/extracted/pem/directory-hash/0179095f.0': Permission denied

/test gke-nongroovy-e2e-tests

An example of a failed overwrite file tls-ca-bundle.pem is owned by the running user (uid=4000) but not with write permissions:
(https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/stackrox_stackrox/15337/pull-ci-stackrox-stackrox-master-gke-ui-e2e-tests/1925667877780721664/artifacts/gke-ui-e2e-tests/stackrox-stackrox-e2e-test/artifacts/central-6d6f78c77c-kxzrh-central-previous.log)

/etc/pki/ca-trust/extracted/pem:
total 912
drwxr-sr-x 3 4000 4000   4096 May 22 22:15 .
drwxr-sr-x 6 4000 4000   4096 May 22 22:09 ..
-rw-r--r-- 1 4000 4000    898 May 22 22:15 README
dr-xr-xr-x 2 4000 4000  20480 May 22 22:09 directory-hash
-r--r--r-- 1 4000 4000 165521 May 22 22:15 email-ca-bundle.pem
-r--r--r-- 1 4000 4000 502506 May 22 22:15 objsign-ca-bundle.pem
-r--r--r-- 1 4000 4000 226489 May 22 22:15 tls-ca-bundle.pem

I think the cp should still pass because of the -f and the user has write permissions on the directory. But it doesn't so is the no-write permission may be enforced differently because .. k8s volume? something specific with this cp?

summary of what I understand of the file-perms failure in central startup:

1. we volume mount directories to make them writable and keep the root filesystem read-only
2. the files are saved to /.init-dirs and then copied back into the volumes (emptyDir on k8s, or replacing the contents on a docker run volume)
3. for some of these files, there are other writes (example from update-ca-certificates to any changed ca ssl files)

Problem:
File write errors during central pod startup. I think the errors are all with, /etc/pki/ca-trust/extracted/pem/directory-hash/ files that I did not find on UBI8 and appear to be a new feature for the trusted ca files:
cp: cannot remove '/etc/pki/ca-trust/extracted/pem/directory-hash/ff34af3f.0': Permission denied

/test gke-qa-e2e-tests

@davdhacs: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Full PR test history. Your PR dashboard.