ROX-9350 Use fine-grained host paths for compliance mounts by robbycochran · Pull Request #2479 · stackrox/stackrox

robbycochran · 2022-07-20T22:13:30Z

Description

The compliance container Is mounting the whole host filesystem, including any PV mounts that the node might be mounted which can prevent the unmounting of PV mounts, as the compliance binary will keep the mount points busy. This has been noticed on an OpenShift node when a Pod was mounting a 1TB volume with thousands of files. The compliance container started to scan these files and prevented the unmounting of the PV.

The problem of this behavior has been discussed in detail in https://bugzilla.redhat.com/show_bug.cgi?id=2036055

Checklist

Investigated and inspected CI test results
~~[ ] Unit test and regression tests added~~
Evaluated and added CHANGELOG entry if required
Determined and documented upgrade steps
Documented user facing changes (create PR based on openshift/openshift-docs and merge into rhacs-docs)

If any of these don't apply, please comment below.

Testing Performed

ghost · 2022-07-20T23:01:44Z

Images are ready for the commit at af27644.

To use with deploy scripts, first export MAIN_IMAGE_TAG=3.72.x-626-gaf2764454e.

openshift-ci · 2022-10-26T22:42:36Z

@robbycochran: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/openshift-newest-qa-e2e-tests	`a9b7ea7`	link	false	`/test openshift-newest-qa-e2e-tests`
ci/prow/gke-postgres-upgrade-tests	`b16ba00`	link	false	`/test gke-postgres-upgrade-tests`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

vikin91

Looks good in general - the original issue seems to be addressed.

The PR description does not mention any testing activities - has this been tested?
I also left one question about mounting entire /usr instead of /usr/lib.

I am marking it green, so to avoid that the TZ difference would block shipping this, but please check my comment.

vikin91 · 2022-10-27T13:48:37Z

image/templates/helm/stackrox-secured-cluster/templates/collector.yaml

+        - mountPath: /host/usr
+          name: usr-ro
+          readOnly: true


I am wondering why do we mount now /usr when we used to mount only /usr/lib. Is this intended?

It is intended. The collector and compliance containers now share volumes. The mount directories for the collector container are not changed, just the volumes used to provide those mounts. Previously, collector directly mounted /usr/lib via the host /usr/lib volume, now we mount via subPath on the host /usr volume mount that is shared with the compliance container.

Also note that previously, the compliance container mounted the entire host root.

janisz · 2022-11-15T15:10:59Z

What's the status of this? Is it meant to be included in 3.73

vikin91 · 2022-11-15T16:07:17Z

What's the status of this? Is it meant to be included in 3.73

Friendly ping @robbycochran

RTann · 2022-11-15T16:32:32Z

image/templates/helm/stackrox-secured-cluster/templates/collector.yaml

            path: key.pem
          - key: ca.pem
            path: ca.pem
-      - hostPath:


note: if we do decide to support language-level vulnerabilities in node-scanning, then will will probably need to mount / again, as these packages can, in theory, be anywhere

Yes that's a concern. The key issue is the fact that the compliance container does long-running mounts. Have we thought about node-scanning in an "on-demand" model, where the node scanning occurs with a short-term mount of the root directory?

We could consider that. I'm not familiar with this concept at all, but we can discuss more when/if we need to add language-level vulns

robbycochran · 2022-11-15T16:48:54Z

@theencee Can you take a look at this PR from the perspective of audit log consumption in the compliance container?

theencee · 2022-11-15T17:04:41Z

@theencee Can you take a look at this PR from the perspective of audit log consumption in the compliance container?

I think it should be good because of this:

        - mountPath: /host/var/log
          name: var-log
          readOnly: true

Audit log requires this file path mounted: /host/var/log/kube-apiserver/audit.log

…2479)" This reverts commit 7623dec.

Fix incorrect usage of subpath in #2479 subPath should not be used, and instead, directly create a hostPath volume for the needed volumeMount. Without this change, the collector cannot determine which kernel version to use from /usr/lib/os-release.

…2479)" This reverts commit 7623dec.

…2479)" (#3860) This reverts commit 7623dec.

7ffc6be ROX-13368: Skip failing nongroovy tests on PG (#3721) bbdd7a0 Bump github.com/gofrs/uuid from 4.3.0+incompatible to 4.3.1+incompatible (#3642) 1f253f2 Bump github.com/google/certificate-transparency-go from 1.1.3 to 1.1.4 (#3543) d434c8d [ROX-13030] : Add delete collection API endpoint and service implementation (#3648) f062c21 Dashrews/ROX-13253 wait for central-db to come back after bounce and allow FATAL connection lost error (#3537) edc1174 CI: Fill the gaps for https://testgrid.k8s.io/ (#3715) 86d7c54 ROX-13231: use passed context when non-postgres (#3540) 9093195 Add less specific type for BE collection response string (#3728) 5abb652 Only enable ROX_OBJECT_COLLECTIONS feature flag during gke-postgres-ui-e2e job (#3727) 4f64cd1 Add centralDBOnly mode in render (#3707) d67bbe5 Dashrews/ROX-13082 UUID searcher and common updates to set allow use of postgres UUID PR 1 of 4 (#3679) 6f829d5 ROX-13259: graphInit called during init time (#3705) f3bc50d ROX-13380: Conditional rendering edges for deployments and namespaces (#3641) 3764476 ROX-12319: implement smoke test step with groovy test filter (#3220) f202fd4 ROX-11826: Disable kernel support package uploads for managed central (#3661) 61f03dc ROX-11101: Remove deprecated resources from central (#3115) e6aa6d7 ROX-11101: Restore Role permission in UI (#3428) 3203e04 ROX-11101: Remove deprecated resources (#3036) a35f41e Bump golang.org/x/sys from 0.1.0 to 0.2.0 (#3733) 4120524 Bump snakeyaml from 1.29 to 1.33 in /qa-tests-backend (#3732) e1785c0 Bump github.com/coreos/go-systemd/v22 from 22.4.0 to 22.5.0 (#3724) 870df4a Bump google.golang.org/api from 0.101.0 to 0.102.0 (#3723) 721454c Generalize User-Agent setup (#3672) 6a11bf0 Bumps collector version to 3.11.x-145-gc345f72f5e (#3736) ec5d343 [ROX-12923] Walk retries - remainder work (#3729) 40f3d43 ROX-13440: Replace ambiguous central with sensor in networkGraph integration test (#3730) 281ed22 Bump groovy-xml from 2.5.18 to 2.5.19 in /qa-tests-backend (#3741) 49d1651 Bump github.com/prometheus/client_golang from 1.13.1 to 1.14.0 (#3742) b5544aa Bump cloud.google.com/go/storage from 1.27.0 to 1.28.0 (#3743) 9c61e53 ensure CVSS is present for istio vulns (#3706) ae29d52 ROX-13452: don't always clobber scoped ctx when non-postgres (#3748) 517bf05 ROX-13261: DryRunUpdate on collection datastore (#3687) baf7654 ROX-13378: Group new resources with deprecated in UI (#3690) 569922f ROX-13421: Enable roxctl netpol generate and add tech-preview messages (#3740) 2465fc5 Dashrews/ROX-13082 UUID generator templates PR 2 of 4 (#3681) c093c68 Bump slack-api-client from 1.20.2 to 1.27.0 in /qa-tests-backend (#3752) 2c860bb Bump ubi8-micro from 8.6 to 8.7 in /operator (#3751) 80eb04c Make deploy.sh and deploy-local.sh pass shellcheck (#3582) 2182b43 Dashrews/ROX-13082 UUID test updates PR 3 of 4 (#3694) 6dc6ca5 [ROX-13403] : Fix node -> topVuln sub resolver bug when node cves is empty (#3689) 1b21361 Move integration tests for page title from general to specific containers (#3675) e1a9f31 Bump google.golang.org/api from 0.102.0 to 0.103.0 (#3773) a05ea31 Bump golang.org/x/crypto from 0.1.0 to 0.2.0 (#3772) 65ddf4f ROX-12824: Add roxctl commands to generate Central DB bundle (#3602) c3f1e2f Remove obsolete authProviders request for Integrations page (#3759) 7ccd54d Dashrews/ROX-13082 UUID protos generated PR 4 of 4 (#3698) 9ab5c8f cleanup image digest utilities (#3764) 187ed44 ROX-11931: Convert junit failure artifacts to Slack attachments (#3438) b5d8790 ROX-13432: leaning up unused code copied/pasted from topology demo (#3750) ab05bfc Refactor collection form page for better composition (#3744) c5562f7 Remove babel devDependencies in ui-components (#3761) 2b90b3a Extract collection form from drawer wrapper layout (#3745) a779fc9 [ROX-12625 + ROX-13032] : Add GetCollectionCount and UpdateCollection endpoints and services (#3749) e77f0da Upgrade cypress 11.0.0 devDependencies in ui (#3760) a3fba94 ROX-13068: Use real data for deployment details (#3688) 4c7d90e ROX-12617: Collection to search query converter (#3683) 3e98aec ROX-13067: fill out port configurations section of deployment details (#3714) a48de36 ROX-12835: Add support for NodeScanV2 to Sensor (#3533) 30c5dc7 ROX-13466: Fix deletion of groups with empty properties (#3756) 5cb2470 Add autocomplete for name selector dropdowns (#3676) b9a75ad ROX-13464 adding flows dropdown in NG (#3763) 3217a67 [ROX-13500] Perform type check for V1 CronJob (#3787) af3790d Remove bulk delete from collections table (#3776) dda123b Add more info in migration log (#3788) 179f0c9 ROX-13502: Remove the circular dependency between cluster datastore init and cscc notifier init (#3790) 029d584 Update SCANNER_VERSION (#3774) cbca57c Bump github.com/ckaznocha/protoc-gen-lint from 0.2.4 to 0.3.0 (#3783) 3613b56 Bump golang.org/x/tools from 0.2.0 to 0.3.0 (#3782) 5fc0a6a Bump github.com/google/go-containerregistry from 0.12.0 to 0.12.1 (#3781) 1d1c687 Bump controller-gen version to 0.10.0 (#3754) c3a5290 Untie documentation link from the product version (#3799) ed822aa use correct package for migration (#3784) 397a0b4 Validate that label keys are valid k8s labels and ensure correct key splitting (#3777) edd1050 Rename variable ScannerGRPCEndpoint to ScannerSlimGRPCEndpoint (#3657) 6662c9f ROX-13378: Access Control page permissions (#3720) b0e73c5 fix Operator reconciliation for external Central DB (#3796) b83bc1f ROX-13505: Fix error log scanning the postgres stat collection (#3795) ca660cb Prevent the collection being edited from displaying in its own embedded list (#3778) 3f7b3fc [ROX-13441][POSTGRES] Propagate context correctly in retries (#3793) e0cbc6f ROX-12839: Update changelog to announce removal of in-product docs (#3805) 696e8bc [ROX-12358] Follow up on vulnerability request proto change (#2851) c4b46d8 Change getCollectionCount endpoint and updateCollection request type 5f2efbc remove make proto-fmt (#3804) 0c75540 Remove os.Std* from roxctl/central (#3758) 25a90de Add ability to view embedded collections in a pop up modal (#3747) 5c1bf81 ROX-13240: fix scanner-slim updates when WebSockets are used (#3704) 1d98577 Add more context to jira notifier logging (#3812) da2fd28 ROX-13031: DryRun Collection API (#3766) 1c418d5 Test data migration code in postgres tests (#3803) ed95b37 Update UI Collection requests for BE compatibiltiy (#3762) 09cc188 ROX-11931: Fix junit-parse install in CI (#3811) d2b01e3 ROX-12814: Disable PolicyFieldsTest on openshift. (#3797) d10ce27 ROX-13345: disable 'missing required registry' aspect on openshift (#3798) 3d22396 Update collector to 3.12 (#3809) 1eb33fb ROX-13347: Modify scope queries to included quoted cluster and nameace names, to allow exact matches instead of erroneous and unintended prefix matches. (#3767) 3811a69 ROX-12621: list collection selectors api (#3806) f6d3f9d Add migration for groups with invalid values (#3789) cc21125 Bugfixes for collection autocomplete (#3816) 7623dec ROX-9350 Use fine-grained host paths for compliance mounts (#2479) b4bf5c2 Fix collector volumeMounts (#3826) 0e9be05 ROX-12953: figure out last 4 versions of sensor automatically (#3611) 459c7ae ROX-12814: Add proper todo for reenabling the test (#3817) 9ee40ff ROX-13523: add isEnabled enum to central db spec (#3815) 535bc72 Replace requestConfig with routeMatcherMap in helper functions for integration tests (#3686) 2b75b61 `gosec` G104: Add `ShouldErr(err)` that returns `err` (#3830) fb1b82f WIP: Introduce nodescan call 35f8a8f WIP: Prepare converter 716144b Moved and renamed fake nodescan tests 4748de7 Introduce real node scanner with conversion functions 5e6d9a8 wip: real scanner 0169868 wip: log results 1438d2c wip: Debug Analyze call b09894c wip: Debug Analyze call 3ceed72 wip: Update and improve debug logs d1669fd Remove copied lib, bump scanner version, add debug 14a3f73 Merge branch 'master' into mm/ROX-12967-real-nodescan fbd0450 Fix style issues 17ccb31 Debug: let both scans finish to see what they return

github-actions bot added the area/helm label Jul 20, 2022

robbycochran force-pushed the rc-specific-compliance-host-mounts branch from ecde950 to a775748 Compare July 21, 2022 22:48

robbycochran changed the title ~~Use targeted paths for compliance mounts~~ ROX-9350 Use fine-grained host paths for compliance mounts Jul 21, 2022

robbycochran force-pushed the rc-specific-compliance-host-mounts branch 4 times, most recently from 6ee1f99 to 93bb9e6 Compare July 27, 2022 21:46

robbycochran force-pushed the rc-specific-compliance-host-mounts branch from 93bb9e6 to a9b7ea7 Compare September 6, 2022 22:54

robbycochran added the ci-all-qa-tests Tells CI to run all API tests (not just BAT). label Oct 24, 2022

robbycochran force-pushed the rc-specific-compliance-host-mounts branch 2 times, most recently from 5341bae to b16ba00 Compare October 26, 2022 21:19

vikin91 approved these changes Oct 27, 2022

View reviewed changes

RTann reviewed Nov 15, 2022

View reviewed changes

robbycochran added 5 commits November 15, 2022 19:14

X-Smart-Branch-Parent: origin/master

813ba7b

ROX-9350 - Use targeted paths for compliance mounts

a21297f

use subPath for shared mounts

d611bf7

use relative paths

f868c64

Add /host/lib

cfc82b5

robbycochran force-pushed the rc-specific-compliance-host-mounts branch from b16ba00 to cfc82b5 Compare November 16, 2022 03:15

Add changelog

af27644

robbycochran enabled auto-merge (squash) November 16, 2022 04:12

robbycochran merged commit 7623dec into master Nov 16, 2022

robbycochran deleted the rc-specific-compliance-host-mounts branch November 16, 2022 04:13

robbycochran added a commit that referenced this pull request Nov 16, 2022

Fix incorrect usage of subpath in #2479

ccf462b

robbycochran mentioned this pull request Nov 16, 2022

Fix collector volumeMounts #3826

Merged

1 task

robbycochran added a commit that referenced this pull request Nov 16, 2022

Revert "ROX-9350 Use fine-grained host paths for compliance mounts (#…

2db8aa0

…2479)" This reverts commit 7623dec.

robbycochran mentioned this pull request Nov 16, 2022

Revert ROX-9350 Use fine-grained host paths for compliance mounts #3827

Closed

1 task

robbycochran added a commit that referenced this pull request Nov 18, 2022

Revert "ROX-9350 Use fine-grained host paths for compliance mounts (#…

5d9e7d6

…2479)" This reverts commit 7623dec.

robbycochran added a commit that referenced this pull request Nov 18, 2022

Revert "ROX-9350 Use fine-grained host paths for compliance mounts (#…

c60a902

…2479)" This reverts commit 7623dec.

robbycochran mentioned this pull request Nov 18, 2022

Revert fine-grained host paths for compliance mounts #3850

Merged

5 tasks

janisz added a commit that referenced this pull request Nov 21, 2022

Revert "ROX-9350 Use fine-grained host paths for compliance mounts (#…

72c6bcc

…2479)" This reverts commit 7623dec.

janisz added a commit that referenced this pull request Nov 21, 2022

Revert "ROX-9350 Use fine-grained host paths for compliance mounts (#…

1e0140e

…2479)" This reverts commit 7623dec.

janisz added a commit that referenced this pull request Nov 21, 2022

Revert "ROX-9350 Use fine-grained host paths for compliance mounts (#…

c1e4953

…2479)" (#3860) This reverts commit 7623dec.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ROX-9350 Use fine-grained host paths for compliance mounts#2479

ROX-9350 Use fine-grained host paths for compliance mounts#2479
robbycochran merged 6 commits intomasterfrom
rc-specific-compliance-host-mounts

robbycochran commented Jul 20, 2022 •

edited

Loading

Uh oh!

ghost commented Jul 20, 2022 •

edited by ghost

Loading

Uh oh!

openshift-ci bot commented Oct 26, 2022 •

edited

Loading

Uh oh!

vikin91 left a comment

Uh oh!

vikin91 Oct 27, 2022

Uh oh!

robbycochran Nov 15, 2022 •

edited

Loading

Uh oh!

janisz commented Nov 15, 2022

Uh oh!

vikin91 commented Nov 15, 2022

Uh oh!

RTann Nov 15, 2022

Uh oh!

robbycochran Nov 15, 2022

Uh oh!

RTann Nov 17, 2022

Uh oh!

robbycochran commented Nov 15, 2022

Uh oh!

theencee commented Nov 15, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

robbycochran commented Jul 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Testing Performed

Uh oh!

ghost commented Jul 20, 2022 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci bot commented Oct 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vikin91 left a comment

Choose a reason for hiding this comment

Uh oh!

vikin91 Oct 27, 2022

Choose a reason for hiding this comment

Uh oh!

robbycochran Nov 15, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

janisz commented Nov 15, 2022

Uh oh!

vikin91 commented Nov 15, 2022

Uh oh!

RTann Nov 15, 2022

Choose a reason for hiding this comment

Uh oh!

robbycochran Nov 15, 2022

Choose a reason for hiding this comment

Uh oh!

RTann Nov 17, 2022

Choose a reason for hiding this comment

Uh oh!

robbycochran commented Nov 15, 2022

Uh oh!

theencee commented Nov 15, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

robbycochran commented Jul 20, 2022 •

edited

Loading

ghost commented Jul 20, 2022 •

edited by ghost

Loading

openshift-ci bot commented Oct 26, 2022 •

edited

Loading

robbycochran Nov 15, 2022 •

edited

Loading