Dashrews/ROX-13082 UUID protos generated PR 4 of 4 by dashrews78 · Pull Request #3698 · stackrox/stackrox

dashrews78 · 2022-11-04T10:27:24Z

Description

This is PR 4 of a 4 PR set.
PR 1: #3679
PR 2: #3681
PR 3: #3694

The previous PRs dealt with code generation, searcher changes, test constants and fixtures.
This PR deals with the utilization of all that other work.

Protos were updated to use type(uuid) for fields that are UUIDs.
The stores impacted by those protos were re-generated to deal with the UUID type.
Search code was updated and adjusted to handle postgres UUIDs.
Tests were added that use test proto and such within the searcher code to drive the test cases.
Additionally per a comment from @md2119 on PR Dashrews/ROX-13082 UUID searcher and common updates to set allow use of postgres UUID PR 1 of 4 #3679, I have added the support for regex matching since we can regex match on these columns today and it was essentially as simple as using ::text on the field.

There are a lot of files here, but that vast majority of them are generated.

Checklist

Investigated and inspected CI test results
Unit test and regression tests added
~~- [ ] Evaluated and added CHANGELOG entry if required~~
~~- [ ] Determined and documented upgrade steps~~
~~- [ ] Documented user facing changes (create PR based on [openshift/openshift-docs]~~(https://github.com/openshift/openshift-docs) and merge into rhacs-docs)

If any of these don't apply, please comment below.

Testing Performed

Added some UT to cover various UUID paths.

openshift-ci · 2022-11-04T10:27:29Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

ghost · 2022-11-04T11:03:14Z

Images are ready for the commit at ba01a5a.

To use with deploy scripts, first export MAIN_IMAGE_TAG=3.72.x-576-gba01a5a6bf.

dashrews78 · 2022-11-04T16:46:41Z

/test

openshift-ci · 2022-11-04T16:46:45Z

@dashrews78: No presubmit jobs available for stackrox/stackrox@dashrews/uuid-test-updates-pr3

Details

In response to this:

/test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

c-du

Overall looks good to me.

rhybrillou

Good job overall. Please check the comments.

rhybrillou · 2022-11-07T18:35:38Z

proto/storage/process_indicator.proto

 message ProcessIndicator {
-    // A unique uuid for the Indicator message
-    string id                                      = 1 [(gogoproto.moretags) = 'search:"Process ID,store,hidden"  sql:"pk"'];
+    // A unique type(uuid) for the Indicator message


// A unique uuid for the Indicator message.

ah yes, replace all got me on one of the iterations.

rhybrillou · 2022-11-07T18:36:08Z

proto/storage/process_indicator.proto


 message ProcessSignal {
-    // A unique UUID for identifying the message
+    // A unique type(uuid) for identifying the message


The old version is probably better here.

rhybrillou · 2022-11-07T18:47:30Z

migrator/migrations/n_30_to_n_31_postgres_network_flows/postgres/flow_impl.go


 	deleteStmt = "DELETE FROM network_flows WHERE Props_SrcEntity_Type = $1 AND Props_SrcEntity_Id = $2 AND Props_DstEntity_Type = $3 AND Props_DstEntity_Id = $4 AND Props_DstPort = $5 AND Props_L4Protocol = $6 AND ClusterId = $7"
-	walkStmt   = "SELECT nf.Props_SrcEntity_Type, nf.Props_SrcEntity_Id, nf.Props_DstEntity_Type, nf.Props_DstEntity_Id, nf.Props_DstPort, nf.Props_L4Protocol, nf.LastSeenTimestamp, nf.ClusterId FROM network_flows nf " + joinStmt
+	walkStmt   = "SELECT nf.Props_SrcEntity_Type, nf.Props_SrcEntity_Id, nf.Props_DstEntity_Type, nf.Props_DstEntity_Id, nf.Props_DstPort, nf.Props_L4Protocol, nf.LastSeenTimestamp, nf.ClusterId ::text FROM network_flows nf " + joinStmt + " WHERE nf.ClusterId = $1"


nf.ClusterId::text rather than with a whitespace between the column name and the cast.

rhybrillou · 2022-11-07T18:52:13Z

pkg/search/postgres/common_test.go

-			var actualJoins []string
-			for _, join := range actual.InnerJoins {
-				actualJoins = append(actualJoins, join.rightTable)
+			log.Infof("SHREWS -- selectedFields = %v", actual)


The debug data can be cleaned in the eventual state of the PR.

rhybrillou · 2022-11-07T18:54:35Z

proto/storage/alert.proto

        string       namespace      = 2 [(gogoproto.moretags) = 'search:"Namespace,store"'];
-        string       cluster_id     = 3 [(gogoproto.moretags) = 'search:"Cluster ID,store"'];
+        string       cluster_id     = 3 [(gogoproto.moretags) = 'search:"Cluster ID,store" sql:"type(uuid)"'];
        string       namespace_id   = 4 [(gogoproto.moretags) = 'search:"Namespace ID"'];


Should this (namespace_id) be a UUID too ?

rhybrillou · 2022-11-07T18:57:46Z

proto/storage/process_indicator.proto


    string container_name                          = 3 [(gogoproto.moretags) = 'search:"Container Name,hidden" policy:",prefer-parent"'];
    // Pod name
    string pod_id                                  = 4 [(gogoproto.moretags) = 'search:"Pod ID,hidden"'];


I think it is the pod_id that should be a UUID. I'm not sure what type the pod_uid should be.

@rhybrillou I had checked that out before. I'm not entirely sure why there are both fields, but pod_id is not a UUID and pod_uid is. In the code pod_uid is the one used to reference the id in the pods table.

id | podid | poduid --------------------------------------+--------------------------------------------------+-------------------------------------- 64a96d48-a800-5c7d-9379-1fe2668e2a29 | d6d1f550603c32897ae8b43c341d780ed816b6196fa353fa | 0185ac3d-6fff-5bfc-890c-a84a406fd652```

Is the comment pod name correct?

@md2119 yes, it appears the comment is correct. Obviously I'm not sure why someone called the name the podid.

central_active=# select p.id, p.name, pi.podid, pi.poduid from pods p inner join process_indicators pi on p.id = pi.poduid; id | name | podid | poduid --------------------------------------+--------------------------------------------------+--------------------------------------------------+-------------------------------------- 234e886d-86d9-58bd-bfda-2c2f7ea31725 | c8e42380aff61c233a68766a8d8047f92e5691f775c12117 | c8e42380aff61c233a68766a8d8047f92e5691f775c12117 | 234e886d-86d9-58bd-bfda-2c2f7ea31725

dcbc47c protos and generated stuff 3126391 group check failure c5f2f97 some testing 7d32a74 rebase fixes d5aae72 test fix feb6500 regenerate after rebase a5fb8a0 PR responses f331b66 generate after rebase 4d73ac4 some test stuff 004a595 working on some tests 15ccff3 some testing 334ff00 put regex support back 36d4e04 some test work 39126fa more tests

dashrews78 · 2022-11-09T16:18:05Z

/test go-unit-tests-release
/test go-unit-tests
/test go-postgres-tests

dashrews78 · 2022-11-09T16:23:28Z

/test go-postgres-tests

md2119 · 2022-11-09T22:19:31Z

proto/storage/process_indicator.proto


    string container_name                          = 3 [(gogoproto.moretags) = 'search:"Container Name,hidden" policy:",prefer-parent"'];
    // Pod name
    string pod_id                                  = 4 [(gogoproto.moretags) = 'search:"Pod ID,hidden"'];


Is the comment pod name correct?

md2119 · 2022-11-09T22:21:33Z

pkg/search/postgres/query/test/uuid/index_test.go

+	log = logging.LoggerForModule()
+)
+
+type SingleUUIDIndexSuite struct {


dashrews78 · 2022-11-10T10:14:30Z

/test gke-nongroovy-e2e-tests

7ffc6be ROX-13368: Skip failing nongroovy tests on PG (#3721) bbdd7a0 Bump github.com/gofrs/uuid from 4.3.0+incompatible to 4.3.1+incompatible (#3642) 1f253f2 Bump github.com/google/certificate-transparency-go from 1.1.3 to 1.1.4 (#3543) d434c8d [ROX-13030] : Add delete collection API endpoint and service implementation (#3648) f062c21 Dashrews/ROX-13253 wait for central-db to come back after bounce and allow FATAL connection lost error (#3537) edc1174 CI: Fill the gaps for https://testgrid.k8s.io/ (#3715) 86d7c54 ROX-13231: use passed context when non-postgres (#3540) 9093195 Add less specific type for BE collection response string (#3728) 5abb652 Only enable ROX_OBJECT_COLLECTIONS feature flag during gke-postgres-ui-e2e job (#3727) 4f64cd1 Add centralDBOnly mode in render (#3707) d67bbe5 Dashrews/ROX-13082 UUID searcher and common updates to set allow use of postgres UUID PR 1 of 4 (#3679) 6f829d5 ROX-13259: graphInit called during init time (#3705) f3bc50d ROX-13380: Conditional rendering edges for deployments and namespaces (#3641) 3764476 ROX-12319: implement smoke test step with groovy test filter (#3220) f202fd4 ROX-11826: Disable kernel support package uploads for managed central (#3661) 61f03dc ROX-11101: Remove deprecated resources from central (#3115) e6aa6d7 ROX-11101: Restore Role permission in UI (#3428) 3203e04 ROX-11101: Remove deprecated resources (#3036) a35f41e Bump golang.org/x/sys from 0.1.0 to 0.2.0 (#3733) 4120524 Bump snakeyaml from 1.29 to 1.33 in /qa-tests-backend (#3732) e1785c0 Bump github.com/coreos/go-systemd/v22 from 22.4.0 to 22.5.0 (#3724) 870df4a Bump google.golang.org/api from 0.101.0 to 0.102.0 (#3723) 721454c Generalize User-Agent setup (#3672) 6a11bf0 Bumps collector version to 3.11.x-145-gc345f72f5e (#3736) ec5d343 [ROX-12923] Walk retries - remainder work (#3729) 40f3d43 ROX-13440: Replace ambiguous central with sensor in networkGraph integration test (#3730) 281ed22 Bump groovy-xml from 2.5.18 to 2.5.19 in /qa-tests-backend (#3741) 49d1651 Bump github.com/prometheus/client_golang from 1.13.1 to 1.14.0 (#3742) b5544aa Bump cloud.google.com/go/storage from 1.27.0 to 1.28.0 (#3743) 9c61e53 ensure CVSS is present for istio vulns (#3706) ae29d52 ROX-13452: don't always clobber scoped ctx when non-postgres (#3748) 517bf05 ROX-13261: DryRunUpdate on collection datastore (#3687) baf7654 ROX-13378: Group new resources with deprecated in UI (#3690) 569922f ROX-13421: Enable roxctl netpol generate and add tech-preview messages (#3740) 2465fc5 Dashrews/ROX-13082 UUID generator templates PR 2 of 4 (#3681) c093c68 Bump slack-api-client from 1.20.2 to 1.27.0 in /qa-tests-backend (#3752) 2c860bb Bump ubi8-micro from 8.6 to 8.7 in /operator (#3751) 80eb04c Make deploy.sh and deploy-local.sh pass shellcheck (#3582) 2182b43 Dashrews/ROX-13082 UUID test updates PR 3 of 4 (#3694) 6dc6ca5 [ROX-13403] : Fix node -> topVuln sub resolver bug when node cves is empty (#3689) 1b21361 Move integration tests for page title from general to specific containers (#3675) e1a9f31 Bump google.golang.org/api from 0.102.0 to 0.103.0 (#3773) a05ea31 Bump golang.org/x/crypto from 0.1.0 to 0.2.0 (#3772) 65ddf4f ROX-12824: Add roxctl commands to generate Central DB bundle (#3602) c3f1e2f Remove obsolete authProviders request for Integrations page (#3759) 7ccd54d Dashrews/ROX-13082 UUID protos generated PR 4 of 4 (#3698) 9ab5c8f cleanup image digest utilities (#3764) 187ed44 ROX-11931: Convert junit failure artifacts to Slack attachments (#3438) b5d8790 ROX-13432: leaning up unused code copied/pasted from topology demo (#3750) ab05bfc Refactor collection form page for better composition (#3744) c5562f7 Remove babel devDependencies in ui-components (#3761) 2b90b3a Extract collection form from drawer wrapper layout (#3745) a779fc9 [ROX-12625 + ROX-13032] : Add GetCollectionCount and UpdateCollection endpoints and services (#3749) e77f0da Upgrade cypress 11.0.0 devDependencies in ui (#3760) a3fba94 ROX-13068: Use real data for deployment details (#3688) 4c7d90e ROX-12617: Collection to search query converter (#3683) 3e98aec ROX-13067: fill out port configurations section of deployment details (#3714) a48de36 ROX-12835: Add support for NodeScanV2 to Sensor (#3533) 30c5dc7 ROX-13466: Fix deletion of groups with empty properties (#3756) 5cb2470 Add autocomplete for name selector dropdowns (#3676) b9a75ad ROX-13464 adding flows dropdown in NG (#3763) 3217a67 [ROX-13500] Perform type check for V1 CronJob (#3787) af3790d Remove bulk delete from collections table (#3776) dda123b Add more info in migration log (#3788) 179f0c9 ROX-13502: Remove the circular dependency between cluster datastore init and cscc notifier init (#3790) 029d584 Update SCANNER_VERSION (#3774) cbca57c Bump github.com/ckaznocha/protoc-gen-lint from 0.2.4 to 0.3.0 (#3783) 3613b56 Bump golang.org/x/tools from 0.2.0 to 0.3.0 (#3782) 5fc0a6a Bump github.com/google/go-containerregistry from 0.12.0 to 0.12.1 (#3781) 1d1c687 Bump controller-gen version to 0.10.0 (#3754) c3a5290 Untie documentation link from the product version (#3799) ed822aa use correct package for migration (#3784) 397a0b4 Validate that label keys are valid k8s labels and ensure correct key splitting (#3777) edd1050 Rename variable ScannerGRPCEndpoint to ScannerSlimGRPCEndpoint (#3657) 6662c9f ROX-13378: Access Control page permissions (#3720) b0e73c5 fix Operator reconciliation for external Central DB (#3796) b83bc1f ROX-13505: Fix error log scanning the postgres stat collection (#3795) ca660cb Prevent the collection being edited from displaying in its own embedded list (#3778) 3f7b3fc [ROX-13441][POSTGRES] Propagate context correctly in retries (#3793) e0cbc6f ROX-12839: Update changelog to announce removal of in-product docs (#3805) 696e8bc [ROX-12358] Follow up on vulnerability request proto change (#2851) c4b46d8 Change getCollectionCount endpoint and updateCollection request type 5f2efbc remove make proto-fmt (#3804) 0c75540 Remove os.Std* from roxctl/central (#3758) 25a90de Add ability to view embedded collections in a pop up modal (#3747) 5c1bf81 ROX-13240: fix scanner-slim updates when WebSockets are used (#3704) 1d98577 Add more context to jira notifier logging (#3812) da2fd28 ROX-13031: DryRun Collection API (#3766) 1c418d5 Test data migration code in postgres tests (#3803) ed95b37 Update UI Collection requests for BE compatibiltiy (#3762) 09cc188 ROX-11931: Fix junit-parse install in CI (#3811) d2b01e3 ROX-12814: Disable PolicyFieldsTest on openshift. (#3797) d10ce27 ROX-13345: disable 'missing required registry' aspect on openshift (#3798) 3d22396 Update collector to 3.12 (#3809) 1eb33fb ROX-13347: Modify scope queries to included quoted cluster and nameace names, to allow exact matches instead of erroneous and unintended prefix matches. (#3767) 3811a69 ROX-12621: list collection selectors api (#3806) f6d3f9d Add migration for groups with invalid values (#3789) cc21125 Bugfixes for collection autocomplete (#3816) 7623dec ROX-9350 Use fine-grained host paths for compliance mounts (#2479) b4bf5c2 Fix collector volumeMounts (#3826) 0e9be05 ROX-12953: figure out last 4 versions of sensor automatically (#3611) 459c7ae ROX-12814: Add proper todo for reenabling the test (#3817) 9ee40ff ROX-13523: add isEnabled enum to central db spec (#3815) 535bc72 Replace requestConfig with routeMatcherMap in helper functions for integration tests (#3686) 2b75b61 `gosec` G104: Add `ShouldErr(err)` that returns `err` (#3830) fb1b82f WIP: Introduce nodescan call 35f8a8f WIP: Prepare converter 716144b Moved and renamed fake nodescan tests 4748de7 Introduce real node scanner with conversion functions 5e6d9a8 wip: real scanner 0169868 wip: log results 1438d2c wip: Debug Analyze call b09894c wip: Debug Analyze call 3ceed72 wip: Update and improve debug logs d1669fd Remove copied lib, bump scanner version, add debug 14a3f73 Merge branch 'master' into mm/ROX-12967-real-nodescan fbd0450 Fix style issues 17ccb31 Debug: let both scans finish to see what they return

openshift-ci bot added the do-not-merge/work-in-progress label Nov 4, 2022

github-actions bot added area/central area/postgres labels Nov 4, 2022

dashrews78 changed the base branch from dashrews/uuid-test-updates-pr3 to master November 4, 2022 10:28

dashrews78 marked this pull request as ready for review November 4, 2022 10:28

dashrews78 requested review from a team as code owners November 4, 2022 10:28

openshift-ci bot removed the do-not-merge/work-in-progress label Nov 4, 2022

github-actions bot added the area/auth label Nov 4, 2022

dashrews78 force-pushed the dashrews/uuid-protos-generated-pr4 branch from 2f9395e to 280681b Compare November 4, 2022 13:44

dashrews78 changed the base branch from master to dashrews/uuid-test-updates-pr3 November 4, 2022 15:05

dashrews78 changed the title ~~Dashrews/UUID protos generated PR 4 of 4~~ Dashrews/ROX-13082 UUID protos generated PR 4 of 4 Nov 4, 2022

dashrews78 changed the base branch from dashrews/uuid-test-updates-pr3 to master November 4, 2022 16:47

dashrews78 force-pushed the dashrews/uuid-protos-generated-pr4 branch from de4672f to 2f2057b Compare November 4, 2022 18:59

dashrews78 changed the base branch from master to dashrews/uuid-test-updates-pr3 November 4, 2022 19:46

dashrews78 requested review from c-du, md2119 and rhybrillou November 4, 2022 19:48

janisz approved these changes Nov 4, 2022

View reviewed changes

c-du reviewed Nov 4, 2022

View reviewed changes

dashrews78 force-pushed the dashrews/uuid-test-updates-pr3 branch from 551e716 to ce36928 Compare November 7, 2022 14:33

dashrews78 force-pushed the dashrews/uuid-protos-generated-pr4 branch from a7fb199 to feb6500 Compare November 7, 2022 15:18

dashrews78 changed the base branch from dashrews/uuid-test-updates-pr3 to master November 7, 2022 15:32

dashrews78 changed the base branch from master to dashrews/uuid-test-updates-pr3 November 7, 2022 16:34

rhybrillou reviewed Nov 7, 2022

View reviewed changes

dashrews78 force-pushed the dashrews/uuid-test-updates-pr3 branch from ce36928 to b2a2dbd Compare November 8, 2022 15:19

Base automatically changed from dashrews/uuid-test-updates-pr3 to master November 9, 2022 16:09

dashrews78 added 7 commits November 9, 2022 11:09

X-Smart-Branch-Parent: master

18479e5

search changes

a44e94e

PR feedback

5ebdb6f

X-Smart-Branch-Parent: dashrews/uuid-search-pr1

291b6e0

X-Smart-Branch-Parent: dashrews/uuid-search-pr2

f435344

X-Smart-Branch-Parent: dashrews/uuid-test-updates-pr3

9267efd

dashrews78 force-pushed the dashrews/uuid-protos-generated-pr4 branch from 39126fa to 366fa4f Compare November 9, 2022 16:12

dashrews78 added 7 commits November 9, 2022 11:54

style

59e0d7c

try to fix build prob

da2149c

moving some stuff for style

a48f4d3

fix graph test to use parent 4 with a uuid

14e5a38

try to see what all fails migration on upgrade

b76cf48

move policy back to string to ensure imports are ok

e6da1d2

deal with cluster health migration

f15496b

dashrews78 requested review from c-du, md2119 and rhybrillou and removed request for rhybrillou November 9, 2022 21:17

style

ba01a5a

md2119 approved these changes Nov 9, 2022

View reviewed changes

c-du approved these changes Nov 9, 2022

View reviewed changes

dashrews78 merged commit 7ccd54d into master Nov 10, 2022

dashrews78 deleted the dashrews/uuid-protos-generated-pr4 branch November 10, 2022 17:39

Conversation

dashrews78 commented Nov 4, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Testing Performed

Uh oh!

openshift-ci bot commented Nov 4, 2022

Uh oh!

ghost commented Nov 4, 2022 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dashrews78 commented Nov 4, 2022

Uh oh!

openshift-ci bot commented Nov 4, 2022

Uh oh!

c-du left a comment

Choose a reason for hiding this comment

Uh oh!

rhybrillou left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dashrews78 Nov 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dashrews78 Nov 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dashrews78 commented Nov 9, 2022

Uh oh!

dashrews78 commented Nov 9, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dashrews78 commented Nov 10, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

dashrews78 commented Nov 4, 2022 •

edited

Loading

ghost commented Nov 4, 2022 •

edited by ghost

Loading

dashrews78 Nov 7, 2022 •

edited

Loading

dashrews78 Nov 10, 2022 •

edited

Loading