Skip to content

ROX-11070: Add migration for groups bucket.#2066

Merged
dhaus67 merged 4 commits intomasterfrom
dh/06-14-Add_migration_for_groups
Jul 5, 2022
Merged

ROX-11070: Add migration for groups bucket.#2066
dhaus67 merged 4 commits intomasterfrom
dh/06-14-Add_migration_for_groups

Conversation

@dhaus67
Copy link
Contributor

@dhaus67 dhaus67 commented Jun 14, 2022 •
edited
Loading

Description

This is the fourth PR in a series of PRs regarding removing the composite key from the groups datastore, allowing users to specify different roles for the same group properties.

For full context, you can navigate to the previous PRs by using the comment created below.

This PR will create a migration for the groups bucket, adding a ID to all groups which don't have one associated with them as well as using the ID as the new key within the bucket instead of the composite key.

Testing Performed

  • see CI.

Manual testing:

  • create some auth providers (number as you like), create a couple of groups per auth provider:
curl -X POST https://<endpoint>/v1/groupsbatch -H "Authorization: Bearer <token>" -d ‘{"previous_groups":[{"props":{"authProviderId”:”<auth-provider-id>”},”roleName":"Admin"}],"required_groups":[{"props":{"authProviderId":"<auth-provider-id>","key":"name","value":"someday"},"roleName":"None"},{"props":{"authProviderId":"<auth-provider-id>","key":"name","value":"someway"},"roleName":"Scope Manager"},{"props":{"authProviderId":"<auth-provider-id>","key":"email","value":"someone"},"roleName":"Vulnerability Management Requester"},{"props":{"authProviderId":"<auth-provider-id>","key":"email","value":"somewhen"},"roleName":"Admin"},{"props":{"authProviderId":"<auth-provider-id>","key":"email","value":"something"},"roleName":"Vulnerability Report Creator"},{"props":{"authProviderId":"<auth-provider-id>","key":"groups","value":"somehow"},"roleName":"Vulnerability Management Approver"},{"props":{"authProviderId":"<auth-provider-id>","key":"groups","value":"somewhere"},"roleName":"Sensor Creator"},{"props":{"authProviderId":"<auth-provider-id>","key":"userid","value":"someday"},"roleName":"None"},{"props":{"authProviderId":"<auth-provider-id>","key":"userid","value":"sometime"},"roleName":"Continuous Integration"},{"props":{"authProviderId":"<auth-provider-id>","key":"userid","value":"somewhat"},"roleName":"Analyst"},{"props":{"authProviderId":"<auth-provider-id>"},"roleName":"Admin"}]}’
  • Ensure groups are added correctly within UI as well as returned via API:
curl https://dh0701wildlimittype.demo.stackrox.com/v1/groups  -H "Authorization: Bearer <token>"
  • Start the new tag of the image, ensure that the log indicates a successful migration + contains the sequence number 106:
pkg/migrations: 2022/07/01 02:59:55.857584 migration_version.go:54: Info: Migration version of database at /var/lib/stackrox/.db-64328521-bef9-4cbe-a584-87de0c203566: &{/var/lib/stackrox/.db-64328521-bef9-4cbe-a584-87de0c203566 3.68.1 92}
Migrator: 2022/07/01 02:59:55.857777 log.go:13: Info: starting DB compaction
Migrator: 2022/07/01 02:59:55.858396 log.go:18: Info: Free fraction of 0.0938 (24576/262144) is < 0.7500. Will not compact
Migrator: 2022/07/01 02:59:56.330146 log.go:18: Info: Found DB at version 92, which is less than what we expect (106). Running migrations...
Migrator: 2022/07/01 02:59:56.336617 log.go:18: Info: Removed 0 service accounts that don't belong to a valid cluster
Migrator: 2022/07/01 02:59:56.349655 log.go:18: Info: Removed 0 K8S roles that don't belong to a valid cluster
Migrator: 2022/07/01 02:59:56.356511 log.go:18: Info: Removed 0 K8S role bindings that don't belong to a valid cluster
Migrator: 2022/07/01 02:59:56.370269 log.go:18: Info: Successfully updated DB from version 92 to 93
Migrator: 2022/07/01 02:59:56.373519 log.go:18: Info: Successfully updated DB from version 93 to 94
Migrator: 2022/07/01 02:59:56.375939 log.go:18: Info: Successfully updated DB from version 94 to 95
Migrator: 2022/07/01 02:59:56.413153 log.go:18: Info: Successfully updated DB from version 95 to 96
Migrator: 2022/07/01 02:59:56.416439 log.go:18: Info: Successfully updated DB from version 96 to 97
Migrator: 2022/07/01 02:59:56.430744 log.go:18: Info: Successfully updated DB from version 97 to 98
Migrator: 2022/07/01 02:59:56.435394 log.go:18: Info: Successfully updated DB from version 98 to 99
Migrator: 2022/07/01 02:59:56.441711 log.go:18: Info: Successfully updated DB from version 99 to 100
Migrator: 2022/07/01 02:59:56.445257 log.go:18: Info: Successfully updated DB from version 100 to 101
Migrator: 2022/07/01 02:59:56.449060 log.go:18: Info: Successfully updated DB from version 101 to 102
Migrator: 2022/07/01 02:59:56.452615 log.go:18: Info: Successfully updated DB from version 102 to 103
Migrator: 2022/07/01 02:59:56.454663 log.go:18: Info: no policy exists for ID 38bf79e7-48bf-4ab1-b72f-38e8ad8b4ec3 in policy migration. Continuing
Migrator: 2022/07/01 02:59:56.458572 log.go:18: Info: Successfully updated DB from version 103 to 104
Migrator: 2022/07/01 02:59:56.461076 log.go:18: Info: Successfully updated DB from version 104 to 105
Migrator: 2022/07/01 02:59:56.474212 log.go:18: Info: Successfully updated DB from version 105 to 106
replica: 2022/07/01 02:59:56.475100 db_replica_manager.go:249: Info: Persisting upgraded replica: temp
version: 2022/07/01 02:59:57.043526 ensure.go:49: Info: Version found in the DB was current. We're good to go!
pkg/migrations: 2022/07/01 02:59:57.044053 migration_version.go:54: Info: Migration version of database at /var/lib/stackrox/current: &{/var/lib/stackrox/current 3.68.1 92}
  • After the migration, list the groups and verify all of the groups have now an ID associated with them.
  • Check within the UI that the groups are correctly listed within each auth provider, deletion and adding of new groups should work + no groups should be lost.

@openshift-ci
Copy link

openshift-ci bot commented Jun 14, 2022

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@dhaus67
Copy link
Contributor Author

dhaus67 commented Jun 14, 2022 •
edited
Loading

Current dependencies on/for this PR:

This comment was auto-generated by Graphite.

@dhaus67 dhaus67 force-pushed the dh/06-14-Add_migration_for_groups branch from 0bf669d to 7663a59 Compare June 14, 2022 19:24
@dhaus67 dhaus67 mentioned this pull request Jun 14, 2022
Merged
5 tasks
@ghost
Copy link

ghost commented Jun 14, 2022 •
edited by ghost
Loading

Tag for build #730863 is 3.70.x-572-gdee1f4b9a3.

💻 For deploying this image using the dev scripts, run the following first:

export MAIN_IMAGE_TAG='3.70.x-572-gdee1f4b9a3'

🕹️ A roxctl binary can be downloaded from the CircleCI artifacts.

@dhaus67 dhaus67 force-pushed the dh/06-14-Add_migration_for_groups branch from 7663a59 to 4efd782 Compare June 14, 2022 23:37
@dhaus67 dhaus67 force-pushed the dh/06-14-Update_service_and_datastore branch from 7239011 to f4894c6 Compare June 14, 2022 23:39
@dhaus67 dhaus67 force-pushed the dh/06-14-Add_migration_for_groups branch from 4efd782 to 42dce65 Compare June 14, 2022 23:39
@dhaus67 dhaus67 changed the base branch from dh/06-14-Update_service_and_datastore to dh/06-15-Handle_ID_of_groups_within_UI June 14, 2022 23:39
This was referenced Jun 14, 2022
Merged
Merged
@dhaus67 dhaus67 marked this pull request as ready for review June 14, 2022 23:58
@dhaus67 dhaus67 requested review from rukletsov and theencee June 14, 2022 23:59
@dhaus67 dhaus67 force-pushed the dh/06-15-Handle_ID_of_groups_within_UI branch from 98627b2 to d71d74b Compare June 15, 2022 14:04
@dhaus67 dhaus67 force-pushed the dh/06-14-Add_migration_for_groups branch from 42dce65 to d5f91cc Compare June 15, 2022 14:04
@dhaus67 dhaus67 force-pushed the dh/06-15-Handle_ID_of_groups_within_UI branch from d71d74b to bfbce16 Compare June 15, 2022 18:48
@dhaus67 dhaus67 force-pushed the dh/06-14-Add_migration_for_groups branch from d5f91cc to 20da2d4 Compare June 15, 2022 18:48
@janisz
Copy link
Contributor

janisz commented Jun 22, 2022

/test all

@dhaus67 dhaus67 force-pushed the dh/06-14-Add_migration_for_groups branch from cf7339b to 9984d29 Compare June 23, 2022 23:52
@dhaus67 dhaus67 requested a review from a team June 30, 2022 00:58
rukletsov
rukletsov requested changes Jun 30, 2022
View reviewed changes
@dhaus67 dhaus67 force-pushed the dh/06-15-Handle_ID_of_groups_within_UI branch from 437d113 to 9381fe0 Compare June 30, 2022 21:03
@dhaus67 dhaus67 force-pushed the dh/06-14-Add_migration_for_groups branch from 9984d29 to b33646c Compare June 30, 2022 21:03
@dhaus67 dhaus67 requested a review from rukletsov July 1, 2022 04:53
rukletsov
rukletsov reviewed Jul 1, 2022
View reviewed changes
Copy link
Member

@rukletsov rukletsov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have one question but beyond that this is beautiful. Thanks for timely addressing my feedback.

@dhaus67 dhaus67 requested a review from rukletsov July 4, 2022 01:11
@dhaus67 dhaus67 force-pushed the dh/06-15-Handle_ID_of_groups_within_UI branch from 9381fe0 to 666fb17 Compare July 4, 2022 02:42
@dhaus67 dhaus67 force-pushed the dh/06-14-Add_migration_for_groups branch from 9d640bb to de1901d Compare July 4, 2022 02:42
@dhaus67 dhaus67 mentioned this pull request Jul 4, 2022
Merged
rukletsov
rukletsov approved these changes Jul 4, 2022
View reviewed changes
@dhaus67 dhaus67 force-pushed the dh/06-15-Handle_ID_of_groups_within_UI branch from 666fb17 to 7db4450 Compare July 4, 2022 12:13
@dhaus67 dhaus67 force-pushed the dh/06-14-Add_migration_for_groups branch 2 times, most recently from a08a85d to 9f3e942 Compare July 4, 2022 17:07
Base automatically changed from dh/06-15-Handle_ID_of_groups_within_UI to master July 4, 2022 20:41
@dhaus67 dhaus67 force-pushed the dh/06-14-Add_migration_for_groups branch from 9f3e942 to dee1f4b Compare July 4, 2022 20:52
@ghost
Copy link

ghost commented Jul 4, 2022

Images are ready for the commit at dee1f4b.

To use with deploy scripts, first export MAIN_IMAGE_TAG=3.70.x-573-ga383e3045b.

@openshift-ci
Copy link

openshift-ci bot commented Jul 4, 2022

@dhaus67: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/gke-upgrade-tests dee1f4b link false /test gke-upgrade-tests

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@dhaus67
Copy link
Contributor Author

dhaus67 commented Jul 5, 2022

The CI failure is also occurring in the nightlies, I'll go ahead and merge this one for now.

@dhaus67 dhaus67 merged commit 4a01e82 into master Jul 5, 2022
@dhaus67 dhaus67 deleted the dh/06-14-Add_migration_for_groups branch July 5, 2022 12:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rukletsov rukletsov rukletsov approved these changes

@theencee theencee Awaiting requested review from theencee

Assignees

No one assigned

Labels

area/postgres team/merlin

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@dhaus67 @janisz @rukletsov