Skip to content

ROX-9128: Create certSecretsRepo type to store certificates in k8s secrets#457

Merged
juanrh merged 49 commits intomasterfrom
juanrh/ROX-9128
Feb 14, 2022
Merged

ROX-9128: Create certSecretsRepo type to store certificates in k8s secrets#457
juanrh merged 49 commits intomasterfrom
juanrh/ROX-9128

Conversation

@juanrh
Copy link
Contributor

@juanrh juanrh commented Jan 26, 2022

Checklist

  • Investigated and inspected CI test results
  • Unit test and regression tests added
  • Evaluated and added CHANGELOG entry if required
  • Determined and documented upgrade steps

CHANGELOG entry and upgrade steps not required.

Testing Performed

Added unit test. New test are passing with go test -count=100 -race -v -p=1 github.com/stackrox/rox/sensor/kubernetes/localscanner -run TestServiceCertificatesRepoSecretsImpl

@ghost
Copy link

ghost commented Jan 26, 2022

Tag for build #210093 is 3.68.x-190-g0bc9854aaa.

💻 For deploying this image using the dev scripts, run the following first:

export MAIN_IMAGE_TAG='3.68.x-190-g0bc9854aaa'

📦 You can also generate an installation bundle with:

docker run -i --rm stackrox/main:3.68.x-190-g0bc9854aaa central generate interactive > bundle.zip

🕹️ A roxctl binary artifact can be downloaded from CircleCI.

Copy link
Contributor

@porridge porridge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The names and comments here are a bit abstract... I think I can see that this is a wrapper over a secrets client, but what value does it provide? Reading and writing collections of secrets? Perhaps mention that in the docstrings?

Copy link
Contributor

@SimonBaeumer SimonBaeumer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good but would like to keep it simpler. In this case removing the exponential backoff logic because it is already taken care of by the ticker.

Btw, great to see the effort you put into testing!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why the amount of service types a limit? I think the benefit is that the implementation is dynamic right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we pass more secret names to newFixture than the size of serviceTypes then serviceTypes[i] will get out of bounds, this is a check to ensure we don't write tests wrong. We could change newFixture to replace secretNames ...string with a map[storage.ServiceType]string to prevent that, but newFixture is just a quick test utils function so I wanted to make it easy to call.
We cannot use serviceTypes[i %len(serviceTypes)] because then we would be overriding the value for a service type in the secretsMap map[storage.ServiceType]*v1.Secret that the repo is persisting

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've finally removed the s.Require().LessOrEqual because newFixture now only accepts a single secret name, which is what the tests are using. See next comment for more context

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to be more explicit here. It's a bit off that the first element of the given string array represents a specific service. Also your tests only ever use the first element.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when I wrote newFixture I wasn't sure how many secrets we'd be adding to the fake client set. So I'll rewrite this so newFixture accepts a single secret name, and simplify the creation of these maps removing the look and the s.Require().LessOrEqual above. This should make the test easier to read

@juanrh
Copy link
Contributor Author

juanrh commented Jan 31, 2022

The names and comments here are a bit abstract... I think I can see that this is a wrapper over a secrets client, but what value does it provide? Reading and writing collections of secrets? Perhaps mention that in the docstrings?

The value is basically in making the certificate refresher a bit simpler to understand by moving the responsibility of persisting the secrets to this type, by implementing the repository pattern. It's not a big win, but I think separating this is not a big effort either. I'll add a docstring to NewCertSecretsRepo to make this more clear

Copy link
Contributor

@porridge porridge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we're trying to abstract away kubernetes by using the repository pattern, I wonder whether it would make sense to match this abstraction level in the function signatures, and accept/return the keys/values that are stored in a secret, rather than the whole Secret type?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this a problem? Perhaps the caller does not want to update all previously declared secrets this time?
What worries me more is that we're silently ignoring entries in secrets that were not declared at repo initialization time.

@juanrh
Copy link
Contributor Author

juanrh commented Feb 1, 2022

Since we're trying to abstract away kubernetes by using the repository pattern, I wonder whether it would make sense to match this abstraction level in the function signatures, and accept/return the keys/values that are stored in a secret, rather than the whole Secret type?

The motivation for having a separate certSecretsRepo type is splitting the code into cohesive pieces, and the usual feature of a repository of abstracting the specific storage system is not very useful in this case, because we are already coupled to k8s in many ways. In order to change this repository to handle values of type map[storage.ServiceType]secretData for type secretData map[string][]byte then we'd need need to add more information to secretData like ObjectMeta and all the other fields in type Secret struct that would be an additional development effort that I don't see would give much valuable, as we are already coupled with k8s. On the other hand, a less ambitious repository that handle values of type map[storage.ServiceType]*v1.Secret already helps splitting the coding work into sensible units. Maybe calling this "repository" is the problem here? What about certSecretCRUD instead?

@juanrh juanrh requested a review from porridge February 1, 2022 18:18
@porridge
Copy link
Contributor

porridge commented Feb 2, 2022

FTR, we talked with Juan offline and he agreed to remove the use of v1.Secret from the API of the repository. This way versioning concerns and packing/unpacking data into/out of Secrets will be constrained to that are of the code which additionally helps modularity.

@juanrh
Copy link
Contributor Author

juanrh commented Feb 2, 2022

FTR, we talked with Juan offline and he agreed to remove the use of v1.Secret from the API of the repository. This way versioning concerns and packing/unpacking data into/out of Secrets will be constrained to that are of the code which additionally helps modularity.

I've changed this to a repository of *storage.TypedServiceCertificateSet. I think this makes sense because:

Please let me know what you think

Copy link
Contributor

@SimonBaeumer SimonBaeumer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did not reviewed the test until now. S
I think this repo should be concrete for scanner atm. It can be extended following the next iterations.
My main argument is that the differences are in the data structures and logic how to persist the certificate rather than having a "real" generic solution here.
I fear introducing the abstractions here leads to a lot of unnecessary development with little value.

In contrast the ticker and synchronous messaging are patterns which could be repeated savely without breaking its abstractions.
(Whereas we decided against making sync messaging over gRPC streams because of the efforts)

Comment on lines 45 to 69
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be honest I don't understand this, can you explain why this is necessary?
I would expect the secrets repo knows how to convert between v1.Secret <> storage.TypedServiceCertificateSet and communicating to the k8s API.

Copy link
Contributor Author

@juanrh juanrh Feb 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is to ensure the invariant "All secrets with non nil Data store the same CA PEM." A difference between storage.TypedServiceCertificateSet and v1.Secret is that the same CA is repeated in different secrets, while it appears only once in storage.TypedServiceCertificateSet.
But on second though that check is not really needed here because we just use secrets for the ObjectMeta but we ignore the Data in those secrets. So I'll remove it from here and turn it into an error in getServiceCertificates when different CAs are found for different secrets

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
func newServiceCertificatesRepoWithSecretsPersistence(secrets map[storage.ServiceType]*v1.Secret,
func newServiceCertificatesRepo(secrets map[storage.ServiceType]*v1.Secret,

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also I think the repository can be initialized with all service types currently supported instead of passing them directly to it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea here is that the repository will use secrets as a template for storing secrets in k8s, and it will only modify the secret Data with the certificates received in storage.TypedServiceCertificateSet. That way the sensor component can handle the logic of setting up ObjectMeta with suitable OwnerReferences, and the repository doesn't care about that.
This is documented in the invariant "secrets and secretsClient are read-only, except the field Data of the entries in secrets" in the docstring for type serviceCertificatesRepoSecretsImpl

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rename done

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why checking here the secrets? The repo should be stateless and not know about the underlying data.
Imho it is up to the application layer to ensure that the CA cert are configured correctly.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same reason as I mentioned above: we are using the secrets as templates, be able to use the secret names for reading from k8s and to persist in k8s with the right ObjectMeta

@juanrh
Copy link
Contributor Author

juanrh commented Feb 3, 2022

Did not reviewed the test until now. S I think this repo should be concrete for scanner atm. It can be extended following the next iterations. My main argument is that the differences are in the data structures and logic how to persist the certificate rather than having a "real" generic solution here. I fear introducing the abstractions here leads to a lot of unnecessary development with little value.

In contrast the ticker and synchronous messaging are patterns which could be repeated savely without breaking its abstractions. (Whereas we decided against making sync messaging over gRPC streams because of the efforts)

I agree this is localscanner code in the end. However, as seen in my answers above, we are using secrets map[storage.ServiceType]*v1.Secret to create the repo in order to have access to the secrets name and ObjectMeta so the scanner component that crates the repository takes care of the ownership of the secrets. So that parameter is there not to be general but to leave that responsibility to the sensor component.

@juanrh juanrh requested a review from SimonBaeumer February 3, 2022 16:42
Copy link
Contributor

@SimonBaeumer SimonBaeumer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still think this repo should not support other services like it does it now. We can also discuss this offline if we don't agree.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I lost track on the discussions, so commenting here.
Why should the serviceCertificateSecret be passed to the repo in the constructor?
To me it provides no value (expect DI) because we know upfront which storage.ServiceType are supported and which not.
Now every time the repo is used I need to construct the map[storage.ServiceType]serviceCertificateSecret and pass it to the repo? 🤔

I still think this should only be implemented for scanner certs for now and avoiding pre-optimizations.

The repo currently only wraps the k8s api and adds type conversion logic, for this the current implementation introduces complexity which is not necessary.
Let's keep it simple, if we need to support other services we still can add them.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why should the serviceCertificateSecret be passed to the repo in the constructor?

As discussed in #457 (comment) and #457 (comment) this repository is only concerned with modifying the secret data, not with creating secrets, or the secret name or ownership. By keeping that responsebility out of the repo we are making the repo simpler.
serviceCertificateSecret contains that information which is not created by the repository as a *v1.Secret, and also the paths for the certificates in the secret. All that information is constant for the life of the repository and that is why it is passed in the constructor. That way getServiceCertificates and putServiceCertificates have less parameters and are easier to use, and harder to use wrong.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we know upfront which storage.ServiceType are supported and which not

I understand that you mean we could instead pass two serviceCertificateSecret one for scanner and another for scannerDB, or we could use instead

type serviceCertificateSecrets struct { 
    scannerSecret serviceCertificateSecret
    scannerDBSecret serviceCertificateSecret
}

That is the same information in map[storage.ServiceType]serviceCertificateSecret, but with a type that is less flexible. I can change map[storage.ServiceType]serviceCertificateSecret to accept two serviceCertificateSecrets, but I see no value in changing the field secrets map[storage.ServiceType]serviceCertificateSecret in serviceCertificatesRepoSecretsImpl.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

expect DI

I don't know what "DI" means, can you please explain?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DI = Dependency Injection

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still think this is not necessary, instead a valid secret must be passed on every put to the repo.
If the ownerRef is necessary it should be injected instead.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

requiring passing additional parameters that would be constant for the life of the repository would make putServiceCertificates harder to use, which might lead to more errors

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// we don't know where to persist this.
// we don't know how to persist this.

@juanrh
Copy link
Contributor Author

juanrh commented Feb 10, 2022

Some additional tests are missing, I'll complete the tests when we agree on the production code

@juanrh
Copy link
Contributor Author

juanrh commented Feb 10, 2022

A problem with the current implementation of createSecrets is that if we get a k8s api error creating one of the secrets, after we have created the other secret, then retrying when using this with the refresher will always fail because Create will return an error if the secret already existed. Something similar will happen if a customer deletes one secret and tries to restart Sensor as a workaround, unless the customer deletes both secrets. Deleting both secrets is a workaround, so you think this would work as it is? Another alternative is checking IsAlreadyExists in createSecrets to not fail if the secret already existed, or use createSecret in the caller, which breaks the repository abstraction a bit more, although we could live with that

Copy link
Contributor

@SimonBaeumer SimonBaeumer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks very good! Only smaller style suggestions after that ready to merge 💪

sensorDeployment *appsApiv1.Deployment, secretName string) (*v1.Secret, error) {
secret, err := r.secretsClient.Get(ctx, secretName, metav1.GetOptions{})

if k8sErrors.IsNotFound(err) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use secretClient.Create directly here, no need to check existence beforehand.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this comment seems to be outdated, at 8456e48 createSecret uses Create directly

// service type.
// - if the data for a secret is missing some expecting key then the corresponding field in the TypedServiceCertificate
// for that secret will contain a zero value.
func (r *serviceCertificatesRepoSecretsImpl) GetServiceCertificates(ctx context.Context) (*storage.TypedServiceCertificateSet, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Return error here if secrets were not owned by the deployment. The GetServiceCertificates is then used to determine whether the ticker stops or not.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this comment seems to be outdated, at 8456e48 getServiceCertificates returns ErrUnexpectedSecretsOwner if the owner is not as expected

secretsClient corev1.SecretInterface
}

// serviceCertSecretSpec species the name of the secret where certificates for a service are stored, and
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// serviceCertSecretSpec species the name of the secret where certificates for a service are stored, and
// serviceCertSecretSpec specifies the name of the secret where certificates for a service are stored, and

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

// - Otherwise, fails with ErrUnexpectedSecretsOwner in case the owner specified in the constructor is not the sole owner of all secrets.
// - Otherwise, fails with ErrMissingSecretData in case any secret has no data.
// - Otherwise, fails with ErrDifferentCAForDifferentServiceTypes in case the CA is not the same in all secrets.
// - Otherwise, fails with any k8s API error.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this doc line can be removed, at least to me it is clear that this happens in all cases where a connection is necessary.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment on lines +110 to +119
// TODO: if this works abstract as "highestPriorityError" func
if _, ok := errors[ErrUnexpectedSecretsOwner]; ok {
return nil, ErrUnexpectedSecretsOwner
}
if _, ok := errors[ErrMissingSecretData]; ok {
return nil, ErrMissingSecretData
}
if _, ok := errors[ErrDifferentCAForDifferentServiceTypes]; ok {
return nil, ErrDifferentCAForDifferentServiceTypes
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not using multierror and returning all errors at the same time?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just investigated and I learned that errors.Is can be used to determine whether or not an error appears inside a multierror, so I'll follow your suggestion and use errors.Is in the client code

Comment on lines +134 to +136
"successful put": {nil, s.newFixture("")},
"failed put": {errForced, s.newFixture("patch")},
"cancelled put": {context.Canceled, s.newFixture("")},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Named params please

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment on lines +64 to +66
"successful get": {nil, s.newFixture("")},
"failed get": {errForced, s.newFixture("get")},
"cancelled get": {context.Canceled, s.newFixture("")},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

named params please

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

fixture *certSecretsRepoFixture
}{
"successful put": {nil, s.newFixture("")},
"failed put": {errForced, s.newFixture("patch")},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why should it fail here? Can you described that in the test description?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These errors are forced in the k8s API mock, by passing "patch" to the verbToError parameter of newFixture, which uses clientSet.CoreV1().(*fakecorev1.FakeCoreV1).PrependReactor to force the error for the verb. I've renamed the test name here to "failed put due to k8s API error", and also in TestGet that was doing something similar

s.Error(repo.createSecrets(ctx, certificates.Clone()))
}

func (s *serviceCertificatesRepoSecretsImplSuite) checkExpectedError(expectedErr, err error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did you created this function?
If errors are checked I expected either checking the type with s.ErrorIs or if not available checking the error msg with s.Contains(err.Error(), "expected string").

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

replaced with s.ErrorIs, nice to know about this

return s.newFixtureAdvancedOpts(verbToError, false)
}

func (s *serviceCertificatesRepoSecretsImplSuite) newFixtureAdvancedOpts(verbToError string, emptySecretData bool,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I don't like about that signature is that verbToError magically gets empty strings passed which disturbes the reading flow of a test.
I need to first look at newFixtureAdvancedOpts and understand what's going on under the hood.

Can you use a struct as the input parameter which is more expressive instead of multiple parameters?
This makes it more obvious because the passed parameters become named automatically (or using constants for verbToError and emptySecretData).

I know a lot of our unit tests don't do that - and that's not good because it gets frustrating debugging these.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

verbToError is the HTTP verb of the k8s API for which we force an error in the fake k8s client set, so by passing "" we specify that all verbs that we'll use will work fine.
I created a certSecretsRepoFixtureSpec struct as suggested, and unified newFixtureAdvancedOpts and newFixture, into just newFixture that takes that struct. I've also added comments to both newFixture and certSecretsRepoFixtureSpec to clarify how this behaves

@SimonBaeumer
Copy link
Contributor

A problem with the current implementation of createSecrets is that if we get a k8s api error creating one of the secrets, after we have created the other secret, then retrying when using this with the refresher will always fail because Create will return an error if the secret already existed. Something similar will happen if a customer deletes one secret and tries to restart Sensor as a workaround, unless the customer deletes both secrets. Deleting both secrets is a workaround, so you think this would work as it is? Another alternative is checking IsAlreadyExists in createSecrets to not fail if the secret already existed, or use createSecret in the caller, which breaks the repository abstraction a bit more, although we could live with that

I think we may want to combine Update and Create functions into a Ensure function instead. We are not interested in Update or Delete, only "Hey Kubernetes, Give me my desired state".
For this a func Ensure which determines between Create and Put on every secret.

@juanrh
Copy link
Contributor Author

juanrh commented Feb 11, 2022

I think we may want to combine Update and Create functions into a Ensure function instead. We are not interested in Update or Delete, only "Hey Kubernetes, Give me my desired state".
For this a func Ensure which determines between Create and Put on every secret.

I have replaced put and create with ensure, please let me know what you think. Some additional tests are required, but I'm leaving that for when we reach and agreement for the production code

@juanrh juanrh requested a review from SimonBaeumer February 11, 2022 16:55
func (r *serviceCertificatesRepoSecretsImpl) ensureServiceCertificate(ctx context.Context, caPem []byte,
cert *storage.TypedServiceCertificate, secretSpec serviceCertSecretSpec) error {
patchErr := r.patchServiceCertificate(ctx, caPem, cert, secretSpec)
if k8sErrors.IsNotFound(patchErr) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the simplicity and readability of this function!

}
}

type certSecretsRepoFixtureSpec struct {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spec is misleading because typically Spec is reserved for k8s objects (i.e. PodSpec). For this Config or Options is better suited.

Suggested change
type certSecretsRepoFixtureSpec struct {
type certSecretsRepoFixtureConfig struct {

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

emptySecretData bool
// These keys will be removed from the data keys of the secret used to initialize the fake k8s client set.
missingSecretDataKeys []string
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding docs!

@juanrh
Copy link
Contributor Author

juanrh commented Feb 14, 2022

completed tests

@juanrh juanrh requested a review from SimonBaeumer February 14, 2022 09:36
Copy link
Contributor

@SimonBaeumer SimonBaeumer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/LGTM!

@juanrh juanrh merged commit 2d1cfb4 into master Feb 14, 2022
@juanrh juanrh deleted the juanrh/ROX-9128 branch February 14, 2022 12:54
RTann pushed a commit that referenced this pull request Apr 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants