Skip to content

ROX-13596: Determine user-agent in gRPC-gateway and HTTP#3924

Merged
parametalol merged 2 commits intomasterfrom
ivan/ROX-13596-extract-user-agent-http-grpc
Dec 21, 2022
Merged

ROX-13596: Determine user-agent in gRPC-gateway and HTTP#3924
parametalol merged 2 commits intomasterfrom
ivan/ROX-13596-extract-user-agent-http-grpc

Conversation

@ivan-degtiarenko
Copy link
Contributor

@ivan-degtiarenko ivan-degtiarenko commented Nov 28, 2022

Description

This is fix to telemetry interceptors to correctly determine User-Agent.

The reason why the request user-agent value was ignored in gRPC case is that:

  1. To transform from HTTP to GRPC request, we connect to our gRPC server as a client from within the central

    stackrox/pkg/grpc/server.go

    Lines 352 to 353 in 0529062

    dialCtxFunc := a.listenOnLocalEndpoint(grpcServer)
    localConn, err := a.connectToLocalEndpoint(dialCtxFunc)

    stackrox/pkg/grpc/server.go

    Lines 311 to 317 in 0529062

    if localConn != nil {
    for _, service := range a.apiServices {
    if err := service.RegisterServiceHandler(context.Background(), gwMux, localConn); err != nil {
    log.Panicf("failed to register API service: %v", err)
    }
    }
    }
  2. Connection for this client is created once during central initialization, thus we cannot override it on every request.
  3. Value for user-agent dial option is not specified in our code, thus we use the default

    stackrox/pkg/grpc/server.go

    Lines 226 to 233 in 0529062

    func (a *apiImpl) connectToLocalEndpoint(dialCtxFunc pipeconn.DialContextFunc) (*grpc.ClientConn, error) {
    return grpc.Dial("", grpc.WithTransportCredentials(insecure.NewCredentials()),
    grpc.WithContextDialer(func(ctx context.Context, endpoint string) (net.Conn, error) {
    return dialCtxFunc(ctx)
    }),
    grpc.WithDefaultCallOptions(grpc.MaxCallRecvMsgSize(maxResponseMsgSize())),
    )
    }

Checklist

  • Investigated and inspected CI test results
  • Unit test and regression tests added
  • Evaluated and added CHANGELOG entry if required
  • Determined and documented upgrade steps
  • Documented user facing changes (create PR based on openshift/openshift-docs and merge into rhacs-docs)

If any of these don't apply, please comment below.

Testing Performed

  1. UI calls are tracked with browser User-Agent

Before in telemetry(UI)

{
  "context": {
    "Central ID": "ff82bc70-a9e9-471e-bd8b-93a02f20caf2",
    "library": {
      "name": "analytics-go",
      "version": "3.0.0"
    }
  },
  "event": "API Call",
  "integrations": {},
  "messageId": "191a6bd1-fc65-4f4a-9553-0652ad5f795b",
  "originalTimestamp": "2022-11-29T18:09:24.161145124Z",
  "properties": {
    "Code": 200,
    "Path": "/v1/clusters",
    "User-Agent": "grpc-go/1.51.0"
  },
  "receivedAt": "2022-11-29T18:10:03.809Z",
  "sentAt": "2022-11-29T18:10:03.715Z",
  "timestamp": "2022-11-29T18:09:24.254Z",
  "type": "track",
  "userId": "local:ff82bc70-a9e9-471e-bd8b-93a02f20caf2:admin",
  "writeKey": "R5fMyO9n0gibSGzOXtlP2qCFWCGb8uoW"
}

After in telemetry(UI)

{
  "context": {
    "Central ID": "d88edb55-30e3-4cea-9ead-7cfac8b381d8",
    "Tenant ID": "",
    "library": {
      "name": "analytics-go",
      "version": "3.0.0"
    }
  },
  "event": "API Call",
  "integrations": {},
  "messageId": "7c677284-cd9a-4b83-aca7-27f434449262",
  "originalTimestamp": "2022-11-29T18:10:24.099219Z",
  "properties": {
    "Code": 200,
    "Path": "/v1/clusters",
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36"
  },
  "receivedAt": "2022-11-29T18:11:19.819Z",
  "sentAt": "2022-11-29T18:11:17.238Z",
  "timestamp": "2022-11-29T18:10:26.679Z",
  "type": "track",
  "userId": "sso:4df1b98c-24ed-4073-a9ad-356aec6bb62d:admin",
  "writeKey": "R5fMyO9n0gibSGzOXtlP2qCFWCGb8uoW"
}
  1. roxctl is tracked with roxctl/<version user agent
  2. /v1/ping is tracked with kube-probe user agent

Comment on lines +65 to +74

if throughGrpcGateway {
// By default, all permanent HTTP headers are added grpcgateway- prefix:
// https://github.com/grpc-ecosystem/grpc-gateway/blob/master/runtime/mux.go#L106-L114
// User-Agent header is occupied with internal grpc-go value:
// https://github.com/grpc/grpc-go/blob/0238b6e1cec37b55820b461d3d30652c54efe2c4/clientconn.go#L211-L215
userAgentKey := fmt.Sprintf("%suser-agent", runtime.MetadataPrefix)
userAgent = strings.Join(ri.Metadata.Get(userAgentKey), ", ")
}
// Fallback to HTTP
if !throughGrpcGateway || userAgent == "" {
userAgent = strings.Join(ri.Metadata.Get("User-Agent"), ", ")
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May we just try to find the grpcgatewayuser-agent key and use it if found or fallback to User-Agent otherwise?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the code here has clearer intent: if grpc-gateway is used, use grpc-gateway header; if it does not work or we are not on gRPC protocol, use HTTP header. It would be unexpected to see grpcgateway-user-agent in the HTTP interceptor, and I would think we should ignore such header in such context.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there is too much code for this intent: you even had to add more parameters to two functions and compute the normal User-Agent twice.
Instead, you could move this grpc related check (i.e. try to read the grpcgatewayuser-agent header, and you don't need any bool parameter) to the beginning of the function, and put the existing User-Agent retrieval under else.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rukletsov rukletsov requested review from a team and rukletsov November 29, 2022 08:44
}
// Fallback to HTTP
if !throughGrpcGateway || userAgent == "" {
userAgent = strings.Join(ri.Metadata.Get("User-Agent"), ", ")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW, with such branching you don't need userAgent declaration at L43. If you keep the current structure, pull L43 right before branching and make it var userAgent string. However, I'm in favor of Michaël's suggestion below.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment on lines +65 to +74

if throughGrpcGateway {
// By default, all permanent HTTP headers are added grpcgateway- prefix:
// https://github.com/grpc-ecosystem/grpc-gateway/blob/master/runtime/mux.go#L106-L114
// User-Agent header is occupied with internal grpc-go value:
// https://github.com/grpc/grpc-go/blob/0238b6e1cec37b55820b461d3d30652c54efe2c4/clientconn.go#L211-L215
userAgentKey := fmt.Sprintf("%suser-agent", runtime.MetadataPrefix)
userAgent = strings.Join(ri.Metadata.Get(userAgentKey), ", ")
}
// Fallback to HTTP
if !throughGrpcGateway || userAgent == "" {
userAgent = strings.Join(ri.Metadata.Get("User-Agent"), ", ")
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

@parametalol parametalol left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

"google.golang.org/grpc"
)

const grpcGatewayUserAgentHeader = runtime.MetadataPrefix + "user-agent"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
const grpcGatewayUserAgentHeader = runtime.MetadataPrefix + "user-agent"
const grpcGatewayUserAgentHeader = runtime.MetadataPrefix + "User-Agent"

I understand that metadata.MD converts keys to lowercase but this is not obvious here, is not common knowledge, and who knows when it changes. Canonical form is "User-Agent".

}

// By default, all permanent HTTP headers in grpc-gateway are added grpcgateway- prefix:
// https://github.com/grpc-ecosystem/grpc-gateway/blob/master/runtime/mux.go#L106-L114
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// https://github.com/grpc-ecosystem/grpc-gateway/blob/master/runtime/mux.go#L106-L114
// https://github.com/grpc-ecosystem/grpc-gateway/blob/8952e38d5addd28308e29c272c696a578aa8ace8/runtime/mux.go#L106-L114

Always press y in GitHub to get a link with pinned line references.

// By default, all permanent HTTP headers in grpc-gateway are added grpcgateway- prefix:
// https://github.com/grpc-ecosystem/grpc-gateway/blob/master/runtime/mux.go#L106-L114
// User-Agent header is occupied with internal grpc-go value:
// https://github.com/grpc/grpc-go/blob/0238b6e1cec37b55820b461d3d30652c54efe2c4/clientconn.go#L211-L215
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's actually not clear to me what this code reference proves. cc.dopts.copts.UserAgent += " " + grpcUA hints that User-Agent value is not overwritten but appended to, so it will be something like ACS roxctl user agent blah grpc-go/1.50.1. Which sort of contradicts what's claimed in ROX-13596.

Could you please split the "Testing Performed" section into before / after?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding cc.dopts.copts.UserAgent: this shows where the user agent's value grpc-go/1.51.0 is initially taken from.

Answering the question of why there's no other user-agent value:

  1. To transform from HTTP to GRPC request, we connect to our gRPC server as a client from within the central

    stackrox/pkg/grpc/server.go

    Lines 352 to 353 in 0529062

    dialCtxFunc := a.listenOnLocalEndpoint(grpcServer)
    localConn, err := a.connectToLocalEndpoint(dialCtxFunc)

    stackrox/pkg/grpc/server.go

    Lines 311 to 317 in 0529062

    if localConn != nil {
    for _, service := range a.apiServices {
    if err := service.RegisterServiceHandler(context.Background(), gwMux, localConn); err != nil {
    log.Panicf("failed to register API service: %v", err)
    }
    }
    }
  2. Connection for this client is created once during central initialization, thus we cannot override it on every request.
  3. Value for user-agent dial option is not specified in our code, thus we use the default

    stackrox/pkg/grpc/server.go

    Lines 226 to 233 in 0529062

    func (a *apiImpl) connectToLocalEndpoint(dialCtxFunc pipeconn.DialContextFunc) (*grpc.ClientConn, error) {
    return grpc.Dial("", grpc.WithTransportCredentials(insecure.NewCredentials()),
    grpc.WithContextDialer(func(ctx context.Context, endpoint string) (net.Conn, error) {
    return dialCtxFunc(ctx)
    }),
    grpc.WithDefaultCallOptions(grpc.MaxCallRecvMsgSize(maxResponseMsgSize())),
    )
    }

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I see, didn't know that we persist the connection — thanks. This makes sense.

It does not make sense to capture your reply as a code comment because it builds on non-local knowledge which might change any time and render the comment misleading, but it does make sense to explain that in the PR description.

One last thing: do you want to follow up and augment grpc server init code with User-Agent for HTTP muxer?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One last thing: do you want to follow up and augment grpc server init code with User-Agent for HTTP muxer?

If I understand correctly what you suggest, that would add some additional user-agent value to all requests going through the muxer. Essentially, what @0x656b694d did here:
https://github.com/stackrox/stackrox/pull/3672/files#diff-af62cb3a3a7c777bd1b8feed18657c3ecae951d2dbdea55527f08c8c67967c71R233

I think it works as a fallback option. It's better than displaying grpc-go/1.51.0, so why not?

I had a small concern regarding introducing more variety to potential user-agent values in Segment, but Segment supports contains filter, so that shouldn't be a problem.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, essentially what's done in #3672, so my comment should be irrelevant. Why do we still see "User-Agent": "grpc-go/1.51.0" in the Before section then?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"grpc-go/1.51.0" is always appended by the gogrpc library to the User-Agent, should we set or not anything with WithUserAgent.

Copy link
Member

@rukletsov rukletsov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, Ivan. Please update the PR description per my recent comment and 🚢 it.

@parametalol parametalol force-pushed the ivan/ROX-13596-extract-user-agent-http-grpc branch from 0529062 to 00170b0 Compare December 12, 2022 16:40
@ivan-degtiarenko ivan-degtiarenko changed the base branch from michael/ROX-12713-built-in-segment to michael/ROX-13582-phonehome-interceptor December 13, 2022 10:24
@ivan-degtiarenko ivan-degtiarenko force-pushed the ivan/ROX-13596-extract-user-agent-http-grpc branch from 00170b0 to 63c3298 Compare December 13, 2022 11:20
@parametalol parametalol force-pushed the michael/ROX-13582-phonehome-interceptor branch from b88ffc0 to 8e09a92 Compare December 14, 2022 15:31
@ivan-degtiarenko ivan-degtiarenko force-pushed the ivan/ROX-13596-extract-user-agent-http-grpc branch from 5cfdcb9 to 101fdee Compare December 14, 2022 16:56
Copy link
Contributor

@parametalol parametalol left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@parametalol parametalol force-pushed the michael/ROX-13582-phonehome-interceptor branch 3 times, most recently from bf1637b to 3b370cf Compare December 19, 2022 17:11
Base automatically changed from michael/ROX-13582-phonehome-interceptor to master December 20, 2022 15:53
Co-authored-by: Ivan Degtiarenko <78353299+ivan-degtiarenko@users.noreply.github.com>
@parametalol parametalol force-pushed the ivan/ROX-13596-extract-user-agent-http-grpc branch from 728c7e3 to 800e342 Compare December 21, 2022 09:36
@ghost
Copy link

ghost commented Dec 21, 2022

Images are ready for the commit at 5bb6250.

To use with deploy scripts, first export MAIN_IMAGE_TAG=3.73.x-288-g5bb6250b64.

@parametalol parametalol merged commit 479eed5 into master Dec 21, 2022
@parametalol parametalol deleted the ivan/ROX-13596-extract-user-agent-http-grpc branch December 21, 2022 14:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants