Skip to content

Datadog Agent → Vector TCP intake (port 8282): verifying tag transparency / no mutation #24459

@sathvik09

Description

@sathvik09

A note for the community

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Summary

We use Vector as a metrics aggregator between Datadog Agents and Datadog intake. After upgrading the Datadog Agent from 7.50.3 to 7.71.0, certain metric tags (e.g. process_shard:integrations-*) disappear downstream when metrics are forwarded through Vector.

Based on our investigation so far, it is unclear whether this behavior is the result of an upstream change in the Datadog Agent or something occurring downstream. We are opening this issue to maybe get a set of eyes on any Vector-side influence in this case.

Environment

  • Vector version: 0.44.0
  • Source type: socket (TCP)
  • Port: 8282
  • Protocol: non-HTTP, binary (Datadog Agent internal format)
  • Deployment: Kubernetes
  • Vector role: metrics aggregator / forwarder to Datadog

note: No Vector configuration changes were made between working and broken states.

Architecture (simplified)

Application Pods
→ DogStatsD (8125)
→ Datadog Agent
→ TCP (8282)
→ Vector
→ Datadog

note: Multiple Datadog Agent DaemonSets forward metrics to the same Vector instance.

Observed Behavior

Working state

Datadog Agent version: 7.50.3-jmx

Metrics forwarded through Vector include tags such as:

  • process_shard:integrations-*
  • service_family
  • customer_tier

Broken state

Datadog Agent version: 7.71.0-jmx (also reproducible on 7.68.0+)

Metrics still arrive, but certain tags are missing
Issue only occurs when Vector forwarding is enabled
Disabling Vector restores all tags

Vector logs and vector tap confirm that the missing tags are not present in events as received by Vector.

  • No Vector config changes between versions
  • Same Vector instance handles both working and broken agents
  • Tags are present inside the Datadog Agent prior to forwarding
  • Tags are missing by the time metrics reach Vector
  • Rolling back Agent immediately restores tags

Given this setup:

Can Vector’s TCP source or any default metrics handling mutate, filter, or drop tags from Datadog Agent–originated payloads without explicit transforms?

Configuration


Version

0.44.0

Debug Output


Example Data

No response

Additional Context

No response

References

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    type: bugA code related bug.

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions