Description

This PR implements solutions for the following goals:

• Stop accumulating network flows and endpoints in Sensor's memory when running with fake workloads
• Periodically purge network flows and endpoints from Sensor's memory that were never removed by the enrichment process.
• Implement & refactor a large group of metrics around the enrichment to get a better picture around the process.

How this part of the code works

The enrichment process is there to add data known to Sensor to the data sent by Collector.
Sensor keeps track of open connections (network flows) and endpoints that participate in connections.
When Collector sends an update (every 30s), Sensor analyzes the data and does the following procedure:

1. Stores the data from Collector in the connectionsByHost map. This map can be understood as a queue of updates to be enriched.
2. The data is periodically (every 30s) checked by Sensor according to the following schema:

• calculate a diff between the update from Collector and the state of connections and endpoints known in Sensor.
• if there is any new information from Collector or the result of the enrichment is different than in the last cycle: send update to Central.
• all connections and endpoints that are active (not closed) are stored in Sensor's memory (activeConnections & activeEndpoints), because further updates from Collector are expected.
• if there is data missing required for enrichment, Sensor puts back the item to the connectionsByHost map, so that the enrichment can be tried again in the next cycle.
• all successfully enriched entities are removed from the connectionsByHost map.
• additionally, Sensor may delete the data from the connectionsByHost map if it knows that the data is faulty or incomplete and there is no chance for it to work in the next cycle.

The goal is that at the end of each 30s cycle, the connectionsByHost map contains only those entries that should be retried again in the next cycle. It is expected that over longer time periods the size of connectionsByHost map fluctuates around a known (relatively small) number.

Adding the purger

Unfortunately, we observed that in some cases connections and endpoints were never deleted from the connectionsByHost map. Moreover, in each enrichment cycle they were copied to the activeConnections & activeEndpoints slices and were also stuck there. The enrichment process is expected to handle those data structures so that data stays there only for a limited time, but that was not the case.

To make sure that no data is stuck in Sensor, I add a purger in this PR. The purger periodically (30m) checks the three data structures (activeConnections, activeEndpoints, and connectionsByHost) and checks a list of rules whether a given entry can be removed. The ultimate rule is the max-age where the purger checks the time the data resides in Sensor and deletes it if it has not been enriched for a given time (default 4h).

In clusters with high workload, the purger takes up to 2s (based on metrics collected for a long-running cluster) to complete the cleanup. During that time, it locks two mutexes, so I made sure that it runs rarely (twice per hour). On smaller clusters the runtime of purger is negligible.

User-facing documentation

• CHANGELOG is updated OR update is not needed
• documentation PR is created and is linked above OR is not needed

Testing and quality

• the change is production ready: the change is GA or otherwise the functionality is gated by a feature flag
• CI results are inspected

Automated testing

• added unit tests
• added e2e tests
• added regression tests
• added compatibility tests
• modified existing tests

How I validated my change

Mainly by long-running cluster and checking metrics. I have been observing long-running clusters for about 2 weeks until I got to the point where:

• activeEndpoints, activeConnections, and connectionsByHost in the networkFlowManager are not constantly growing
• memory consumption of Sensor over longer time periods is not constantly growing (the way it used to before).

BEFORE: Applying purger only to activeConnections & activeEndpoints

AFTER: Purger cleans activeConnections, activeEndpoints, and connectionsByHost

BEFORE: Typical CPU & memory consumption for fake workloads long-running cluster

AFTER: CPU & memory consumption for fake workloads long-running cluster (with purger)

Summary by Sourcery

Implement a purger mechanism to prevent memory accumulation in Sensor's network flow tracking by periodically cleaning up stale network flows, endpoints, and connections

New Features:

• Introduced a purger mechanism that periodically checks and removes stale network flows, endpoints, and connections from Sensor's memory
• Added configurable max age and cycle time for the purger via environment variables
• Implemented detailed metrics tracking for purger events and performance

Enhancements:

• Refactored network flow tracking to support historical container tracking
• Added more granular metrics for network flow enrichment and tracking
• Improved memory management in Sensor's network flow processing

CI:

• Updated CI configurations to support new environment variables for purger settings

Tests:

• Added unit tests for purger functionality across different scenarios
• Implemented test cases for purging host connections, active endpoints, and active connections

Chores:

• Updated interfaces and mocks to support historical container tracking
• Added comprehensive test coverage for the new purger mechanism