Executive Summary

The TestPod E2E test failed because process filtering introduced in March 2026 is too aggressive and drops short-lived processes (/bin/date, /bin/sleep) that the test expects to observe. This is a regression introduced by commit df1f1e84aa (ROX-32679) after the test was previously "unflaked" in October 2025.

Timeline of TestPod History

Phase 1: Early Flakiness (Pre-Oct 2025)

• Test was historically flaky due to unreliable process event detection
• Multiple skip/unskip cycles: cab7af93bb, c3a9e96f1c added skips

Phase 2: Major Unflake Effort (Oct 2025)

• Oct 24, 2025 - 8c2c913c9f - ROX-29771: Unflake TestPods (major refactor)

  • Removed testT.Skip()
  • Added setupMultiContainerPodTest() helper for robust setup
  • Added retry logic (30 attempts over 150 seconds)
  • Changed from exact match to "at least" semantics for process events
  • Initially only required /usr/sbin/nginx (commented out /bin/sh, /bin/date, /bin/sleep)

• Oct 28, 2025 - 39ca19f7da - ROX-29771: Ensure Sensor is up

  • Added waitForSensorHealthy() check before test
  • Prevents pipeline delays from previous tests

• Oct 30, 2025 - 76e375a63a - ROX-31331: Require Collector reporting all processes

  • Made test STRICTER: removed workaround, now requires ALL 4 processes:

    requiredProcesses := []string{"/usr/sbin/nginx", "/bin/sh", "/bin/date", "/bin/sleep"}

  • This assumed Collector would reliably detect short-lived processes

Phase 3: Flakiness Returns (Feb 2026)

• Feb 10, 2026 - 5515da9706 - Re-disabled test as flaky
  • Added testT.Skip("Flaky: https://issues.redhat.com/browse/ROX-29771")
  • Test was failing again despite October's unflake efforts

Phase 4: Breaking Change (Mar 2026)

• Mar 5, 2026 - df1f1e84aa - ROX-32679: Preset process filter levels
  • Introduced ROX_PROCESS_FILTER_MODE with three presets:
    • aggressive: MaxExactPathMatches=1, FanOutLevels=[]
    • default: MaxExactPathMatches=5, FanOutLevels=[8,6,4,2]
    • minimal: MaxExactPathMatches=100, FanOutLevels=[20,15,10,5]
  • DEFAULT MODE is too aggressive for short-lived processes
  • Designed to prevent "container spamming" by limiting repeated process executions
  • This is the likely culprit for the current failures

Phase 5: Current State (Apr 2026)

• Apr 3, 2026 - Build b61d01eaddcb8e79eb758847bfaeec191a37efef tested
  • Test was ENABLED (skip removed at some point between Feb 10 and Apr 3)
  • Test FAILED with missing processes: /bin/date and /bin/sleep
  • Only saw: /usr/sbin/nginx, /bin/sh, /usr/sbin/nginx (duplicate)

Root Cause Analysis

Primary Issue: Process Filter Too Aggressive

The default process filter configuration introduced in ROX-32679:

• MaxExactPathMatches: 5 - limits exact path matches
• FanOutLevels: [8,6,4,2] - limits fan-out tree depth
• Designed to prevent spam from containers that spawn many short-lived processes

Why it breaks the test:

1. /bin/date executes for milliseconds in a loop
2. /bin/sleep 1 executes for 1 second repeatedly
3. Filter sees these as "spam" and drops them after initial detections
4. Test expects to see these processes but they're filtered out

Secondary Issue: Event Collection Delays

The investigation showed a ~100-second delay before ANY events appeared in Central:

• Pod created: 13:50:33 UTC
• First events: ~13:52:12 UTC (~1min 39sec delay)
• Test timeout: 13:53:03 UTC (exhausted all retries)

This suggests pipeline issues beyond just filtering.

Test Expectations vs Reality

Test expects (from tests/yamls/multi-container-pod.yaml):

• Container "1st" (nginx): /usr/sbin/nginx (long-lived) ✅
• Container "2nd" (ubuntu): /bin/sh running loop ✅
  • Loop executes: date >> /html/index.html; sleep 1 repeatedly
  • Expected: /bin/date ❌ FILTERED
  • Expected: /bin/sleep ❌ FILTERED

Responsible Team

@stackrox/sensor-ecosystem

This involves:

• Runtime process event collection (Collector → Sensor → Central pipeline)
• Process filtering implementation (sensor/common/processfilter/, pkg/process/filter/)
• Event ingestion and processing

Permanent Solution

Immediate Fix (Option A): Disable Filtering in CI

# In .openshift-ci/dispatch.sh or test deployment configuration
export ROX_PROCESS_FILTER_MODE=minimal

This ensures E2E tests capture all process events for validation.

Immediate Fix (Option B): Make Test More Robust

Change test to use longer-lived processes that won't be filtered:

# In tests/yamls/multi-container-pod.yaml
command: ["/bin/sh"]
args:
  - -c
  - |
    while true; do
      /usr/bin/uptime >> /html/index.html
      sleep 30
    done

Then update expected processes to match.

Long-term Improvements

1. Test Improvements (tests/pods_test.go):

   • Add diagnostics to verify process filter mode is compatible with test
   • Add explicit pre-check that event pipeline is working
   • Better error messages showing which processes are missing and why
   • Consider testing with different filter modes

2. Process Filter Improvements (pkg/process/filter/):

   • Add debug logging when processes are filtered (at configurable level)
   • Add metrics for filter hit rates and dropped processes
   • Consider exempting certain "test" namespaces from aggressive filtering
   • Document filter behavior and tuning guidance

3. Event Pipeline Investigation:

   • Investigate 100-second delay in event collection
   • Check for buffering, resource constraints, or backpressure issues
   • Add metrics for end-to-end event latency (Collector → Central)

4. Documentation:

   • Document process filtering behavior in AGENTS.md or test docs
   • Add guidance on when to use different filter modes
   • Explain CI environment configuration

Key Files and Locations

Test Code:

• tests/pods_test.go:134-169 - TestPod function
• tests/yamls/multi-container-pod.yaml - Test pod definition
• tests/common.go - Test helper functions

Process Filtering:

• pkg/env/process_filter_mode.go - Filter mode configuration (ROX_PROCESS_FILTER_MODE)
• pkg/env/process_filter.go - Individual filter settings
• pkg/process/filter/filter.go - Core filter implementation
• sensor/common/processfilter/filter.go - Sensor filter singleton
• central/processindicator/filter/singleton.go - Central filter singleton

CI Configuration:

• .openshift-ci/dispatch.sh - CI test dispatch script
• Environment variables for test configuration

Related Issues

• ROX-29771 - Original flaky test tracking issue
• ROX-31331 - Collector process detection improvements
• ROX-32679 - Process filter preset modes (introduced regression)

Recommendations

1. Immediate: Set ROX_PROCESS_FILTER_MODE=minimal in CI test environments
2. Short-term: Update test to use longer-lived processes or verify filter compatibility
3. Medium-term: Add observability to process filtering (logs, metrics)
4. Long-term: Improve filter logic to handle test scenarios better

Investigation Artifacts

• Build ID: 2040047831439380480
• Build URL: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/branch-ci-stackrox-stackrox-master-merge-gke-nongroovy-e2e-tests/2040047831439380480
• Investigation directory: /tmp/investigation-2040047831439380480/
• Tested commit: b61d01eaddcb8e79eb758847bfaeec191a37efef
• Test failure time: ~13:53:03 UTC on April 3, 2026