This PR replaces pod-level retry logic with HTTP client-level retries using hashicorp/go-retryablehttp to provide better network
resilience for GKE API connectivity issues while working within existing 10-second context timeouts.

Problem

Our CI tests are experiencing frequent network timeout failures, particularly in GKE environments:

• ROX-30485, ROX-30486, ROX-30470: context deadline exceeded errors on GKE endpoints (34.133.244.129, 34.56.5.71)
• ROX-29713, ROX-29356: GRPC connection resolution failures and TLS establishment issues
• General network instability: Transient network issues causing test flakiness

Current approach was to add retry logic at the pod creation/deletion level, but this is less efficient than handling retries at the HTTP
transport layer.

Solution

HTTP Client Retry Implementation

• go-retryablehttp integration into the Kubernetes client HTTP transport using the existing WrapTransport pattern
• Fast retry strategy designed to work within existing 10-second context timeouts
• Smart retry logic that only retries on network errors and server errors (5xx, 429)

Key Configuration

// Individual request timeout: 3 seconds
retryClient.HTTPClient.Timeout = 3 * time.Second

// Retry policy: 3 retries with exponential backoff
retryClient.RetryMax = 3
retryClient.RetryWaitMin = 500 * time.Millisecond
retryClient.RetryWaitMax = 2 * time.Second

Timeline within 10-second context:

• Attempt 1: 0s → 3s (timeout) → 0.5s delay
• Attempt 2: 3.5s → 6.5s (timeout) → 1s delay
• Attempt 3: 7.5s → 10s (context deadline)

This allows up to 3 retry attempts within the existing 10-second context window.

Implementation Details

Custom Retry Policy

  retryClient.CheckRetry = func(ctx context.Context, resp *http.Response, err error) (bool, error) {
      if err != nil {
          // Retry on network errors (timeouts, connection refused, etc.)
          logf(t, "K8s API network error, will retry: %v", err)
          return true, nil
      }

      if resp != nil {
          // Retry on 5xx server errors and 429 (rate limit)
          if resp.StatusCode >= 500 || resp.StatusCode == 429 {
              logf(t, "K8s API server error %d, will retry", resp.StatusCode)
              return true, nil
          }
      }

      return false, nil
  }

Transport Integration

Uses the established WrapTransport pattern found in:

• pkg/k8sintrospect/collector.go:58-64
• sensor/kubernetes/upgrade/process.go
• sensor/upgrader/upgradectx/context.go

Benefits

✅ Network Resilience: Automatic retry for transient network issues
✅ Backward Compatibility: Works with existing 10-second context timeouts
✅ Smart Retry Logic: Only retries appropriate errors, not client mistakes
✅ Comprehensive Logging: Debug visibility into retry attempts
✅ Existing Dependency: Uses hashicorp/go-retryablehttp v0.7.8 already in go.mod
✅ Proven Pattern: Follows established WrapTransport usage in codebase

Testing

• ✅ Code compiles successfully with go build -tags test_e2e
• ✅ Uses existing dependency (hashicorp/go-retryablehttp v0.7.8)
• ✅ Follows established patterns in the codebase
• 🧪 Ready for CI testing against problematic GKE endpoints

Related Issues

This implementation directly addresses:

• ROX-30485: GKE API timeout with endpoint 34.133.244.129
• ROX-30486: GKE API timeout with endpoint 34.56.5.71
• ROX-30470: Context deadline exceeded in GKE tests
• ROX-29713: GRPC connection resolution failures
• ROX-29356: Service discovery and TLS establishment issues

Test Plan

1. Monitor CI runs for reduced timeout failures in GKE environments
2. Check retry logs to verify appropriate retry behavior
3. Verify no regression in test execution times
4. Confirm fast failure for genuine API errors (4xx responses)