Skip to content

ci: split Go unit test job into parallel shards #19702

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
guzalv wants to merge 3 commits into
base: master
Choose a base branch
from
Draft

ci: split Go unit test job into parallel shards #19702

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
41df802
ci: split Go unit test job into parallel component shards
guzalv Mar 30, 2026
5d63808
test: add deliberate sensor test failure for shard isolation validation
guzalv Mar 30, 2026
38dbc41
test: revert deliberate sensor test failure
guzalv Mar 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 65 additions & 4 deletions .github/workflows/unit-tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,25 @@ concurrency:
cancel-in-progress: true

jobs:
go:
go-unit-tests:
name: "Go Unit Tests (${{ matrix.component.name }}, ${{ matrix.gotags }})"
strategy:
fail-fast: false
matrix:
gotags: [ 'GOTAGS=""', 'GOTAGS=release' ]
component:
- name: central
pkg_include: 'stackrox/rox/central/'
pkg_exclude: '^$'
- name: pkg
pkg_include: 'stackrox/rox/pkg/'
pkg_exclude: '^$'
- name: sensor
pkg_include: 'stackrox/rox/(sensor|compliance)/'
pkg_exclude: '^$'
- name: other
pkg_include: '.'
pkg_exclude: 'stackrox/rox/(central|pkg|sensor|compliance)/'
runs-on: ubuntu-latest
outputs:
new-jiras: ${{ steps.junit2jira.outputs.new-jiras }}
Comment on lines 40 to 41
Copy link
Copy Markdown

@coderabbitai coderabbitai bot Mar 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Check GitHub Actions documentation or existing patterns in the repo for matrix output handling
rg -n "outputs:" --type yaml -A5 .github/workflows/ | head -60

Repository: stackrox/stackrox

Length of output: 3776

🏁 Script executed:

cat -n .github/workflows/unit-tests.yaml | head -150

Repository: stackrox/stackrox

Length of output: 5288

🏁 Script executed:

cat -n .github/workflows/unit-tests.yaml | tail -n +150 | head -200

Repository: stackrox/stackrox

Length of output: 7078

🏁 Script executed:

cat -n .github/workflows/unit-tests.yaml | tail -n +350

Repository: stackrox/stackrox

Length of output: 8503

🏁 Script executed:

# Verify how matrix job outputs work in GitHub Actions
# Search for any documentation or comments about this limitation
rg -i "matrix.*output|last.*completed" .github/workflows/ -A3 -B3

Repository: stackrox/stackrox

Length of output: 14667

🌐 Web query:

GitHub Actions matrix job outputs limitation last completed shard only

💡 Result:

Yes, GitHub Actions matrix jobs have a limitation where job-level outputs (defined in jobs.<job_id>.outputs) are overwritten by each matrix shard, and dependent jobs only receive the output from the last completed shard. This is a known behavior since matrix jobs share the same job ID, and outputs are not preserved per shard. Official documentation on job outputs and matrix strategies does not mention per-shard output preservation, confirming the limitation. A 2023 runner PR (#2477) aimed to support matrix context in output keys (e.g., outputs: ${{ matrix.os }}_result: ...), but server-side changes were pending, and no changelog entry confirms full rollout by 2026-03-30. Workarounds include: - Upload per-shard data as artifacts from matrix jobs, download and aggregate in a dependent job. - Use third-party actions like cloudposse/github-action-matrix-outputs-{write,read}. - Predefine fixed output names matching matrix values (not dynamic). Example artifact workaround: jobs: test: strategy: matrix: runner: [DEV, TEST] steps: - run: jq -cn --arg runner ${{ matrix.runner }} --arg outcome success '$ARGS.named' > outcome.json - uses: actions/upload-artifact@v4 with: name: outcome-${{ matrix.runner }} path: outcome.json aggregate: needs: test steps: - uses: actions/download-artifact@v4 - id: collect run: | jq -s . outcomes/*.json > all-outcomes.json echo "outcomes=$(cat all-outcomes.json)" >> $GITHUB_OUTPUT

Citations:

Matrix job outputs capture only the last completed shard, not all shards.

For matrix jobs, GitHub Actions only captures the output from the last completed matrix job, not an aggregation of all. Since go-unit-tests runs 8 parallel jobs (4 components × 2 gotags), the new-jiras output will only reflect one shard's result, potentially missing Jira tickets created by other shards. The same applies to go-extra-tests and go-postgres.

Consider implementing an aggregation job that collects outputs from all matrix shards using artifact uploads and aggregation, rather than relying on the single final shard output.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/unit-tests.yaml around lines 40 - 41, The workflow
currently exposes outputs like new-jiras from the junit2jira step but for matrix
jobs (jobs named go-unit-tests, go-extra-tests, go-postgres) GitHub Actions only
preserves the last completed shard’s outputs; change this by having each matrix
shard upload its junit2jira result (e.g., JSON or text) as a per-shard artifact
(use unique names including matrix variables), then add a non-matrix aggregation
job that runs after the matrix (needs depends-on: the matrix job), downloads all
shard artifacts, merges/concatenates them to produce the combined new-jiras
value, and sets that aggregated value as the workflow output (replacing the
current direct step output reference to junit2jira.outputs.new-jiras). Ensure
you reference the junit2jira step name in each shard for artifact creation and
the aggregation job reads those artifacts to compute the final output.

Expand All @@ -30,6 +44,9 @@ jobs:
volumes:
- /usr:/mnt/usr
- /opt:/mnt/opt
env:
UNIT_TEST_PKG_INCLUDE: ${{ matrix.component.pkg_include }}
UNIT_TEST_PKG_EXCLUDE: ${{ matrix.component.pkg_exclude }}
steps:
- name: Checkout
uses: actions/checkout@v6
Expand All @@ -44,14 +61,18 @@ jobs:
- name: Cache Go dependencies
uses: ./.github/actions/cache-go-dependencies

- name: Go Unit Tests
- name: Go Unit Tests (${{ matrix.component.name }})
run: ${{ matrix.gotags }} make go-unit-tests

- uses: codecov/codecov-action@v3
with:
token: ${{ secrets.CODECOV_TOKEN }}
flags: go-unit-tests

- name: Go Log Level Tests
if: matrix.component.name == 'pkg'
run: ${{ matrix.gotags }} make go-log-level-tests

- name: Generate junit report
if: always()
run: make generate-junit-reports
Expand All @@ -62,6 +83,45 @@ jobs:
with:
paths: 'junit-reports/report.xml'

- name: Report test failures to Jira
if: (!cancelled())
id: junit2jira
uses: ./.github/actions/junit2jira
with:
create-jiras: ${{ github.event_name == 'push' }}
jira-user: ${{ secrets.JIRA_USER }}
jira-token: ${{ secrets.JIRA_TOKEN }}
gcp-account: ${{ secrets.GCP_SERVICE_ACCOUNT_STACKROX_CI }}
directory: 'junit-reports'

go-extra-tests:
name: "Go Extra Tests (${{ matrix.gotags }})"
strategy:
fail-fast: false
matrix:
gotags: [ 'GOTAGS=""', 'GOTAGS=release' ]
runs-on: ubuntu-latest
outputs:
new-jiras: ${{ steps.junit2jira.outputs.new-jiras }}
container:
image: quay.io/stackrox-io/apollo-ci:stackrox-test-0.5.3
volumes:
- /usr:/mnt/usr
- /opt:/mnt/opt
steps:
- name: Checkout
uses: actions/checkout@v6
with:
fetch-depth: 0

- uses: ./.github/actions/job-preamble
with:
gcp-account: ${{ secrets.GCP_SERVICE_ACCOUNT_STACKROX_CI }}
free-disk-space: 30

- name: Cache Go dependencies
uses: ./.github/actions/cache-go-dependencies

- name: Go Integration Unit Tests
run: ${{ matrix.gotags }} make integration-unit-tests

Expand Down Expand Up @@ -481,7 +541,8 @@ jobs:
name: Post failure message to Slack
runs-on: ubuntu-latest
needs:
- go
- go-unit-tests
- go-extra-tests
- go-bench
- go-postgres
- local-roxctl-tests
Expand All @@ -499,7 +560,7 @@ jobs:
- name: Slack message
env:
GITHUB_CONTEXT: ${{ toJSON(github) }}
mention_author: ${{ needs.go.outputs.new-jiras || needs.go-postgres.outputs.new-jiras || needs.local-roxctl-tests.outputs.new-jiras || needs.ui.outputs.new-jiras || needs.go.outputs.new-jiras || needs.shell-unit-tests.outputs.new-jiras || needs.sensor-integration-tests.outputs.new-jiras }}
mention_author: ${{ needs.go-unit-tests.outputs.new-jiras || needs.go-extra-tests.outputs.new-jiras || needs.go-postgres.outputs.new-jiras || needs.local-roxctl-tests.outputs.new-jiras || needs.ui.outputs.new-jiras || needs.shell-unit-tests.outputs.new-jiras || needs.sensor-integration-tests.outputs.new-jiras }}
run: |
source scripts/ci/lib.sh
slack_workflow_failure
19 changes: 14 additions & 5 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,12 @@ SILENT ?= @
# TODO: [ROX-19070] Update postgres store test generation to work for foreign keys
UNIT_TEST_IGNORE := "stackrox/rox/sensor/tests|stackrox/rox/operator/tests|stackrox/rox/central/reports/config/store/postgres|stackrox/rox/central/complianceoperator/v2/scanconfigurations/store/postgres|stackrox/rox/central/auth/store/postgres|stackrox/rox/scanner/e2etests"

# UNIT_TEST_PKG_INCLUDE/EXCLUDE allow filtering the package list for parallel CI sharding.
# UNIT_TEST_PKG_INCLUDE: grep -E pattern to include only matching packages (default: match all)
# UNIT_TEST_PKG_EXCLUDE: grep -Ev pattern to exclude matching packages (default: match none)
UNIT_TEST_PKG_INCLUDE ?= .
UNIT_TEST_PKG_EXCLUDE ?= ^$$

GOBUILD := $(CURDIR)/scripts/go-build.sh
DOCKERBUILD := $(CURDIR)/scripts/docker-build.sh
GO_TEST_OUTPUT_PATH=$(CURDIR)/test-output/test.log
Expand Down Expand Up @@ -533,9 +539,12 @@ test-prep:
go-unit-tests: build-prep test-prep
set -o pipefail ; \
CGO_ENABLED=1 GOEXPERIMENT=cgocheck2 MUTEX_WATCHDOG_TIMEOUT_SECS=30 GOTAGS=$(GOTAGS),test scripts/go-test.sh -timeout 25m -race -cover -coverprofile test-output/coverage.out -v \
$(shell git ls-files -- '*_test.go' | sed -e 's@^@./@g' | xargs -n 1 dirname | sort | uniq | xargs go list| grep -v '^github.com/stackrox/rox/tests$$' | grep -Ev $(UNIT_TEST_IGNORE)) \
$(shell git ls-files -- '*_test.go' | sed -e 's@^@./@g' | xargs -n 1 dirname | sort | uniq | xargs go list| grep -v '^github.com/stackrox/rox/tests$$' | grep -Ev $(UNIT_TEST_IGNORE) | grep -E "$(UNIT_TEST_PKG_INCLUDE)" | grep -Ev "$(UNIT_TEST_PKG_EXCLUDE)") \
| tee $(GO_TEST_OUTPUT_PATH)
# Exercise the logging package for all supported logging levels to make sure that initialization works properly

# Exercise the logging package for all supported logging levels to make sure that initialization works properly.
.PHONY: go-log-level-tests
go-log-level-tests:
Comment on lines +545 to +547
Copy link
Copy Markdown
Contributor

@sourcery-ai sourcery-ai bot Mar 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: The new go-log-level-tests target does not depend on build-prep/test-prep, which may cause issues when run directly.

When this logic lived under go-unit-tests, it inherited build-prep and test-prep. Now that it’s a separate go-log-level-tests target, that prep only runs when invoked via a higher-level target like make test. Invoking make go-log-level-tests directly could fail due to missing setup (e.g., env, test-output/, or other prep side effects). Please add build-prep test-prep as prerequisites (or factor out a shared prerequisite target) so it behaves correctly when run standalone.

Suggested change
# Exercise the logging package for all supported logging levels to make sure that initialization works properly.
.PHONY: go-log-level-tests
go-log-level-tests:
# Exercise the logging package for all supported logging levels to make sure that initialization works properly.
.PHONY: go-log-level-tests
go-log-level-tests: build-prep test-prep

@echo "Run log tests"
for encoding in console json; do \
for level in debug info warn error fatal panic; do \
Expand All @@ -555,12 +564,12 @@ sensor-pipeline-benchmark: build-prep test-prep
go-postgres-unit-tests: build-prep test-prep
set -o pipefail ; \
CGO_ENABLED=1 GOEXPERIMENT=cgocheck2 MUTEX_WATCHDOG_TIMEOUT_SECS=30 GOTAGS=$(GOTAGS),test,sql_integration scripts/go-test.sh -timeout 15m -race -cover -coverprofile test-output/coverage.out -v \
$(shell git grep -rl "//go:build sql_integration" central pkg tools | sed -e 's@^@./@g' | xargs -n 1 dirname | sort | uniq | xargs go list -tags sql_integration | grep -v '^github.com/stackrox/rox/tests$$' | grep -Ev $(UNIT_TEST_IGNORE)) \
$(shell git grep -rl "//go:build sql_integration" central pkg tools | sed -e 's@^@./@g' | xargs -n 1 dirname | sort | uniq | xargs go list -tags sql_integration | grep -v '^github.com/stackrox/rox/tests$$' | grep -Ev $(UNIT_TEST_IGNORE) | grep -E "$(UNIT_TEST_PKG_INCLUDE)" | grep -Ev "$(UNIT_TEST_PKG_EXCLUDE)") \
| tee $(GO_TEST_OUTPUT_PATH)
@# The -p 1 passed to go test is required to ensure that tests of different packages are not run in parallel, so as to avoid conflicts when interacting with the DB.
set -o pipefail ; \
CGO_ENABLED=1 GOEXPERIMENT=cgocheck2 MUTEX_WATCHDOG_TIMEOUT_SECS=30 GOTAGS=$(GOTAGS),test,sql_integration scripts/go-test.sh -p 1 -race -cover -coverprofile test-output/migrator-coverage.out -v \
$(shell git grep -rl "//go:build sql_integration" migrator | sed -e 's@^@./@g' | xargs -n 1 dirname | sort | uniq | xargs go list -tags sql_integration | grep -v '^github.com/stackrox/rox/tests$$' | grep -Ev $(UNIT_TEST_IGNORE)) \
$(shell git grep -rl "//go:build sql_integration" migrator | sed -e 's@^@./@g' | xargs -n 1 dirname | sort | uniq | xargs go list -tags sql_integration | grep -v '^github.com/stackrox/rox/tests$$' | grep -Ev $(UNIT_TEST_IGNORE) | grep -E "$(UNIT_TEST_PKG_INCLUDE)" | grep -Ev "$(UNIT_TEST_PKG_EXCLUDE)") \
| tee -a $(GO_TEST_OUTPUT_PATH)

.PHONY: go-postgres-bench-tests
Expand Down Expand Up @@ -595,7 +604,7 @@ ui-component-tests:
make -C ui test-component

.PHONY: test
test: go-unit-tests ui-test shell-unit-tests
test: go-unit-tests go-log-level-tests ui-test shell-unit-tests

.PHONY: integration-unit-tests
integration-unit-tests: build-prep test-prep
Expand Down
Loading