perf(ci): parallelize style-slim via dynamic matrix#19312
Draft
perf(ci): parallelize style-slim via dynamic matrix#19312
Conversation
Remove the container: directive from all 4 container-based jobs in the Style workflow (check-generated-files, misc-checks, style-check, openshift-ci-lint) and run them directly on the ubuntu-latest runner. Container initialization consistently takes ~55s per job. Replacing it with on-host tool setup (setup-go, setup-node, yq, xmlstarlet, pip) costs ~5-15s, saving ~40-50s per job. Most tools (Go, Node.js, Make, GCC, Git, Helm, shellcheck, jq) are already preinstalled on the runner. For style-check, the shell-based tool installations (yq, xmlstarlet, pip) run as parallel background subshells to minimize wall-clock time. Also updates cache-ui-dependencies to use portable ~/. paths instead of container-specific /github/home/ paths, with a cache key bump (v2->v3). Generated with the assistance of AI. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Address sourcery-ai review feedback: - Add SHA256 checksum verification after downloading yq binary to mitigate supply-chain risk - Use `python3 -m pip` instead of bare `pip` for deterministic Python interpreter resolution across runner image updates Generated with the assistance of AI. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
actions/setup-go sets GOTOOLCHAIN=local by default, which prevents Go from auto-downloading a newer toolchain when a dependency (e.g. buf) requires one. The CI container didn't have this restriction. Set GOTOOLCHAIN=auto at the workflow level so Go tools that require a newer patch version (e.g. buf needing go1.25.6 when go.mod says 1.25.0) can download the required toolchain automatically. Generated with the assistance of AI. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
actions/setup-go writes GOTOOLCHAIN=local to GITHUB_ENV, which overrides any workflow-level env setting. This prevents Go tools in sub-modules with newer go.mod directives (e.g. tools/proto requires go 1.25.6) from auto-downloading the required toolchain. Write GOTOOLCHAIN=auto to GITHUB_ENV in a step after setup-go so it takes effect in all subsequent steps. This matches the behavior in the CI container where GOTOOLCHAIN was unset (defaulting to auto). The project uses GOTOOLCHAIN=local only in targeted make targets (go mod tidy) as a compatibility guard — not as a global setting. Generated with the assistance of AI. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The operator's bundle_helpers virtualenv uses packages (pip, pytest) that rely on pkgutil.ImpImporter, which was removed in Python 3.12. The CI container had Python 3.9 (UBI8) where this worked. The ubuntu-latest runner has Python 3.12. Add actions/setup-python with python-version 3.9 to match the container's Python version until operator/bundle_helpers is updated for Python 3.12 compatibility. Generated with the assistance of AI. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace the monolithic style-check job (which ran ~11 make targets sequentially in ~26 min) with a dynamic matrix that: 1. A lightweight discover job parses style-slim prerequisites from the Makefile, so adding/removing targets in the Makefile automatically updates the CI matrix without workflow changes. 2. Each target runs as an independent parallel matrix cell with conditional setup — only ui-lint gets Node.js, only shell-style gets xmlstarlet, only openshift-ci-style gets Python linters, etc. Expected wall-clock improvement: from ~26 min (sum of all targets) to ~15 min (longest single target, likely roxvet) plus ~2 min setup. Generated with the assistance of AI. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
Skipping CI for Draft Pull Request. |
Contributor
There was a problem hiding this comment.
Hey - I've left some high level feedback:
- The Makefile parsing in the
style-targetsjob (sed+tr+xargs+sed) is quite brittle against formatting changes (e.g., comments, reordering, extra whitespace); consider tightening this (e.g., anchoring on a dedicated variable or pattern, or adding a small script with clearer parsing logic) so a benign Makefile edit doesn’t silently skew the matrix. - The
yqinstallation logic is duplicated in multiple jobs with nearly identical shell blocks; consider extracting this into a small composite action or reusable step to avoid divergence when updating versions or flags later.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- The Makefile parsing in the `style-targets` job (`sed` + `tr` + `xargs` + `sed`) is quite brittle against formatting changes (e.g., comments, reordering, extra whitespace); consider tightening this (e.g., anchoring on a dedicated variable or pattern, or adding a small script with clearer parsing logic) so a benign Makefile edit doesn’t silently skew the matrix.
- The `yq` installation logic is duplicated in multiple jobs with nearly identical shell blocks; consider extracting this into a small composite action or reusable step to avoid divergence when updating versions or flags later.Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
Contributor
|
Images are ready for the commit at 9ae6147. To use with deploy scripts, first |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #19312 +/- ##
==========================================
+ Coverage 49.62% 49.64% +0.01%
==========================================
Files 2680 2689 +9
Lines 202231 202505 +274
==========================================
+ Hits 100362 100534 +172
- Misses 94382 94474 +92
- Partials 7487 7497 +10
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
actionlint runs shellcheck on inline scripts and flags the backslash in tr '\\' ' ' as SC1003. This is a false positive — the double backslash is intentional (tr needs a literal backslash argument). Generated with the assistance of AI. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Replace the monolithic
style-checkjob with a dynamic matrix that runs eachstyle-slimsub-target as an independent parallel job.Problem:
make style-slimruns ~11 targets sequentially in ~26 minutes. The targets are independent (roxvet, proto-style, shell-style, ui-lint, etc.) and can run in parallel.Approach:
style-targetsdiscover job parses thestyle-slimprerequisite list from the Makefile (sed/jq), so adding or removing targets in the Makefile automatically updates the CI matrix — no workflow changes needed.ui-lintgets Node.js, onlyshell-stylegets xmlstarlet, etc.Expected improvement: wall-clock from ~26 min (sum of all targets) to ~15 min (longest single target, likely
roxvet) + ~2 min setup overhead.Builds on #19309 (remove container from style jobs).
User-facing documentation
Testing and quality
Automated testing
How I validated my change
CI validation — this PR's own Style workflow run validates that all matrix cells pass independently.