From 0bf7adb5dd3da077d41b936b96252d3c312b2e13 Mon Sep 17 00:00:00 2001 From: Lee Dogeon Date: Wed, 21 Jan 2026 10:09:03 +0900 Subject: [PATCH 1/8] Simplify upgrade-pylib.md by removing manual copy steps handled by quick command --- .claude/commands/upgrade-pylib.md | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/.claude/commands/upgrade-pylib.md b/.claude/commands/upgrade-pylib.md index ba4cef525a..4cce1fd420 100644 --- a/.claude/commands/upgrade-pylib.md +++ b/.claude/commands/upgrade-pylib.md @@ -23,24 +23,19 @@ This helps improve the tooling for future upgrades. ## Steps -1. **Delete existing library in Lib/** - - If `Lib/$ARGUMENTS.py` exists, delete it - - If `Lib/$ARGUMENTS/` directory exists, delete it - -2. **Copy from cpython/Lib/** - - If `cpython/Lib/$ARGUMENTS.py` exists, copy it to `Lib/$ARGUMENTS.py` - - If `cpython/Lib/$ARGUMENTS/` directory exists, copy it to `Lib/$ARGUMENTS/` - -3. **Upgrade tests (quick upgrade with update_lib)** - - Run: `python3 scripts/update_lib quick cpython/Lib/test/test_$ARGUMENTS.py` (single file) - - Or: `python3 scripts/update_lib quick cpython/Lib/test/test_$ARGUMENTS/` (directory) +1. **Run quick upgrade with update_lib** + - Run: `python3 scripts/update_lib quick $ARGUMENTS` (module name) + - Or: `python3 scripts/update_lib quick cpython/Lib/$ARGUMENTS.py` (library file path) + - Or: `python3 scripts/update_lib quick cpython/Lib/$ARGUMENTS/` (library directory path) - This will: + - Copy library files (delete existing `Lib/$ARGUMENTS.py` or `Lib/$ARGUMENTS/`, then copy from `cpython/Lib/`) - Patch test files preserving existing RustPython markers - Run tests and auto-mark new test failures (not regressions) - Remove `@unittest.expectedFailure` from tests that now pass - - **Handle warnings**: If you see warnings like `WARNING: TestCFoo does not exist in remote file`, it means the class structure changed and markers couldn't be transferred automatically. These need to be manually restored in step 4 or added in step 5. + - Create a git commit with the changes + - **Handle warnings**: If you see warnings like `WARNING: TestCFoo does not exist in remote file`, it means the class structure changed and markers couldn't be transferred automatically. These need to be manually restored in step 2 or added in step 3. -4. **Review git diff and restore RUSTPYTHON-specific changes** +2. **Review git diff and restore RUSTPYTHON-specific changes** - Run `git diff Lib/test/test_$ARGUMENTS` to review all changes - **Only restore changes that have explicit `RUSTPYTHON` comments**. Look for: - `# XXX: RUSTPYTHON` or `# XXX RUSTPYTHON` - Comments marking RustPython-specific code modifications @@ -49,7 +44,7 @@ This helps improve the tooling for future upgrades. - **Do NOT restore other diff changes** - these are likely upstream CPython changes, not RustPython-specific modifications - When restoring, preserve the original context and formatting -5. **Verify tests** +3. **Verify tests** - Run: `cargo run --release -- -m test test_$ARGUMENTS -v` - The `-v` flag shows detailed output to identify which tests fail and why - For each new failure, add appropriate markers based on the failure type: @@ -86,9 +81,14 @@ This helps improve the tooling for future upgrades. ## Example Usage ``` +# Using module names (recommended) /upgrade-pylib inspect /upgrade-pylib json /upgrade-pylib asyncio + +# Using library paths (alternative) +/upgrade-pylib cpython/Lib/inspect.py +/upgrade-pylib cpython/Lib/json/ ``` ## Example: Restoring RUSTPYTHON changes From 3e33e7c9f2526b33f2780911b9407b8db400c43c Mon Sep 17 00:00:00 2001 From: Lee Dogeon Date: Wed, 21 Jan 2026 14:41:13 +0900 Subject: [PATCH 2/8] Let upgrade-pylib run auto-mark instead marking manually --- .claude/commands/upgrade-pylib-next.md | 27 ++++++++++++++++ .claude/commands/upgrade-pylib.md | 43 ++++++++++++++------------ 2 files changed, 50 insertions(+), 20 deletions(-) create mode 100644 .claude/commands/upgrade-pylib-next.md diff --git a/.claude/commands/upgrade-pylib-next.md b/.claude/commands/upgrade-pylib-next.md new file mode 100644 index 0000000000..f10f199d9d --- /dev/null +++ b/.claude/commands/upgrade-pylib-next.md @@ -0,0 +1,27 @@ +--- +allowed-tools: Bash(git add:*), Bash(git commit:*), Bash(python3 scripts/update_lib quick:*), Skill(upgrade-pylib) +--- + +# Upgrade Next Python Library + +Find the next Python library module ready for upgrade and run `/upgrade-pylib` for it. + +## Current TODO Status + +!`cargo run --release -- scripts/update_lib todo 2>/dev/null` + +## Instructions + +From the TODO list above, find modules matching these patterns (in priority order): + +1. `[ ] [no deps]` - Modules with no dependencies (can be upgraded immediately) +2. `[ ] [n/n]` - Modules where all dependencies are already upgraded (e.g., `[3/3]`, `[5/5]`) + +These patterns indicate modules that are ready to upgrade without blocking dependencies. + +**After identifying a suitable module**, run: +``` +/upgrade-pylib +``` + +If no modules match these criteria, inform the user that all eligible modules have dependencies that need to be upgraded first. diff --git a/.claude/commands/upgrade-pylib.md b/.claude/commands/upgrade-pylib.md index 4cce1fd420..e9221199fb 100644 --- a/.claude/commands/upgrade-pylib.md +++ b/.claude/commands/upgrade-pylib.md @@ -1,3 +1,7 @@ +--- +allowed-tools: Bash(python3 scripts/update_lib auto-mark:*) +--- + # Upgrade Python Library from CPython Upgrade a Python standard library module from CPython to RustPython. @@ -44,13 +48,25 @@ This helps improve the tooling for future upgrades. - **Do NOT restore other diff changes** - these are likely upstream CPython changes, not RustPython-specific modifications - When restoring, preserve the original context and formatting -3. **Verify tests** - - Run: `cargo run --release -- -m test test_$ARGUMENTS -v` - - The `-v` flag shows detailed output to identify which tests fail and why - - For each new failure, add appropriate markers based on the failure type: - - **Test assertion failure** → `@unittest.expectedFailure` with `# TODO: RUSTPYTHON` comment - - **Panic/crash** → `@unittest.skip("TODO: RUSTPYTHON; ")` - - **Class-specific markers**: If a test fails only in the C implementation (TestCFoo) but passes in the Python implementation (TestPyFoo), or vice versa, add the marker to the specific subclass, not the base class: +3. **Mark remaining test failures with auto-mark** + - Run: `python3 scripts/update_lib auto-mark Lib/test/test_$ARGUMENTS.py --mark-failure` + - Or for directory: `python3 scripts/update_lib auto-mark Lib/test/test_$ARGUMENTS/ --mark-failure` + - This will: + - Run tests and mark ALL failing tests with `@unittest.expectedFailure` + - Remove `@unittest.expectedFailure` from tests that now pass + - **Note**: The `--mark-failure` flag marks all failures including regressions. Review the changes before committing. + +4. **Handle panics manually** + - If any tests cause panics/crashes (not just assertion failures), they need `@unittest.skip` instead: + ```python + @unittest.skip("TODO: RUSTPYTHON; panics with 'index out of bounds'") + def test_crashes(self): + ... + ``` + - auto-mark cannot detect panics automatically - check the test output for crash messages + +5. **Handle class-specific failures** + - If a test fails only in the C implementation (TestCFoo) but passes in the Python implementation (TestPyFoo), or vice versa, move the marker to the specific subclass: ```python # Base class - no marker here class TestFoo: @@ -65,19 +81,6 @@ This helps improve the tooling for future upgrades. def test_something(self): return super().test_something() ``` - - **New tests from CPython**: The upgrade may bring in entirely new tests that didn't exist before. These won't have any RUSTPYTHON markers in the diff - they just need to be tested and marked if they fail. - - Example markers: - ```python - # TODO: RUSTPYTHON - @unittest.expectedFailure - def test_something(self): - ... - - # TODO: RUSTPYTHON - @unittest.skip("TODO: RUSTPYTHON; panics with 'index out of bounds'") - def test_crashes(self): - ... - ``` ## Example Usage ``` From 6bd6d7f4f5e9cf0b6e3fbbb5d4eb2a1e42c6610d Mon Sep 17 00:00:00 2001 From: Lee Dogeon Date: Wed, 21 Jan 2026 14:56:48 +0900 Subject: [PATCH 3/8] Let upgrade-pylib a separate commit from updating module --- .claude/commands/upgrade-pylib.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.claude/commands/upgrade-pylib.md b/.claude/commands/upgrade-pylib.md index e9221199fb..7d80ccf29a 100644 --- a/.claude/commands/upgrade-pylib.md +++ b/.claude/commands/upgrade-pylib.md @@ -82,6 +82,10 @@ This helps improve the tooling for future upgrades. return super().test_something() ``` +6. **Commit the test fixes** + - Run: `git add -u && git commit -m "Mark failing tests"` + - This creates a separate commit for the test markers added in steps 2-5 + ## Example Usage ``` # Using module names (recommended) From 7024060e5efeaac508a2ff74fa5bdd0f32953898 Mon Sep 17 00:00:00 2001 From: Lee Dogeon Date: Wed, 21 Jan 2026 15:19:24 +0900 Subject: [PATCH 4/8] Correct commands' permissions --- .claude/commands/upgrade-pylib-next.md | 2 +- .claude/commands/upgrade-pylib.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.claude/commands/upgrade-pylib-next.md b/.claude/commands/upgrade-pylib-next.md index f10f199d9d..4427306928 100644 --- a/.claude/commands/upgrade-pylib-next.md +++ b/.claude/commands/upgrade-pylib-next.md @@ -1,5 +1,5 @@ --- -allowed-tools: Bash(git add:*), Bash(git commit:*), Bash(python3 scripts/update_lib quick:*), Skill(upgrade-pylib) +allowed-tools: Skill(upgrade-pylib) --- # Upgrade Next Python Library diff --git a/.claude/commands/upgrade-pylib.md b/.claude/commands/upgrade-pylib.md index 7d80ccf29a..2a7ddaae7a 100644 --- a/.claude/commands/upgrade-pylib.md +++ b/.claude/commands/upgrade-pylib.md @@ -1,5 +1,5 @@ --- -allowed-tools: Bash(python3 scripts/update_lib auto-mark:*) +allowed-tools: Bash(git add:*), Bash(git commit:*), Bash(python3 scripts/update_lib quick:*), Bash(python3 scripts/update_lib auto-mark:*) --- # Upgrade Python Library from CPython From 5ae4ed1693a506aa1ab42e5842e377cc2b39ab67 Mon Sep 17 00:00:00 2001 From: Lee Dogeon Date: Wed, 21 Jan 2026 15:44:06 +0900 Subject: [PATCH 5/8] Add open PR check to upgrade-pylib-next command --- .claude/commands/upgrade-pylib-next.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/.claude/commands/upgrade-pylib-next.md b/.claude/commands/upgrade-pylib-next.md index 4427306928..e4d0a24d9e 100644 --- a/.claude/commands/upgrade-pylib-next.md +++ b/.claude/commands/upgrade-pylib-next.md @@ -1,5 +1,5 @@ --- -allowed-tools: Skill(upgrade-pylib) +allowed-tools: Skill(upgrade-pylib), Bash(gh pr list:*) --- # Upgrade Next Python Library @@ -10,6 +10,10 @@ Find the next Python library module ready for upgrade and run `/upgrade-pylib` f !`cargo run --release -- scripts/update_lib todo 2>/dev/null` +## Open Upgrade PRs + +!`gh pr list --search "Update in:title" --json number,title --template '{{range .}}#{{.number}} {{.title}}{{"\n"}}{{end}}'` + ## Instructions From the TODO list above, find modules matching these patterns (in priority order): @@ -19,6 +23,8 @@ From the TODO list above, find modules matching these patterns (in priority orde These patterns indicate modules that are ready to upgrade without blocking dependencies. +**Important**: Skip any modules that already have an open PR in the "Open Upgrade PRs" list above. + **After identifying a suitable module**, run: ``` /upgrade-pylib From 2904859bc52d0041878d28a50c5aca5999281a9e Mon Sep 17 00:00:00 2001 From: Lee Dogeon Date: Wed, 21 Jan 2026 23:40:32 +0900 Subject: [PATCH 6/8] Fix dependency pattern example in upgrade-pylib-next command --- .claude/commands/upgrade-pylib-next.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.claude/commands/upgrade-pylib-next.md b/.claude/commands/upgrade-pylib-next.md index e4d0a24d9e..712b79433b 100644 --- a/.claude/commands/upgrade-pylib-next.md +++ b/.claude/commands/upgrade-pylib-next.md @@ -19,7 +19,7 @@ Find the next Python library module ready for upgrade and run `/upgrade-pylib` f From the TODO list above, find modules matching these patterns (in priority order): 1. `[ ] [no deps]` - Modules with no dependencies (can be upgraded immediately) -2. `[ ] [n/n]` - Modules where all dependencies are already upgraded (e.g., `[3/3]`, `[5/5]`) +2. `[ ] [0/n]` - Modules where all dependencies are already upgraded (e.g., `[0/3]`, `[0/5]`) These patterns indicate modules that are ready to upgrade without blocking dependencies. From d5834123407553183a3e3c73fc9e56b44b418ce4 Mon Sep 17 00:00:00 2001 From: Lee Dogeon Date: Thu, 22 Jan 2026 00:52:42 +0900 Subject: [PATCH 7/8] Let upgrade-pylib command to invetigate failing tests --- .claude/commands/investigate-test-failure.md | 41 ++++++++++++++++++++ .claude/commands/upgrade-pylib.md | 22 ++++++++--- 2 files changed, 58 insertions(+), 5 deletions(-) create mode 100644 .claude/commands/investigate-test-failure.md diff --git a/.claude/commands/investigate-test-failure.md b/.claude/commands/investigate-test-failure.md new file mode 100644 index 0000000000..143e4e3753 --- /dev/null +++ b/.claude/commands/investigate-test-failure.md @@ -0,0 +1,41 @@ +--- +allowed-tools: Bash(python3:*), Bash(cargo run:*), Read, Grep, Glob, Bash(git add:*), Bash(git commit:*) +--- + +# Investigate Test Failure + +Investigate why a specific test is failing and determine if it can be fixed or needs an issue. + +## Arguments +- `$ARGUMENTS`: Failed test identifier (e.g., `test_inspect.TestGetSourceBase.test_getsource_reload`) + +## Steps + +1. **Analyze failure cause** + - Read the test code + - Analyze failure message/traceback + - Check related RustPython code + +2. **Verify behavior in CPython** + - Run the test with `python3 -m unittest` to confirm expected behavior + - Document the expected output + +3. **Determine fix feasibility** + - **Simple fix** (import issues, small logic bugs): Fix and commit + - **Complex fix** (major unimplemented features): Collect issue info and report to user + +4. **For complex issues - Collect issue information** + Following `.github/ISSUE_TEMPLATE/report-incompatibility.md` format: + + - **Feature**: Description of missing/broken Python feature + - **Minimal reproduction code**: Smallest code that reproduces the issue + - **CPython behavior**: Result when running with python3 + - **RustPython behavior**: Result when running with cargo run + - **Python Documentation link**: Link to relevant CPython docs + + Report collected information to the user. Issue creation is done only upon user request. + + Example issue creation command: + ``` + gh issue create --template report-incompatibility.md --title "..." --body "..." + ``` diff --git a/.claude/commands/upgrade-pylib.md b/.claude/commands/upgrade-pylib.md index 2a7ddaae7a..5c09f84af6 100644 --- a/.claude/commands/upgrade-pylib.md +++ b/.claude/commands/upgrade-pylib.md @@ -48,7 +48,19 @@ This helps improve the tooling for future upgrades. - **Do NOT restore other diff changes** - these are likely upstream CPython changes, not RustPython-specific modifications - When restoring, preserve the original context and formatting -3. **Mark remaining test failures with auto-mark** +3. **Investigate test failures with subagent** + - First, run tests to collect the list of failures: + ``` + cargo run --release -- -m unittest test.$ARGUMENTS -v 2>&1 | grep -E "^(FAIL|ERROR):" + ``` + - For each failure, use the Task tool with `general-purpose` subagent to investigate: + - Subagent should follow the `/investigate-test-failure` skill workflow + - Pass the failed test identifier as the argument (e.g., `test_inspect.TestGetSourceBase.test_getsource_reload`) + - If subagent can fix the issue easily: fix and commit + - If complex issue: subagent collects issue info and reports back (issue creation on user request only) + - Using subagent prevents context pollution in the main conversation + +4. **Mark remaining test failures with auto-mark** - Run: `python3 scripts/update_lib auto-mark Lib/test/test_$ARGUMENTS.py --mark-failure` - Or for directory: `python3 scripts/update_lib auto-mark Lib/test/test_$ARGUMENTS/ --mark-failure` - This will: @@ -56,7 +68,7 @@ This helps improve the tooling for future upgrades. - Remove `@unittest.expectedFailure` from tests that now pass - **Note**: The `--mark-failure` flag marks all failures including regressions. Review the changes before committing. -4. **Handle panics manually** +5. **Handle panics manually** - If any tests cause panics/crashes (not just assertion failures), they need `@unittest.skip` instead: ```python @unittest.skip("TODO: RUSTPYTHON; panics with 'index out of bounds'") @@ -65,7 +77,7 @@ This helps improve the tooling for future upgrades. ``` - auto-mark cannot detect panics automatically - check the test output for crash messages -5. **Handle class-specific failures** +6. **Handle class-specific failures** - If a test fails only in the C implementation (TestCFoo) but passes in the Python implementation (TestPyFoo), or vice versa, move the marker to the specific subclass: ```python # Base class - no marker here @@ -82,9 +94,9 @@ This helps improve the tooling for future upgrades. return super().test_something() ``` -6. **Commit the test fixes** +7. **Commit the test fixes** - Run: `git add -u && git commit -m "Mark failing tests"` - - This creates a separate commit for the test markers added in steps 2-5 + - This creates a separate commit for the test markers added in steps 2-6 ## Example Usage ``` From 50abecae74e75581077a8b596df17dd1ac020348 Mon Sep 17 00:00:00 2001 From: Lee Dogeon Date: Fri, 23 Jan 2026 22:19:57 +0900 Subject: [PATCH 8/8] Add pre-commit review to invetigate-test-failure command --- .claude/commands/investigate-test-failure.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/.claude/commands/investigate-test-failure.md b/.claude/commands/investigate-test-failure.md index 143e4e3753..e9d6b2d2d2 100644 --- a/.claude/commands/investigate-test-failure.md +++ b/.claude/commands/investigate-test-failure.md @@ -1,5 +1,5 @@ --- -allowed-tools: Bash(python3:*), Bash(cargo run:*), Read, Grep, Glob, Bash(git add:*), Bash(git commit:*) +allowed-tools: Bash(python3:*), Bash(cargo run:*), Read, Grep, Glob, Bash(git add:*), Bash(git commit:*), Bash(cargo fmt:*), Bash(git diff:*), Task --- # Investigate Test Failure @@ -21,9 +21,17 @@ Investigate why a specific test is failing and determine if it can be fixed or n - Document the expected output 3. **Determine fix feasibility** - - **Simple fix** (import issues, small logic bugs): Fix and commit + - **Simple fix** (import issues, small logic bugs): Fix code → Run `cargo fmt --all` → Pre-commit review → Commit - **Complex fix** (major unimplemented features): Collect issue info and report to user + **Pre-commit review process**: + - Run `git diff` to see the changes + - Use Task tool with `general-purpose` subagent to review: + - Compare implementation against cpython/ source code + - Verify the fix aligns with CPython behavior + - Check for any missed edge cases + - Proceed to commit only after review passes + 4. **For complex issues - Collect issue information** Following `.github/ISSUE_TEMPLATE/report-incompatibility.md` format: