Proposal: Use mcp-conformance-action for conformance testing #1826
Description
Summary
I've created a reusable GitHub Action for MCP server conformance testing: mcp-conformance-action. I believe this could simplify your conformance workflow while maintaining the same coverage.
Current State
Your script/conformance-test script is comprehensive and tests 17 flag configurations. The action I've built supports similar functionality through a configurations input that can specify different start_command options with various flags.
Proposed Benefits
- Reusable - Other MCP servers can use the same pattern
- Simpler workflow - No custom bash script maintenance
- Language-agnostic - Built-in setup for Go, Node, Python, Rust, C#
- Job summaries - Automatic markdown report generation
- SHA-pinned - Security hardened with dependabot
What Would Change
The action can handle most of your test configurations by using the configurations input with different CLI flags in each start_command. See this gist for a proposed workflow:
https://gist.github.com/SamMorrowDrums/e5fd6e76adf452741c130edde1846b22
Gap Analysis
| Feature | Your Script | Action |
|---|---|---|
| Multi-config testing | ✅ 17 configs | ✅ Supported via configurations JSON |
| Schema comparison | ✅ | ✅ |
| Timing comparisons | ✅ | ✅ |
| Job summary | ✅ | ✅ |
| Dynamic tool calls | ✅ (list_available_toolsets, enable_toolset) | ❌ Not yet |
| Custom JSON normalization | ✅ | ✅ |
Main Gap: The action currently only tests listing endpoints (tools/list, resources/list, prompts/list), not tool invocations. Your dynamic toolset testing (dynamic-tool-calls config) wouldn't be covered yet.
Options
- Partial adoption - Use the action for standard configs, keep custom script for dynamic testing only
- Full migration - I can extend the action to support custom JSON-RPC calls
- Pass - If custom script works well for you, no change needed!
Would you be interested in exploring this? Happy to create a PR or extend the action's capabilities.