ci: run test suite in 3 parallel shards + make suite shard-safe
Completes the test-sharding half of #3197 (Docker-cache half shipped v0.51.177). Adds pytest-shard 3-way split to tests.yml (3 shards x 3 Python = 9 jobs, fail-fast: false). pytest-shard is 0-indexed so the matrix uses [0,1,2] — the original #3197 used [1,2,3] which would have crashed the out-of-range job and silently skipped shard 0's tests. Made the suite shard-safe by fixing 4 cross-test state-pollution bugs that passed sequentially but failed when sharded: - test_onboarding_mvp: reset onboarding_completed flag (settings.json) in the autouse fixture; the config-cleanup only cleared config.yaml/.env. - test_issue693_system_health_panel: invalidate the process-wide password-hash cache before/after so a prior test's "no password" cache doesn't defeat the auth-gate assertion. - test_auth_session_persistence: assert against auth._SESSIONS_FILE (where auth actually writes) instead of a local _TEST_STATE path that only matched under a lucky import order. - test_profile_env_isolation (root cause of the worst leak): stop deleting + re-importing api.profiles under a temp HERMES_BASE_HOME — that swapped the module object and poisoned the cached _DEFAULT_HERMES_HOME for every later test (broke test_title_aux_routing's load_config). Now points the cached path via monkeypatch.setattr (auto-restored, no module swap). - conftest: autouse fixture restores HERMES_HOME/HERMES_BASE_HOME after each test as defense-in-depth against future switch_profile leaks. Verified: all 3 shards green (6912 passed, 0 failed); full sequential run still green (6957 passed, 0 failed). Slowest shard ~70s vs ~180s sequential.
This commit is contained in:
22
.github/workflows/tests.yml
vendored
22
.github/workflows/tests.yml
vendored
@@ -10,8 +10,19 @@ jobs:
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
# Don't cancel the other shards/versions when one fails — we want the full
|
||||
# failure picture across the matrix in a single run.
|
||||
fail-fast: false
|
||||
matrix:
|
||||
python-version: ['3.11', '3.12', '3.13']
|
||||
# Split the suite across 3 parallel shards per Python version. pytest-shard
|
||||
# partitions tests deterministically by test-id hash; the suite was made
|
||||
# shard-safe (no cross-test state leakage) so every shard passes
|
||||
# independently. See docs/agent-memory note on test-suite shard-safety.
|
||||
# NOTE: pytest-shard is 0-indexed — shard ids must be 0..num_shards-1.
|
||||
# Using 1-based ids would crash the out-of-range job AND silently skip
|
||||
# shard 0's tests.
|
||||
shard: [0, 1, 2]
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
@@ -20,11 +31,16 @@ jobs:
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: ${{ matrix.python-version }}
|
||||
cache: 'pip'
|
||||
cache-dependency-path: |
|
||||
**/setup.cfg
|
||||
**/requirements*.txt
|
||||
**/pyproject.toml
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
pip install "pyyaml>=6.0" pytest pytest-timeout pytest-asyncio
|
||||
pip install "pyyaml>=6.0" pytest pytest-timeout pytest-asyncio pytest-shard
|
||||
# Install the `mcp` package so tests/test_mcp_server.py runs in CI.
|
||||
# The package is an optional runtime dep of mcp_server.py — users
|
||||
# who run the MCP integration install it themselves; CI installs
|
||||
@@ -33,5 +49,5 @@ jobs:
|
||||
# importorskip and the matrix stays green.
|
||||
pip install mcp || echo "mcp install failed — test_mcp_server.py will importorskip"
|
||||
|
||||
- name: Run tests
|
||||
run: pytest tests/ -v --timeout=60
|
||||
- name: Run tests (shard ${{ matrix.shard }} of 3)
|
||||
run: pytest tests/ -v --timeout=60 --shard-id=${{ matrix.shard }} --num-shards=3
|
||||
|
||||
Reference in New Issue
Block a user