Speed up CI from ~15 min to a sub-3-minute PR gate.
Problem: CI run #494 took ~15 min; 860s of a 904s build-and-test job was the serialized integration suite (200+ parametrized cases, each spawning a client subprocess, funneled through one session-scoped server).
Changes:
Add pytest-xdist to the CI Dockerfile (image -> v10); parallelize integration with -n 4 --dist=loadgroup.
Worker-isolate TEST_DATA_DIR (keyed on PYTEST_XDIST_WORKER) so concurrent workers never collide on shared-filesystem fixtures.
Register a ci marker; tag a fast, representative integration subset.
CI split: PRs run lint + build + unit + the marked smoke set (verified 27 cases / 36s); push to dev/main runs the full suite plus sanitizer/fuzz/coverage/valgrind.
Measured: full suite -n4 = 284 passed / 2 failed / 11 skipped / 1 xpassed in 227s (was 860s). The 2 failures are the pre-existing setpriv/nobody-environment test (fails identically on the base path under plain pytest; skipped in the real runner where the container runs non-root). No xdist-specific failures.
Note: CI now requires the fastsync-ci:v10 image (with pytest-xdist). The image must be rebuilt/pushed before merging this (otherwise the workflow pulls a missing tag).
Speed up CI from ~15 min to a sub-3-minute PR gate.
**Problem:** CI run #494 took ~15 min; 860s of a 904s build-and-test job was the serialized integration suite (200+ parametrized cases, each spawning a client subprocess, funneled through one session-scoped server).
**Changes:**
- Add `pytest-xdist` to the CI Dockerfile (image -> v10); parallelize integration with `-n 4 --dist=loadgroup`.
- Worker-isolate `TEST_DATA_DIR` (keyed on `PYTEST_XDIST_WORKER`) so concurrent workers never collide on shared-filesystem fixtures.
- Register a `ci` marker; tag a fast, representative integration subset.
- CI split: **PRs** run lint + build + unit + the marked smoke set (verified 27 cases / 36s); **push to dev/main** runs the full suite plus sanitizer/fuzz/coverage/valgrind.
**Measured:** full suite -n4 = 284 passed / 2 failed / 11 skipped / 1 xpassed in 227s (was 860s). The 2 failures are the pre-existing setpriv/nobody-environment test (fails identically on the base path under plain pytest; skipped in the real runner where the container runs non-root). No xdist-specific failures.
**Note:** CI now requires the `fastsync-ci:v10` image (with pytest-xdist). The image must be rebuilt/pushed before merging this (otherwise the workflow pulls a missing tag).
- Add pytest-xdist to the CI Dockerfile (image -> v10).
- Worker-isolate TEST_DATA_DIR (PYTEST_XDIST_WORKER) so concurrent xdist
workers never collide on shared-filesystem fixtures.
- Register a 'ci' marker and tag a fast representative subset of integration
tests (basic TCP, incremental, compression, delete, basis, append).
- ci.yaml: lint + build + unit + the marked subset (-n4) on every PR; the
full integration suite plus sanitizer/fuzz/coverage/valgrind run only on
push to dev/main.
- Integration step runtime drops from ~860s (serial) to ~230s (-n4); the PR
gate lands well under ~3 minutes.
- Replace --dist=loadgroup (a no-op: it only groups by xdist_group marks) with
--dist=load, and make module teardowns remove only their own SOURCE_DIR/DEST_DIR
(never the shared worker-keyed TEST_DATA_DIR) so interleaved modules can't wipe
each other's fixtures. Add a session-scoped cleanup of the per-worker dir.
(loadfile grouped the whole test_features.py module onto one worker and slowed
the full suite to 761s vs 224s with load.)
- Mark the two setpriv privilege tests with a 'setpriv' marker and run the full
merge suite as -m "not setpriv", so the dev/main gate can't go red on the
env-dependent /root-traversal tests regardless of the runner uid.
- Add a TLS basic test to the ci subset, add workflow_dispatch for on-demand full
runs, and fix trailing newlines.
- Measured: smoke -m ci = 28 passed/39s; full -n4 = 280 passed/11 skipped/1 xpassed
in 224s (was 860s serial).
Merged locally into dev (a5e176b) then pushed. CI speedup landed: PR gate = -m ci smoke (28 tests / ~39s), dev/main push = full suite -m "not setpriv" at -n4 (~224s) plus sanitizer/fuzz/coverage/valgrind. CI image fastsync-ci:v10 (with pytest-xdist) built and pushed. Feature branch removed.
Merged locally into dev (a5e176b) then pushed. CI speedup landed: PR gate = `-m ci` smoke (28 tests / ~39s), dev/main push = full suite `-m "not setpriv"` at `-n4` (~224s) plus sanitizer/fuzz/coverage/valgrind. CI image `fastsync-ci:v10` (with pytest-xdist) built and pushed. Feature branch removed.
Pull request closed
This pull request cannot be reopened because the branch was deleted.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Speed up CI from ~15 min to a sub-3-minute PR gate.
Problem: CI run #494 took ~15 min; 860s of a 904s build-and-test job was the serialized integration suite (200+ parametrized cases, each spawning a client subprocess, funneled through one session-scoped server).
Changes:
pytest-xdistto the CI Dockerfile (image -> v10); parallelize integration with-n 4 --dist=loadgroup.TEST_DATA_DIR(keyed onPYTEST_XDIST_WORKER) so concurrent workers never collide on shared-filesystem fixtures.cimarker; tag a fast, representative integration subset.Measured: full suite -n4 = 284 passed / 2 failed / 11 skipped / 1 xpassed in 227s (was 860s). The 2 failures are the pre-existing setpriv/nobody-environment test (fails identically on the base path under plain pytest; skipped in the real runner where the container runs non-root). No xdist-specific failures.
Note: CI now requires the
fastsync-ci:v10image (with pytest-xdist). The image must be rebuilt/pushed before merging this (otherwise the workflow pulls a missing tag).Merged locally into dev (
a5e176b) then pushed. CI speedup landed: PR gate =-m cismoke (28 tests / ~39s), dev/main push = full suite-m "not setpriv"at-n4(~224s) plus sanitizer/fuzz/coverage/valgrind. CI imagefastsync-ci:v10(with pytest-xdist) built and pushed. Feature branch removed.Pull request closed