Implements the Phase-3 delete-policy row: --delete-excluded, --max-delete=NUM, --ignore-errors, --force, --prune-empty-dirs.
Default-behavior parity decision
rsync does NOT delete filter-excluded destination files under plain --delete; FastSync previously did (deletion = "not in the keep-set"). This PR changes the default to rsync parity: the sender now records every path its source scan prunes by user-selection rules (--filter/-F/-C, --exclude/--include, --max-size/--min-size) and transmits them as protected prefixes in the delete-manifest frame; the walker never deletes them. --delete-excluded opts back in (empty protected list → excluded mirrors become extras). Divergence: protection is sender-derived only (FastSync never re-applies rules to the destination), --files-from subset pruning and -R relative paths are never protected, and --max-size/--min-size mirrors are protected (verified against real rsync 3.4.1).
Per-flag realization
--delete-excluded – default protects excluded mirrors (above); the flag makes the sender send no protected prefixes so they are deleted. Covered single-thread, -m, early (--delete-before) and commit timing, and whole excluded-dir subtrees.
--max-delete=NUM – all-or-nothing (rsync doc semantics): the receiver rehearses the deletion (an identical fd-relative count walk) and if the extras would exceed NUM deletes NOTHING and fails with a distinct --max-delete error; at/below the limit it deletes exactly the extras. Directories count. Inert without --delete. The hard MAX_SERVER_DELETE_COUNT (100000) also became all-or-nothing with its own error (previously it truncated at 100000 then failed).
--ignore-errors – a source-scan I/O error (unreadable directory) now records a non-fatal scanner error; default aborts the run (no deletion), with the flag the scan continues, the readable tree transfers, deletion still runs, and the run exits non-zero. Verified end-to-end by running the client as an unprivileged user against a mode-000 source dir. Divergence: without the flag FastSync aborts the whole run (rsync transfers the rest and merely skips deletion); both leave deletion undone.
--force – rsync's file-over-directory replacement: an incoming regular file may clear a (non-empty) destination directory first, via a new confined, symlink-safe file_remove_tree_secure() (O_NOFOLLOW fd walk, symlinks removed by name, never followed). Without --force the write/run fails. Applies on the immediate-install path (not under --delay-updates, documented).
--prune-empty-dirs – long-only (FastSync -m stays multithreading; recorded divergence). Recursive transfers never emit directory entries, so empty dirs are inherently never transferred (rsync -m parity) and truly-empty destination chains are removed by --delete with or without the flag; the flag's additional effect is on the --dirs generator (an empty source dir's explicit entry is omitted → not created, no -i/--out-format line, existing empty mirror pruned by --delete). Explicitly --files-from-listed dirs still pass through.
Integration (TestDeletePolicy, 19 tests): delete-excluded default vs opt-out (st/-m/early), --max-delete over/at limit, --force replacement, --prune-empty-dirs (dirs + recursion parity), --ignore-errors across a real scan I/O error.
Local results: ./build/tests all pass; full tests/integration/test_features.py 190 passed; preflight/tcp/tls pass (ssh tests auto-skip locally — no ssh binary); clang-format 18 and cppcheck clean in the CI image; strict-warnings build clean.
Docs
RSYNC_COMPAT.md rows 106–115 updated (--delete default note, all five policy flags → ✅) plus the Phase-3 implementation-notes block (two-section frame, all-or-nothing walker, 2.9.0 bump, policy-flags-don't-imply-delete). Summary counts intentionally left for the orchestrator to recount.
Implements the Phase-3 delete-policy row: `--delete-excluded`, `--max-delete=NUM`, `--ignore-errors`, `--force`, `--prune-empty-dirs`.
## Default-behavior parity decision
rsync does NOT delete filter-excluded destination files under plain `--delete`; FastSync previously did (deletion = "not in the keep-set"). This PR changes the default to rsync parity: the sender now records every path its source scan prunes by user-selection rules (`--filter`/`-F`/`-C`, `--exclude`/`--include`, `--max-size`/`--min-size`) and transmits them as **protected prefixes** in the delete-manifest frame; the walker never deletes them. `--delete-excluded` opts back in (empty protected list → excluded mirrors become extras). Divergence: protection is sender-derived only (FastSync never re-applies rules to the destination), `--files-from` subset pruning and `-R` relative paths are never protected, and `--max-size`/`--min-size` mirrors are protected (verified against real rsync 3.4.1).
## Per-flag realization
- **`--delete-excluded`** – default protects excluded mirrors (above); the flag makes the sender send no protected prefixes so they are deleted. Covered single-thread, `-m`, early (`--delete-before`) and commit timing, and whole excluded-dir subtrees.
- **`--max-delete=NUM`** – all-or-nothing (rsync doc semantics): the receiver rehearses the deletion (an identical fd-relative count walk) and if the extras would exceed NUM deletes NOTHING and fails with a distinct `--max-delete` error; at/below the limit it deletes exactly the extras. Directories count. Inert without `--delete`. The hard `MAX_SERVER_DELETE_COUNT` (100000) also became all-or-nothing with its own error (previously it truncated at 100000 then failed).
- **`--ignore-errors`** – a source-scan I/O error (unreadable directory) now records a non-fatal scanner error; default aborts the run (no deletion), with the flag the scan continues, the readable tree transfers, deletion still runs, and the run exits non-zero. Verified end-to-end by running the client as an unprivileged user against a mode-000 source dir. Divergence: without the flag FastSync aborts the whole run (rsync transfers the rest and merely skips deletion); both leave deletion undone.
- **`--force`** – rsync's file-over-directory replacement: an incoming regular file may clear a (non-empty) destination directory first, via a new confined, symlink-safe `file_remove_tree_secure()` (O_NOFOLLOW fd walk, symlinks removed by name, never followed). Without `--force` the write/run fails. Applies on the immediate-install path (not under `--delay-updates`, documented).
- **`--prune-empty-dirs`** – long-only (FastSync `-m` stays multithreading; recorded divergence). Recursive transfers never emit directory entries, so empty dirs are inherently never transferred (rsync `-m` parity) and truly-empty destination chains are removed by `--delete` with or without the flag; the flag's additional effect is on the `--dirs` generator (an empty source dir's explicit entry is omitted → not created, no `-i`/`--out-format` line, existing empty mirror pruned by `--delete`). Explicitly `--files-from`-listed dirs still pass through.
## Protocol
- Two-section `STATUS_MANIFEST` frame (keep-set + protected prefixes); receiver `DeleteManifest` threaded through commit/early paths.
- New wire bool `force_delete`; `max_delete` default changed to -1; `ignore_errors` is client-only (not serialized).
- `PROTOCOL_VERSION` **2.8.0 → 2.9.0** (mirrored version strings).
## Tests
- Unit: walker all-or-nothing bounds (exceeded → `DELETE_WALK_LIMIT_EXCEEDED`, nothing removed; exact bound → deletes), protected-prefix skipping + symlink/kept-dir leave-behind; config wire round-trip for `force_delete`/`delete_excluded`/`prune_empty_dirs`/`max_delete`; CLI parse/validation incl. invalid `--max-delete`.
- Integration (`TestDeletePolicy`, 19 tests): delete-excluded default vs opt-out (st/-m/early), `--max-delete` over/at limit, `--force` replacement, `--prune-empty-dirs` (dirs + recursion parity), `--ignore-errors` across a real scan I/O error.
- Local results: `./build/tests` all pass; full `tests/integration/test_features.py` 190 passed; preflight/tcp/tls pass (ssh tests auto-skip locally — no ssh binary); clang-format 18 and cppcheck clean in the CI image; strict-warnings build clean.
## Docs
`RSYNC_COMPAT.md` rows 106–115 updated (`--delete` default note, all five policy flags → ✅) plus the Phase-3 implementation-notes block (two-section frame, all-or-nothing walker, 2.9.0 bump, policy-flags-don't-imply-delete). Summary counts intentionally left for the orchestrator to recount.
Adds ignore_errors (client-only) and force_delete (wire) booleans, makes
max_delete default -1 (no client limit), and parses --delete-excluded,
--max-delete=NUM, --ignore-errors, --force, --prune-empty-dirs. Bumps
PROTOCOL_VERSION 2.8.0 -> 2.9.0 for the new on-the-wire force_delete field.
delete_extras_limited now rehearses a finite-capped deletion before unlinking
anything (an identical fd-relative walk that counts files and directories) and
returns DELETE_WALK_LIMIT_EXCEEDED with nothing removed when the run would
exceed the cap, so --max-delete is enforced per run instead of truncating the
deletion. A directory that still holds entries the walker leaves in place
(protected excluded prefix, manifest-kept file, symlink) is left behind rather
than failing the whole deletion, matching rsync's leave-non-empty-dirs
behavior. Rehearsal/delete each open an independent file description so a
prior pass cannot drain the directory stream.
The STATUS_MANIFEST frame now carries two count-delimited sections: the kept
paths and a protected-prefix list (excluded-on-source paths the walker must not
delete unless --delete-excluded opted out). The receiver's DeleteManifest is
passed through the commit/early paths unchanged. manifest_delete_extras
honors a client --max-delete (all-or-nothing) and produces a distinct error for
it versus the 100000-entry server bound. --force clears a non-empty directory
that blocks an incoming regular file (confined, symlink-safe) via a new
file_remove_tree_secure helper.
The scanners now record every entry pruned by user-selection rules
(--filter/-C/per-dir, --exclude/--include, --max-size/--min-size) as a
destination-relative protected path on a caller-supplied sink (thread-safe in
the parallel scanner); --files-from subset pruning and -R relative wire paths
are never recorded. The sender transmits these as manifest protected prefixes,
giving rsync's default --delete behavior (excluded mirrors survive) with
--delete-excluded opting back into deleting them. --ignore-errors makes an
unreadable source directory a recorded, non-fatal scan error: the run continues,
the deletion still runs, and the exit code reports the ignored error.
--prune-empty-dirs omits an empty source directory's explicit --dirs entry.
Empty directories were never transferred by recursive scans (rsync -m parity).
Unit: walker all-or-nothing bounds (exceeded -> nothing removed + distinct
result; exact bound -> deletes), protected-prefix skipping, config wire
round-trip for force_delete/delete_excluded/prune_empty_dirs/max_delete, CLI
parse/validation for the new flags. Integration (TestDeletePolicy): default
delete-excluded protection and --delete-excluded opt-out (single-thread, -m,
early --delete-before, excluded-dir subtrees), --max-delete all-or-nothing over
and at the limit, --force file-over-nonempty-dir replacement, --prune-empty-dirs
(--dirs mode + recursion-mode parity), and --ignore-errors keeping deletion
active across a genuine scan I/O error (run as an unprivileged user).
--delete-excluded/--max-delete/--ignore-errors/--force/--prune-empty-dirs now
✅ with precise notes: the rsync-parity default (plain --delete protects
filter-excluded destination mirrors), the -m divergence (FastSync -m stays
multithreading, so --prune-empty-dirs is long-only), the all-or-nothing
max-delete/hard-bound error model, the 2.8.0 -> 2.9.0 protocol bump, and the
new two-section STATUS_MANIFEST frame. Summary counts left for the orchestrator
to recount after the wave.
file_remove_tree_secure() opens the final component O_NOFOLLOW below the
authorized root and recursively wipes it with an fd-relative walk (symlinks are
removed by name, never followed); --force uses it to clear a destination
directory that blocks an incoming regular file.
A sequential scanner records an opendir failure of its seed/root directory as a
skippable io_error and would complete an EMPTY scan, whose keep-set manifest
would then delete every destination entry. The seed directory that maps to the
transfer root (relative path "") is now fatal regardless of --ignore-errors;
only subdirectories discovered during an otherwise-successful root scan are
skippable. The -m path never had this hole (its root open failure aborts
scanner creation), so sequential and -m now agree.
scan_directory_multithreaded read directory_scanner_had_io_error() /
parallel_scanner_had_io_error() AFTER destroying the scanner object (a
heap-use-after-free on every successful -m run that recorded an io_error); the
flag is now captured before the destroy. As a second line of defense against
an empty-keep-set wipe, all four manifest send sites (sequential and -m, early
pre-scan and commit data pass) now refuse to transmit a keep-set manifest when
the scan that built it recorded an io_error and produced no keep entries: a
source that merely LOOKS empty because part of it was unreadable must never
delete the whole destination. A genuinely empty source (no io_error) still
sends its empty keep-set and prunes extras.
- ignore-errors scan-error test now runs single-threaded and under -m (the
exact scan_directory_multithreaded path that had the use-after-free);
- new integration test: --delete/--delete-before --ignore-errors with an
unreadable SOURCE ROOT (sequential and -m) must fail and delete NOTHING;
- new integration test: a destination-only file that merely matches an exclude
rule is deleted under plain --delete (protection is sender-derived), while a
source-excluded mirror is protected;
- delete-protects/delete-excluded coverage extended to --delete-after and
--delete-delay;
- new unit test: the 100000-entry server hard bound is all-or-nothing (more
extras than the bound -> nothing removed);
- new integration test: --force is inert under --delay-updates (documented);
- fixed the ignore-errors assertion message that stated the opposite of what it
asserted.
The walker's all-or-nothing guarantee holds only while the destination is not
concurrently modified (rehearsal and delete are separate walks). The
protected-prefix list shares the 16 MB MAX_MANIFEST_BYTES budget with the
keep-set and each section is capped at MAX_MANIFEST_ENTRIES; an over-budget
frame is rejected on the receiver with STATUS_ERROR rather than truncated.
Merged into dev via local merge (2FA blocks server-side merge). delete-policy ef13a70 + fuzzy ebab335. Independent c-review: delete-policy REQUEST CHANGES (2 blockers fixed: -m use-after-free, unreadable-root wipe under --ignore-errors) ; fuzzy APPROVE WITH NITS (perf/confinement/CLI findings fixed). dev CI run #487: all jobs success. Closing without server merge.
Merged into dev via local merge (2FA blocks server-side merge). delete-policy ef13a70 + fuzzy ebab335. Independent c-review: delete-policy REQUEST CHANGES (2 blockers fixed: -m use-after-free, unreadable-root wipe under --ignore-errors) ; fuzzy APPROVE WITH NITS (perf/confinement/CLI findings fixed). dev CI run #487: all jobs success. Closing without server merge.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Implements the Phase-3 delete-policy row:
--delete-excluded,--max-delete=NUM,--ignore-errors,--force,--prune-empty-dirs.Default-behavior parity decision
rsync does NOT delete filter-excluded destination files under plain
--delete; FastSync previously did (deletion = "not in the keep-set"). This PR changes the default to rsync parity: the sender now records every path its source scan prunes by user-selection rules (--filter/-F/-C,--exclude/--include,--max-size/--min-size) and transmits them as protected prefixes in the delete-manifest frame; the walker never deletes them.--delete-excludedopts back in (empty protected list → excluded mirrors become extras). Divergence: protection is sender-derived only (FastSync never re-applies rules to the destination),--files-fromsubset pruning and-Rrelative paths are never protected, and--max-size/--min-sizemirrors are protected (verified against real rsync 3.4.1).Per-flag realization
--delete-excluded– default protects excluded mirrors (above); the flag makes the sender send no protected prefixes so they are deleted. Covered single-thread,-m, early (--delete-before) and commit timing, and whole excluded-dir subtrees.--max-delete=NUM– all-or-nothing (rsync doc semantics): the receiver rehearses the deletion (an identical fd-relative count walk) and if the extras would exceed NUM deletes NOTHING and fails with a distinct--max-deleteerror; at/below the limit it deletes exactly the extras. Directories count. Inert without--delete. The hardMAX_SERVER_DELETE_COUNT(100000) also became all-or-nothing with its own error (previously it truncated at 100000 then failed).--ignore-errors– a source-scan I/O error (unreadable directory) now records a non-fatal scanner error; default aborts the run (no deletion), with the flag the scan continues, the readable tree transfers, deletion still runs, and the run exits non-zero. Verified end-to-end by running the client as an unprivileged user against a mode-000 source dir. Divergence: without the flag FastSync aborts the whole run (rsync transfers the rest and merely skips deletion); both leave deletion undone.--force– rsync's file-over-directory replacement: an incoming regular file may clear a (non-empty) destination directory first, via a new confined, symlink-safefile_remove_tree_secure()(O_NOFOLLOW fd walk, symlinks removed by name, never followed). Without--forcethe write/run fails. Applies on the immediate-install path (not under--delay-updates, documented).--prune-empty-dirs– long-only (FastSync-mstays multithreading; recorded divergence). Recursive transfers never emit directory entries, so empty dirs are inherently never transferred (rsync-mparity) and truly-empty destination chains are removed by--deletewith or without the flag; the flag's additional effect is on the--dirsgenerator (an empty source dir's explicit entry is omitted → not created, no-i/--out-formatline, existing empty mirror pruned by--delete). Explicitly--files-from-listed dirs still pass through.Protocol
STATUS_MANIFESTframe (keep-set + protected prefixes); receiverDeleteManifestthreaded through commit/early paths.force_delete;max_deletedefault changed to -1;ignore_errorsis client-only (not serialized).PROTOCOL_VERSION2.8.0 → 2.9.0 (mirrored version strings).Tests
DELETE_WALK_LIMIT_EXCEEDED, nothing removed; exact bound → deletes), protected-prefix skipping + symlink/kept-dir leave-behind; config wire round-trip forforce_delete/delete_excluded/prune_empty_dirs/max_delete; CLI parse/validation incl. invalid--max-delete.TestDeletePolicy, 19 tests): delete-excluded default vs opt-out (st/-m/early),--max-deleteover/at limit,--forcereplacement,--prune-empty-dirs(dirs + recursion parity),--ignore-errorsacross a real scan I/O error../build/testsall pass; fulltests/integration/test_features.py190 passed; preflight/tcp/tls pass (ssh tests auto-skip locally — no ssh binary); clang-format 18 and cppcheck clean in the CI image; strict-warnings build clean.Docs
RSYNC_COMPAT.mdrows 106–115 updated (--deletedefault note, all five policy flags → ✅) plus the Phase-3 implementation-notes block (two-section frame, all-or-nothing walker, 2.9.0 bump, policy-flags-don't-imply-delete). Summary counts intentionally left for the orchestrator to recount.--delete-excluded/--max-delete/--ignore-errors/--force/--prune-empty-dirs now ✅ with precise notes: the rsync-parity default (plain --delete protects filter-excluded destination mirrors), the -m divergence (FastSync -m stays multithreading, so --prune-empty-dirs is long-only), the all-or-nothing max-delete/hard-bound error model, the 2.8.0 -> 2.9.0 protocol bump, and the new two-section STATUS_MANIFEST frame. Summary counts left for the orchestrator to recount after the wave.Merged into dev via local merge (2FA blocks server-side merge). delete-policy
ef13a70+ fuzzyebab335. Independent c-review: delete-policy REQUEST CHANGES (2 blockers fixed: -m use-after-free, unreadable-root wipe under --ignore-errors) ; fuzzy APPROVE WITH NITS (perf/confinement/CLI findings fixed). dev CI run #487: all jobs success. Closing without server merge.Pull request closed