HANDOFF.md

@@ -40,7 +40,7 @@
 5. **Preserve-attribute split (protocol 2.22.0)** landed on `feat/preserve-attr-split`: per-attribute `-p/-t/-o/-g` + `--no-*` negations, `-a` = `-rlptgoD`, and the 2.21.0 → 2.22.0 wire bump.
 6. **Rsync-parity wave (protocol 2.23.0)** on `feat/rsync-parity`: rsync short options/clustering/attached values (`-r`/`-b`/`-L`/`-B`, `-av`, `-aAX`, `-B1000`, `-essh`, `-MOPT`), `-c` checksum quick-check, `--checksum-choice`/`--compress-choice` validation and seed randomization, rsync timeout/max-alloc defaults, temp-dir confinement + `EXDEV` fallback, ownership/mapping parity (numeric-ids modifier, map ranges/`*`/empty-FROM, `--chown`+map conflicts, fake-super resolved-owner record), verbatim symlink storage with rsync `--safe-links`/`--munge-links`, socket recreation under `--specials`, `--chmod` 3.4.1 semantics, and delete scoping + `--max-delete` partial/exit-25. Wire: appended delete-manifest synchronized-directory section and `STATUS_DELETE_LIMIT`.
 7. **Parity-completion wave (protocol 2.24.0 → 2.26.0)** on `feat/parity-completion`: per-directory delete plans (`STATUS_DELETE_PLAN`) for `--delete-during`/`--delete-delay`; receiver `STATUS_STATS` counters feeding `--stats`/`--progress` and `--out-format %b/%c/%C`, plus `-n --delete` lines; `lz4`/`zlib`/`zlibx` compression and `md4`/`sha1`/`none` checksums with `auto` negotiation (default `xxh128`/`zstd`); general `-R`/`--no-implied-dirs`/`-d`; the full filter grammar (`merge`/`dir-merge`/`hide`/`show`/`protect`/`risk`/`clear` + modifiers) and corrected `-F`/`-FF`; receiver-side `--chown`/map TO-name resolution; absolute basis dirs + `--link-dest` relink; receiver-side `--ignore-existing` short-circuit; `--preallocate` over `--sparse` via `fallocate(2)`; `--iconv=.`/`-`/`--no-iconv`; lone `-h` help; aliases `--ignore-non-existing`/`--protect-args`/`--msgs2stderr`; and the full `--info`/`--debug` vocabulary. `RSYNC_COMPAT.md` reclassifies the matrix to 106 ✅ / 27 ⚠️ / 23 ❌; the later rsync-parity-stats pass (`fix/parity-stats`) moves it to 107 ✅ / 25 ⚠️ / 24 ❌ (see item 8).
-8. **rsync-parity-stats pass** on `fix/parity-stats` (no wire change, `PROTOCOL_VERSION` stays `2.26.0`): `--delete-delay` now counts/budgets only entries actually removed (a refilled deferred directory that survives `ENOTEMPTY` is not counted; `--max-delete` partial-delete count matches rsync); `--stats` gained the `(reg/dir/link/special)` `Number of files` breakdown and now counts only regular files actually stored for `Number of regular files transferred`/transferred size/literal data (up-to-date re-runs report 0); `Total file size` includes symlink target lengths; `--progress` prints the leading `./` root line and counts it in `to-chk` so a single-file transfer matches rsync; and `%C` uses the selected transfer checksum with `checksum_digest_file` supporting md4/sha1/none, byte-identical to rsync for every algorithm. `--out-format` reclassified ❌ (`%b`/delta-`%c` are protocol-specific). Differential + regression tests added; full suite + ASan + clang-format + cppcheck clean.
+8. **rsync-parity-stats pass** on `fix/parity-stats` (no wire change, `PROTOCOL_VERSION` stays `2.26.0`): `--delete-delay` now reports only entries it actually removes, while the `--max-delete` budget is charged at plan/snapshot time (`planned`, via `defer_add`) to bound the deferred list (a refilled deferred directory that survives `ENOTEMPTY` is not reported but still consumes budget); `--stats` gained the `(reg/dir/link/special)` `Number of files` breakdown and now counts only regular files actually stored for `Number of regular files transferred`/transferred size/literal data (up-to-date re-runs report 0); `Total file size` includes symlink target lengths; `--progress` prints the leading `./` root line and counts it in `to-chk` so a single-file transfer matches rsync; and `%C` uses the selected transfer checksum with `checksum_digest_file` supporting md4/sha1/none, byte-identical to rsync for every algorithm. `--out-format` reclassified ❌ (`%b`/delta-`%c` are protocol-specific). Differential + regression tests added; full suite + ASan + clang-format + cppcheck clean.
 9. **Option-parity wave (protocol 2.26.0 → 2.27.0, on `fix/parity-options`):**
    `--bwlimit` now ports rsync 3.4.1's units/quantization and paces like its
    leaky bucket; `--ignore-errors` reproduces rsync's default (an I/O error
@@ -58,7 +58,7 @@
    holds the rsync differentials (bwlimit parse+rate, info lines, real-setpriv
    `--ignore-errors`, rsync-daemon `-M`, filter-protect pin).
 
-9. **rsync-parity-fs pass** on `fix/parity-fs` (no wire change of its own; integrated
+10. **rsync-parity-fs pass** on `fix/parity-fs` (no wire change of its own; integrated
    on top of the 2.27.0 options wave): recursive transfers now recreate empty source directories (and
    `-m/--prune-empty-dirs` still suppresses them), a directory entry replaces a
    blocking destination regular file, and `-R --no-implied-dirs --files-from`
@@ -67,12 +67,18 @@
    now reproduces rsync's push direction (destination charset = the spec's REMOTE
    half; a server `--iconv` overrides), and `-T/--temp-dir` relative semantics are
    confirmed identical while the absolute-path confinement is a deliberate
-   divergence. The basis-dir options, `--delay-updates`, `--fuzzy` and `--dry-run`
-   were reclassified to ❌ after a differential test reproduced each exact residual
-   (basis content verification, fixed staging-name collision, heuristic size
-   window, and dry-run would-delete over-report). Differential-gate allowlist
+   divergence. The basis-dir options, `--delay-updates` and `--dry-run` were
+   reclassified to ❌ after a differential test reproduced each exact residual
+   (basis content verification, fixed staging-name collision, and dry-run
+   would-delete over-report). `--fuzzy` was also reclassified to ❌ (deterministic
+   heuristic with a 10× size window, not rsync's matcher), but its residual is the
+   candidate-selection heuristic itself: the final tree is byte-exact by design, so
+   it is pinned by the `TestFuzzy` threshold suite rather than a byte-level rsync
+   differential. The parity-review pass then moved `--delete-delay` to ⚠️ (the
+   plan-time `--max-delete` charge and non-recursive deferred removal differ from
+   rsync when a snapshotted entry fails removal). Differential-gate allowlist
    entries `min_size`/`empty_dirs_recursive`/`dirs_plain` were removed. The
-   integrated stats+options+fs branch stands at **112 ✅ / 12 ⚠️ / 33 ❌ = 157**;
+   integrated stats+options+fs branch stands at **111 ✅ / 13 ⚠️ / 33 ❌ = 157**;
    full suite + ASan + clang-format + cppcheck clean.
 
 ## Next steps

RSYNC_COMPAT.md

@@ -6,8 +6,8 @@ This document maps rsync's full feature set to FastSync's current implementation
 
 | Status | Count | Description |
 |--------|-------|-------------|
-| ✅ Parity | 112 | Reproduces rsync's semantics for this option's scope |
-| ⚠️ Caveat | 12 | Wired and tested, but carries a documented behavioral difference from rsync (named in the row and/or the wave notes) |
+| ✅ Parity | 111 | Reproduces rsync's semantics for this option's scope |
+| ⚠️ Caveat | 13 | Wired and tested, but carries a documented behavioral difference from rsync (named in the row and/or the wave notes) |
 | ❌ Divergent | 33 | Rejected, an accepted no-op, deliberately non-rsync (native config/auth/batch, privileged namespaces, safe-subset privilege), or impossible on any portable filesystem call |
 | **Total** | **157** | One row per rsync option/feature group; a row may name several spellings |
 
@@ -33,8 +33,9 @@ genuinely non-interoperable residuals are reclassified divergent (`-M` over a
 daemon/TCP connection, which FastSync's binary config handshake has no argv
 channel for, and receiver-side `protect`/`risk` re-derivation for
 destination-only entries, which would need a receiver filter engine). After the
-combined `fix/parity-stats` + `fix/parity-options` + `fix/parity-fs` passes the
-matrix is **112 ✅ / 12 ⚠️ / 33 ❌ = 157**.
+combined `fix/parity-stats` + `fix/parity-options` + `fix/parity-fs` passes (and
+the later `fix/parity-review` correction that moved `--delete-delay` to ⚠️) the
+matrix is **111 ✅ / 13 ⚠️ / 33 ❌ = 157**.
 
 **Parity completion wave (protocol 2.23.0 → 2.26.0).** This wave closed the
 remaining gaps the rsync-parity wave left open (delete timing, wire counters and
@@ -153,7 +154,7 @@ Every one of those has an entry below with its remaining caveats.
 | `--delete` | Delete extraneous files from dest | ⚠️ Caveat | `use_delete` config field. Deletion is always derived from the transmitted keep-set manifest of the paths the sender sent/keeps (never from unchecked input), runs through the symlink-safe walker bounded by `MAX_SERVER_DELETE_COUNT`, and skips the `.fastsync-stage` staging dir under `--delay-updates`. FastSync's default timing when no timing flag is given is **delete-after** (extras are removed only once the whole transfer succeeded) — intentionally NOT rsync's `--del`/delete-during default, to preserve FastSync's commit-style safety. By default the destination mirror of a path the source scan pruned (filter/exclude/size rules) is **protected** from deletion — matching rsync, which does not delete excluded files under `--delete`; `--delete-excluded` opts back into deleting them (see below). Deletion is scoped to the **synchronized directories** sent in the manifest (protocol 2.23.0), so a `--files-from` subset no longer deletes untransmitted paths outside the listed directory subtrees. The walk is bounded: a client `--max-delete=NUM` (or the 100000-entry server bound) makes it **partial** — entries up to the bound are removed, the rest are skipped, and the client exits **25** (`RERR_PARTIAL`), matching rsync, rather than failing the transfer. Extraneous destination symlinks are unlinked by name (never followed); a directory still holding a kept/protected entry is left behind rather than failing |
 | `--delete-before` | Delete before transfer | ⚠️ Caveat | Implies `--delete`. The sender runs a full source pre-scan (paths only) and transmits the keep-set manifest BEFORE any file data; the receiver validates it, removes every destination entry not listed (bounded walk, staging-dir skip, protected prefixes honored), then acks `STATUS_OK`. The sender only starts streaming after the deletion committed, or aborts if the receiver reported a deletion error. By definition the deletions already happened when a later transfer phase fails — rsync's delete-before is destructive the same way; a subsequent failure does not restore the removed files. Divergence: the keep-set is the pre-scan snapshot, so a file that appears on the source between the pre-scan and the data pass is still transferred but was not protected from deletion |
 | `--del`, `--delete-during` | Delete during transfer | ⚠️ Caveat | Both spellings accepted; imply `--delete`. **Protocol 2.24.0 implements per-directory delete plans:** as the sender finishes each source directory it streams a `STATUS_DELETE_PLAN` for that directory and the receiver removes that directory's extras before applying the next directory's data, so a mid-transfer failure has already removed the extras of the directories reached (verified with a byte-slicing proxy). **Remaining divergence:** the exact abort boundary and the progressive ordering of removals versus rsync's generator can differ, and `-d`/`--dirs` (no descent) falls back to the end-of-transfer commit. `-R` plans are scoped to the transferred prefix subtree |
-| `--delete-delay` | Find deletions during, delete after | ✅ Parity | Implies `--delete`. **Protocol 2.24.0 implements rsync's delete-delay timing:** the sender records each directory's delete plan while scanning and the receiver commits those removals only after the whole transfer succeeds (per plan), so an extra created in the destination after its directory's plan survives while `--delete-after` re-scans and removes it, and a failed transfer removes nothing. The deleted count and `--max-delete` budget now advance only on an actual removal: a directory snapshotted into the plan that is refilled before the commit and survives `ENOTEMPTY` is **not** reported, and a `--max-delete=2` partial delete reports exactly 2 (differential test vs rsync 3.4.1, exit 25 both). Unit tests cover the refilled-directory and removed-file accounting. Exact ordering of which extras are removed first can still differ from rsync's generator |
+| `--delete-delay` | Find deletions during, delete after | ⚠️ Caveat | Implies `--delete`. **Protocol 2.24.0 implements rsync's delete-delay timing:** the sender records each directory's delete plan while scanning and the receiver commits those removals only after the whole transfer succeeds (per plan), so an extra created in the destination after its directory's plan survives while `--delete-after` re-scans and removes it, and a failed transfer removes nothing. The **reported** deleted count advances only on an actual removal; the `--max-delete` budget is charged at plan/snapshot time (the `planned` counter, incremented by `defer_add`) to bound the deferred list, so a snapshotted extra that is later skipped or fails removal still consumes budget. A directory snapshotted into the plan that is refilled before the commit and survives `ENOTEMPTY` is **not** reported, and a `--max-delete=2` partial delete reports exactly 2 (differential test vs rsync 3.4.1, exit 25 both). **Caveat:** rsync charges `--max-delete` on actual removals and its deferred removal recurses into a queued directory, so when a snapshotted entry fails removal FastSync skips a later extra that rsync would still delete, and content created in a refilled extra directory is removed by rsync but left in place by FastSync (the differential test pins both sides). Unit tests cover the refilled-directory, removed-file, and plan-time budget accounting. Exact ordering of which extras are removed first can still differ from rsync's generator |
 | `--delete-after` | Delete after transfer | ✅ Parity | Implies `--delete`. The delete-after timing is also what plain `--delete` does: the keep-set manifest closes the data stream and the receiver commits the bounded deletion only after the terminal `STATUS_FINISHED` proves the whole transfer (every data frame received and stored) succeeded. A failed or aborted transfer removes nothing |
 | `--delete-excluded` | Also delete excluded files | ⚠️ Caveat | `delete_excluded` config field. Under `--delete` FastSync protects (rsync's default) the destination mirror of paths the sender's source scan pruned by the user-selection rules — the `--filter`/`-F`/`-C` layer and the legacy `--exclude`/`--include` layer. The sender transmits those concrete pruned paths as **protected prefixes** in the delete-manifest frame (see the Phase-3 notes below); the walker never descends into or removes them. `--delete-excluded` opts back in: the sender sends an empty protected list, so the excluded destination mirrors become ordinary extras and are removed. **`--max-size`/`--min-size` pruned mirrors are a separate, always-on protection** (protocol 2.23.0, rsync parity): size-pruned source mirrors survive `--delete` even with `--delete-excluded`. Divergences (documented): protection is derived only from what the source scan actually pruned — a stray destination-only file that happens to match an exclude rule is not protected (FastSync never re-applies rules to the destination, keeping deletion sender-derived) |
 | `--max-delete=NUM` | Max files to delete | ✅ Parity | `max_delete` config field (default -1 = no client limit; 0 = delete nothing). **Protocol 2.23.0 matches rsync's partial semantics:** the receiver deletes up to NUM entries (regular files, symlinks and empty directories; each directory removal counts as one) and then **stops deleting, skips the rest, and reports the run as partial**. The client prints a "deletions stopped due to `--max-delete` limit" message and exits **25** (rsync's `RERR_PARTIAL`), not a hard failure — the transfer itself succeeded. NUM only applies together with `--delete` (it is inert otherwise, matching rsync). A client NUM below the server hard bound `MAX_SERVER_DELETE_COUNT` (100000) replaces it; a NUM above it never raises that cap. Deleting an entire destination with no limit is still bounded by the server's 100000-entry ceiling. `--delete-missing-args` exact-path deletions and the ordinary extras walk draw from the same budget, matching rsync |
@@ -908,7 +909,7 @@ These are the last compatibility items and the closing phase toward rsync flag p
 
 **Wire:** two trailing config-frame blocks after the `--iconv` spec, in fixed order — `send_privilege_options`/`receive_privilege_options` (one `super_mode` int, validated `0..2`), then `send_copy_as_options`/`receive_copy_as_options` (presence int + two int32 ids, validated `>= 0`, with `copy_as_set ⇒ use_metadata`). `PROTOCOL_VERSION` bumped **2.17.0 → 2.18.0**. **Divergences from rsync:** rsync's `--super` elevates the receiver and `--copy-as` actually switches its credentials; FastSync never elevates and only permits/forwards confined attempts, and `--copy-as` forces ownership rather than switching identity.
 
-**Honest status after the parity-completion wave (protocol 2.27.0), updated by the rsync-parity-stats, rsync-parity-options, and rsync-parity-fs passes.** ✅ Parity 112 / ⚠️ Caveat 12 / ❌ Divergent 33 = 157 rows. Earlier revisions of this document reported "143 ✅ / 0 divergence / 0 partial"; that conflated "parsed and tested" with "rsync parity", because many rows carried documented behavioral differences and some short options were not parsed at all. This reclassification makes every difference explicit. The completion wave closed 23 previously-caveated rows (9 that triage showed were already parity, plus 14 genuine fixes) and turned the 17 inherently non-rsync rows — native daemon config/auth, the FastSync batch container, the safe-subset device/privilege flags, `-X`'s privileged namespaces, `--fake-super`'s native xattr format, and the `--old-args` no-op — into explicit ❌ divergences. The stats pass flipped `--delete-delay` to ✅ (actual-removal accounting) and reclassified `--out-format` to ❌ (protocol-specific `%b`/delta-`%c`), and sharpened the `--stats`/`--progress`/`--checksum-choice` residuals. The options pass flipped `--bwlimit` and `--ignore-errors` to ✅ (rsync-exact size parsing and ~100 ms leaky-bucket throttling, and rsync's skip-unreadable-subdir plus IO-error-suppressed deletion with exit 23) and emits rsync-format `--info=name/flist/del/remove/nonreg/progress` lines (real-run `deleting`/`*deleting` carried over a new trailing `report_deletes` wire bool, `PROTOCOL_VERSION` 2.26.0 → 2.27.0), while reclassifying `-M` over daemon/TCP and receiver-side `protect`/`risk` re-derivation to ❌ (no argv channel / receiver filter engine). The fs pass flipped `-d/--dirs` and `--iconv` to ✅ — recursive transfers now recreate empty source directories (and replace a blocking destination non-directory with an incoming directory); `-R --no-implied-dirs --files-from` places a listed file under a missing implied parent with default attributes instead of refusing; and `--iconv` now reproduces rsync's push direction (destination charset = the spec's REMOTE half) — and reclassified seven rows to ❌ after reproducing their exact residual with differential tests: `--temp-dir` (the receiver confines the scratch dir to the receive root, so an absolute temp dir is deliberately rejected although standalone rsync follows it), the three basis-dir options (FastSync xxHash-verifies a basis hit while rsync's `--size-only` quick check installs the wrong basis content), `--delay-updates` (fixed staging name wipes an unrelated destination entry of that name), `--fuzzy` (deterministic heuristic with a 10× size window, not rsync's matcher), and `--dry-run` (would-delete report over-reports). The remaining ⚠️ rows are the ones with a documented residual (see the row notes and the **Parity Completion Wave (protocol 2.26.0)** section below).
+**Honest status after the parity-completion wave (protocol 2.27.0), updated by the rsync-parity-stats, rsync-parity-options, rsync-parity-fs, and parity-review passes.** ✅ Parity 111 / ⚠️ Caveat 13 / ❌ Divergent 33 = 157 rows. Earlier revisions of this document reported "143 ✅ / 0 divergence / 0 partial"; that conflated "parsed and tested" with "rsync parity", because many rows carried documented behavioral differences and some short options were not parsed at all. This reclassification makes every difference explicit. The completion wave closed 23 previously-caveated rows (9 that triage showed were already parity, plus 14 genuine fixes) and turned the 17 inherently non-rsync rows — native daemon config/auth, the FastSync batch container, the safe-subset device/privilege flags, `-X`'s privileged namespaces, `--fake-super`'s native xattr format, and the `--old-args` no-op — into explicit ❌ divergences. The stats pass flipped `--delete-delay` to ✅ (actual-removal accounting), but the parity-review pass moved it back to ⚠️: FastSync charges the `--max-delete` budget at plan/snapshot time and leaves a refilled snapshotted directory in place, whereas rsync charges on actual removals and recursively removes a queued directory (including content created after its plan), so the two diverge when a snapshotted entry fails removal (see the `--delete-delay` row and the differential test). The stats pass also reclassified `--out-format` to ❌ (protocol-specific `%b`/delta-`%c`), and sharpened the `--stats`/`--progress`/`--checksum-choice` residuals. The options pass flipped `--bwlimit` and `--ignore-errors` to ✅ (rsync-exact size parsing and ~100 ms leaky-bucket throttling, and rsync's skip-unreadable-subdir plus IO-error-suppressed deletion with exit 23) and emits rsync-format `--info=name/flist/del/remove/nonreg/progress` lines (real-run `deleting`/`*deleting` carried over a new trailing `report_deletes` wire bool, `PROTOCOL_VERSION` 2.26.0 → 2.27.0), while reclassifying `-M` over daemon/TCP and receiver-side `protect`/`risk` re-derivation to ❌ (no argv channel / receiver filter engine). The fs pass flipped `-d/--dirs` and `--iconv` to ✅ — recursive transfers now recreate empty source directories (and replace a blocking destination non-directory with an incoming directory); `-R --no-implied-dirs --files-from` places a listed file under a missing implied parent with default attributes instead of refusing; and `--iconv` now reproduces rsync's push direction (destination charset = the spec's REMOTE half) — and reclassified six rows to ❌ after reproducing their exact residual with differential tests: `--temp-dir` (the receiver confines the scratch dir to the receive root, so an absolute temp dir is deliberately rejected although standalone rsync follows it), the three basis-dir options (FastSync xxHash-verifies a basis hit while rsync's `--size-only` quick check installs the wrong basis content), `--delay-updates` (fixed staging name wipes an unrelated destination entry of that name), and `--dry-run` (would-delete report over-reports). `--fuzzy` was also reclassified to ❌ (deterministic heuristic with a 10× size window, not rsync's matcher), but its residual is the candidate-selection heuristic itself: the final tree is byte-exact by design, so no destination differential can expose it and the row is pinned by the `TestFuzzy` threshold suite rather than a byte-level rsync differential. The remaining ⚠️ rows are the ones with a documented residual (see the row notes and the **Parity Completion Wave (protocol 2.26.0)** section below).
 
 **Preserve-attribute split (protocol 2.21.0 → 2.22.0) — ✅ implemented.** FastSync splits the former single metadata bundle into four independent, rsync-compatible per-attribute flags — `-p/--perms`, `-t/--times`, `-o/--owner`, `-g/--group` — each with a negation (`--no-perms`/`--no-times`/`--no-owner`/`--no-group`, short `--no-p`/`--no-t`/`--no-o`/`--no-g`), plus `--no-preserve` clearing all four. `-a/--archive` is now full rsync `-rlptgoD` (owner and group included, though their application stays privilege-gated), `-A/--acls` implies `-p`, `-X/--xattrs` does not, `-E/--executability` sets only executability, and `-U`/`-N` do not imply `-t`. `--incremental`/`--delta` still auto-preserve perms+times unless the user explicitly negated them. Wire: the binary config frame gains four appended booleans (`preserve_perms`/`preserve_times`/`preserve_owner`/`preserve_group`) after `omit_link_times`, so `PROTOCOL_VERSION` is bumped **2.21.0 → 2.22.0**; the fixed-width `FileMetadata` layout is unchanged and the receiver gates the metadata frame on a derived `use_metadata`. Receiver behavior: each attribute is applied independently, directory modes are applied under `-p` (at the end of the transfer, alongside dir times), symlink mode under `-p`, and `-O/--omit-dir-times` suppresses directory times only. Documented divergences as of 2.22.0, **all but (d)/(e) removed by the rsync-parity wave (protocol 2.23.0)**: (a) the mode-masking divergence is **gone** — under `-p` the source mode is now copied exactly, including `S_IWGRP`/`S_IWOTH` and setuid/setgid/sticky; (b) a brand-new file without `-p` still gets `source_mode & ~umask` when metadata is present (else the historical fixed `0644`), and a new *directory* without `-p` still uses FastSync's `0755` default; (c) the `--chmod`-implies-`-p` divergence is **gone** — `--chmod` no longer implies `-p` (rsync parity); (d) `-o`/`-g` map by name on the receiver with a raw-numeric fallback (only numeric ids cross the wire); (e) a daemon module without `client owner = yes` does not refuse a plain `-a`/`-o`/`-g` — it forces super off, applies no ownership, and logs a warning, while explicit `--chown`/`--usermap`/`--groupmap`/`--numeric-ids`/`--copy-as`/`--super` are still refused.
 

src/client/client_send.c

@@ -72,7 +72,7 @@ static void log_server_rejection(const char* context) {
  * continues past an unreadable subdirectory so the readable tree transfers, and
  * always reports the partial transfer (exit 23); this only decides whether the
  * deletion phase is skipped.  Returns true when deletion may proceed. */
-static bool ignore_errors_allows_delete(const Config* config, bool had_io_error) {
+bool ignore_errors_allows_delete(const Config* config, bool had_io_error) {
   return !had_io_error || (config && config->ignore_errors);
 }
 

src/client/client_send.h

@@ -23,6 +23,12 @@ void client_set_abort_armed(bool armed);
  * config_delete() once the call returns). */
 int send_files(Config* config);
 int send_files_multithreaded(Config** config);
+/* rsync's --ignore-errors deletion gate: with no I/O error during the scan the
+ * deletion phase always proceeds; with one it is suppressed unless
+ * `--ignore-errors` was given.  Exposed so the decision can be unit-tested
+ * without a privileged (mode-000) source directory.  See client_send.c. */
+bool ignore_errors_allows_delete(const Config* config, bool had_io_error);
+
 /* Phase 6 residual-batch (client-only).  See client_send.c. */
 int write_batch_from_source(const Config* config, const char* batch_path);
 int apply_batch_to_dest(const Config* config, const char* batch_path, const char* dest_root);

src/shared/delete_plan.h

@@ -77,10 +77,10 @@ int delete_plan_session_receive(DeletePlanSession* session, const Config* config
 DeleteCommitResult delete_plan_session_commit(DeletePlanSession* session, const Config* config);
 /* True once the shared --max-delete budget stopped part of a deletion. */
 bool delete_plan_session_limit_reached(const DeletePlanSession* session);
-/* Number of destination entries the session's plans ACTUALLY removed, for the
-   end-of-transfer stats.  A --delete-delay entry only snapshotted for removal
-   that survives the commit (e.g. a directory refilled before it is removed, so
-   unlink/rmdir fails with ENOTEMPTY) is not counted. */
+/* Number of destination entries the session actually removed, for the
+   end-of-transfer stats.  For --delete-delay this excludes a snapshotted entry
+   that survived (e.g. a refilled directory that failed ENOTEMPTY), even though
+   that entry already consumed --max-delete budget at snapshot time. */
 size_t delete_plan_session_deleted(const DeletePlanSession* session);
 /* Install an observer invoked for every destination-relative path the session
    truly removes (including the deferred --delete-delay commit), so the receiver

tests/integration/README.md

@@ -38,11 +38,17 @@ FastSync mirrors the absolute source path under its receive root (see
 # Fast subset that guards the ✅ surface on pull requests
 python3 -m pytest tests/integration/test_differential_parity.py -n 4 --dist=load -m parity_ci
 
-# Full set (all ✅ cases plus the documented ⚠️/❌ residuals)
+# Full differential case table (`_CASES`): every row is marked `parity`, and a
+# case with an allowlisted residual in `parity_caveats.py` is included too.
 python3 -m pytest tests/integration/test_differential_parity.py -n 4 --dist=load -m parity
 ```
 
-The suite skips cleanly when `rsync` is not installed.
+`-m parity` selects only the `_CASES` table in this module. Differential
+coverage for options outside that table (`--temp-dir`, `--delay-updates`,
+`--dry-run`, `--fuzzy`, the basis-dir options, `-M` over daemon/TCP, and
+receiver filter-protect) lives in dedicated modules (`test_option_parity.py`,
+`test_parity_blockers.py`, `test_parity_quickwins.py`, ...) and is not part of
+this gate. The suite skips cleanly when `rsync` is not installed.
 
 ## Allowlist (`parity_caveats.py`)
 
@@ -52,10 +58,11 @@ Each entry maps a case id to the aspects that may differ (`tree`, `stdout`,
 
 ```python
 CAVEATS = {
-    "min_size": {
-        "tree": "recursive transfer does not create a source directory that "
-                "becomes empty after --min-size filtering. ref: RSYNC_COMPAT.md "
-                "`-d/--dirs` row and completion-wave residual.",
+    "max_delete": {
+        "tree": "which destination extras survive a partial --max-delete abort "
+                "is deletion-order dependent and unspecified; rc=25 and the "
+                "number of survivors match rsync. ref: RSYNC_COMPAT.md "
+                "`--max-delete=NUM` row.",
     },
 }
 ```

tests/integration/test_delete_delay_budget_parity.py

@@ -0,0 +1,191 @@
+"""Differential coverage for ``--delete-delay`` + ``--max-delete`` with a
+refilled deferred directory.
+
+FastSync snapshots a directory's extras at plan time (``defer_add``) and charges
+``--max-delete`` then, and its deferred commit only removes the snapshot path, so
+a directory refilled before the commit survives ``ENOTEMPTY``.  rsync computes
+the deferred deletions during the transfer, charges ``--max-delete`` on actual
+removals, and recursively removes a queued directory -- so content created after
+the plan inside an extra directory is removed too.
+
+These tests run both tools on the same fixture and pin the shared budget bound
+(the later extra survives, both exit 25) plus the documented residual (the
+refilled directory's late content survives under FastSync, not rsync).  They are
+not part of the fast PR gate because the rsync side needs a wide real-time
+injection window (a throttled transfer), while the FastSync side uses the
+existing byte-deterministic slicing proxy.
+
+The refilled directory sits at the transfer ROOT, whose delete plan is always
+processed before any subdirectory's, so the budget is deterministically charged
+to the refilled entry; the second extra lives under ``b`` and is skipped.
+"""
+import os
+import shutil
+import subprocess
+import sys
+import threading
+import time
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(__file__))
+from common import (  # noqa: E402
+    TEST_DATA_DIR,
+    ServerManager,
+    clean_dir,
+    get_dest_received_dir,
+    run_client,
+)
+from test_delete_timing_parity import _SlicingProxy  # noqa: E402
+
+RSYNC = shutil.which("rsync")
+requires_rsync = pytest.mark.skipif(RSYNC is None, reason="rsync 3.4.1 not installed")
+
+BIG_BYTES = 8 * 1024 * 1024
+MID_TRANSFER_BYTES = 256 * 1024
+PROXY_THROTTLE = 0.001
+# rsync is driven locally, so the refill is injected on a wall-clock delay while
+# a throttled ~8 s transfer is in flight.  1.5 s is safely after rsync's plan
+# scan (t=0) and well before the deferred commit at the end.
+RSYNC_BWLIMIT = 1024  # 1 MiB/s
+RSYNC_INJECT_DELAY = 1.5
+
+
+def _write(path, content):
+    os.makedirs(os.path.dirname(path), exist_ok=True)
+    with open(path, "wb") as fh:
+        fh.write(content)
+
+
+def _seed_source(tag):
+    source = os.path.join(TEST_DATA_DIR, f"ddb_{tag}_src")
+    clean_dir(source)
+    _write(os.path.join(source, "a", "keep.bin"), b"B" * BIG_BYTES)
+    _write(os.path.join(source, "b", "keep.txt"), b"keep\n")
+    return source
+
+
+def _seed_fastsync(tag):
+    """FastSync mirrors the absolute source path under its receive root, so the
+    extras live below ``received``."""
+    source = _seed_source(tag)
+    dest = os.path.join(TEST_DATA_DIR, f"ddb_{tag}_dst")
+    clean_dir(dest)
+    received = get_dest_received_dir(dest, source)
+    os.makedirs(os.path.join(received, "xdir"), exist_ok=True)
+    os.makedirs(os.path.join(received, "b", "ydir"), exist_ok=True)
+    return source, dest, received
+
+
+def _seed_rsync(tag):
+    """rsync mirrors the source contents directly into the destination, so the
+    extras are flat under ``rsync_dst``."""
+    source = _seed_source(tag)
+    rsync_dst = os.path.join(TEST_DATA_DIR, f"ddb_{tag}_dst")
+    clean_dir(rsync_dst)
+    os.makedirs(os.path.join(rsync_dst, "xdir"), exist_ok=True)
+    os.makedirs(os.path.join(rsync_dst, "b", "ydir"), exist_ok=True)
+    return source, rsync_dst
+
+
+def _deleted_count(text):
+    for line in text.splitlines():
+        if line.startswith("Number of deleted files:"):
+            return int(line.split(":", 1)[1].split()[0])
+    return None
+
+
+def _rsync(args, timeout=120):
+    env = dict(os.environ, LC_ALL="C")
+    return subprocess.run([RSYNC] + args, capture_output=True, text=True, env=env, timeout=timeout)
+
+
+class TestDeleteDelayRefilledDirVsRsync:
+    """The shared budget bound and the documented recursive-removal residual."""
+
+    def _fastsync_refilled(self, tag, max_delete=None):
+        """Run FastSync with the refill injected deterministically by the proxy
+        hook (fired once the receiver has processed the plan frames)."""
+        source, dest, received = _seed_fastsync(tag)
+        late = os.path.join(received, "xdir", "new.txt")
+
+        def hook():
+            _write(late, b"created mid-transfer\n")
+
+        flags = ["--delete-delay", "--incremental", "--ignore-times", "--stats"]
+        if max_delete is not None:
+            flags.append(f"--max-delete={max_delete}")
+        with ServerManager() as server:
+            server.start(extra_args=["--allow-delete"])
+            proxy = _SlicingProxy(server.port, hook=hook, hook_after=MID_TRANSFER_BYTES,
+                                  throttle=PROXY_THROTTLE, wait_for_reply=True)
+            result, _ = run_client(source, dest, flags=flags, port=proxy.port)
+            proxy.finish()
+        assert proxy.hook_called.is_set(), "refill hook never fired"
+        return result, received, late
+
+    @requires_rsync
+    def test_max_delete_budget_bound_matches_and_residual_pinned(self):
+        # --- FastSync: budget charged at snapshot; late content preserved ---
+        result, received, late = self._fastsync_refilled("budget_fs", max_delete=1)
+        assert result.returncode == 25, (result.stderr or result.stdout)[:300]
+        assert _deleted_count(result.stdout) == 0, result.stdout
+        assert os.path.exists(late), "FastSync removed the late content of a snapshotted dir"
+        assert os.path.isdir(os.path.join(received, "xdir"))
+        assert os.path.isdir(os.path.join(received, "b", "ydir")), (
+            "FastSync did not charge the plan-time budget: b/ydir was removed"
+        )
+
+        # --- rsync: budget charged on actual removals; dirs removed recursively ---
+        source, rsync_dst = _seed_rsync("budget_rsync")
+
+        def inject():
+            time.sleep(RSYNC_INJECT_DELAY)
+            _write(os.path.join(rsync_dst, "xdir", "new.txt"), b"created mid-transfer\n")
+
+        t = threading.Thread(target=inject)
+        t.start()
+        rsync_result = _rsync(
+            ["-a", "--delete-delay", "--max-delete=1", "--stats",
+             f"--bwlimit={RSYNC_BWLIMIT}", source + "/", rsync_dst + "/"]
+        )
+        t.join()
+        assert rsync_result.returncode == 25, rsync_result.stderr
+        # rsync removes the late content (recursive deferred removal); FastSync
+        # keeps it and charges the snapshot directive instead.
+        assert not os.path.exists(os.path.join(rsync_dst, "xdir", "new.txt")), (
+            "rsync kept late content inside a queued directory"
+        )
+        # The shared observable: the later extra survives in both tools under
+        # --max-delete=1, and both report the capped run with exit 25.
+        assert os.path.isdir(os.path.join(rsync_dst, "b", "ydir")), (
+            "rsync did not bound the deletion with --max-delete=1"
+        )
+        assert os.path.isdir(os.path.join(received, "b", "ydir"))
+
+    @requires_rsync
+    def test_refilled_extra_dir_recursive_removal_residual(self):
+        """Without --max-delete the residual is a plain tree difference: rsync
+        removes the refilled extra directory (and its late content), FastSync
+        leaves the snapshot path in place on ENOTEMPTY."""
+        result, received, late = self._fastsync_refilled("recur_fs")
+        assert result.returncode == 0, (result.stderr or result.stdout)[:300]
+        assert os.path.exists(late), "FastSync removed the refilled directory's late content"
+
+        source, rsync_dst = _seed_rsync("recur_rsync")
+
+        def inject():
+            time.sleep(RSYNC_INJECT_DELAY)
+            _write(os.path.join(rsync_dst, "xdir", "new.txt"), b"created mid-transfer\n")
+
+        t = threading.Thread(target=inject)
+        t.start()
+        rsync_result = _rsync(
+            ["-a", "--delete-delay", "--stats", f"--bwlimit={RSYNC_BWLIMIT}",
+             source + "/", rsync_dst + "/"]
+        )
+        t.join()
+        assert rsync_result.returncode == 0, rsync_result.stderr
+        assert not os.path.exists(os.path.join(rsync_dst, "xdir")), (
+            "rsync did not recursively remove the refilled extra directory"
+        )

tests/integration/test_delete_timing_parity.py

@@ -229,10 +229,13 @@ class TestDeleteTimingFinalStateParity:
     @pytest.mark.parametrize("timing", ["--delete-during", "--delete-delay"])
     @requires_rsync
     def test_success_final_state_matches_rsync(self, timing):
+        # Worker-safe names: xdist may run both parametrizations concurrently, so
+        # the timing is part of every fixture path.
+        label = timing.lstrip("-")
         # Build the rsync fixture from the same seed so both sides start equal.
-        source, dest, received = _seed_pair("parity_rsync")
+        source, dest, received = _seed_pair(f"parity_rsync_{label}")
         source2 = source
-        rsync_dst = os.path.join(TEST_DATA_DIR, "dtp_parity_rsync_dst")
+        rsync_dst = os.path.join(TEST_DATA_DIR, f"dtp_rsync_{label}_dst")
         clean_dir(rsync_dst)
         # rsync mirrors src/ into dst/; seed the same extra.
         _write(os.path.join(rsync_dst, "d", "old_extra"), b"stale extra\n")
@@ -258,7 +261,8 @@ class TestDeleteTimingTypeConflictParity:
     @pytest.mark.parametrize("timing", ["--delete-during", "--delete-delay"])
     @requires_rsync
     def test_type_conflicts_match_rsync(self, timing):
-        source = os.path.join(TEST_DATA_DIR, "dtc_src")
+        label = timing.lstrip("-")
+        source = os.path.join(TEST_DATA_DIR, f"dtc_{label}_src")
         clean_dir(source)
         _write(os.path.join(source, "foo"), b"now a file\n")
         _write(os.path.join(source, "bar", "inner.txt"), b"now a dir\n")
@@ -268,13 +272,13 @@ class TestDeleteTimingTypeConflictParity:
             _write(os.path.join(root, "foo", "inner.txt"), b"was a dir\n")
             _write(os.path.join(root, "bar"), b"was a file\n")
 
-        rsync_dst = os.path.join(TEST_DATA_DIR, "dtc_rsync_dst")
+        rsync_dst = os.path.join(TEST_DATA_DIR, f"dtc_{label}_rsync_dst")
         seed_dest(rsync_dst)
         rsync_result = _rsync(["-a", timing, source + "/", rsync_dst + "/"])
         assert rsync_result.returncode == 0, rsync_result.stderr
         rsync_tree = _tree(rsync_dst)
 
-        dest = os.path.join(TEST_DATA_DIR, "dtc_dst")
+        dest = os.path.join(TEST_DATA_DIR, f"dtc_{label}_dst")
         clean_dir(dest)
         received = get_dest_received_dir(dest, source)
         seed_dest(received)
@@ -292,7 +296,7 @@ class TestDeleteTimingFailure:
 
     @pytest.mark.parametrize("mt", [False, True])
     def test_during_removes_delay_preserves_on_failure(self, mt):
-        source, dest, received = _seed_pair("failure", big=True)
+        source, dest, received = _seed_pair(f"failure_mt{int(mt)}", big=True)
         extra = os.path.join(received, "d", "old_extra")
         assert os.path.exists(extra)
         with ServerManager() as server:
@@ -404,7 +408,7 @@ class TestDeleteDelayVsAfterSnapshot:
 
     @pytest.mark.parametrize("mt", [False, True])
     def test_late_created_extra_survives_delay_not_after(self, mt):
-        source, dest, received = _seed_pair("latecreate", big=True)
+        source, dest, received = _seed_pair(f"latecreate_mt{int(mt)}", big=True)
         old_extra = os.path.join(received, "d", "old_extra")
         new_extra = os.path.join(received, "d", "new_extra")
         with ServerManager() as server:
@@ -462,3 +466,47 @@ class TestDeleteAfterThreadsKeepSet:
         assert not os.path.exists(extra), (
             f"{delete_flag} --threads did not remove an extra: keep-set manifest was suppressed"
         )
+
+class TestDeleteDelayMaxDeleteRefilledDir:
+    """--delete-delay charges the --max-delete budget at plan/snapshot time, so a
+    refilled snapshotted directory that survives ENOTEMPTY still spends its slot
+    and a later extra is skipped, while the reported count stays at actual
+    removals.
+
+    The refilled directory is at the destination ROOT (its plan is always sent
+    first) and the skipped extra is under a separate source directory, so the
+    ordering that decides the budget charge is deterministic -- not readdir
+    order.  The refill is injected through the byte-barrier proxy so it is
+    causally after the plan frame."""
+
+    def test_budget_charged_at_plan_time(self):
+        source = os.path.join(TEST_DATA_DIR, "ddmb_src")
+        dest = os.path.join(TEST_DATA_DIR, "ddmb_dst")
+        clean_dir(source)
+        clean_dir(dest)
+        _write(os.path.join(source, "a", "keep.bin"), b"B" * BIG_BYTES)
+        _write(os.path.join(source, "b", "keep.txt"), b"keep\n")
+        received = get_dest_received_dir(dest, source)
+        refilled_dir = os.path.join(received, "xdir")
+        os.makedirs(refilled_dir, exist_ok=True)
+        later_dir = os.path.join(received, "b", "ydir")
+        os.makedirs(later_dir, exist_ok=True)
+
+        def hook():
+            _write(os.path.join(refilled_dir, "new.txt"), b"created mid-transfer\n")
+
+        with ServerManager() as server:
+            server.start(extra_args=["--allow-delete"])
+            proxy = _SlicingProxy(server.port, hook=hook, hook_after=MID_TRANSFER_BYTES,
+                                  throttle=PROXY_THROTTLE, wait_for_reply=True)
+            flags = ["--delete-delay", "--max-delete=1", "--incremental", "--ignore-times", "--stats"]
+            result, _ = run_client(source, dest, flags=flags, port=proxy.port)
+            proxy.finish()
+        assert result.returncode == 25, (result.stderr or result.stdout)[:400]
+        assert proxy.hook_called.is_set(), "hook never fired"
+        # The refilled directory still consumes the plan-time budget, so the
+        # later extra is skipped...
+        assert os.path.exists(os.path.join(refilled_dir, "new.txt")), "late file vanished"
+        assert os.path.isdir(later_dir), "later extra was not skipped by the plan-time budget"
+        # ...while the reported count reflects only actual removals (none here).
+        assert _deleted_count(result.stdout) == 0, result.stdout

tests/test_client_cli.c

@@ -4456,6 +4456,26 @@ static void test_parse_args_rejects_unsupported_short() {
   }
 }
 
+/* The --ignore-errors deletion gate, unit-tested without a privileged source
+ * directory: a clean scan always deletes; an I/O error suppresses deletion
+ * unless --ignore-errors is set.  (The end-to-end mode-000 differential lives in
+ * the setpriv integration test; this pins the decision itself in the PR gate.) */
+static void test_ignore_errors_allows_delete(void) {
+  Config* cfg = config_create();
+  EXPECT_NOT_NULL(cfg);
+  EXPECT_TRUE(ignore_errors_allows_delete(cfg, false));
+  EXPECT_FALSE(ignore_errors_allows_delete(cfg, true));
+
+  cfg->ignore_errors = true;
+  EXPECT_TRUE(ignore_errors_allows_delete(cfg, false));
+  EXPECT_TRUE(ignore_errors_allows_delete(cfg, true));
+
+  /* A NULL config cannot opt into --ignore-errors. */
+  EXPECT_TRUE(ignore_errors_allows_delete(NULL, false));
+  EXPECT_FALSE(ignore_errors_allows_delete(NULL, true));
+  config_delete(cfg);
+}
+
 void test_client_cli() {
   test_validate_config_required_paths();
   test_parse_args_numeric_ids();
@@ -4650,4 +4670,5 @@ void test_client_cli() {
   test_parse_args_noop_does_not_consume_argv();
   test_parse_args_backup_copy_links_shorts();
   test_parse_args_rejects_unsupported_short();
+  test_ignore_errors_allows_delete();
 }

tests/test_delete_plan.c

@@ -149,8 +149,77 @@ static void test_delete_delay_max_delete_bounds_actual(void) {
   config_delete(config);
 }
 
+/* --max-delete is charged at plan/snapshot time, not at actual removal: a
+ * deferred entry that survives ENOTEMPTY still consumes its budget slot, so a
+ * later directory's extra is skipped even though nothing was actually removed.
+ * The reported count stays 0 (actual removals) while the run is partial.  The
+ * two plans are sent as separate frames for "a" then "b", so the ordering that
+ * decides which entry gets the budget is deterministic (unlike readdir order). */
+static void test_delete_delay_refilled_dir_charges_budget_at_plan(void) {
+  char root[] = "/tmp/fastsync_dp_planbudget_XXXXXX";
+  EXPECT_TRUE(mkdtemp(root) != NULL);
+  char adir[1024], bdir[1024], xdir[1024], ydir[1024];
+  snprintf(adir, sizeof(adir), "%s/a", root);
+  snprintf(bdir, sizeof(bdir), "%s/b", root);
+  snprintf(xdir, sizeof(xdir), "%s/a/x", root);
+  snprintf(ydir, sizeof(ydir), "%s/b/y", root);
+  EXPECT_EQ_INT(mkdir(adir, 0700), 0);
+  EXPECT_EQ_INT(mkdir(bdir, 0700), 0);
+  EXPECT_EQ_INT(mkdir(xdir, 0700), 0);
+  EXPECT_EQ_INT(mkdir(ydir, 0700), 0);
+
+  Config* config = config_create();
+  EXPECT_NOT_NULL(config);
+  config->receive_root_directory = str_dup(root);
+  config->use_delete = true;
+  config->delete_delay = true;
+  config->max_delete = 1;
+
+  int p[2];
+  EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
+
+  DeletePlanSession* session = delete_plan_session_create(config);
+  EXPECT_NOT_NULL(session);
+  /* Plan "a" first: its empty extra dir snapshots and charges the budget. */
+  send_plan_frame(p[1], "a");
+  EXPECT_EQ_INT(delete_plan_session_receive(session, config, p[0]), 0);
+  EXPECT_FALSE(delete_plan_session_limit_reached(session));
+  /* Plan "b": the budget is already spent at snapshot time, so b/y is skipped
+     even though a/x has not (and will not) be removed. */
+  send_plan_frame(p[1], "b");
+  EXPECT_EQ_INT(delete_plan_session_receive(session, config, p[0]), 0);
+  EXPECT_TRUE(delete_plan_session_limit_reached(session));
+  EXPECT_EQ_INT((int)delete_plan_session_deleted(session), 0);
+
+  /* Refill a/x so its deferred rmdir fails ENOTEMPTY. */
+  char refill[1200];
+  snprintf(refill, sizeof(refill), "%s/new.txt", xdir);
+  int fd = open(refill, O_WRONLY | O_CREAT | O_TRUNC, 0600);
+  EXPECT_TRUE(fd >= 0);
+  close(fd);
+
+  EXPECT_EQ_INT(delete_plan_session_commit(session, config), DELETE_COMMIT_LIMIT_REACHED);
+  /* Nothing was actually removed, and the plan-time budget still stopped b/y. */
+  EXPECT_EQ_INT((int)delete_plan_session_deleted(session), 0);
+  struct stat st;
+  EXPECT_EQ_INT(lstat(xdir, &st), 0);
+  EXPECT_EQ_INT(lstat(ydir, &st), 0);
+
+  delete_plan_session_destroy(session);
+  close(p[0]);
+  close(p[1]);
+  unlink(refill);
+  rmdir(xdir);
+  rmdir(ydir);
+  rmdir(adir);
+  rmdir(bdir);
+  rmdir(root);
+  config_delete(config);
+}
+
 void test_delete_plan(void) {
   test_delete_delay_refilled_dir_not_counted();
   test_delete_delay_removed_file_counted();
   test_delete_delay_max_delete_bounds_actual();
+  test_delete_delay_refilled_dir_charges_budget_at_plan();
 }