diff --git a/HANDOFF.md b/HANDOFF.md index 5eba8d9..34f2057 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -75,7 +75,8 @@ heuristic with a 10× size window, not rsync's matcher), but its residual is the candidate-selection heuristic itself: the final tree is byte-exact by design, so it is pinned by the `TestFuzzy` threshold suite rather than a byte-level rsync - differential. The parity-review pass then moved `--delete-delay` to ⚠️ (the + differential. (Track 5b later found the name heuristic is rsync's own and moved + the row ❌ → ⚠️, leaving only the narrower delta size window; see entry 15.) The parity-review pass then moved `--delete-delay` to ⚠️ (the plan-time `--max-delete` charge and non-recursive deferred removal differ from rsync when a snapshotted entry fails removal). Differential-gate allowlist entries `min_size`/`empty_dirs_recursive`/`dirs_plain` were removed. The @@ -157,6 +158,22 @@ resolution base and over-limit MISS refusal): matrix now **116 ✅ / 13 ⚠️ / 28 ❌ = 157**. +15. **No-wire parity track 5b** on `feat/parity-2.28` (`PROTOCOL_VERSION` stays + `2.28.0` by project decision): `-y`/`--fuzzy` reclassified ❌ → ⚠️. A probe + against real rsync 3.4.1 (pinned `-B8192`, repeated-content 64 KiB corpus) + showed the name heuristic is already rsync's (`util1.c fuzzy_distance` / + `find_filename_suffix` + the exact size+mtime pass) and the output is always + byte-exact; the only residual is candidate ELIGIBILITY, because FastSync's + `delta_should_attempt` gate caps the size ratio at 10× and requires both + files ≥ 16 KiB while rsync will reuse a basis from 0.25× to 10000× and below + 16 KiB. The choice is observable only as `--stats` bandwidth counters. Added + differential case `fuzzy_basis` (same-suffix sibling, one name edit, + identical content, block size pinned) asserting tree **and** normalized + `--stats` parity where the choices coincide, plus `TestFuzzy` pinning the + window boundary on both sides (>10× and <16 KiB siblings declined by + FastSync while rsync uses them, both trees byte-identical). Matrix now + **116 ✅ / 14 ⚠️ / 27 ❌ = 157**. + ## Next steps 1. **Merge PR #284** (`dev` -> `main`) once reviewed (protected branch). 2. **Deferred security items** (documented, not implemented): diff --git a/RSYNC_COMPAT.md b/RSYNC_COMPAT.md index cf08bac..20fd46e 100644 --- a/RSYNC_COMPAT.md +++ b/RSYNC_COMPAT.md @@ -7,8 +7,8 @@ This document maps rsync's full feature set to FastSync's current implementation | Status | Count | Description | |--------|-------|-------------| | ✅ Parity | 116 | Reproduces rsync's semantics for this option's scope | -| ⚠️ Caveat | 13 | Wired and tested, but carries a documented behavioral difference from rsync (named in the row and/or the wave notes) | -| ❌ Divergent | 28 | Rejected, an accepted no-op, deliberately non-rsync (native config/auth/batch, privileged namespaces, safe-subset privilege), or impossible on any portable filesystem call | +| ⚠️ Caveat | 14 | Wired and tested, but carries a documented behavioral difference from rsync (named in the row and/or the wave notes) | +| ❌ Divergent | 27 | Rejected, an accepted no-op, deliberately non-rsync (native config/auth/batch, privileged namespaces, safe-subset privilege), or impossible on any portable filesystem call | | **Total** | **157** | One row per rsync option/feature group; a row may name several spellings | This matrix reports honest rsync parity, not "implemented" as a synonym for @@ -51,6 +51,8 @@ matrix is **111 ✅ / 13 ⚠️ / 33 ❌ = 157**. **Parity track 5a (wire, on `feat/parity-2.28`; `PROTOCOL_VERSION` stays 2.28.0 by project decision).** The three basis-dir options (`--compare-dest`/`--copy-dest`/`--link-dest`) now default to rsync's metadata quick-check instead of FastSync's historical xxHash64 content equality: a basis hit is accepted on equal size plus equal mtime (or size alone under `--size-only`; `-I` disables matching), so a same-size/different-content basis is trusted exactly as rsync trusts it. A new FastSync-only, long-only `--verify-basis` flag restores the stricter whole-file content equality, and its bool is appended to the basis block of the config frame (the golden wire frame grew by one int to 886 bytes; still 2.28.0). `--verify-basis` hashes the basis by streaming its confined descriptor, so an arbitrarily large basis is verified without buffering. Basis materialization is also no longer capped at the 256 MiB whole-file payload bound: a `--copy-dest` hit streams the basis through a bounded buffer, a `--link-dest` copy fallback streams from the basis, and a hit of any size is materialized (a basis MISS still falls back to the normal transfer, which keeps its own bound). A `--copy-dest` hit re-applies the SOURCE attributes (the sender now transmits the source metadata with the basis check frame), matching rsync's "copy then fix attributes"; a `--link-dest` success keeps the shared inode's own attributes exactly as before (writing through the shared inode would mutate the basis). The `--compare-dest`/`--copy-dest`/`--link-dest` rows move ❌ → ⚠️ (residuals: the relative-DIR resolution base and the over-limit MISS refusal). The matrix is now **116 ✅ / 13 ⚠️ / 28 ❌ = 157**. +**Parity track 5b (no-wire, on `feat/parity-2.28`; `PROTOCOL_VERSION` stays 2.28.0 by project decision).** `-y`/`--fuzzy` is reclassified ❌ → ⚠️: the receiver-side similar-file basis is an internal bandwidth optimization (the config `fuzzy` bool over the existing receiver-driven delta handshake, unchanged since 2.9.0), and the transferred tree is byte-exact by design regardless of which basis — or no basis — is chosen. A probe against real rsync 3.4.1 showed the remaining difference is not the name rule (FastSync already ports `util1.c fuzzy_distance`/`find_filename_suffix` plus the exact size+mtime pass) but candidate ELIGIBILITY: rsync will pick a fuzzy basis whose size ratio to the source is unrestricted (empirically from 0.25× to 10000×, and for files as small as 300 B), while FastSync's `delta_should_attempt` gate caps the ratio at 10× and requires both files ≥ 16 KiB, so an out-of-window sibling is declined and the file is sent whole. The choice is observable only as bandwidth (`Matched data`/`Literal data`/`Total transferred file size` in `--stats`); the destination tree and exit code are identical either way. A new differential case (`fuzzy_basis`: same-suffix sibling one name-edit away, content identical, block size pinned to 8192) asserts tree **and** normalized `--stats` parity where the two tools' choices coincide; `TestFuzzy` pins the window boundary on both sides (a >10× and a <16 KiB sibling are declined by FastSync while rsync uses them, both trees byte-identical). No wire field changed. The matrix is now **116 ✅ / 14 ⚠️ / 27 ❌ = 157**. + **Parity completion wave (protocol 2.23.0 → 2.26.0).** This wave closed the remaining gaps the rsync-parity wave left open (delete timing, wire counters and output, codec breadth, general `-R`/`-d`, the full filter grammar, receiver-side @@ -663,7 +665,7 @@ targets verbatim, matching rsync. | `--compare-dest=DIR` | Compare dest files relative to DIR | ⚠️ Caveat | DIR is a receiver-side basis; protocol 2.26.0 uses an absolute path verbatim (rsync semantics) and resolves a relative path below the destination root (`..` components are rejected, `//` collapsed and trailing `/` dropped) — note rsync resolves a relative DIR against the destination directory while FastSync resolves it below the receive root and appends the mirrored source path, so the same relative spelling addresses a different tree (use an absolute DIR for exact parity). On the receiver's per-file check (implies `--incremental`) an exact match is rsync's metadata quick-check: same size and mtime (unless `--size-only`; `-I` disables matching), with NO content digest required by default (track 5a). A match suppresses the data transfer. The FastSync-only `--verify-basis` restores the stricter whole-file content equality. compare-dest never copies: it only skips a file the destination does **not** already hold (sparse destination, rsync parity), and is consulted before the normal delta/full paths. Repeatable; searched in command-line order, first match wins. Differential-tested against rsync 3.4.1 (`compare_dest`, and `test_verify_basis_restores_strict_content`). Residual: a basis MISS above the 256 MiB whole-file payload bound is refused up front (FastSync's general whole-file limit, not basis-specific); rsync applies basis dirs to arbitrary sizes. Wire: a basis-count field plus the `verify_basis` bool are present on the config frame (protocol 2.9.0/2.28.0) | | `--copy-dest=DIR` | Include copies of unchanged files | ⚠️ Caveat | Same basis rules as `--compare-dest`, but an exact match materializes a **local copy** of the DIR file into the destination (via the atomic temp+rename store path, so `--existing`/`--ignore-existing`/`--update`/`--backup`/`--delay-updates` all still apply) instead of transferring data. Track 5a re-applies the SOURCE attributes on the copy (rsync's "copy then fix attributes"): the sender transmits the source metadata with the basis check frame, so the copy's mode/uid/gid/mtime match the source rather than the basis inode (differential `copy_dest` compares modes). The copy streams the basis file through a bounded buffer, so a basis larger than the whole-file payload bound still materializes. Repeatable; command-line order = priority. Requires `--incremental` (implied); incompatible with `-s`. Wire: protocol 2.9.0 | | `--link-dest=DIR` | Hardlink to files when unchanged | ⚠️ Caveat | Same basis rules as `--copy-dest`, but an exact match installs an atomic **hard link** to the DIR file (temp hard link + rename) so no data or disk space is used; where the link is impossible (basis on another filesystem, filesystem refuses links) it falls back cleanly to a byte-identical local copy (streamed from the basis, so an over-limit basis still works), never a corrupt/partial file. `--delay-updates` stages the link and publishes by rename, so the final entry stays a real hard link. Repeatable (searched in command-line order, first match wins). Differential-tested against rsync 3.4.1 (`link_dest`). Inherent shared-inode semantics (identical to rsync): a link keeps the basis inode's own mode/uid/gid and mtime — metadata is never written through the shared inode (that would mutate the basis file), so a later `--inplace` run that rewrites such a destination path **will mutate the basis snapshot** through the shared inode (use `--copy-dest` when the destination must stay independently writable); protocol 2.26.0 re-links an already up-to-date destination file to the basis; a `--remove-source-files` source satisfied by a basis dir is treated as skipped and therefore **retained** (never removed); basis dirs are excluded from `--delete`. Residual: a basis MISS above the 256 MiB whole-file payload bound is refused (FastSync's general whole-file limit). Requires `--incremental` (implied); incompatible with `-s`. Wire: protocol 2.9.0 | -| `-y`, `--fuzzy`, `--no-fuzzy` | Find similar file for basis | ❌ Divergent | `-y/--fuzzy` is a pure bandwidth optimization on the existing receiver-driven delta path: when a file must be transferred and the destination holds no usable content at the exact path (file absent, or the destination file is outside the delta engine's size bounds), the receiver searches the SAME destination directory for an existing regular file whose basename is similar to the incoming name and uses it as the delta basis, so the sender transmits only the differences instead of the whole file. The output is always byte-exact regardless of which (or whether any) basis is chosen. Decision location: the receiver performs the candidate search inside `receive_incremental_check` and sends the normal `STATUS_DELTA_SIGNATURE`; the sender never learns the basis was a different file, so no new frame type or sender logic was needed — only the config frame grew a `fuzzy` boolean, so `PROTOCOL_VERSION` was bumped **2.8.0 → 2.9.0** (peers must match). Similarity heuristic (deterministic, simpler than rsync's deliberately-fuzzy matching, and documented precisely): candidates are the target's sibling entries in its destination directory, opened `O_NOFOLLOW`/`AT_SYMLINK_NOFOLLOW` under the confined root (symlinks never followed; nothing outside the destination root is ever read or hashed); dotfiles, directories, the target's own name, and the `.fastsync-stage`/temp scratch names are excluded; like the ordinary delta path, the block signature the receiver transmits is derived from on-disk content it may not otherwise send, so a negotiated `--fuzzy` run exposes the destination's sibling files (at block granularity) to the sender as a known-plaintext oracle — the same information class as the normal delta handshake over the file being replaced; the size gate is the delta engine's own bounds (both files ≥ 16 KiB, ≤ `--delta-max`, ratio ≤ 10×) rather than rsync's ~1.5× size window; protocol 2.26.0 uses a name-distance/suffix heuristic modelled on rsync's plus an exact size+mtime pass, and reads a single best candidate; the exact tie-break order can still differ from rsync's; the directory scan is capped at 4096 entries so a pathological directory cannot stall a transfer. When fuzzy applies: only to files the receiver would otherwise send whole — the destination's own file is always preferred as the delta basis when it exists and fits the delta size bounds, so fuzzy does NOT replace an existing-but-different destination basis; FastSync's 10× delta size-ratio bound means an existing destination file that is too far away in size still lets the fuzzy search run. When no similar candidate exists the transfer falls back to the normal whole-file transfer. rsync-divergence note: rsync's own matching uses a fuzzy name/size rule set; FastSync implements the closest safe deterministic approximation above. Because FastSync's delta machinery is off by default (rsync's is on), `--fuzzy` implies `--incremental` + `--delta` (unless `--whole-file`/`-W` or an explicit `--no-delta` switched delta off, in which case fuzzy is inert — matching rsync where `--whole-file` makes fuzzy irrelevant). Unlike the basis-dir options, `--fuzzy` honors an explicit `--no-incremental` (it does not force the handshake back on); an explicit `--no-incremental` also suppresses the delta implication so no invalid `--delta requires --incremental` config results. `--no-fuzzy` negates it. All surrounding semantics are untouched: a fuzzy-reconstructed file is stored as a normal file, so `--remove-source-files`, itemize/`-i`, `--stats`, `--backup`, `--delay-updates`, `--existing`/`--ignore-existing`/`--update` behave exactly as for a whole-file transfer (the fuzzy delta does not skip the file). **Reclassified Divergent:** because the output is always byte-exact, the residual is the candidate-selection heuristic itself — a deterministic name-distance/suffix rule with a 10× size-ratio window (vs rsync's ~1.5× window), not rsync's deliberately fuzzy matcher, so the chosen basis (and thus the wire bytes) can differ from rsync even though the final tree cannot. The Integration fuzzy suite (`TestFuzzy`) pins FastSync's thresholds (exact-size+mtime pass, name-distance rejection, 10× unsuitable-destination fallback); a bit-identical basis choice is not achievable without porting rsync's matcher | +| `-y`, `--fuzzy`, `--no-fuzzy` | Find similar file for basis | ⚠️ Caveat | `-y/--fuzzy` is a pure bandwidth optimization on the existing receiver-driven delta path: when a file must be transferred and the destination holds no usable content at the exact path (file absent, or the destination file is outside the delta engine's size bounds), the receiver searches the SAME destination directory for an existing regular file whose basename is similar to the incoming name and uses it as the delta basis, so the sender transmits only the differences instead of the whole file. The output is always byte-exact regardless of which (or whether any) basis is chosen. Decision location: the receiver performs the candidate search inside `receive_incremental_check` and sends the normal `STATUS_DELTA_SIGNATURE`; the sender never learns the basis was a different file, so no new frame type or sender logic was needed — only the config frame grew a `fuzzy` boolean, so `PROTOCOL_VERSION` was bumped **2.8.0 → 2.9.0** (peers must match). Similarity heuristic (a deterministic port of rsync 3.4.1's matcher — `util1.c` `fuzzy_distance`/`find_filename_suffix` plus `generator.c find_fuzzy`'s exact size+mtime pass — documented precisely): candidates are the target's sibling entries in its destination directory, opened `O_NOFOLLOW`/`AT_SYMLINK_NOFOLLOW` under the confined root (symlinks never followed; nothing outside the destination root is ever read or hashed); dotfiles, directories, the target's own name, and the `.fastsync-stage`/temp scratch names are excluded; like the ordinary delta path, the block signature the receiver transmits is derived from on-disk content it may not otherwise send, so a negotiated `--fuzzy` run exposes the destination's sibling files (at block granularity) to the sender as a known-plaintext oracle — the same information class as the normal delta handshake over the file being replaced; the size gate is the delta engine's own bounds (both files ≥ 16 KiB, ≤ `--delta-max`, ratio ≤ 10×); rsync's fuzzy matcher is not tied to a delta size bound and empirically reuses a basis well outside FastSync's window (a 64 KiB source against a repeated-content sibling from 0.25× to 10000×, and files as small as 300 B), so candidate ELIGIBILITY — and hence the chosen basis — can differ even though the name heuristic is the same; protocol 2.26.0 uses rsync's weighted-Levenshtein name/suffix distance plus an exact size+mtime pass and reads a single best candidate; the tie-break (smallest size gap, then lexical name) is deterministic where rsync leaves equal distances to its file-list order; the directory scan is capped at 4096 entries so a pathological directory cannot stall a transfer. When fuzzy applies: only to files the receiver would otherwise send whole — the destination's own file is always preferred as the delta basis when it exists and fits the delta size bounds, so fuzzy does NOT replace an existing-but-different destination basis; FastSync's 10× delta size-ratio bound means an existing destination file that is too far away in size still lets the fuzzy search run. When no similar candidate exists the transfer falls back to the normal whole-file transfer. rsync-divergence note: the name matching is rsync's own rule; the residual is eligibility bounded by FastSync's delta engine, so no name-matcher port can widen it. Because FastSync's delta machinery is off by default (rsync's is on), `--fuzzy` implies `--incremental` + `--delta` (unless `--whole-file`/`-W` or an explicit `--no-delta` switched delta off, in which case fuzzy is inert — matching rsync where `--whole-file` makes fuzzy irrelevant). Unlike the basis-dir options, `--fuzzy` honors an explicit `--no-incremental` (it does not force the handshake back on); an explicit `--no-incremental` also suppresses the delta implication so no invalid `--delta requires --incremental` config results. `--no-fuzzy` negates it. All surrounding semantics are untouched: a fuzzy-reconstructed file is stored as a normal file, so `--remove-source-files`, itemize/`-i`, `--stats`, `--backup`, `--delay-updates`, `--existing`/`--ignore-existing`/`--update` behave exactly as for a whole-file transfer (the fuzzy delta does not skip the file). **Reclassified Caveat (track 5b):** the output is always byte-exact regardless of the basis, and the name rule is rsync's, so the residual is candidate ELIGIBILITY: FastSync's 10× ratio / 16 KiB delta gates make its size window strictly narrower than rsync's, and when eligibility differs the chosen basis — and therefore the `--stats` `Matched data`/`Literal data`/`Total transferred file size` counters — can differ even though the tree cannot. Where the two tools' choices coincide and the block size is pinned, both the tree and the counters match rsync (differential `fuzzy_basis`); `TestFuzzy` pins the window boundary on both sides (a >10× sibling and a <16 KiB sibling are declined, with a byte-exact whole-file fallback). A wider window would require loosening the delta engine's bounds, not changing the name heuristic. | ## 12. Compression @@ -923,10 +925,10 @@ These are the last compatibility items and the closing phase toward rsync flag p **Wire:** two trailing config-frame blocks after the `--iconv` spec, in fixed order — `send_privilege_options`/`receive_privilege_options` (one `super_mode` int, validated `0..2`), then `send_copy_as_options`/`receive_copy_as_options` (presence int + two int32 ids, validated `>= 0`, with `copy_as_set ⇒ use_metadata`). `PROTOCOL_VERSION` bumped **2.17.0 → 2.18.0**. **Divergences from rsync:** rsync's `--super` elevates the receiver and `--copy-as` actually switches its credentials; FastSync never elevates and only permits/forwards confined attempts, and `--copy-as` forces ownership rather than switching identity. -**Honest status after the parity-completion wave (protocol 2.27.0), updated by the rsync-parity-stats, rsync-parity-options, rsync-parity-fs, parity-review, no-wire parity-track-1/2b and wire parity-track-4a/5a passes.** ✅ Parity 116 / ⚠️ Caveat 13 / ❌ Divergent 28 = 157 rows. Earlier revisions of this document reported "143 ✅ / 0 divergence / 0 partial"; that conflated "parsed and tested" with "rsync parity", because many rows carried documented behavioral differences and some short options were not parsed at all. This reclassification makes every difference explicit. The completion wave closed 23 previously-caveated rows (9 that triage showed were already parity, plus 14 genuine fixes) and turned the 17 inherently non-rsync rows — native daemon config/auth, the FastSync batch container, the safe-subset device/privilege flags, `-X`'s privileged namespaces, `--fake-super`'s native xattr format, and the `--old-args` no-op — into explicit ❌ divergences. The stats pass flipped `--delete-delay` to ✅ (actual-removal accounting), but the parity-review pass moved it back to ⚠️ because FastSync charged the `--max-delete` budget at plan/snapshot time and left a refilled snapshotted directory in place, whereas rsync charges on actual removals and recursively removes a queued directory (including content created after its plan). The no-wire parity-track-1 pass fixed both (actual-removal charging plus recursive deferred removal with an independent deferred-list cap), narrowing the caveat to the partial-delete ordering. The stats pass also reclassified `--out-format` to ❌ (protocol-specific `%b`/delta-`%c`), and sharpened the `--stats`/`--progress`/`--checksum-choice` residuals. The options pass flipped `--bwlimit` and `--ignore-errors` to ✅ (rsync-exact size parsing and ~100 ms leaky-bucket throttling, and rsync's skip-unreadable-subdir plus IO-error-suppressed deletion with exit 23) and emits rsync-format `--info=name/flist/del/remove/nonreg/progress` lines (real-run `deleting`/`*deleting` carried over a new trailing `report_deletes` wire bool, `PROTOCOL_VERSION` 2.26.0 → 2.27.0), while reclassifying `-M` over daemon/TCP +**Honest status after the parity-completion wave (protocol 2.27.0), updated by the rsync-parity-stats, rsync-parity-options, rsync-parity-fs, parity-review, no-wire parity-track-1/2b and wire parity-track-4a/5a passes.** ✅ Parity 116 / ⚠️ Caveat 14 / ❌ Divergent 27 = 157 rows. Earlier revisions of this document reported "143 ✅ / 0 divergence / 0 partial"; that conflated "parsed and tested" with "rsync parity", because many rows carried documented behavioral differences and some short options were not parsed at all. This reclassification makes every difference explicit. The completion wave closed 23 previously-caveated rows (9 that triage showed were already parity, plus 14 genuine fixes) and turned the 17 inherently non-rsync rows — native daemon config/auth, the FastSync batch container, the safe-subset device/privilege flags, `-X`'s privileged namespaces, `--fake-super`'s native xattr format, and the `--old-args` no-op — into explicit ❌ divergences. The stats pass flipped `--delete-delay` to ✅ (actual-removal accounting), but the parity-review pass moved it back to ⚠️ because FastSync charged the `--max-delete` budget at plan/snapshot time and left a refilled snapshotted directory in place, whereas rsync charges on actual removals and recursively removes a queued directory (including content created after its plan). The no-wire parity-track-1 pass fixed both (actual-removal charging plus recursive deferred removal with an independent deferred-list cap), narrowing the caveat to the partial-delete ordering. The stats pass also reclassified `--out-format` to ❌ (protocol-specific `%b`/delta-`%c`), and sharpened the `--stats`/`--progress`/`--checksum-choice` residuals. The options pass flipped `--bwlimit` and `--ignore-errors` to ✅ (rsync-exact size parsing and ~100 ms leaky-bucket throttling, and rsync's skip-unreadable-subdir plus IO-error-suppressed deletion with exit 23) and emits rsync-format `--info=name/flist/del/remove/nonreg/progress` lines (real-run `deleting`/`*deleting` carried over a new trailing `report_deletes` wire bool, `PROTOCOL_VERSION` 2.26.0 → 2.27.0), while reclassifying `-M` over daemon/TCP and receiver-side `protect`/`risk` re-derivation to ❌ (no argv channel / receiver filter engine); the wire parity-track-4a pass later added that -receiver filter engine, flipping `--filter=RULE` back to ✅ (see above). The fs pass flips `-d/--dirs` and `--iconv` to ✅ — recursive transfers now recreate empty source directories (and replace a blocking destination non-directory with an incoming directory); `-R --no-implied-dirs --files-from` places a listed file under a missing implied parent with default attributes instead of refusing; and `--iconv` now reproduces rsync's push direction (destination charset = the spec's REMOTE half) — and reclassified six rows to ❌ after reproducing their exact residual with differential tests: `--temp-dir` (the receiver confines the scratch dir to the receive root, so an absolute temp dir is deliberately rejected although standalone rsync follows it), the three basis-dir options (FastSync xxHash-verifies a basis hit while rsync's `--size-only` quick check installs the wrong basis content), `--delay-updates` (fixed staging name wipes an unrelated destination entry of that name), and `--dry-run` (would-delete report over-reports). `--fuzzy` was also reclassified to ❌ (deterministic heuristic with a 10× size window, not rsync's matcher), but its residual is the candidate-selection heuristic itself: the final tree is byte-exact by design, so no destination differential can expose it and the row is pinned by the `TestFuzzy` threshold suite rather than a byte-level rsync differential. The remaining ⚠️ rows are the ones with a documented residual (see the row notes and the **Parity Completion Wave (protocol 2.26.0)** section below). +receiver filter engine, flipping `--filter=RULE` back to ✅ (see above). The fs pass flips `-d/--dirs` and `--iconv` to ✅ — recursive transfers now recreate empty source directories (and replace a blocking destination non-directory with an incoming directory); `-R --no-implied-dirs --files-from` places a listed file under a missing implied parent with default attributes instead of refusing; and `--iconv` now reproduces rsync's push direction (destination charset = the spec's REMOTE half) — and reclassified six rows to ❌ after reproducing their exact residual with differential tests: `--temp-dir` (the receiver confines the scratch dir to the receive root, so an absolute temp dir is deliberately rejected although standalone rsync follows it), the three basis-dir options (FastSync xxHash-verifies a basis hit while rsync's `--size-only` quick check installs the wrong basis content), `--delay-updates` (fixed staging name wipes an unrelated destination entry of that name), and `--dry-run` (would-delete report over-reports). `--fuzzy` was also reclassified to ❌ (deterministic heuristic with a 10× size window, not rsync's matcher), but its residual is the candidate-selection heuristic itself: the final tree is byte-exact by design, so no destination differential can expose it and the row is pinned by the `TestFuzzy` threshold suite rather than a byte-level rsync differential. (Track 5b later showed the name heuristic is in fact rsync's own and moved the row ❌ → ⚠️, leaving only the narrower delta size window as the residual; see the track 5b paragraph above.) The remaining ⚠️ rows are the ones with a documented residual (see the row notes and the **Parity Completion Wave (protocol 2.26.0)** section below). **Preserve-attribute split (protocol 2.21.0 → 2.22.0) — ✅ implemented.** FastSync splits the former single metadata bundle into four independent, rsync-compatible per-attribute flags — `-p/--perms`, `-t/--times`, `-o/--owner`, `-g/--group` — each with a negation (`--no-perms`/`--no-times`/`--no-owner`/`--no-group`, short `--no-p`/`--no-t`/`--no-o`/`--no-g`), plus `--no-preserve` clearing all four. `-a/--archive` is now full rsync `-rlptgoD` (owner and group included, though their application stays privilege-gated), `-A/--acls` implies `-p`, `-X/--xattrs` does not, `-E/--executability` sets only executability, and `-U`/`-N` do not imply `-t`. `--incremental`/`--delta` still auto-preserve perms+times unless the user explicitly negated them. Wire: the binary config frame gains four appended booleans (`preserve_perms`/`preserve_times`/`preserve_owner`/`preserve_group`) after `omit_link_times`, so `PROTOCOL_VERSION` is bumped **2.21.0 → 2.22.0**; the fixed-width `FileMetadata` layout is unchanged and the receiver gates the metadata frame on a derived `use_metadata`. Receiver behavior: each attribute is applied independently, directory modes are applied under `-p` (at the end of the transfer, alongside dir times), symlink mode under `-p`, and `-O/--omit-dir-times` suppresses directory times only. Documented divergences as of 2.22.0, **all but (d)/(e) removed by the rsync-parity wave (protocol 2.23.0)**: (a) the mode-masking divergence is **gone** — under `-p` the source mode is now copied exactly, including `S_IWGRP`/`S_IWOTH` and setuid/setgid/sticky; (b) a brand-new file without `-p` still gets `source_mode & ~umask` when metadata is present (else the historical fixed `0644`), and a new *directory* without `-p` still uses FastSync's `0755` default; (c) the `--chmod`-implies-`-p` divergence is **gone** — `--chmod` no longer implies `-p` (rsync parity); (d) `-o`/`-g` map by name on the receiver with a raw-numeric fallback (only numeric ids cross the wire); (e) a daemon module without `client owner = yes` does not refuse a plain `-a`/`-o`/`-g` — it forces super off, applies no ownership, and logs a warning, while explicit `--chown`/`--usermap`/`--groupmap`/`--numeric-ids`/`--copy-as`/`--super` are still refused. @@ -1183,8 +1185,11 @@ These remain after the wave; the individual rows carry the precise wording. - **Basis dirs** now use rsync's metadata quick-check by default (track 5a) and stream a hit of any size; the FastSync-only `--verify-basis` restores the stricter content equality. Remaining residuals: the relative-DIR resolution - base and the over-limit basis-MISS refusal. **`--fuzzy`** - has a different tie-break order; and **`--bwlimit`** rejects rsync's + base and the over-limit basis-MISS refusal. **`--fuzzy`** uses rsync's + name heuristic, but its candidate eligibility is bounded by the delta + engine (both files ≥ 16 KiB, size ratio ≤ 10×), a narrower window than + rsync's, so the selected basis — and the `--stats` bandwidth counters — + can differ while the tree stays byte-exact; and **`--bwlimit`** rejects rsync's `0`/decimal/suffixed rates. - **`--inc-recursive`/`--no-inc-recursive`** are not implemented (rejected). diff --git a/tests/integration/parity_harness.py b/tests/integration/parity_harness.py index 39d2ec6..23c2ea7 100644 --- a/tests/integration/parity_harness.py +++ b/tests/integration/parity_harness.py @@ -229,6 +229,18 @@ def corpus_iconv(root: str) -> None: os.utime(full, (_SRC_MTIME, _SRC_MTIME)) +# Payload for the --fuzzy basis corpus: large enough for the delta engine's +# 16 KiB minimum and with repeated content so a coinciding basis yields a +# non-zero (and identical) Matched data count in both tools. +FUZZY_PAYLOAD = (b"the quick brown fox jumps over the lazy dog\n" * 2000)[:65536] + + +def corpus_fuzzy(root: str) -> None: + """A named regular file; the `fuzzy` seed adds the similar-suffix sibling.""" + clean_dir(root) + _write(os.path.join(root, "report_v2.txt"), FUZZY_PAYLOAD) + + CORPORA: Dict[str, Callable[[str], None]] = { "basic": corpus_basic, "unicode": corpus_unicode, @@ -240,6 +252,7 @@ CORPORA: Dict[str, Callable[[str], None]] = { "multidir": corpus_multidir, "relative": corpus_relative, "iconv": corpus_iconv, + "fuzzy": corpus_fuzzy, } diff --git a/tests/integration/test_differential_parity.py b/tests/integration/test_differential_parity.py index 0164ea7..cf44b82 100644 --- a/tests/integration/test_differential_parity.py +++ b/tests/integration/test_differential_parity.py @@ -133,6 +133,16 @@ def seed_max_delete(_src, rroot, froot): _mk(os.path.join(root, "extra2.txt"), b"e2\n", _OLD_MTIME) +def fuzzy_basis_seed(_src, rroot, froot): + """Seed a same-suffix sibling whose name is one edit from the source and + whose content matches it, with a DIFFERENT mtime so rsync's exact + size+mtime pass cannot fire: both tools must select it via the + name-distance pass. Where the two tools' basis choices coincide the + block-level results are identical when the block size is pinned.""" + for root in (rroot, froot): + _mk(os.path.join(root, "report_v1.txt"), H.FUZZY_PAYLOAD, _OLD_MTIME) + + def max_delete_count_check(_src, rroot, froot, _rs, _fs): """The exact survivor set is order-dependent; the count must still match.""" r = H.snapshot(rroot) @@ -208,6 +218,18 @@ _CASES = [ H.Case("chmod", "basic", ["-a", "--chmod=Fu+rwx"], compare_modes=True, ci=True, ref="--chmod"), + # --- delta / similar-file basis (--fuzzy) ----------------------------- + # Basis choices coincide here (same-suffix sibling, name distance one edit, + # content identical); with the block size pinned both tools report the same + # Matched/Literal/transferred counters. The residual (FastSync's narrower + # delta size window) is covered by TestFuzzy in test_parity_quickwins.py. + H.Case("fuzzy_basis", "fuzzy", + ["-a", "--no-whole-file", "--fuzzy", "--stats", "-B8192"], + fastsync_flags=["-a", "--incremental", "--delta", "--fuzzy", + "--stats", "--delta-block=8192"], + seed=fuzzy_basis_seed, stdout=H.STDOUT_STATS, ci=True, + ref="-y/--fuzzy similar-file basis"), + # --- deletion --------------------------------------------------------- H.Case("delete", "basic", ["-a", "--delete"], seed=seed_extras, server_args=DELETE, ci=True, ref="--delete"), diff --git a/tests/integration/test_parity_quickwins.py b/tests/integration/test_parity_quickwins.py index d4a30d6..596b468 100644 --- a/tests/integration/test_parity_quickwins.py +++ b/tests/integration/test_parity_quickwins.py @@ -847,6 +847,92 @@ class TestVerifyAndFlip: "--verify-basis must reject the same-size/different-content basis" +def _stat_bytes(output, key): + """Parse a --stats byte counter (e.g. ``Matched data: 65,536 bytes``).""" + for line in output.splitlines(): + if line.startswith(key + ":"): + raw = line.split(":", 1)[1].strip().split()[0] + return int(raw.replace(",", "")) + return None + + +class TestFuzzy: + """Track 5b: `-y`/`--fuzzy` is an internal bandwidth optimization with a + byte-exact result. FastSync ports rsync 3.4.1's weighted-Levenshtein name + heuristic, so where both delta engines admit the candidate the tools pick + the same basis (the ``fuzzy_basis`` differential asserts the tree and the + Matched/Literal counters match with the block size pinned). The residual is + candidate ELIGIBILITY: FastSync's delta size gate (both files >= 16 KiB and + a <= 10x size ratio) is narrower than rsync's, which empirically uses a + fuzzy basis well beyond 10x and below 16 KiB. These tests pin the window + boundary and prove the byte-exact fallback on both sides of it.""" + + _BASE = b"the quick brown fox jumps over the lazy dog\n" * 4000 + + def _src(self, tag): + source = os.path.join(TEST_DATA_DIR, f"fz_{tag}_src") + clean_dir(source) + return source + + def _dst(self, tag): + d = os.path.join(TEST_DATA_DIR, f"fz_{tag}_dst") + clean_dir(d) + return d + + def _run_both(self, shared_server, source, dest, rdst, payload, sibling, + rs_extra=(), fs_extra=()): + with open(os.path.join(source, "report_v2.txt"), "wb") as fh: + fh.write(payload) + for root in (rdst, get_dest_received_dir(dest, source)): + os.makedirs(root, exist_ok=True) + with open(os.path.join(root, "report_v1.txt"), "wb") as fh: + fh.write(sibling) + rs = _rsync(["-a", "--no-whole-file", "--fuzzy", "--stats"] + + list(rs_extra) + [source + "/", rdst + "/"]) + assert rs.returncode == 0, rs.stderr[:300] + result, _ = run_client( + source, dest, + flags=["-a", "--incremental", "--delta", "--fuzzy", "--stats"] + + list(fs_extra), + port=shared_server.port) + assert result.returncode == 0, result.stderr[:300] + _assert_same_tree(rdst, get_dest_received_dir(dest, source), "(--fuzzy)") + return rs, result + + @requires_rsync + def test_fuzzy_above_size_window_declines_but_tree_exact(self, shared_server): + """A sibling >10x the source is used by rsync but declined by FastSync's + delta size-ratio gate; both destinations stay byte-identical.""" + n = 65536 + payload = (self._BASE * ((n // len(self._BASE)) + 1))[:n] + sibling = (self._BASE * 200)[: n * 20] + source, dest, rdst = (self._src("big"), self._dst("big"), + self._dst("big_r")) + rs, result = self._run_both(shared_server, source, dest, rdst, + payload, sibling) + assert _stat_bytes(rs.stdout, "Matched data") > 0, \ + "rsync should still use a >10x fuzzy basis" + assert _stat_bytes(result.stdout, "Matched data") == 0, \ + "FastSync's 10x delta size-ratio gate must decline the oversized basis" + assert _stat_bytes(result.stdout, "Literal data") == n + + @requires_rsync + def test_fuzzy_below_delta_minimum_declines_but_tree_exact(self, shared_server): + """A sibling below the 16 KiB delta minimum is used by rsync but never + enters FastSync's delta/fuzzy path; both trees stay byte-identical.""" + n = 8192 + payload = (self._BASE * ((n // len(self._BASE)) + 1))[:n] + source, dest, rdst = (self._src("small"), self._dst("small"), + self._dst("small_r")) + rs, result = self._run_both(shared_server, source, dest, rdst, + payload, payload) + assert _stat_bytes(rs.stdout, "Matched data") > 0, \ + "rsync applies --fuzzy below 16 KiB" + assert _stat_bytes(result.stdout, "Matched data") == 0, \ + "FastSync's 16 KiB delta minimum must bypass the fuzzy basis" + assert _stat_bytes(result.stdout, "Literal data") == n + + class TestIgnoreExistingShortCircuit: """#9: --ignore-existing is decided by the receiver during the per-file check, before the sender streams any payload. A large destination file that