feat(parity): rsync parity tracks 1-6 (protocol 2.28.0) #303

Merged
TapTap merged 14 commits from feat/parity-2.28 into dev 2026-09-19 17:14:37 +02:00
Owner

Summary

Closes the implementable rsync-parity gaps from the A/B plan. PROTOCOL_VERSION stays 2.28.0 (one batched bump for the whole cycle). Matrix: 116 ✅ / 14 ⚠️ / 27 ❌ = 157 (was 111/13/33).

Tracks

  • Track 1 (no-wire): -n --delete sends real protected/size-skipped/scope so would-delete no longer over-reports; --delete-delay charges --max-delete on actual removals and recursively removes a refilled deferred directory; --info=name root line + name2 is uptodate.
  • Track 2a (wire): STATUS_STATS gains receiver-observed literal_bytes + created reg/dir/link/special counters; Number of created files carries rsync's breakdown and Literal data is exact for delta.
  • Track 2b: opt-in paths-only pre-count when progress is requested → rsync to-chk denominator and per-directory/symlink/special name lines.
  • Track 3a: per-codec level defaults (zstd 3 / zlib 6 / lz4 ignored) + RSYNC_COMPRESS_LIST/RSYNC_CHECKSUM_LIST for auto; zlibx reclassified.
  • Track 3b: --checksum-choice proven observable-equivalent; reclassified.
  • Track 4a/4b (wire): bounded BLOCK_PROTECT_RULES + receiver-side protect/risk engine, so a destination-only entry matching P survives like rsync; --filter and --delete-excluded flip to parity.
  • Track 5a (wire): basis dirs default to rsync's metadata quick-check; FastSync-only --verify-basis restores content equality; copy-dest attributes + streaming removes the basis-hit size cap.
  • Track 5b: --fuzzy name heuristic matches rsync; residual is the narrower eligibility window.
  • Track 6: plain --delete defaults to rsync's delete-during; FastSync-only --delete-commit keeps the old atomic timing.

Review wave (fixed)

  • Blocker: basis-dir materializations were counted as created/literal in --stats; now excluded (differential test_link_dest_stats_matches_rsync).
  • Filter wire block: reject over-long patterns and cap the rule list at 1024 to avoid delete-walk glob amplification; negative tests added.
  • README stale --delete/--stats/--progress docs corrected; new flags documented.

Verification

  • Unit 44/44; ASan 44/44; full integration -m "not setpriv" 854 passed / 23 skipped / 1 xpassed; strict differential parity 62 passed; clang-format + cppcheck clean.

New FastSync-only flags

--verify-basis, --delete-commit (both long-only; documented in RSYNC_COMPAT.md/README.md).

## Summary Closes the implementable rsync-parity gaps from the A/B plan. `PROTOCOL_VERSION` stays **2.28.0** (one batched bump for the whole cycle). Matrix: **116 ✅ / 14 ⚠️ / 27 ❌ = 157** (was 111/13/33). ## Tracks - **Track 1 (no-wire):** `-n --delete` sends real protected/size-skipped/scope so would-delete no longer over-reports; `--delete-delay` charges `--max-delete` on actual removals and recursively removes a refilled deferred directory; `--info=name` root line + `name2 is uptodate`. - **Track 2a (wire):** `STATUS_STATS` gains receiver-observed `literal_bytes` + created reg/dir/link/special counters; `Number of created files` carries rsync's breakdown and `Literal data` is exact for delta. - **Track 2b:** opt-in paths-only pre-count when progress is requested → rsync `to-chk` denominator and per-directory/symlink/special name lines. - **Track 3a:** per-codec level defaults (zstd 3 / zlib 6 / lz4 ignored) + `RSYNC_COMPRESS_LIST`/`RSYNC_CHECKSUM_LIST` for `auto`; zlibx reclassified. - **Track 3b:** `--checksum-choice` proven observable-equivalent; reclassified. - **Track 4a/4b (wire):** bounded `BLOCK_PROTECT_RULES` + receiver-side `protect`/`risk` engine, so a destination-only entry matching `P` survives like rsync; `--filter` and `--delete-excluded` flip to parity. - **Track 5a (wire):** basis dirs default to rsync's metadata quick-check; FastSync-only `--verify-basis` restores content equality; copy-dest attributes + streaming removes the basis-hit size cap. - **Track 5b:** `--fuzzy` name heuristic matches rsync; residual is the narrower eligibility window. - **Track 6:** plain `--delete` defaults to rsync's delete-during; FastSync-only `--delete-commit` keeps the old atomic timing. ## Review wave (fixed) - **Blocker:** basis-dir materializations were counted as created/literal in `--stats`; now excluded (differential `test_link_dest_stats_matches_rsync`). - Filter wire block: reject over-long patterns and cap the rule list at 1024 to avoid delete-walk glob amplification; negative tests added. - README stale `--delete`/`--stats`/`--progress` docs corrected; new flags documented. ## Verification - Unit 44/44; ASan 44/44; full integration `-m "not setpriv"` 854 passed / 23 skipped / 1 xpassed; strict differential parity 62 passed; clang-format + cppcheck clean. ## New FastSync-only flags `--verify-basis`, `--delete-commit` (both long-only; documented in `RSYNC_COMPAT.md`/`README.md`).
TapTap added 14 commits 2026-09-19 17:11:04 +02:00
- -n/--delete sends the same protected/size-skipped/scope as a real run, so
  the read-only would-delete walk no longer over-reports (row -> caveat)
- --delete-delay charges --max-delete on actual removals and recursively
  re-scans a refilled deferred directory at commit; independent deferred cap
- --info=name emits the leading ./ root line and name2 'is uptodate' lines
- differential tests promoted from residual pins to rsync parity assertions
- PROTOCOL_VERSION 2.27.0 -> 2.28.0; STATUS_STATS gains literal_bytes and
  the created reg/dir/link/special counters (golden wire updated)
- receiver reports which destination entries it newly created, including
  implicitly-created parent directories below the logical transfer root, so
  Number of created files carries rsync's per-type breakdown
- Literal data is now exact for a delta transfer (receiver counts the literal
  fragments it stored)
- differential-tested vs rsync 3.4.1 for fresh-create, update and delta
- when --progress/--info=progress is requested, a metadata-only pre-scan
  builds the full file-list total and directory names so the to-chk
  denominator counts every regular/dir/link/special entry like rsync
- per-directory/symlink/special name lines emitted; sequential and --threads
- reuses the --delete-during/delay pre-scan when present; non-progress runs
  take no extra pass
- --progress stays a caveat (emission order still differs); single-file output
  remains byte-identical
- rsync 3.4.1 per-codec defaults (zstd 3, zlib/zlibx 6, lz4 level ignored)
  and per-codec clamping; explicit --zl still wins
- auto resolves via whitespace-separated RSYNC_COMPRESS_LIST /
  RSYNC_CHECKSUM_LIST (first supported wins; all-unknown exits 4)
- zlibx reclassified: FastSync's zlib stream already excludes matched data,
  so its tree/stdout/exit match zlib
- --compress/-z and --compress-choice flip to parity (113/12/32)
Probe shows the block-checksum choice is not observable in the parity surface:
%c, Matched/Literal data and the destination tree are invariant across
xxh64/xxh128/xxh3/md5/md4/sha1 and the transfer,pre-transfer form; only %C
changes, and it is byte-identical to rsync. FastSync's fixed xxHash32 block
strong sum is collision-safe within the payload cap. Row -> parity (114/11/32).
- new bounded config-wire block (BLOCK_PROTECT_RULES) serializes the sender's
  compiled filter rules to the receiver (bounded count + 256 KiB patterns;
  strict action/sides validation)
- receiver evaluates protect/risk in the whole-tree extras walk and the
  per-directory delete plans, so a dest-only entry matching 'P' is kept like
  rsync; dry-run would-delete enumeration also honours it
- --filter flips to parity (115/11/31); per-dir merge receiver re-derivation
  remains the documented residual
Adds the exclude_protect_dest_only differential and flips --delete-excluded
to parity now that receiver-side rules protect a destination-only excluded
entry like rsync; tally 116/10/31.
- default basis match is rsync's metadata quick-check (size + mtime; size-only
  drops mtime; -I disables), no mandatory content digest
- new long-only --verify-basis (wire bool, protocol stays 2.28.0) restores the
  strict whole-file content equality
- --copy-dest re-applies source attributes; basis-hit 256 MiB cap removed by
  streaming the copy/hash; basis miss keeps the normal payload bound
- compare/copy/link-dest rows -> caveat; tally 116/13/28
Probe shows the candidate choice is observable only as --stats bandwidth
counters (tree/exit always identical). FastSync already uses rsync's
fuzzy_distance/find_filename_suffix name heuristic; the residual is the
narrower delta eligibility window (>=16 KiB, <=10x) vs rsync's wider one.
Row -> caveat; tally 116/14/27.
- plain --delete with no timing flag now selects delete-during (progressive
  deletion, matching rsync and avoiding the full old+new tree peak)
- new long-only FastSync --delete-commit restores the old atomic behavior
  (delete only after the whole transfer succeeds); timing-identical to
  --delete-after, implemented via the same wire bool
- timing flags are mutually exclusive; --delete-commit conflicts with other
  timings; --delete-before/--delete-during rows reworded per Phase-0 probes
- CHANGELOG migration note; tally unchanged 116/14/27
- receiver stats: exclude basis-dir materializations (--link-dest/--copy-dest)
  from created/literal tallies; rsync reports 0 for a basis hit, so a fresh
  --link-dest --stats run now matches (differential test_link_dest_stats_matches_rsync)
- filter wire block: reject a pattern above the glob evaluation bound and lower
  MAX_FILTER_RULES to 1024, so a crafted rule list cannot amplify delete-walk
  glob work or install a rule that silently never protects
- negative tests for over-cap count and over-long pattern
- README: correct --delete default, --stats/--progress description, add
  --delete-commit; RSYNC_COMPAT stale version labels/overclaim fixed
Adds a unit test proving the config-only STATUS_DELETE_PLAN frame applies
--delete-missing-args exact deletions while walking no directory (the
--files-from-with-no-synced-dir fix).
chore: remove manual-test scratch trees from branch
CI / lint (pull_request) Successful in 1m59s
CI / parity-full (pull_request) Skipped
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
CI / parity-fast (pull_request) Successful in 17s
CI / build-and-test (pull_request) Successful in 49s
cd7b96d0bb
TapTap merged commit 10159dc120 into dev 2026-09-19 17:14:37 +02:00
TapTap deleted branch feat/parity-2.28 2026-09-19 17:14:38 +02:00
Sign in to join this conversation.
No Reviewers
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: TapTap/FastSync#303