100 Commits
Author SHA1 Message Date
TapTap ee6523afac Release v2.28.0
CI / lint (push) Successful in 2m0s
CI / parity-fast (push) Skipped
CI / lint (pull_request) Successful in 2m0s
CI / parity-full (pull_request) Skipped
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
CI / parity-full (push) Successful in 19s
CI / sanitizers (address) (push) Successful in 50s
CI / sanitizers (undefined) (push) Successful in 44s
CI / build-and-test (push) Successful in 1m8s
CI / coverage (push) Successful in 42s
CI / fuzz-build (push) Successful in 47s
CI / parity-fast (pull_request) Successful in 19s
CI / build-and-test (pull_request) Successful in 52s
CI / valgrind (push) Successful in 2m13s
- rsync-parity cycle: differential parity gate + tracks 1-6
- protocol 2.28.0 (batched wire changes)
- --delete defaults to delete-during; new --delete-commit, --verify-basis
- parity matrix 116/14/27 of 157
- tested: unit, integration, ASan, strict differential parity
2026-09-20 01:11:33 +02:00
TapTap 4163caa1d3 docs(handoff): record the rsync-parity 1-6 cycle (PR #303, protocol 2.28.0)
CI / lint (push) Successful in 1m57s
CI / parity-fast (push) Skipped
CI / parity-full (push) Successful in 20s
CI / sanitizers (address) (push) Successful in 49s
CI / sanitizers (undefined) (push) Successful in 42s
CI / build-and-test (push) Failing after 1m8s
CI / fuzz-build (push) Successful in 46s
CI / coverage (push) Successful in 42s
CI / valgrind (push) Successful in 2m13s
2026-09-19 17:21:26 +02:00
TapTap 10159dc120 Merge pull request 'feat(parity): rsync parity tracks 1-6 (protocol 2.28.0)' (#303) from feat/parity-2.28 into dev
CI / lint (push) Successful in 2m0s
CI / parity-fast (push) Skipped
CI / parity-full (push) Successful in 21s
CI / sanitizers (address) (push) Successful in 51s
CI / build-and-test (push) Successful in 1m7s
CI / sanitizers (undefined) (push) Successful in 43s
CI / fuzz-build (push) Successful in 47s
CI / coverage (push) Successful in 43s
CI / valgrind (push) Successful in 2m13s
2026-09-19 17:14:37 +02:00
TapTap cd7b96d0bb chore: remove manual-test scratch trees from branch
CI / lint (pull_request) Successful in 1m59s
CI / parity-full (pull_request) Skipped
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
CI / parity-fast (pull_request) Successful in 17s
CI / build-and-test (pull_request) Successful in 49s
2026-09-19 17:10:12 +02:00
TapTap f6f49d536e test(delete): cover apply=false config-only carrier frame
Adds a unit test proving the config-only STATUS_DELETE_PLAN frame applies
--delete-missing-args exact deletions while walking no directory (the
--files-from-with-no-synced-dir fix).
2026-09-19 17:07:56 +02:00
TapTap 38d304103c fix(parity): review-wave fixes (basis stats over-report, filter-rule bounds)
- receiver stats: exclude basis-dir materializations (--link-dest/--copy-dest)
  from created/literal tallies; rsync reports 0 for a basis hit, so a fresh
  --link-dest --stats run now matches (differential test_link_dest_stats_matches_rsync)
- filter wire block: reject a pattern above the glob evaluation bound and lower
  MAX_FILTER_RULES to 1024, so a crafted rule list cannot amplify delete-walk
  glob work or install a rule that silently never protects
- negative tests for over-cap count and over-long pattern
- README: correct --delete default, --stats/--progress description, add
  --delete-commit; RSYNC_COMPAT stale version labels/overclaim fixed
2026-09-19 17:04:12 +02:00
TapTap 4b09213b88 chore: remove accidental scratch tree from branch 2026-09-19 16:30:21 +02:00
TapTap 36fd0774e8 feat(delete): default --delete to rsync delete-during; add --delete-commit
- plain --delete with no timing flag now selects delete-during (progressive
  deletion, matching rsync and avoiding the full old+new tree peak)
- new long-only FastSync --delete-commit restores the old atomic behavior
  (delete only after the whole transfer succeeds); timing-identical to
  --delete-after, implemented via the same wire bool
- timing flags are mutually exclusive; --delete-commit conflicts with other
  timings; --delete-before/--delete-during rows reworded per Phase-0 probes
- CHANGELOG migration note; tally unchanged 116/14/27
2026-09-19 16:29:48 +02:00
TapTap 711b7e50b3 docs(parity): --fuzzy name heuristic matches rsync; reclassify to caveat
Probe shows the candidate choice is observable only as --stats bandwidth
counters (tree/exit always identical). FastSync already uses rsync's
fuzzy_distance/find_filename_suffix name heuristic; the residual is the
narrower delta eligibility window (>=16 KiB, <=10x) vs rsync's wider one.
Row -> caveat; tally 116/14/27.
2026-09-19 16:06:24 +02:00
TapTap 67076bf218 feat(basis): rsync quick-check default + FastSync-only --verify-basis
- default basis match is rsync's metadata quick-check (size + mtime; size-only
  drops mtime; -I disables), no mandatory content digest
- new long-only --verify-basis (wire bool, protocol stays 2.28.0) restores the
  strict whole-file content equality
- --copy-dest re-applies source attributes; basis-hit 256 MiB cap removed by
  streaming the copy/hash; basis miss keeps the normal payload bound
- compare/copy/link-dest rows -> caveat; tally 116/13/28
2026-09-19 15:55:51 +02:00
TapTap 8ec8cb7203 test(parity): dest-only excluded entry protected under default --delete
Adds the exclude_protect_dest_only differential and flips --delete-excluded
to parity now that receiver-side rules protect a destination-only excluded
entry like rsync; tally 116/10/31.
2026-09-19 13:53:50 +02:00
TapTap 82959395fb feat(filter): receiver-side protect/risk engine for dest-only entries
- new bounded config-wire block (BLOCK_PROTECT_RULES) serializes the sender's
  compiled filter rules to the receiver (bounded count + 256 KiB patterns;
  strict action/sides validation)
- receiver evaluates protect/risk in the whole-tree extras walk and the
  per-directory delete plans, so a dest-only entry matching 'P' is kept like
  rsync; dry-run would-delete enumeration also honours it
- --filter flips to parity (115/11/31); per-dir merge receiver re-derivation
  remains the documented residual
2026-09-19 13:52:48 +02:00
TapTap c9f94ea46e docs(parity): --checksum-choice observable-equivalent to rsync; reclassify
Probe shows the block-checksum choice is not observable in the parity surface:
%c, Matched/Literal data and the destination tree are invariant across
xxh64/xxh128/xxh3/md5/md4/sha1 and the transfer,pre-transfer form; only %C
changes, and it is byte-identical to rsync. FastSync's fixed xxHash32 block
strong sum is collision-safe within the payload cap. Row -> parity (114/11/32).
2026-09-19 13:23:02 +02:00
TapTap eff9852038 feat(codecs): per-codec level defaults, RSYNC_*_LIST auto, zlibx reclassify
- rsync 3.4.1 per-codec defaults (zstd 3, zlib/zlibx 6, lz4 level ignored)
  and per-codec clamping; explicit --zl still wins
- auto resolves via whitespace-separated RSYNC_COMPRESS_LIST /
  RSYNC_CHECKSUM_LIST (first supported wins; all-unknown exits 4)
- zlibx reclassified: FastSync's zlib stream already excludes matched data,
  so its tree/stdout/exit match zlib
- --compress/-z and --compress-choice flip to parity (113/12/32)
2026-09-19 13:14:48 +02:00
TapTap c80098623f feat(progress): opt-in paths-only pre-count for rsync to-chk parity
- when --progress/--info=progress is requested, a metadata-only pre-scan
  builds the full file-list total and directory names so the to-chk
  denominator counts every regular/dir/link/special entry like rsync
- per-directory/symlink/special name lines emitted; sequential and --threads
- reuses the --delete-during/delay pre-scan when present; non-progress runs
  take no extra pass
- --progress stays a caveat (emission order still differs); single-file output
  remains byte-identical
2026-09-19 12:49:56 +02:00
TapTap 6119e1e75c feat(stats): receiver-observed created/literal counters (protocol 2.28.0)
- PROTOCOL_VERSION 2.27.0 -> 2.28.0; STATUS_STATS gains literal_bytes and
  the created reg/dir/link/special counters (golden wire updated)
- receiver reports which destination entries it newly created, including
  implicitly-created parent directories below the logical transfer root, so
  Number of created files carries rsync's per-type breakdown
- Literal data is now exact for a delta transfer (receiver counts the literal
  fragments it stored)
- differential-tested vs rsync 3.4.1 for fresh-create, update and delta
2026-09-19 10:45:23 +02:00
TapTap 25062352f6 fix(parity): dry-run delete protections, delete-delay actual-removal budget, --info name2
- -n/--delete sends the same protected/size-skipped/scope as a real run, so
  the read-only would-delete walk no longer over-reports (row -> caveat)
- --delete-delay charges --max-delete on actual removals and recursively
  re-scans a refilled deferred directory at commit; independent deferred cap
- --info=name emits the leading ./ root line and name2 'is uptodate' lines
- differential tests promoted from residual pins to rsync parity assertions
2026-09-19 09:25:07 +02:00
TapTap 00d628d4ba Merge pull request 'test(parity): empty allowlist, assert max_delete invariants' (#302) from fix/parity-allowlist into dev
CI / lint (push) Successful in 1m40s
CI / parity-fast (push) Skipped
CI / parity-full (push) Successful in 20s
CI / sanitizers (address) (push) Successful in 51s
CI / sanitizers (undefined) (push) Successful in 41s
CI / build-and-test (push) Successful in 1m8s
CI / fuzz-build (push) Successful in 47s
CI / coverage (push) Successful in 42s
CI / valgrind (push) Successful in 2m13s
2026-09-18 22:31:46 +02:00
TapTap 3545d88905 test(parity): assert max_delete invariants, empty the parity allowlist
CI / lint (pull_request) Successful in 1m40s
CI / parity-full (pull_request) Skipped
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
CI / parity-fast (pull_request) Successful in 18s
CI / build-and-test (pull_request) Successful in 52s
The max_delete case allowlisted the tree aspect because the surviving extras after a partial --max-delete abort are deletion-order dependent. Under FASTSYNC_PARITY_STRICT a run where the orders coincide was reported as a stale entry (CI failure), while removing the entry made the order-dependent tree mismatch fail. Add a per-case 'compare_tree' flag: max_delete now asserts rc=25 plus the survivor count via extra_check instead of exact tree identity, so the allowlist can be empty. Burn-down reached zero.
2026-09-18 22:28:38 +02:00
TapTap 596a039454 Merge pull request 'fix(parity): ⚠️ residual burn-down + review fixes (protocol 2.27.0)' (#301) from feat/parity-fixes into dev
CI / lint (push) Successful in 1m42s
CI / parity-fast (push) Skipped
CI / parity-full (push) Failing after 17s
CI / build-and-test (push) Successful in 1m4s
CI / sanitizers (address) (push) Successful in 49s
CI / sanitizers (undefined) (push) Successful in 43s
CI / fuzz-build (push) Successful in 40s
CI / coverage (push) Successful in 40s
CI / valgrind (push) Successful in 2m12s
2026-09-18 22:19:25 +02:00
TapTap ae037cc27b Merge pull request 'test(parity): differential rsync 3.4.1 parity gate + CI' (#300) from feat/parity-gate into dev
CI / lint (push) Successful in 1m40s
CI / parity-fast (push) Skipped
CI / parity-full (push) Failing after 19s
CI / build-and-test (push) Successful in 58s
CI / sanitizers (address) (push) Successful in 51s
CI / sanitizers (undefined) (push) Successful in 45s
CI / fuzz-build (push) Successful in 43s
CI / coverage (push) Successful in 41s
CI / valgrind (push) Successful in 2m15s
2026-09-18 22:18:53 +02:00
TapTap 3d0672a721 docs(agents): note FASTSYNC_UNDER_VALGRIND for manual valgrind runs
CI / lint (pull_request) Successful in 1m40s
CI / parity-full (pull_request) Skipped
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
CI / parity-fast (pull_request) Successful in 16s
CI / build-and-test (pull_request) Successful in 51s
2026-09-18 22:15:38 +02:00
TapTap b82aab72c5 Merge branch 'fix/parity-review-c' into feat/parity-fixes 2026-09-18 22:11:07 +02:00
TapTap 8e7764007d Merge branch 'fix/parity-review-b' into feat/parity-fixes
# Conflicts:
#	src/shared/delete_plan.h
#	tests/integration/test_delete_timing_parity.py
2026-09-18 22:11:02 +02:00
TapTap 1042d15db7 docs(parity): correct delete-delay budget, fuzzy, and test-review gaps
- --delete-delay: state that the reported count advances on actual removal
  while --max-delete is charged at plan/snapshot time (defer_add/planned).
  A new differential shows rsync instead charges on actual removals and
  recursively removes a queued directory, so the row moves to Caveat
  (matrix 111/13/33) and the residual is pinned by tests.
- Add a deterministic unit test (plan-time budget charge), a FastSync
  integration test (byte-barrier refill + --max-delete), and an rsync
  differential for a refilled deferred directory.
- Soften the --fuzzy summary: the tree is byte-exact by design, so it is
  pinned by the threshold suite, not a byte-level differential.
- README: describe what -m parity actually selects; fix the allowlist
  example to the real max_delete entry.
- xdist-safe delete-timing fixture names (timing and --threads mode).
- Renumber the duplicate HANDOFF item 9 to 10; add the missing final
  newline to test_checksum.c.
- Expose ignore_errors_allows_delete and unit-test the deletion gate
  without a privileged source directory; update the stale deleted-count
  doc comment.

No production behavior changes.
2026-09-18 22:10:14 +02:00
TapTap cbe37a77dd refactor(parity): address code-quality review findings
- cli: remove UB in --bwlimit scaling (range-check the double product before
  casting, drop atoi for the +/-1 form) and add huge/boundary unit tests
- test: widen the CI throttle wall-clock band to [1.5, 4.5]s with a 2s
  cross-tolerance so a loaded runner cannot flake it
- log: drop the unused LOG_INFO_BACKUP bit; --info=backup is accepted-but-
  silent like the other rsync-only categories
- client_send: remove the duplicate delete_display_path forward declaration
- utils: add non-allocating utils_strip_transfer_root and use it from
  scanner_note_nonreg and delete_display_path (was duplicated logic)
- scanner: lstat() instead of stat() when re-reading an empty dir's metadata
- file: drop the no-op else-if and the redundant ELOOP arm in
  file_ensure_directory_secure (symlinks are refused anyway)
- format/stats: document literal_data as whole-file accurate (delta upper
  bound) instead of claiming literal bytes sent
- docs: refresh stale protocol 2.26.0 labels to 2.27.0
2026-09-18 22:00:32 +02:00
TapTap a5d45ef266 fix(parity): init delete_suppressed; gate deleted-path retention
- Initialize PipelineContextSender.delete_suppressed=false: an uninitialized
  true silently skipped the --delete keep-set manifest under -m/--threads,
  so destination extras were never removed.
- Allocate/install the receiver deleted-path observer only when
  report_deletes is set (--info=del / -i / --out-format under --delete), cap
  the retained list at MAX_MANIFEST_ENTRIES, and free already-created lists
  on the receiver-pipeline create failure path.
- Validate report_deletes/report_stats/report_dest_info on receive.
- Correct stale comments (config.h report_deletes, delete_plan.h deleted
  count, multiprocessing.h stats locking, utils.h observer placement).
- Tests: sender delete_suppressed init, report_deletes gating (unit), and
  -m/--delete default delete-after keep-set removal (integration).
2026-09-18 21:56:44 +02:00
TapTap a960391b34 fix(cli): guard parse_bwlimit_value against NULL (cppcheck) 2026-09-18 21:26:28 +02:00
TapTap 134f8b027a Merge branch 'fix/parity-fs' into feat/parity-fixes
# Conflicts:
#	RSYNC_COMPAT.md
2026-09-18 21:19:31 +02:00
TapTap eb7e3fd2e0 test(parity): option-wave rsync differentials + docs (109/21/26)
- tests/integration/test_option_parity.py: bwlimit parse matrix + throttle
  rate, --info flist/name/nonreg/del(dry+real+itemize)/remove, real-setpriv
  --ignore-errors, rsync-daemon -M forwarding evidence, filter protect/risk
  destination-only divergence pin.
- RSYNC_COMPAT.md: tally 109/21/26; --bwlimit and --ignore-errors -> Parity,
  --filter and -M -> Divergent with differential rationale, --info residual
  narrowed to the categories with no client-observable event.
- HANDOFF/README: protocol 2.27.0, option-wave summary.
2026-09-18 21:16:12 +02:00
TapTap 6a129b54d4 feat(parity): real --info=del deletion lines + rsync throttle pacing
- Wire: config frame gains report_deletes (protocol 2.26.0 -> 2.27.0); the
  receiver lists actually-removed paths in the STATUS_STATS path list, so the
  sender prints rsync's `deleting PATH` / `*deleting   PATH` lines for a real
  --delete run (and -i/out-format).  Observers threaded through the manifest,
  missing-args and per-directory delete engines; golden wire len/hash updated.
- bwlimit: throttle now paces like rsync 3.4.1 -- ~100ms burst capacity and the
  sleep is no longer credited as refill, so 4 MiB at 1024/2048 KiB/s matches
  rsync within ~4% (was ~2x too fast).
2026-09-18 21:06:20 +02:00
TapTap c7b2c7eb2b docs(parity): dir-time entry is record-only; empty dirs carried by STATUS_MKDIR 2026-09-18 20:50:22 +02:00
TapTap e77dfbec70 fix(fs): differential-test and reclassify basis dirs, delay-updates, fuzzy, dry-run
Each row's exact residual reproduced against rsync 3.4.1:
- basis dirs (--compare/copy/link-dest): FastSync xxHash-verifies a basis hit
  while rsync --size-only installs the wrong same-size basis content.
- --delay-updates: the fixed .fastsync-stage name wipes an unrelated
  destination entry of that name even without --delete; rsync leaves it.
- --fuzzy: deterministic name/size heuristic (10x window), not rsync's matcher.
- --dry-run: would-delete report over-reports the updated file and an
  excluded-but-protected extra, and ordering differs.
Docs: RSYNC_COMPAT tally 109/16/31; HANDOFF item 9. clang-format + cppcheck +
ASan + full suite clean.
2026-09-18 20:49:51 +02:00
TapTap f3ac4df4d0 fix(parity): rsync bwlimit units, --info categories, --ignore-errors deletion semantics
- --bwlimit: faithful port of rsync 3.4.1 parse_size_arg (default KiB/s,
  binary K/M/G/T/P, decimal KB/MB, KiB/MiB, decimals, 0 = unlimited, 512-byte
  floor, (size+512)/1024 quantization).  Unit tests + docs.
- --info: wire del/remove/name/flist/nonreg/progress to real FastSync events in
  rsync's line format (deleting PATH, sender removed NAME, name lines,
  'sending incremental file list', skipping non-regular file "NAME"); name no
  longer aliases copy; --info=progress drives the progress path and report_stats.
- --ignore-errors: match rsync's default -- a source I/O error skips deletion
  unless --ignore-errors, while the readable tree still transfers and the run
  exits 23.  Covers all delete timings and both send paths.
2026-09-18 20:44:24 +02:00
TapTap 91197fd7cf fix(fs): rsync push iconv direction; gate empty-dir emission; reclassify --temp-dir
- --iconv now matches rsync's push direction: the destination charset is the
  client spec's REMOTE half, so a default receiver writes wire names verbatim;
  a server's own --iconv LOCAL overrides it (daemon charset analog). Updated
  unit + integration tests and added a default-server differential gate case.
- Empty-directory emission is gated behind a new ScannerOptions.emit_empty_dirs
  set only by the real sender, so low-level scanner helpers keep the historical
  file-only list.
- --temp-dir reclassified to Divergent: relative dirs match rsync exactly
  (resolved under the destination), but an absolute path is deliberately
  rejected by the confined receiver; differential test added.
- Docs/tally: 109 Parity / 22 Caveat / 25 Divergent.
2026-09-18 20:39:20 +02:00
TapTap 13708352ec fix(fs): recreate empty dirs, replace blocking file; -R --no-implied-dirs implied parents
- Recursive scans emit a directory entry for every traversed directory that
  produced no transferred child, so empty source dirs (and dirs emptied by
  filtering) are recreated like rsync; -m prunes them, --files-from/--list-only
  never emit implicit dirs.
- A directory entry now replaces a destination regular file (rsync removes the
  non-directory) instead of aborting; confined to the secure parent fd.
- -R --no-implied-dirs --files-from: stop refusing a listed file whose parent
  is not listed; create the implied parent with default attributes (no source
  metadata is captured for it), matching rsync 3.4.1.
- Differential gate: drop min_size/empty_dirs_recursive/dirs_plain allowlist
  entries (dirs_plain now hands FastSync the same trailing-slash source as
  rsync); add differential test for the files-from implied parent.
- Docs: -d row -> Parity, tally 108/24/24.

No wire change (PROTOCOL_VERSION stays 2.26.0).
2026-09-18 20:25:42 +02:00
TapTap d162d93570 Merge branch 'feat/parity-gate' into fix/parity-fs 2026-09-18 20:20:25 +02:00
TapTap 9fa1696eff fix(parity): actual-removal delete-delay counts, rsync-accurate stats/progress/%C
- --delete-delay: count/track only entries actually removed; a directory
  refilled before commit (ENOTEMPTY) no longer inflates Number of deleted
  files or the --max-delete budget (unit + integration + rsync differential).
- --stats: per-type Number of files breakdown; only stored regular files
  count as transferred; transferred/literal byte totals and Total file size
  (symlink target lengths) now match rsync for whole-file transfers.
- --progress: print the leading ./ root line and include the root entry in
  the to-chk denominator (single-file output byte-identical to rsync).
- --out-format %C: use the selected transfer checksum and render every
  algorithm exactly like rsync; checksum_digest_file gains md4/sha1/none.
  Reclassify --out-format to Divergent (protocol-specific %b/delta-%c).
- Docs: RSYNC_COMPAT tally 107/25/24, HANDOFF update. No wire change.
2026-09-18 20:13:29 +02:00
TapTap b705fb807f test(parity): add differential rsync-parity CI gate
CI / lint (pull_request) Successful in 1m42s
CI / parity-full (pull_request) Skipped
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
CI / parity-fast (pull_request) Successful in 19s
CI / build-and-test (pull_request) Successful in 48s
Add tests/integration/test_differential_parity.py plus a shared
parity_harness.py and a data-driven parity_caveats.py allowlist.  The gate
runs real rsync 3.4.1 and FastSync over the same corpora, compares the
destination trees (paths, hashes, symlink targets, modes, hard-link
grouping) and the normalized -i/--stats/--out-format output, and fails on
any difference not listed in the allowlist.  Stale allowlist entries warn
(or fail under FASTSYNC_PARITY_STRICT=1) so the residual list shrinks.

Register parity/parity_ci markers and wire a fast PR job (parity_ci) plus a
push-only full job (parity, strict) into .gitea/workflows/ci.yaml.
Document the gate and the allowlist workflow in tests/integration/README.md.
2026-09-18 19:56:25 +02:00
TapTap cee281ff55 Merge pull request 'fix(test): stop the valgrind hang, init per_dir_filter_count' (#299) from fix/valgrind-hang into dev
CI / lint (push) Successful in 1m41s
CI / lint (pull_request) Successful in 1m40s
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
CI / sanitizers (undefined) (push) Successful in 44s
CI / sanitizers (address) (push) Successful in 50s
CI / build-and-test (push) Successful in 54s
CI / fuzz-build (push) Successful in 45s
CI / coverage (push) Successful in 41s
CI / build-and-test (pull_request) Successful in 44s
CI / valgrind (push) Successful in 2m12s
2026-09-17 23:47:47 +02:00
TapTap 5c509831b8 fix(test): robust valgrind detection + per-suite io-fd reset; init per_dir_filter_count
CI / lint (pull_request) Successful in 1m41s
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
CI / build-and-test (pull_request) Successful in 46s
The post-merge valgrind job on dev hangs.  Root cause: the CI valgrind step
exports FASTSYNC_UNDER_VALGRIND=1, but nothing read it, and the
/proc/self/maps "vgpreload" probe is unreliable on valgrind 3.22 (the guest's
maps no longer list the tool's own libraries).  So the fork-based unit tests
ran under valgrind anyway; tests that call io_set_fds() left the thread-local
read/write descriptors pointing at a closed test pipe, and a later
send_n_data()/receive_n_data() call was silently redirected to those stale fds
(legacy_session() prefers the globals, which the stdin/stdout SSH server
requires).  Later tests only worked by fd-reuse luck; under valgrind the fd
numbers no longer coincide, so the read blocked forever on an empty pipe.

- test_utils.h: honor FASTSYNC_UNDER_VALGRIND (already set by ci.yaml) and keep
  the maps scan as a best-effort fallback.  Reset io_set_fds(-1, -1) at the
  start of every RUN_TEST so one suite cannot leak descriptor redirection into
  the next.
- test_iconv.c: skip the forking wire-string roundtrip under valgrind like the
  other fork-based tests.
- config.c: initialize per_dir_filter_count in config_set_defaults.  The field
  was never initialized, so -F/-FF counting read uninitialized heap (valgrind:
  conditional jump on uninitialised value at client_cli.c:1464) and could count
  from garbage.

Verified with the CI-equivalent command (FASTSYNC_UNDER_VALGRIND=1 valgrind
--leak-check=full --show-leak-kinds=definite --error-exitcode=1): completes
with 0 errors (previously hung >80 min).  Unit 43/43; full integration 729
passed; cppcheck and clang-format clean.
2026-09-17 23:44:41 +02:00
TapTap ee57aeea4a Merge pull request 'rsync 3.4.1 drop-in parity (#285-#297) + parity completion (protocol 2.26.0)' (#298) from feat/rsync-parity into dev
CI / lint (pull_request) Successful in 1m46s
CI / lint (push) Successful in 1m47s
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
CI / build-and-test (pull_request) Successful in 47s
CI / sanitizers (address) (push) Successful in 50s
CI / build-and-test (push) Successful in 59s
CI / sanitizers (undefined) (push) Successful in 47s
CI / fuzz-build (push) Successful in 45s
CI / coverage (push) Successful in 44s
CI / valgrind (push) Failing after 3h0m1s
2026-09-17 20:33:16 +02:00
TapTap 803c1d3385 fix: review pass — uninit stats, append crash, filter rollback, ASan leak
CI / lint (pull_request) Successful in 1m45s
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
CI / build-and-test (pull_request) Successful in 45s
Address findings from the four-agent review of PR #298:

- file_create: zero the new File.matched_bytes.  It was uninitialized
  malloc memory, so the receiver could sum a garbage value into
  STATUS_STATS "Matched data" (nondeterministic --stats divergence and
  an uninitialized-heap disclosure on the wire).
- send_append: load the source into memory before hashing/copying the
  prefix and tail.  Files >64 MiB without compression (and --sendfile
  runs) are streamed without loading, so --append/--append-verify
  dereferenced a NULL data->data and crashed.
- filter_file_append: clamp the rollback to the surviving rule count.
  A "clear" rule in a merge file frees every rule including the
  caller's; the old rollback rewound count to rules_before and
  resurrected freed pointers for a double free / UAF.  Also roll back
  when set_rule_owner fails instead of leaving owner-less rules.
- filter_rule_parse: reject the xattr-name filter modifier (x), which
  was parsed and silently reinterpreted as a filename rule (affecting
  what --delete protects).  The p modifier stays accepted (existing
  grammar test).
- Remove two dead functions: compression_default_algo and
  change_render_itemize_code.
- tests: free ctx->would_delete in the two test_multiprocessing manual
  teardowns (ASan leak, 1648 bytes/run).
- docs: correct the RSYNC_COMPAT/HANDOFF tally (156 rows: 106/27/23),
  downgrade --info/--debug to caveat with their silent categories, add
  %C-vs-xxh64 and --delete-delay count caveats, refresh stale xattr
  mode comments, and document -p special-bit (setuid/setgid/sticky)
  parity plus its mitigations.
2026-09-17 20:30:15 +02:00
TapTap b8ec62beef fix(file): create implicit directories with 0777 & ~umask (rsync parity)
CI / lint (pull_request) Successful in 1m58s
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
CI / build-and-test (pull_request) Successful in 45s
Implicit parent directories were created with a hardcoded 0755, diverging from rsync's 0777 & ~umask whenever the process umask is not 022 (the CI runner uses 0). Matches rsync under any umask; verified with umask 0.
2026-09-17 19:39:21 +02:00
TapTap 59bfd32b9e docs(usage): correct --temp-dir help to the confined receive-root behavior
CI / lint (pull_request) Successful in 1m40s
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
CI / build-and-test (pull_request) Failing after 57s
2026-09-17 19:33:16 +02:00
TapTap 3432a33d9a docs(parity): finalize rsync-parity docs for protocol 2.26.0
Reclassify the RSYNC_COMPAT matrix after the parity-completion wave (protocol
2.23.0 -> 2.26.0): 9 already-parity rows to parity, 17 inherently non-rsync
rows to divergent, and the genuine fixes to parity, recounting to
109 parity / 25 caveat / 23 divergent of 157 rows. Add rows for --bwlimit,
--partial, --partial-dir, --no-whole-file, --inc-recursive/--no-inc-recursive,
--protect-args and --msgs2stderr; fix the documented -f/filter, -F/.rsync-filter,
empty --files-from, and --preallocate/--sparse precedence bugs; add the Parity
Completion Wave section.

Refresh README/HANDOFF/release skill/cmake-expert to protocol 2.26.0 and the
zlib/lz4 + md4/sha1/none codecs, add the CHANGELOG 2.26.0 entry, and correct
the stale --max-delete --help wording.
2026-09-17 19:25:06 +02:00
TapTap a1b081d328 Merge branch 'fix/parity-tests' into feat/parity-completion 2026-09-17 01:38:19 +02:00
TapTap bbecff9c04 fix(delete): keep the empty-scan safety guard file-only
Adding every traversed directory to the per-directory plan keep set must not
make an I/O-errored partial scan look non-empty.  Count only transmitted file
entries for delete_plan_sender_empty(), so a scan that hit an unreadable
directory and found no files still refuses to delete.
2026-09-17 01:34:26 +02:00
TapTap c1553bd5d6 style(delete): simplify redundant root[0] check (cppcheck) 2026-09-17 01:31:19 +02:00
TapTap 1a550bda24 test: pin delta-mode %c divergence against rsync
Add test_out_format_c_delta_mode_divergence documenting that FastSync's
delta %c (its own handshake bytes) cannot match rsync's (16-byte sum header
plus per-block checksums); only the whole-file case is aligned.  The test
pins both values so a future parity improvement is noticed.
2026-09-17 01:30:09 +02:00
TapTap 902f86192d test: stats file-count residual, --threads coverage, deterministic delete timing
- Output parity: add `File list size` to the strict --stats differential and
  document the row-#3 residual with test_stats_file_count_breakdown_residual
  (rsync's `Number of files`/`Number of created files` type breakdown is not
  reproducible from what the sender knows: no directory accounting and no
  per-entry destination-created state).  The row stays a caveat.
- Add --threads variants for the --stats and --progress/-P differentials.
  The `-n --delete --threads` variant is a documented xfail: the threaded
  dry-run path does not consume the receiver's STATUS_STATS delete list yet.
- Replace the 0.2s sleep flake in the delete-timing proxy with a socket
  barrier: the hook now fires only after the server sends a reply (proving it
  processed the preceding per-directory delete plan), using --incremental +
  --ignore-times to guarantee a mid-transfer handshake reply.
2026-09-17 01:29:29 +02:00
TapTap 6a40ac86e5 test(parity): cover --threads for -R delete scope and empty-dir retention 2026-09-17 01:25:37 +02:00
TapTap 410ba6e992 style: clang-format receiver.c and server.c 2026-09-17 01:23:47 +02:00
TapTap 6c6f02e5dd fix(parity): empty-dir delete, per-dir filter errors, -R protect, stats parser
Blockers addressed together (shared scanner/delete-plan plumbing):

* #10: an empty in-scope source directory produced no plan keep entry, so the
  receiver deleted the destination directory itself.  The scanner now records
  every traversed directory into a delete-plan sink, the plan sender keeps them,
  and any directory whose plan the data stream never triggered is emitted after
  the data so its extras are still removed.  Differential tests cover
  --delete-during and --delete-delay.
* #8: an invalid per-directory filter file was silently ignored when an earlier
  merge file in the same directory existed; key the failure off the error text
  (both sequential and parallel scanners) and fail the scan.
* #9: -R + --files-from receiver-protect rules recorded the source-relative
  path; record the bare relative wire path in both scanners so the protected
  destination mirror survives --delete.
* #5: the STATUS_STATS would-delete parser now validates each retained path and
  enforces the shared MAX_MANIFEST_BYTES budget, and the --out-format dry-run
  delete line is escaped like the itemize line.
* #11: drop the unused DELETE_PLAN_MAX_NAMES macro, log the delete-limit
  warning once per session, roll back dir-merge names from a per-directory file
  that fails to parse, and guard every filter error snprintf against err==NULL.

#10 leaves the empty directory itself kept and its extras removed, matching
rsync's final state on both per-directory timings.
2026-09-17 01:21:06 +02:00
TapTap d9006d1fda client: report rsync's 16-byte %c sum header for whole-file transfers
rsync's %c counts the block-checksum bytes received: even a whole-file
transfer with no basis receives rsync's 16-byte sum header (append and
inplace included), while a dry run receives nothing.  FastSync's
whole-file path has no equivalent header, so report 16 for parity when
delta is inactive, keep 0 for dry runs, and keep the real received bytes
when delta is active (FastSync's signature framing differs, so delta %c
stays divergent).  Add a strict %c/%l/%n differential against rsync and
turn the %b check into a real rsync differential (semantics: both count
wire bytes and exceed %l; the exact values are protocol-specific).
2026-09-17 01:19:07 +02:00
TapTap 36375010d3 client: accept rsync 3.4.1 --info/--debug category vocabulary
Accept the full rsync 3.4.1 --info (backup, del, flist, mount, nonreg,
progress, remove, symsafe) and --debug (acl, backup, bind, chdir, connect,
cmd, del, deltasum, dup, exit, filter, flist, fuzzy, genr, hash, hlink,
iconv, nstr, own, recv, send, time) vocabularies, plus the historical
syms/hl/owner aliases, with level suffixes.  Categories FastSync already
emits (copy/name/misc/skip/stats and io/proto/pack/util) still set their
log flags; the rest are accepted but silent.  Unknown names remain
rejected by name, matching rsync.  Update the CLI unit tests and the
rsync-parity integration tests (previously they required del/filter to be
rejected).
2026-09-17 01:16:39 +02:00
TapTap 5efa0dba7c test: differential st_blocks for --sparse/--preallocate and --ignore-existing wire volume
- Assert FastSync's sparse/preallocate block accounting matches rsync 3.4.1
  for a hole file (preallocate wins over sparse, exactly like rsync).
- Assert --ignore-existing is answered by the receiver before the sender
  transmits the payload (CountingProxy wire volume near-zero).
- CountingProxy.run gains a bounded join_timeout so tests that only need the
  client->server count do not wait for the server's idle socket.
- Fix the stale protocol 2.24.0 comment in test_config.c (golden is 2.26.0).
2026-09-17 01:13:05 +02:00
TapTap e32733fbf6 feat(stats): populate receiver wire counters on both receive paths
The single-threaded and -m receivers never populated ReceiverStats.matched_data
or .deleted_files, so --stats always printed 0 for both even when rsync
reported nonzero.  Track the bytes reconstructed from the basis file while
applying a delta, and tally the delete-commit counts (manifest and
per-directory sessions) into the receiver stats.  The -m pipeline now carries
its own stats/would-delete fields and emits the STATUS_STATS frame before the
terminal success, so --threads finally reports the counters and renders
-n --delete  lines.

Also normalize the -n --delete would-delete enumeration's absolute basis
prefixes exactly like the real commit path (fixing an over-report) and fix the
basis_delete_relative off-by-one when the receive root is '/'.  Unit tests
cover the root mapping and the basis protection; integration tests cover
matched/deleted stats for both receivers and the --threads dry-run delete
lines.
2026-09-17 01:08:32 +02:00
TapTap b02799327d fix(delete): guard the per-directory delete commit against dry-run
delete_plan_session_commit() lacked the central no-mutation guard that
manifest_delete_all() has, so a server-contacting -n run (or a hostile plan
frame) could still remove --delete-missing-args mirrors on the per-directory
timing path.  Return DELETE_COMMIT_OK immediately when the session is a
dry-run, and gate the receiver/server commit call sites too.  Add a unit test
that streams a plan naming an existing destination file and asserts it
survives.
2026-09-17 01:02:22 +02:00
TapTap 946aa934cc fix(delete): scope -R per-directory delete walk to the transferred prefix
The -R prefix marker installed in synced_dirs was discarded when finalizing
the per-directory delete sender (--delete-during/--delete-delay), so the
up-front root plan was the receive root '.', whose keep list only held the
first prefix component.  The receiver then deleted destination content
outside the transferred prefix (e.g. unrelated/keep.txt), a data-loss bug;
rsync keeps it.

Confine the walk to the -R prefix: send that prefix's plan as the root plan,
only transmit plans at or below it, and never emit the receive root plan for
a scoped run.  Add a differential test covering both --delete-during and
--delete-delay.
2026-09-17 01:00:55 +02:00
TapTap 125921c11b Merge branch 'feat/parity-codecs' into feat/parity-completion
# Conflicts:
#	src/shared/checksum.h
#	src/shared/config.h
#	tests/integration/test_fault_injection.py
#	tests/integration/test_preflight.py
#	tests/test_client_cli.c
#	tests/test_config.c
#	tests/test_fuzz_smoke.c
2026-09-16 23:49:41 +02:00
TapTap 9dd5288381 Merge branch 'feat/parity-wirestats' into feat/parity-completion
# Conflicts:
#	src/server/receiver_pipeline.c
#	src/shared/config.h
#	src/shared/protocol.h
#	tests/integration/test_fault_injection.py
#	tests/integration/test_preflight.py
#	tests/test_client_cli.c
#	tests/test_config.c
2026-09-16 23:46:28 +02:00
TapTap 3f2c74dd9e Merge branch 'feat/parity-deltiming2' into feat/parity-completion 2026-09-16 23:44:13 +02:00
TapTap 51e41dee2a Merge branch 'feat/parity-leftovers' into feat/parity-completion
# Conflicts:
#	src/client/scanner.c
#	src/client/scanner.h
2026-09-16 23:44:13 +02:00
TapTap dbf1b39d47 fix(delete): share the per-frame manifest byte budget across delete-plan sections 2026-09-16 23:36:46 +02:00
TapTap 845f20a28d docs(delete): describe per-directory timing in usage and config comments 2026-09-16 23:30:26 +02:00
TapTap a5083776da test(delete): cover per-dir timings for files-from scope, max-delete, excluded protection, refuse-delete, type conflicts 2026-09-16 23:29:25 +02:00
TapTap 76a81f1684 test(codec): differential coverage vs rsync and wire-golden updates
Add tests/integration/test_codecs.py (accept/reject matrix and byte
differential against rsync 3.4.1 for every algorithm), extend the
checksum/compression unit tests with MD4/SHA1/none vectors and per-codec
round-trips, pin the new config golden (2.26.0), and update the version
strings and codec acceptance expectations.
2026-09-16 23:27:44 +02:00
TapTap 24b81c7e5a feat(codec): negotiate checksum/compression algorithms (protocol 2.26.0)
Accept the full rsync 3.4.1 --compress-choice set (zstd/lz4/zlib/zlibx/
none/auto) and the two-name --checksum-choice TRANSFER,PRE-TRANSFER form,
including rsync's 'none' rules (rejected with --checksum at exit 4, and
forcing --whole-file as the transfer half) and unknown names at exit 4.
The checksum default becomes the auto-negotiated xxh128.

Negotiation is deterministic and symmetric: both peers run the same
preference resolver (rsync's --version order).  The resolved
compression_algo crosses the wire as a new trailing config-frame int so
the receiver validates and installs the exact codec; an unsupported
choice is refused before STATUS_OK like rsync's failed negotiation.
Bump PROTOCOL_VERSION to 2.26.0.
2026-09-16 23:27:40 +02:00
TapTap 0e33f84f38 feat(codec): implement md4/sha1/none digests and lz4/zlib/zlibx codecs
Add real implementations for the rsync 3.4.1 checksum and compression
breadth: a self-contained MD4 (RFC 1320), OpenSSL-backed SHA1, a
no-digest mode, and LZ4/zlib codecs alongside zstd.  Compressed buffers
are now self-describing (a leading codec id), so every existing
decompression call site keeps working through a process-global codec
selection.  zlibx shares the zlib codec because FastSync compresses only
delta/token bytes (never matched file data), matching the 'x' intent.
2026-09-16 23:27:34 +02:00
TapTap c7ac039523 docs(parity): correct implied-dir walk comment 2026-09-16 23:26:22 +02:00
TapTap e771cc9da6 fix(delete): guard per-dir missing-args by server policy; fall back for --dirs
- Only honor the --delete-missing-args exact paths when the server's
  --allow-delete policy left delete_missing_args set.
- -d/--dirs does not recurse, so a per-directory plan would carry no child
  information and could delete the contents of an untraversed directory; fall
  back to the whole-tree end-of-transfer commit for that mode.
2026-09-16 23:26:14 +02:00
opencode 7f9f82a068 style: clang-format and cppcheck fixes; document filter grammar in usage 2026-09-16 23:24:36 +02:00
TapTap 695b5c8c25 fix(parity): protect -R prefix-relative excluded and size-skipped mirrors
A -R source prune (--exclude/--max-size) must record the destination wire
path below the reconstructed prefix so --delete protects it; the parallel
root scan and the sequential skip path used the source path instead.
2026-09-16 23:21:58 +02:00
TapTap 5b2188d909 fix(parity): scope -R --delete to the transferred prefix subtree
A general -R transfer places its files below the reconstructed prefix, so
marking the whole receive root as the delete scope deleted unrelated
sibling directories (data loss; rsync keeps them).  Use the prefix itself
as the root marker when it is non-empty, in both the single-threaded and
multithreaded pipelines.
2026-09-16 23:20:46 +02:00
TapTap e5da916d54 test(parity): cover -d one-level listing, empty --files-from and negation rejection 2026-09-16 23:19:18 +02:00
TapTap de640bba1b feat(parity): report --stats during server-contacting dry-run
rsync prints the --stats block (with the (DRY RUN) suffix) for -n; route
the dry-run path through report_transfer_stats using the wire counters and
the STATUS_STATS receiver report.
2026-09-16 23:15:53 +02:00
TapTap f0f5719be0 docs(protocol): correct STATUS_STATS field description 2026-09-16 23:14:45 +02:00
TapTap 6200b298ac test(parity): ignore the untransferred source-root line in %C diff 2026-09-16 23:13:22 +02:00
opencode 394a9aae22 feat(parity): rsync filter grammar (merge/dir-merge/hide/show/protect/risk/clear + modifiers) and -F click semantics 2026-09-16 23:10:17 +02:00
TapTap 12d4af1b89 docs(usage): list %c/%C in --out-format help 2026-09-16 23:07:40 +02:00
TapTap 9d7c55d3c0 fix(parity): read STATUS_STATS before --remove-source-files acks
The receiver emits the wire-stats frame before the per-file acks and the
terminal status; the client must consume it in that order or a combined
--stats --remove-source-files run desynchronizes.
2026-09-16 23:06:54 +02:00
TapTap 5a104bfd88 fix(receiver): initialize new sink fields in -m pipeline 2026-09-16 23:05:46 +02:00
TapTap 2ada8f9ad5 style: clang-format wire-stats changes 2026-09-16 23:02:53 +02:00
TapTap 1493f1806d feat(parity): rsync-style per-file --progress and differential tests
Replace the aggregate stderr progress with rsync 3.4.1's per-file progress
block (name, 32 KiB first frame, final frame with (xfr#N, to-chk=X/Y)).
Add differential tests against real rsync for --out-format %C/%b, the
--progress frames, selected --stats lines and -n --delete lines.
2026-09-16 23:00:39 +02:00
TapTap ea28e25535 feat(parity): receiver STATUS_STATS report and -n --delete lines
Add the STATUS_STATS end-of-transfer receiver report (matched/deleted
counters plus a would-delete path list) behind the report_stats wire
bool, and a read-only delete_extras_list walker.  --stats now renders
true wire byte totals and the receiver-reported deleted count; a
server-contacting -n --delete prints transfer-relative '*deleting' lines
matching rsync's itemize layout.
2026-09-16 22:55:26 +02:00
TapTap d6295d62ce test(delete): differential + timing regression tests for per-directory delete plans
- Compare --delete-during/--delete-delay final state against rsync 3.4.1.
- Force a mid-transfer failure through a byte-slicing proxy: --delete-during has
  removed the processed directory's extra, --delete-delay has not.
- Create a destination entry while the transfer is in flight: it survives
  --delete-delay's snapshot but is removed by --delete-after's fresh end scan.
- Cover the --delete-delay type-conflict case now matching rsync.
2026-09-16 22:50:51 +02:00
opencode a9f416ce44 feat(parity): rsync fuzzy distance/suffix heuristic + exact size+mtime pass 2026-09-16 22:45:57 +02:00
TapTap 448edc0432 feat(delete): per-directory delete plans for --delete-during/--delete-delay (protocol 2.24.0)
Stream one delete plan per source directory from sender to receiver instead of
a single whole-tree keep-set manifest:

- --delete-during applies each directory's extras as its plan arrives, before
  that directory's data (rsync's generator-order deletion).
- --delete-delay snapshots each directory's extras while the plan arrives and
  commits the removals only after a fully-successful transfer, so files created
  after the scan survive (matching rsync's delete-delay, not delete-after).
- Type conflicts (a destination file blocking a source directory, or vice
  versa) are cleared immediately in both modes, so the nested write succeeds.

The plan carries the destination-relative directory, its kept child directory
names and its kept child file names; the first frame also carries the global
protected prefixes, size-skipped prefixes and --delete-missing-args paths.
--delete-before keeps the existing whole-tree early manifest; plain --delete and
--delete-after keep the end-of-transfer manifest commit.

Preserves the existing safety surface: protected/size-skipped prefixes and the
--delay-updates/basis skips are honored at any depth, deletion is scoped to the
synchronized directories (--files-from), MAX_SERVER_DELETE_COUNT and
--max-delete (partial + exit 25) are shared across plans, symlinks are never
followed, and paths are confined to the receive root.
2026-09-16 22:45:26 +02:00
TapTap 4a7703b06a feat(parity): -d/--dirs one-level listing for dir/, dir/. and .
A trailing slash (or trailing '/.', or a bare '.') now lists the source's
immediate contents -- files transferred, subdirectories created empty --
without recursing, while a bare directory still sends only its own entry.
The -R prefix applies to the generated entries and to the root entry.
2026-09-16 22:44:22 +02:00
TapTap 6fc297544e test(parity): differential coverage for -R, --no-implied-dirs and client aliases 2026-09-16 22:42:30 +02:00
TapTap 583d3c8edb feat(parity): general -R/--relative path semantics and --no-implied-dirs
Reconstruct the destination-relative prefix from the source spec outside
--files-from: cut at rsync's first '/./' (or a leading './'), normalize
later '.' components and trailing slashes.  Apply it as each File's
send_path in the sequential and parallel scanners (root and worker paths,
files, one-file-system mount entries and directory-time capture).

Transmit the metadata of implied parent directories (prefix components
above the source root), suppressed by --no-implied-dirs, so parent attrs
match rsync in both the single-threaded and -m pipelines.
2026-09-16 22:41:28 +02:00
TapTap 9883757190 fix(parity): accept rsync client aliases, --iconv=. / - and lone -h
- --ignore-non-existing (alias of --existing)
- --protect-args (pre-3.2.6 --secluded-args no-op)
- --msgs2stderr / --no-msgs2stderr (deprecated --stderr=all/client)
- --iconv=. (locale codeset via nl_langinfo), --iconv=- and --no-iconv (disable)
- lone -h prints help and exits 0; -h elsewhere stays human-readable
2026-09-16 22:41:23 +02:00
opencode a690109975 feat(parity): resolve --chown TO names on receiver via map rules 2026-09-16 22:41:12 +02:00
TapTap 36d4d0e43e feat(parity): wire-stats protocol 2.25.0 + out-format %b/%c/%C
Bump PROTOCOL_VERSION to 2.25.0 and append a report_stats bool to the
config frame, add a STATUS_STATS status, and add process-wide wire byte
counters (protocol_bytes_written/read) for the client.

Render the rsync 3.4.1 --out-format %b (wire bytes sent) and %c (wire
bytes read back) tokens from per-file counter deltas, and %C (whole-file
xxh128 checksum, seed 0) via a new streaming checksum_digest_file().
2026-09-16 22:35:43 +02:00
opencode a0b9d9794b feat(parity): absolute basis dirs + link-dest relink of up-to-date dest 2026-09-16 22:34:43 +02:00
opencode 3e9f70d9ba fix(parity): receiver-side --ignore-existing short-circuit before payload 2026-09-16 22:29:46 +02:00
opencode 2b5aaef409 fix(parity): --preallocate wins over --sparse, prefer fallocate(2) 2026-09-16 22:26:58 +02:00
TapTap 478f80be9f test(parity): add --stop-at rsync date-form differential coverage 2026-09-16 22:22:56 +02:00
TapTap e674b25213 fix(parity): client quick wins for rsync 3.4.1 (copy-links exit 23, info/debug flags, empty files-from, -F ordering, delete edges) 2026-09-16 22:21:45 +02:00
96 changed files with 17150 additions and 1732 deletions

No files matched your search

+41
View File
@@ -49,6 +49,47 @@ jobs:
if: github.event_name == 'push'
run: python3 -m pytest tests/integration/ -n 4 --dist=load -m "not setpriv" --durations=25 --tb=short -q
# Differential rsync-parity gate: runs real rsync 3.4.1 and FastSync over the
# same corpora and compares destinations + normalized output. The fast subset
# guards the ✅ surface on every PR; the full set (with FASTSYNC_PARITY_STRICT
# so a fixed caveat must be removed from the allowlist) burns the documented
# ⚠️/❌ residuals down on push. See tests/integration/README.md.
parity-fast:
runs-on: ubuntu-latest
container: gitea.tap-tap.win/taptap/fastsync-ci:v11
needs: lint
if: github.event_name == 'pull_request'
steps:
- name: Checkout
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- name: Configure
run: cmake -B build -S . -DSTRICT_WARNINGS=ON
- name: Build
run: cmake --build build -j$(nproc)
- name: Differential parity (fast subset)
run: python3 -m pytest tests/integration/test_differential_parity.py -n 4 --dist=load -m parity_ci -q
parity-full:
runs-on: ubuntu-latest
container: gitea.tap-tap.win/taptap/fastsync-ci:v11
needs: lint
if: github.event_name == 'push'
steps:
- name: Checkout
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- name: Configure
run: cmake -B build -S . -DSTRICT_WARNINGS=ON
- name: Build
run: cmake --build build -j$(nproc)
- name: Differential parity (full set)
run: FASTSYNC_PARITY_STRICT=1 python3 -m pytest tests/integration/test_differential_parity.py -n 4 --dist=load -m parity -q
sanitizers:
runs-on: ubuntu-latest
container: gitea.tap-tap.win/taptap/fastsync-ci:v11
+21 -5
View File
@@ -55,6 +55,16 @@ if(NOT ZSTD_LIBRARY)
message(FATAL_ERROR "zstd library not found. Ensure it is in your nix-shell!")
endif()
find_library(ZLIB_LIBRARY z)
if(NOT ZLIB_LIBRARY)
message(FATAL_ERROR "zlib library not found. Ensure zlib1g-dev / nix zlib is available!")
endif()
find_library(LZ4_LIBRARY lz4)
if(NOT LZ4_LIBRARY)
message(FATAL_ERROR "lz4 library not found. Ensure liblz4-dev / nix lz4 is available!")
endif()
find_package(OpenSSL REQUIRED)
file(GLOB SHARED_SRCS "src/shared/*.c")
@@ -64,15 +74,15 @@ file(GLOB TEST_SRCS "tests/*.c")
add_executable(server ${SERVER_SRCS} ${SHARED_SRCS})
target_include_directories(server PRIVATE src/shared src/server src/client)
target_link_libraries(server PRIVATE Threads::Threads ${ZSTD_LIBRARY} OpenSSL::SSL OpenSSL::Crypto xxhash)
target_link_libraries(server PRIVATE Threads::Threads ${ZSTD_LIBRARY} ${ZLIB_LIBRARY} ${LZ4_LIBRARY} OpenSSL::SSL OpenSSL::Crypto xxhash)
add_executable(client ${CLIENT_SRCS} ${SHARED_SRCS})
target_include_directories(client PRIVATE src/shared src/server src/client)
target_link_libraries(client PRIVATE Threads::Threads ${ZSTD_LIBRARY} OpenSSL::SSL OpenSSL::Crypto xxhash)
target_link_libraries(client PRIVATE Threads::Threads ${ZSTD_LIBRARY} ${ZLIB_LIBRARY} ${LZ4_LIBRARY} OpenSSL::SSL OpenSSL::Crypto xxhash)
add_executable(tests ${TEST_SRCS} ${SHARED_SRCS} src/client/scanner.c)
target_include_directories(tests PRIVATE tests src/shared src/server src/client)
target_link_libraries(tests PRIVATE Threads::Threads ${ZSTD_LIBRARY} OpenSSL::SSL OpenSSL::Crypto xxhash)
target_link_libraries(tests PRIVATE Threads::Threads ${ZSTD_LIBRARY} ${ZLIB_LIBRARY} ${LZ4_LIBRARY} OpenSSL::SSL OpenSSL::Crypto xxhash)
```
### Source Layout
@@ -85,17 +95,23 @@ tests/integration/ — Python pytest integration tests
```
### Dependencies
- **zstd** — found via `find_library(ZSTD_LIBRARY zstd)`
- **zstd** — found via `find_library(ZSTD_LIBRARY zstd)` (default compression codec)
- **zlib** — found via `find_library(ZLIB_LIBRARY z)` (the `zlib`/`zlibx` codecs)
- **lz4** — found via `find_library(LZ4_LIBRARY lz4)` (the `lz4` codec)
- **OpenSSL** — found via `find_package(OpenSSL REQUIRED)` (TLS 1.2+ transport)
- **xxHash** — fetched via `FetchContent` from the upstream repository (delta transfer hashing, v0.8.3)
- **pthreads** — found via `find_package(Threads REQUIRED)`
- **C11 standard** — required
- **CMake 3.22+** — minimum version
The codec matrix (protocol 2.26.0) uses zstd/zlib/lz4 for compression and
xxHash/OpenSSL for the `xxh128`/`xxh3`/`xxh64`/`md5`/`md4`/`sha1` checksums
(`none` needs no library); both codec families are negotiated per transfer.
## Conventions
- Use `file(GLOB ...)` for source collection (existing pattern).
- All targets link `Threads::Threads`, `${ZSTD_LIBRARY}`, `OpenSSL::SSL`, `OpenSSL::Crypto`, and `xxhash`.
- All targets link `Threads::Threads`, `${ZSTD_LIBRARY}`, `${ZLIB_LIBRARY}`, `${LZ4_LIBRARY}`, `OpenSSL::SSL`, `OpenSSL::Crypto`, and `xxhash`.
- Include directories: `src/shared`, `src/server`, `src/client`, `tests` (for test target).
- Sanitizer support: pass `-DSANITIZER=address`, `-DSANITIZER=thread`, or `-DSANITIZER=undefined` to cmake (live option in CMakeLists.txt).
- Build with `cmake -B build -S . && cmake --build build -j$(nproc)`.
+1 -1
View File
@@ -16,7 +16,7 @@ Ask the user or determine from context:
- **Minor** (x.Y.0) — new features, backward compatible
- **Patch** (x.y.Z) — bug fixes, no protocol changes
Current version: `PROTOCOL_VERSION "2.23.0"` in `src/shared/config.h`
Current version: `PROTOCOL_VERSION "2.26.0"` in `src/shared/config.h`
### Step 2: Check Protocol Version
+15
View File
@@ -56,6 +56,21 @@ cmake -B build -S . && cmake --build build -j$(nproc)
./build/tests # unit tests
python3 -m pytest tests/integration/ -n 4 --dist=load -m "not setpriv" # full integration suite (CI excludes env-dependent privilege tests)
python3 -m pytest tests/integration/ -n 4 --dist=load -m ci # PR-gate subset only
# Differential rsync-parity gate (real rsync 3.4.1 vs FastSync)
python3 -m pytest tests/integration/test_differential_parity.py -n 4 --dist=load -m parity_ci # fast PR subset
python3 -m pytest tests/integration/test_differential_parity.py -n 4 --dist=load -m parity # full set
```
See `tests/integration/README.md` for the differential parity gate and its
`parity_caveats.py` allowlist (the residual burn-down mechanism).
Unit tests under valgrind must set `FASTSYNC_UNDER_VALGRIND=1` (CI does): the
tests use it to skip fork-based tests, because valgrind 3.22 does not expose
`vgpreload` in the guest's `/proc/self/maps`.
```bash
FASTSYNC_UNDER_VALGRIND=1 valgrind --leak-check=full --show-leak-kinds=definite --error-exitcode=1 ./build/tests
```
## CI Workflow — Waiting for Results
+150
View File
@@ -4,6 +4,156 @@ All notable changes to FastSync are documented here. Versions match
`PROTOCOL_VERSION` (printed by `fastsync --version`); the client and server must
run the same version because the handshake is strict.
## [2.28.0] - 2026-09-20
The rsync-parity cycle. `PROTOCOL_VERSION` moves `2.26.0 → 2.27.0 → 2.28.0`;
client and server must run the same version (the handshake is strict). See
`RSYNC_COMPAT.md` for the per-option matrix, now **116 ✅ / 14 ⚠️ / 27 ❌** of
157 rows.
### Added
- **Differential rsync 3.4.1 parity gate** (`tests/integration/
test_differential_parity.py`, `parity_harness.py`, `parity_caveats.py`): runs
real `rsync` and FastSync over generated corpora and diffs the destination
tree, normalized stdout and exit code. A fast subset runs on pull requests and
the full strict set on push; the residual allowlist is empty.
- FastSync-only long option **`--verify-basis`**: require a
`--compare-dest`/`--copy-dest`/`--link-dest` hit to match the source by
whole-file digest instead of trusting the size+mtime quick-check.
- FastSync-only long option **`--delete-commit`** (implies `--delete`): the old
atomic late whole-tree commit.
- `--bwlimit` now parses rsync's units exactly and paces like rsync's leaky
bucket; `--ignore-errors` reproduces rsync's skip-unreadable-subdir and
IO-error-suppressed deletion (exit 23).
- `--info=name/flist/del/remove/nonreg/progress` emit rsync's line format,
including real-run `deleting`/`*deleting` lines carried by a new
`report_deletes` wire bool.
- Receiver-observed `--stats` counters: `Number of created files` now carries
rsync's `(reg/dir/link/special)` breakdown and `Literal data` is exact for a
delta transfer (extended `STATUS_STATS`).
- `--progress` uses an opt-in paths-only pre-count so the `to-chk` denominator
counts every entry like rsync, and emits per-directory/symlink/special names.
- Receiver-side `protect`/`risk` filter engine (new bounded filter-rule wire
block): `--filter='P ...'` now shields a destination-only entry like rsync.
- `auto` for `--compress-choice`/`--checksum-choice` honors
`RSYNC_COMPRESS_LIST`/`RSYNC_CHECKSUM_LIST`, and per-codec compression-level
defaults match rsync.
- Empty source directories are recreated recursively; `-R --no-implied-dirs
--files-from` places listed files under missing implied parents; `--iconv`
matches rsync's push direction; `--delete-delay` reports actual removals and
recursively removes a refilled deferred directory.
### Changed
- **`--delete` now defaults to delete-during (rsync `--del`) timing.** With no
explicit timing flag, a plain `--delete` removes each directory's extras as
that directory is processed instead of committing one whole-tree deletion only
after the entire transfer succeeds. This matches rsync, frees destination
space progressively, and avoids the whole-old+new-tree peak that could
`ENOSPC` a tight destination. The client maps the default onto the existing
`delete_during` wire boolean, so `PROTOCOL_VERSION` stays `2.28.0`.
- Basis directories (`--compare-dest`/`--copy-dest`/`--link-dest`) now default
to rsync's metadata quick-check (equal size and mtime; `--size-only` drops the
mtime leg) instead of FastSync's historical always-verify content hash.
`--copy-dest` re-applies the source attributes, and basis materialization is
streamed so the 256 MiB whole-file cap no longer applies to a basis hit.
- The per-directory `STATUS_DELETE_PLAN` frame gained a one-int `apply` flag:
the one-shot per-run config block (protected prefixes, size-pruned mirrors,
`--delete-missing-args` exact paths) is now always transmitted first on a
config-only carrier (`apply=false`), fixing a latent bug where a
`--delete-missing-args` run whose `--files-from` list synchronized no directory
never sent its exact deletions.
### Notes
- `--delete`/`--delete-during` remain caveats for the mid-transfer abort
boundary (rsync's generator removes all planned extras ahead of its throttled
sender; FastSync removes only reached directories — final trees agree).
`--delete-before`, `--progress`, `--stats`, `--fuzzy` and the basis rows keep
their documented residuals in `RSYNC_COMPAT.md`; `--filter` and
`--delete-excluded` are now parity, including protection of a destination-only
excluded entry under default `--delete`.
### Migration
- Scripts that relied on plain `--delete` deleting nothing until the transfer
fully succeeded must pass **`--delete-commit`** (or `--delete-after`) to keep
that behavior. Plain `--delete` now removes reached directories' extras during
the transfer, exactly like rsync's default; on a completed run the final tree
is unchanged.
- Deployments that relied on FastSync's stricter basis verification should pass
**`--verify-basis`**; the default now trusts the size+mtime quick-check like
rsync.
## [2.26.0] - 2026-09-17
### Added
- **Parity-completion wave.** Closed the remaining rsync-parity gaps against
rsync 3.4.1 and reclassified the inherently non-rsync rows. It moved the wire
protocol three times (`2.23.0 → 2.24.0 → 2.25.0 → 2.26.0`).
- **Delete timing (2.24.0):** per-directory delete plans
(`STATUS_DELETE_PLAN`) for `--delete-during`/`--delete-delay`. An interrupted
during-transfer has already removed the reached directories' extras, while a
delayed transfer commits per directory only after the whole transfer
succeeds (a late-created extra survives `--delete-delay` but not
`--delete-after`). `-R --delete` is scoped to the transferred prefix; empty
in-scope source directories survive; dry-run never deletes.
- **Wire stats (2.25.0):** `STATUS_STATS` carries the receiver counters
(matched data, deleted files) and the dry-run would-delete list. `--stats`
prints rsync's protocol-independent lines; `--progress`/`-P` print per-file
blocks; `--out-format` gains `%b` (wire bytes), `%c` (block-sum bytes) and
`%C` (whole-file digest); `-n --delete` prints escaped `*deleting` lines in
the sequential and `--threads` paths.
- **Codecs (2.26.0):** `lz4`/`zlib`/`zlibx` compression and `md4`/`sha1`/
`none` checksums, with rsync-style `auto` negotiation (default `xxh128` +
`zstd`) and exit-4 rejection of unknown names; the resolved `compression_algo`
crosses the wire.
- General `-R`/`--relative` (including the `/./` cut) and `--no-implied-dirs`;
one-level `-d`/`--dirs` listing for `dir`, `dir/` and `.`; the full filter
grammar (`merge`/`dir-merge`/`hide`/`show`/`protect`/`risk`/`clear` and
modifiers) with `-f` bound to `--filter`; a single `-F` transfers
`.rsync-filter` and `-FF` excludes it.
- Receiver-side `--chown`/`--usermap`/`--groupmap` TO-name resolution; absolute
basis directories and a `--link-dest` relink of an up-to-date destination;
a receiver-side `--ignore-existing` short-circuit before any payload;
`--preallocate` now wins over `--sparse` via `fallocate(2)`.
- Client quick wins: `--iconv=.`/`-`/`--no-iconv`, a lone `-h` prints help, an
empty `--files-from` succeeds (exit 0), a broken referent under
`-L`/`--copy-unsafe-links` exits 23, the full `--info`/`--debug`
vocabularies, and the aliases `--ignore-non-existing`, `--protect-args`,
`--msgs2stderr`.
### Changed
- `PROTOCOL_VERSION` bumped `2.23.0 → 2.24.0` (delete plans),
`2.24.0 → 2.25.0` (`STATUS_STATS` + `report_stats`), and
`2.25.0 → 2.26.0` (codec negotiation + `md4`/`sha1`/`none`).
- `--checksum-choice`/`--cc` now accepts `md4`, `sha1`, `none` and the two-name
form; the negotiated whole-file default is `xxh128`.
- `--compress-choice`/`--zc` now accepts `lz4`, `zlib`, `zlibx`.
- `RSYNC_COMPAT.md` reclassifies the matrix: 9 already-parity rows to ✅, 17
inherently non-rsync rows to ❌ (native daemon config/auth, batch, privileged
xattr namespaces, and the safe-subset device/privilege flags), and the genuine
fixes to ✅; new rows cover `--bwlimit`, `--partial`, `--partial-dir`,
`--no-whole-file`, `--inc-recursive`/`--no-inc-recursive`, `--protect-args`
and `--msgs2stderr`.
- The client `--help` `--max-delete` text now describes the implemented partial
semantics (delete up to N, skip the rest, exit 25).
### Notes
- Remaining documented divergences include the `--stats` per-type file-count
breakdown, `%b`/`%c` being FastSync wire counts, `-n --delete` line ordering,
the default `--delete` timing (delete-after, not rsync's delete-during),
destination-only exclude protection (still sender-derived), `--temp-dir`
absolute paths, basis-dir attribute re-application and the 256 MiB whole-file
cap, `--fuzzy` tie-breaking, `--bwlimit=0`/decimal rates, `zlibx`==`zlib`, and
recursive empty-directory creation.
- Build: adds zlib and lz4 as link dependencies.
## [2.23.0] - 2026-09-16
### Added
+17 -5
View File
@@ -1,6 +1,6 @@
cmake_minimum_required(VERSION 3.22)
project(FastFileTransfer VERSION 2.23.0)
project(FastFileTransfer VERSION 2.28.0)
set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
set(CMAKE_C_STANDARD 11)
@@ -68,6 +68,16 @@ if(NOT ZSTD_LIBRARY)
message(FATAL_ERROR "zstd library not found. Ensure it is in your nix-shell!")
endif()
find_library(ZLIB_LIBRARY z)
if(NOT ZLIB_LIBRARY)
message(FATAL_ERROR "zlib library not found. Ensure zlib1g-dev / nix zlib is available!")
endif()
find_library(LZ4_LIBRARY lz4)
if(NOT LZ4_LIBRARY)
message(FATAL_ERROR "lz4 library not found. Ensure liblz4-dev / nix lz4 is available!")
endif()
find_package(OpenSSL REQUIRED)
# --- Explicit source lists ---
@@ -89,6 +99,7 @@ set(SHARED_SRCS
src/shared/daemon_limits.c
src/shared/data.c
src/shared/delay_updates.c
src/shared/delete_plan.c
src/shared/delta.c
src/shared/file.c
src/shared/file_list.c
@@ -135,8 +146,8 @@ set(CLIENT_MAIN_SRCS src/client/client_cli.c)
# --- Library targets ---
add_library(fastsync_shared STATIC ${SHARED_SRCS})
target_include_directories(fastsync_shared PUBLIC src/shared)
target_link_libraries(fastsync_shared PUBLIC Threads::Threads ${ZSTD_LIBRARY} OpenSSL::SSL
OpenSSL::Crypto xxhash)
target_link_libraries(fastsync_shared PUBLIC Threads::Threads ${ZSTD_LIBRARY} ${ZLIB_LIBRARY}
${LZ4_LIBRARY} OpenSSL::SSL OpenSSL::Crypto xxhash)
add_library(fastsync_client_core STATIC ${CLIENT_CORE_SRCS})
target_include_directories(fastsync_client_core PUBLIC src/client)
@@ -210,6 +221,7 @@ set(TEST_SRCS
tests/test_daemon_limits.c
tests/test_data.c
tests/test_delay_updates.c
tests/test_delete_plan.c
tests/test_delta.c
tests/test_file.c
tests/test_file_list.c
@@ -275,7 +287,7 @@ if(ENABLE_FUZZ)
target_include_directories(${FUZZ_NAME} PRIVATE tests src/shared src/server)
target_compile_options(${FUZZ_NAME} PRIVATE -fsanitize=fuzzer,address,undefined -fno-omit-frame-pointer)
target_link_options(${FUZZ_NAME} PRIVATE -fsanitize=fuzzer,address,undefined)
target_link_libraries(${FUZZ_NAME} PRIVATE Threads::Threads ${ZSTD_LIBRARY} OpenSSL::SSL
OpenSSL::Crypto xxhash)
target_link_libraries(${FUZZ_NAME} PRIVATE Threads::Threads ${ZSTD_LIBRARY} ${ZLIB_LIBRARY}
${LZ4_LIBRARY} OpenSSL::SSL OpenSSL::Crypto xxhash)
endforeach()
endif()
+166 -11
View File
@@ -1,16 +1,19 @@
# FastSync — Session Handoff (2026-09-14)
# FastSync — Session Handoff (2026-09-19)
## Current status
- **Release `v2.21.0`** tagged (`919a729`, "Release v2.21.0"); full CI green
(run 552: lint, build-and-test, ASan, UBSan, fuzz-build, coverage, valgrind).
`dev` has the release commit plus later doc-only merges (a README refresh and
this handoff).
- **Release PR #284 (`dev` -> `main`)** open, CI green (run 553).
`main` is protected: it needs review/approval to merge.
https://gitea.tap-tap.win/TapTap/FastSync/pulls/284
- **`PROTOCOL_VERSION` = `"2.23.0"`** (`src/shared/config.h`); CMake
`project(FastFileTransfer VERSION 2.23.0)`.
- Working tree clean; no wave worktrees remain.
- **rsync-parity tracks 1-6 landed on `dev`** via **PR #303** (`10159dc`,
"feat(parity): rsync parity tracks 1-6 (protocol 2.28.0)"). Dev push CI run
**581** fully green: lint, build-and-test, parity-full, ASan, UBSan,
fuzz-build, coverage, valgrind.
- **`PROTOCOL_VERSION` = `"2.28.0"`** (`src/shared/config.h`); CMake
`project(FastFileTransfer VERSION 2.28.0)`. The cycle batched all wire
changes (stats counters, filter-rule block, `--verify-basis`) under the one
bump.
- **`main` = `ef76c90`** (tag `v2.26.0`); the 2.27.0/2.28.0 work is on `dev`
and not yet released. A `dev -> main` v2.28.0 release PR is the next step.
- Parity matrix: **116 ✅ / 14 ⚠️ / 27 ❌ = 157** (was 111/13/33 at cycle start).
- Working tree clean; feature branch deleted; no scratch trees or worktrees.
## What landed this session
1. **Wave 8 (refactors):** Config X-macro wire table; single-owner `authorized_root`;
@@ -39,6 +42,158 @@
docs state push-only / remote-source unsupported.
5. **Preserve-attribute split (protocol 2.22.0)** landed on `feat/preserve-attr-split`: per-attribute `-p/-t/-o/-g` + `--no-*` negations, `-a` = `-rlptgoD`, and the 2.21.0 → 2.22.0 wire bump.
6. **Rsync-parity wave (protocol 2.23.0)** on `feat/rsync-parity`: rsync short options/clustering/attached values (`-r`/`-b`/`-L`/`-B`, `-av`, `-aAX`, `-B1000`, `-essh`, `-MOPT`), `-c` checksum quick-check, `--checksum-choice`/`--compress-choice` validation and seed randomization, rsync timeout/max-alloc defaults, temp-dir confinement + `EXDEV` fallback, ownership/mapping parity (numeric-ids modifier, map ranges/`*`/empty-FROM, `--chown`+map conflicts, fake-super resolved-owner record), verbatim symlink storage with rsync `--safe-links`/`--munge-links`, socket recreation under `--specials`, `--chmod` 3.4.1 semantics, and delete scoping + `--max-delete` partial/exit-25. Wire: appended delete-manifest synchronized-directory section and `STATUS_DELETE_LIMIT`.
7. **Parity-completion wave (protocol 2.24.0 → 2.26.0)** on `feat/parity-completion`: per-directory delete plans (`STATUS_DELETE_PLAN`) for `--delete-during`/`--delete-delay`; receiver `STATUS_STATS` counters feeding `--stats`/`--progress` and `--out-format %b/%c/%C`, plus `-n --delete` lines; `lz4`/`zlib`/`zlibx` compression and `md4`/`sha1`/`none` checksums with `auto` negotiation (default `xxh128`/`zstd`); general `-R`/`--no-implied-dirs`/`-d`; the full filter grammar (`merge`/`dir-merge`/`hide`/`show`/`protect`/`risk`/`clear` + modifiers) and corrected `-F`/`-FF`; receiver-side `--chown`/map TO-name resolution; absolute basis dirs + `--link-dest` relink; receiver-side `--ignore-existing` short-circuit; `--preallocate` over `--sparse` via `fallocate(2)`; `--iconv=.`/`-`/`--no-iconv`; lone `-h` help; aliases `--ignore-non-existing`/`--protect-args`/`--msgs2stderr`; and the full `--info`/`--debug` vocabulary. `RSYNC_COMPAT.md` reclassifies the matrix to 106 ✅ / 27 ⚠️ / 23 ❌; the later rsync-parity-stats pass (`fix/parity-stats`) moves it to 107 ✅ / 25 ⚠️ / 24 ❌ (see item 8).
8. **rsync-parity-stats pass** on `fix/parity-stats` (no wire change, `PROTOCOL_VERSION` stays `2.26.0`): `--delete-delay` now reports only entries it actually removes, while the `--max-delete` budget is charged at plan/snapshot time (`planned`, via `defer_add`) to bound the deferred list (a refilled deferred directory that survives `ENOTEMPTY` is not reported but still consumes budget); `--stats` gained the `(reg/dir/link/special)` `Number of files` breakdown and now counts only regular files actually stored for `Number of regular files transferred`/transferred size/literal data (up-to-date re-runs report 0); `Total file size` includes symlink target lengths; `--progress` prints the leading `./` root line and counts it in `to-chk` so a single-file transfer matches rsync; and `%C` uses the selected transfer checksum with `checksum_digest_file` supporting md4/sha1/none, byte-identical to rsync for every algorithm. `--out-format` reclassified ❌ (`%b`/delta-`%c` are protocol-specific). Differential + regression tests added; full suite + ASan + clang-format + cppcheck clean.
9. **Option-parity wave (protocol 2.26.0 → 2.27.0, on `fix/parity-options`):**
`--bwlimit` now ports rsync 3.4.1's units/quantization and paces like its
leaky bucket; `--ignore-errors` reproduces rsync's default (an I/O error
skips deletion unless the flag is set; the readable tree still transfers and
the run exits 23) across every delete timing; the `--info` categories with a
FastSync event (`name`/`flist`/`del`/`remove`/`nonreg`/`progress`) emit
rsync's line format, with real-run `deleting`/`*deleting` lines carried over
the new trailing config bool `report_deletes` (golden wire updated by
`tests/test_config.c`). Two residuals were reclassified **divergent**: `-M`
over daemon/TCP (no argv channel in FastSync's binary config handshake;
rsync-daemon differential pins the rsync behavior) and receiver-side
`protect`/`risk` re-derivation for destination-only entries (would need a
receiver filter engine; differential pins the divergence — **reversed by
track 4a below**, which adds that engine). The options pass
stands at **110 ✅ / 21 ⚠️ / 26 ❌**. New `tests/integration/test_option_parity.py`
holds the rsync differentials (bwlimit parse+rate, info lines, real-setpriv
`--ignore-errors`, rsync-daemon `-M`, filter-protect pin).
10. **rsync-parity-fs pass** on `fix/parity-fs` (no wire change of its own; integrated
on top of the 2.27.0 options wave): recursive transfers now recreate empty source directories (and
`-m/--prune-empty-dirs` still suppresses them), a directory entry replaces a
blocking destination regular file, and `-R --no-implied-dirs --files-from`
places a listed file under a missing implied parent with default attributes
instead of refusing (real rsync 3.4.1 parity, differential-tested). `--iconv`
now reproduces rsync's push direction (destination charset = the spec's REMOTE
half; a server `--iconv` overrides), and `-T/--temp-dir` relative semantics are
confirmed identical while the absolute-path confinement is a deliberate
divergence. The basis-dir options, `--delay-updates` and `--dry-run` were
reclassified to ❌ after a differential test reproduced each exact residual
(basis content verification, fixed staging-name collision, and dry-run
would-delete over-report). `--fuzzy` was also reclassified to ❌ (deterministic
heuristic with a 10× size window, not rsync's matcher), but its residual is the
candidate-selection heuristic itself: the final tree is byte-exact by design, so
it is pinned by the `TestFuzzy` threshold suite rather than a byte-level rsync
differential. (Track 5b later found the name heuristic is rsync's own and moved
the row ❌ → ⚠️, leaving only the narrower delta size window; see entry 15.) The parity-review pass then moved `--delete-delay` to ⚠️ (the
plan-time `--max-delete` charge and non-recursive deferred removal differ from
rsync when a snapshotted entry fails removal). Differential-gate allowlist
entries `min_size`/`empty_dirs_recursive`/`dirs_plain` were removed. The
integrated stats+options+fs branch stands at **111 ✅ / 13 ⚠️ / 33 ❌ = 157**;
full suite + ASan + clang-format + cppcheck clean.
11. **No-wire parity track 1** on `feat/parity-2.28` (no protocol change):
`-n --delete` now sends the same filter-excluded + size-pruned protected
prefixes and synchronized-directory scope as a real run (dry-run would-delete
matches rsync for source-derived protections; the destination-only exclude
residual was later closed by track 4a, readdir ordering remains);
`--delete-delay` now charges
`--max-delete` on actual removals and re-scans a queued directory at commit
to remove content created after the plan, with an independent deferred-list
cap (only partial-delete ordering remains); and `--info=name2` emits `NAME is
uptodate` plus the leading `./` root name line for `--info=name` (only the
root-line trigger condition and receiver-side `skip` wording remain). Matrix
now **111 ✅ / 14 ⚠️ / 32 ❌ = 157**; differential + unit tests added in
`test_features.py`, `test_option_parity.py`, `test_delete_plan.c`,
`test_delete_delay_budget_parity.py`, `test_delete_timing_parity.py`.
12. **No-wire parity track 2b** on `feat/parity-2.28` (no protocol change):
`--progress`/`-P`/`--info=progress` (when not `--quiet`) now run an opt-in
paths-only metadata pre-count (no file reads/hashing) that supplies rsync's
full file-list total for the `to-chk` denominator and the directory names,
and emits per-directory/symlink/special name lines, in both the sequential
and `--threads` paths. `--delete-during`/`--delete-delay` reuse their
keep-set pre-scan instead of a second walk; non-progress runs are
unaffected. Differential tests (`progress`/`progress_threads` over a new
`multidir` corpus) match rsync's name set and `to-chk` denominator on a
fresh transfer, and the single-file byte-identical test still passes;
emission order (rsync's sorted depth-first vs FastSync's readdir/BFS stream)
plus re-run over-naming (unconditional `./`, ancestor dirs named with a
transferred child, and no quick-check for symlinks/empty dirs) remain the
caveats, so the row stays ⚠️ and the matrix is unchanged at
**111 ✅ / 14 ⚠️ / 32 ❌ = 157**.
13. **Wire parity track 4a** on `feat/parity-2.28` (`PROTOCOL_VERSION` stays
`2.28.0`): the receiver now has a delete-time filter engine. The sender
compiles its root-level selection rules exactly as the scanner does
(`filter_base_build`) and streams them as one bounded, self-describing
config-frame block (action, sides, anchored, dir-only, negate, owner,
pattern; bounded rule count and pattern bytes, unknown action/sides is a
protocol error). The receiver reconstructs `protect_rules` and applies them
first-match-wins to each extraneous destination path in every delete timing
(the whole-tree commit walker, the `--delete-during`/`--delete-delay`
per-directory plans, and the `-n` would-delete enumeration), so a
`P *.log` rule protects a destination-only `extra.log` like rsync (with
`risk` cancelling); the sender-derived protected-prefix behavior is
preserved when no rules are sent and `--delete-excluded` semantics are
unchanged. Per-directory merge (`:`/`.`) receiver re-derivation remains the
residual. `TestFilterProtect` (real + dry-run) plus differential cases
`filter_protect`, `filter_protect_during`, `filter_protect_delay` added and
the `--filter=RULE` row moves ❌ → ✅: matrix now
**115 ✅ / 11 ⚠️ / 31 ❌ = 157**; unit tests, the three named integration
files, clang-format and cppcheck clean.
14. **Wire parity track 5a** on `feat/parity-2.28` (`PROTOCOL_VERSION` stays
`2.28.0` by project decision): the three basis-dir options now default to
rsync's metadata quick-check (equal size + equal mtime, or size alone under
`--size-only`; `-I` disables matching) instead of FastSync's historical
xxHash64 content equality, so a same-size/different-content basis is trusted
exactly as rsync trusts it. A new FastSync-only, long-only `--verify-basis`
flag restores the strict whole-file content equality; its bool is appended to
the basis block of the config frame (golden wire frame 882 → 886 bytes).
`--verify-basis` streams the confined basis descriptor to hash it, and a
basis hit is no longer capped at the 256 MiB whole-file payload bound:
`--copy-dest` streams the basis through a bounded buffer and `--link-dest`'s
copy fallback streams from the basis, so an over-limit hit materializes (a
basis MISS still falls back to the normal transfer and keeps its own bound).
A `--copy-dest` hit re-applies the SOURCE attributes (the sender transmits
the source metadata with the basis check frame), matching rsync's
"copy then fix attributes"; a `--link-dest` success keeps the shared inode's
attributes (writing through it would mutate the basis). Differential cases
`copy_dest` and `verify_basis` added; `test_basis_dir_size_only_content_residual`
converted to a passing parity assertion; `TestBasisDestDirs` updated for the
new default + `--verify-basis`; unit tests cover the quick-check/verify
decision and the same-size/different-content handshake. The
`--compare-dest`/`--copy-dest`/`--link-dest` rows move ❌ → ⚠️ (relative-DIR
resolution base and over-limit MISS refusal): matrix now
**116 ✅ / 13 ⚠️ / 28 ❌ = 157**.
15. **No-wire parity track 5b** on `feat/parity-2.28` (`PROTOCOL_VERSION` stays
`2.28.0` by project decision): `-y`/`--fuzzy` reclassified ❌ → ⚠️. A probe
against real rsync 3.4.1 (pinned `-B8192`, repeated-content 64 KiB corpus)
showed the name heuristic is already rsync's (`util1.c fuzzy_distance` /
`find_filename_suffix` + the exact size+mtime pass) and the output is always
byte-exact; the only residual is candidate ELIGIBILITY, because FastSync's
`delta_should_attempt` gate caps the size ratio at 10× and requires both
files ≥ 16 KiB while rsync will reuse a basis from 0.25× to 10000× and below
16 KiB. The choice is observable only as `--stats` bandwidth counters. Added
differential case `fuzzy_basis` (same-suffix sibling, one name edit,
identical content, block size pinned) asserting tree **and** normalized
`--stats` parity where the choices coincide, plus `TestFuzzy` pinning the
window boundary on both sides (>10× and <16 KiB siblings declined by
FastSync while rsync uses them, both trees byte-identical). Matrix now
**116 ✅ / 14 ⚠️ / 27 ❌ = 157**.
16. **Lockstep delete-default track 6** on `feat/parity-2.28` (`PROTOCOL_VERSION`
stays `2.28.0`): plain `--delete` now defaults to rsync's delete-during
(`--del`) timing, normalized on the client onto the existing `delete_during`
wire bool. The old late whole-tree commit is opt-in via `--delete-after` or
the FastSync-only long `--delete-commit` (identical `delete_after` timing).
`-d/--dirs` still falls back to the end commit, `--delay-updates` still
deletes before publication, and `--files-from`/`-R` scope is unchanged. The
`STATUS_DELETE_PLAN` frame gained a one-int `apply` flag so the per-run
config block (including `--delete-missing-args` exact paths) is always
transmitted, on a config-only carrier when the scope allows no directory
plan — fixing a latent bug with a file-only `--files-from` list. Differential
cases `delete`/`delete_commit`/`filter_protect_after` plus the extended
`test_delete_timing_parity.py` (plain `--delete` mid-abort removes reached
extras, `--delete-commit` defers) pass; full `-m "not setpriv"` suite,
clang-format and cppcheck clean. Matrix unchanged at
**116 ✅ / 14 ⚠️ / 27 ❌ = 157** (the `--delete`/`--delete-during` rows stay
⚠️ for the abort boundary; `--delete-after` stays ✅).
## Next steps
1. **Merge PR #284** (`dev` -> `main`) once reviewed (protected branch).
+49 -30
View File
@@ -28,7 +28,7 @@ FastSync uses a producer-consumer transfer pipeline and can combine several
optimizations for large or high-latency transfers:
- Multithreaded scanning, loading, and sending.
- Streaming zstd compression with levels 1 through 22.
- Streaming compression (zstd by default, plus lz4/zlib/zlibx) with levels 1 through 22.
- Configurable file chunking and compact chunk serialization.
- `sendfile()` zero-copy transfers over TCP.
- Batched incremental checks to reduce round trips.
@@ -54,7 +54,8 @@ replacement for every rsync feature or protocol mode.
- Dry runs (server-contacting since protocol 2.21.0 for server-routed targets),
excludes, includes, size filters, backups, statistics, and bandwidth
limiting.
- Incremental size/mtime checks and optional xxHash64 content checks.
- Incremental size/mtime checks and optional content checks (`xxh128` by
default, selectable with `--checksum-choice`).
- FastSync-native delta transfer for changed files.
- Optional mode and timestamp preservation.
- Delete manifests with server-side delete authorization.
@@ -111,9 +112,19 @@ matrix is classified as parity, caveat, or divergent in
- Short-option clustering (`-av`, `-aAX`, `-rlpt`) and attached values
(`-B1000`, `-essh`, `-MOPT`, `--opt=value`) are accepted, matching rsync.
- `-r`, `-b`, `-L`, and `-B` are parsed with the rsync short names.
- `--stats` prints the counters FastSync can observe locally; receiver-only
counters (matched data, file-list bytes, deleted count) are reported as 0, and
`--progress` is an aggregate line rather than a per-file block.
- `--stats` prints the counters FastSync can observe plus the receiver-only
counters reported over the wire (`Matched data`, deleted files, and the
created/literal counters); `Number of files` and `Number of created files`
carry rsync's per-type breakdown. `--progress` prints rsync-style per-file
blocks including the leading `./` line, and (when progress is requested) a
paths-only pre-count supplies rsync's `to-chk` denominator.
- Codecs match rsync 3.4.1: `zstd`/`lz4`/`zlib`/`zlibx` compression and
`xxh128`/`xxh3`/`xxh64`/`md5`/`md4`/`sha1`/`none` checksums. `auto` honors
`RSYNC_COMPRESS_LIST`/`RSYNC_CHECKSUM_LIST` and otherwise follows rsync's
compiled-in order. An omitted `--compress-level` uses the codec's rsync
default (zstd 3, zlib/zlibx 6, lz4 ignored); `zlib`/`zlibx` share the
literal-only zlib path (rsync's zlibx semantics), and the transfer checksum is
not separately selectable.
The detailed flag matrix is maintained in
[`RSYNC_COMPAT.md`](RSYNC_COMPAT.md). It reports each row as **parity**,
@@ -137,16 +148,16 @@ This produces `./build/client` and `./build/server`. `compile_commands.json` is
|----------|-------------|
| Positional | `<source> <dest>` — automatic SSH detection if dest contains `:` |
| `-c, --checksum` | Verify content by checksum instead of size+mtime (implies the incremental checksum quick-check) |
| `--checksum-choice <alg>` | Whole-file checksum algorithm: `xxh64`/`xxhash` (default), `xxh3`, `xxh128`, `md5`, or `auto`; `md4`/`sha1`/`none` are rejected by name |
| `-z, --compress [level]` | Enable streaming zstd compression (level 1–22, default 5) |
| `--compress-choice <alg>` | Compression algorithm: `zstd` (default), `none`, or `auto`; `lz4`/`zlib`/`zlibx` are rejected by name |
| `--checksum-choice <alg>` | Whole-file checksum algorithm: `xxh128` (default), `xxh3`, `xxh64`/`xxhash`, `md5`, `md4`, `sha1`, `none`, or `auto` (plus rsync's two-name `transfer,pre-transfer` form) |
| `-z, --compress [level]` | Enable streaming compression (default `zstd`; level 1–22, default 5) |
| `--compress-choice <alg>` | Compression algorithm: `zstd` (default), `lz4`, `zlib`, `zlibx`, `none`, or `auto` |
| `--skip-compress <list>` | Skip compression for suffixes (`/`- or `,`-separated); defaults to rsync 3.4.1's built-in suffix list |
| `-a, --archive` | rsync archive mode (`-rlptgoD`): links, perms, times, owner, group, devices and specials; ownership application stays privilege-gated (not compression/multithreading) |
| `-j, --threads[=N]` | Multithreading mode; `N` (1–256) sets the parallel scanner worker count, bare `-j`/`--threads` uses the default |
| `-m` | rsync `--prune-empty-dirs` (short form now rsync-parity) |
| `-r, --recursive` | Recurse into directories (FastSync is always recursive; accepted for rsync compatibility) |
| `-d, --dirs` | Transfer the named directory entries without recursing into their contents; aliases `--old-dirs`/`--old-d` |
| `-R, --relative` | With `--files-from`, preserve each listed entry's relative path below the destination root |
| `-R, --relative` | Use rsync's relative path semantics (including the `/./` cut); with `--files-from`, preserve each listed entry's relative path below the destination root |
| `--chunk-serialization` | Chunk serialization (batch all files per chunk; long form only) |
| `-s` | rsync `--secluded-args` compatibility no-op (remote SSH argv is already injection-safe) |
| `--sendfile` | Sendfile zero-copy. Incompatible with compression / chunk serialization. TCP only. Long form only. |
@@ -196,11 +207,13 @@ This produces `./build/client` and `./build/server`. `compile_commands.json` is
| `--compare-dest <dir>` | Extra comparison basis: unchanged files are not transferred (requires/implies `--incremental`) |
| `--copy-dest <dir>` | Like `--compare-dest`, but copies the unchanged file from DIR into the destination |
| `--link-dest <dir>` | Like `--copy-dest`, but hard-links the unchanged file from DIR (repeatable; earlier DIRs win) |
| `--delete` | Delete files on receiver not present in source (default timing: delete-after, i.e. only after the whole transfer succeeded). Scoped to the synchronized directories, so `--files-from` subsets are safe |
| `--verify-basis` | FastSync-only: require a basis hit (`--compare-dest`/`--copy-dest`/`--link-dest`) to match the source by whole-file digest instead of trusting the size+mtime quick-check (default matches rsync) |
| `--delete` | Delete files on receiver not present in source (default timing: delete-during, matching rsync, so destination space is freed progressively). Scoped to the synchronized directories, so `--files-from` subsets are safe |
| `--delete-before` | Delete extras before the transfer starts (implies `--delete`) |
| `--delete-during`, `--del` | Delete extras once the keep-set is known, before data is applied (implies `--delete`) |
| `--delete-delay` | Delete extras only after a successful transfer (implies `--delete`) |
| `--delete-after` | Explicit delete-after timing (implies `--delete`) |
| `--delete-commit` | FastSync-only: keep the pre-2.28 atomic timing — delete only after the whole transfer succeeded (identical timing to `--delete-after`) |
| `--delete-excluded` | Also delete filter-excluded destination mirrors (size-pruned mirrors stay protected) |
| `--max-delete <n>` | Delete at most n destination entries; the rest are skipped and the run exits 25 (partial), matching rsync |
| `--delay-updates` | Put updated files into place only at the end of the transfer (`--force` is honored at publication) |
@@ -208,9 +221,9 @@ This produces `./build/client` and `./build/server`. `compile_commands.json` is
| `-n, --dry-run` | Report what would be transferred without mutating the destination. Since protocol 2.21.0 a server-routed target contacts the receiver and reports would-transfer based on receiver state; a plain local destination keeps the client-side scan. Never mutates or deletes. |
| `-v, --verbose` | Enable debug logging |
| `-q, --quiet` | Suppress non-error output |
| `--progress` | Show a periodic aggregate transfer line (bytes sent, current rate); not rsync's per-file progress block |
| `--progress` | Show rsync-style per-file progress blocks from the receiver's wire counters (FastSync does not print rsync's leading `./` line) |
| `-P` | Enables partial-transfer mode + progress output; interrupted writes retain the already-written temp for resumption |
| `--stats` | Print transfer statistics at end (bytes, files, timing). Receiver-only counters (matched data, file-list bytes, deleted count) are reported as 0 |
| `--stats` | Print transfer statistics at end (bytes, files, timing), including the receiver-only counters reported over the wire; rsync's per-type `Number of files` breakdown is not reproduced |
| `-i, --itemize-changes` | Print an rsync-style per-file change line |
| `--out-format=FORMAT` | Output format for changed files (`%f %n %l %b %M %%`) |
| `--list-only` | List source files instead of transferring |
@@ -315,10 +328,10 @@ transfer is never aborted.
4. **Network protocol** — status-code-driven exchange with metadata packing,
keep-alive, and abort support.
5. **Incremental check** — the client sends `STATUS_CHECK` + path + size +
mtime and, with `--checksum`, a whole-file content checksum (xxHash64 by
default, or md5 via `--checksum-choice=md5`/`--cc`, seeded by
`--checksum-seed`); the server compares against the destination. Can be
batched via `STATUS_CHECK_BATCH` for reduced round-trips.
mtime and, with `--checksum`, a whole-file content checksum (`xxh128` by
default; selectable via `--checksum-choice`/`--cc`, seeded by
`--checksum-seed`); the server compares against the destination. Can be
batched via `STATUS_CHECK_BATCH` for reduced round-trips.
6. **Bandwidth limiting** — token-bucket algorithm with sleep throttling on
64 KiB write chunks.
7. **Metadata restoration** — mode via `chmod()`/`fchmod()`, times via
@@ -498,9 +511,9 @@ features without changing the meaning of ordinary compatibility options.
| Option | Purpose |
|---|---|
| `-j`, `--threads[=N]` | Enable the multithreaded scanner/loader/sender pipeline. `N` (1–256) sets the parallel scanner worker count; bare `-j`/`--threads` uses the default. |
| `-z [level]`, `--compress [level]` | Enable streaming zstd compression, levels 1-22. |
| `--compress-level <n>` | Set the zstd compression level. |
| `--zc <alg>` | Alias for `--compress-choice`. FastSync supports `zstd`, `none`, and `auto`; `lz4`/`zlib`/`zlibx` are rejected by name. |
| `-z [level]`, `--compress [level]` | Enable streaming compression (default `zstd`), levels 1-22. |
| `--compress-level <n>` | Set the compression level (1-22). Omitted, each codec uses its rsync default: zstd 3, zlib/zlibx 6, lz4 ignored. |
| `--zc <alg>` | Alias for `--compress-choice`. FastSync supports `zstd` (default), `lz4`, `zlib`, `zlibx`, `none`, and `auto`; `zlib`/`zlibx` share the same literal-only zlib path. |
| `--zl <n>` | Alias for `--compress-level`. |
| `--skip-compress <list>` | Skip compression for `/`- or `,`-separated suffixes; defaults to rsync 3.4.1's built-in list. Incompatible with `--chunk-serialization`. |
| `--compress-threads <n>` | Use `n` zstd compression workers. Requires compression and a zstd build with threaded support; the setting affects sender CPU work only. |
@@ -514,8 +527,8 @@ features without changing the meaning of ordinary compatibility options.
| `--server-port <port>` | Select the TCP server port (`--port <port>` and `--port=<port>` are rsync-friendly aliases). |
| `--tls` | Enable TLS for TCP transport. |
| `--bwlimit <KB/s>` | Apply token-bucket bandwidth limiting. |
| `--progress` | Show a periodic aggregate transfer line (throughput; not a per-file block). |
| `--stats` | Print transfer statistics (receiver-only counters are 0). |
| `--progress` | Show rsync-style per-file progress blocks from the receiver's wire counters (FastSync omits rsync's leading `./` line). |
| `--stats` | Print transfer statistics, including the receiver-only counters reported over the wire; rsync's per-type `Number of files` breakdown is not reproduced. |
| `--timeout <seconds>` | Set the socket **and** per-message protocol I/O timeout. Default `0` = disabled (matching rsync); `0` disables it. |
| `--contimeout <seconds>` | Connection timeout (default 60, matching rsync); `0` disables it. |
@@ -552,12 +565,13 @@ remote SSH argv is already built injection-safe.
| `-W, --whole-file` | Transfer changed files without delta processing (`--no-whole-file` clears it). |
| `-B <n>, --block-size <n>` | Delta block size in bytes (alias `--delta-block`). |
| `-d, --dirs` | Transfer the named directory entries without recursing into their contents (aliases `--old-dirs`/`--old-d`). |
| `-R, --relative` | With `--files-from`, preserve each listed entry's relative path below the destination root. |
| `-R, --relative` | Use rsync's relative path semantics (including the `/./` cut); with `--files-from`, preserve each listed entry's relative path below the destination root. |
| `--files-from <file>` | Read the source file list from FILE (paths relative to the source root). |
| `--delay-updates` | Put updated files into place only at the end of the transfer. |
| `--compare-dest <dir>` | Extra comparison basis: unchanged files are not transferred (requires/implies `--incremental`). |
| `--copy-dest <dir>` | Like `--compare-dest`, but copies the unchanged file from DIR into the destination. |
| `--link-dest <dir>` | Like `--copy-dest`, but hard-links the unchanged file from DIR (repeatable; earlier DIRs win). |
| `--verify-basis` | FastSync-only: require a basis hit to match the source by whole-file digest instead of trusting the size+mtime quick-check (default matches rsync). |
| `--preallocate` | Allocate destination file space up front (fail-fast on a full disk). |
| `--append` | Resume a shorter destination by appending only its tail (prefix not verified; requires `--incremental`). |
| `--append-verify` | Like `--append`, but verifies the retained prefix checksum first (falls back to a full transfer on mismatch). |
@@ -565,6 +579,7 @@ remote SSH argv is already built injection-safe.
| `--delete-before` | Delete extras before the transfer starts (implies `--delete`). |
| `--delete-during`, `--del` | Delete extras once the keep-set manifest is known, before data is applied (implies `--delete`; early mode, same engine behaviour as `--delete-before`). |
| `--delete-delay` | Delete extras only after a successful transfer (implies `--delete`; commit mode, same behaviour as `--delete-after`). |
| `--delete-commit` | FastSync-only: atomic delete-after timing (only after the whole transfer succeeded). |
| `--delete-after` | Explicit delete-after timing: delete only after the transfer succeeded (implies `--delete`). |
| `--delete-excluded` | Also delete filter-excluded destination mirrors (size-pruned mirrors stay protected). |
| `--max-delete <n>` | Delete at most n destination entries; the rest are skipped and the run exits 25 (partial), matching rsync. |
@@ -573,6 +588,7 @@ remote SSH argv is already built injection-safe.
| `--include <pattern>` | Include matching paths. Repeatable. |
| `--exclude-from <file>` | Read exclude patterns from a file. |
| `--include-from <file>` | Read include patterns from a file. |
| `-f, --filter=RULE` | Add an rsync-style filter rule (`+`/`-`, `include`/`exclude`, `merge`/`.`, `dir-merge`/`:`, `hide`/`H`, `show`/`S`, `protect`/`P`, `risk`/`R`, `clear`/`!`, and modifiers; repeatable). |
| `--max-size <bytes>` | Skip files larger than the limit. |
| `--min-size <bytes>` | Skip files smaller than the limit. |
| `--max-alloc <SIZE>` | Maximum single allocation (binary units; default 1G; `0` = no local limit). |
@@ -616,7 +632,7 @@ remote SSH argv is already built injection-safe.
| `--super` | Permit the receiver to attempt confined super-user activities (device nodes). |
| `--no-super` | Forbid those super-user activities even when the receiver is root. |
| `-l`, `--links` | Copy symlinks as symlinks; the target is stored verbatim (absolute and `..`-bearing targets included), matching rsync. |
| `-L`, `--copy-links` | Copy symlink referents (a broken referent exits 0). |
| `-L`, `--copy-links` | Copy symlink referents (a broken referent makes the run exit 23, matching rsync). |
| `--safe-links` | Skip symlinks whose target points outside the transfer tree (applied on the sender). |
| `--copy-unsafe-links` | Copy unsafe symlink referents. |
| `--munge-links` | Rewrite stored symlink targets with rsync's `/rsyncd-munged/` marker. |
@@ -634,8 +650,8 @@ remote SSH argv is already built injection-safe.
|---|---|
| `-v`, `--verbose` | Enable debug logging. |
| `-q`, `--quiet` | Suppress non-error output. |
| `--progress` | Show a periodic aggregate transfer line (not a per-file block). |
| `--stats` | Print transfer statistics (receiver-only counters are reported as 0). |
| `--progress` | Show rsync-style per-file progress blocks (not rsync's leading `./` line). |
| `--stats` | Print transfer statistics, including the receiver-only counters reported over the wire. |
| `-i`, `--itemize-changes` | Print an rsync-style per-file change line. |
| `--out-format=FORMAT` | Output format for changed files (`%f %n %l %b %M %%`). |
| `--list-only` | List source files instead of transferring. |
@@ -782,7 +798,7 @@ before the module list, before authentication, and the connecting peer address
## Protocol and Security
FastSync protocol version `2.23.0` is shared by the client and server. The
FastSync protocol version `2.28.0` is shared by the client and server. The
current protocol is sender-driven and includes configuration negotiation,
including the maximum allocation limit, incremental checks, checksums,
manifests, keep-alives, abort handling, per-file remove-source results, and
@@ -851,14 +867,17 @@ The project will reach the drop-in replacement goal in stages:
`-L`/`-B`, short-option clustering (`-av`, `-aAX`, `-rlpt`), and attached
values (`-B1000`, `-essh`, `-MOPT`) all parse.
2. Add differential tests that compare FastSync and rsync contents, metadata,
links, deletes, filters, dry runs, and exit codes.
links, deletes, filters, dry runs, and exit codes — **done** for the
completion wave's scope; the tests live in `tests/integration/` and skip
cleanly when rsync is unavailable.
3. `-a` implements full rsync `-rlptgoD`; under `-p` the source mode is copied
exactly (no masking). Ownership application stays privilege-gated, as in
rsync.
4. Symlink (verbatim storage), sparse-file, metadata, delete-policy (including
`--max-delete` partial + exit 25), and resumable-write semantics are
implemented; remaining work is the documented edge cases, which the
**Rsync-Parity Wave** section of `RSYNC_COMPAT.md` enumerates honestly.
`--max-delete` partial + exit 25, per-directory `--delete-during`/
`--delete-delay`), codecs, and resumable-write semantics are implemented;
remaining work is the documented edge cases, which the **Parity Completion
Wave** section of `RSYNC_COMPAT.md` enumerates honestly.
5. Add rsync remote-shell and daemon protocol interoperability.
6. Keep FastSync performance options as negotiated, optional extensions.
+307 -106
View File
@@ -6,34 +6,73 @@ This document maps rsync's full feature set to FastSync's current implementation
| Status | Count | Description |
|--------|-------|-------------|
| ✅ Parity | 83 | Reproduces rsync's semantics for this option's scope |
| ⚠️ Caveat | 63 | Fully wired and tested, but carries a documented behavioral difference from rsync (named in the row and/or the wave notes) |
| ❌ Divergent | 4 | Rejected, an accepted no-op, or impossible on any portable filesystem call |
| **Total** | **150** | One row per rsync option/feature group; a row may name several spellings |
| ✅ Parity | 116 | Reproduces rsync's semantics for this option's scope |
| ⚠️ Caveat | 14 | Wired and tested, but carries a documented behavioral difference from rsync (named in the row and/or the wave notes) |
| ❌ Divergent | 27 | Rejected, an accepted no-op, deliberately non-rsync (native config/auth/batch, privileged namespaces, safe-subset privilege), or impossible on any portable filesystem call |
| **Total** | **157** | One row per rsync option/feature group; a row may name several spellings |
This matrix reports honest rsync parity, not "implemented" as a synonym for
"parsed". A ✅ row matches rsync for the option's scope. A ⚠️ row is real and
tested but diverges in at least one documented way — FastSync's push-only model,
its own wire protocol, the delete timings that approximate rsync's engine modes,
the safe-subset privilege model (`--super`/`--copy-as`), the stricter
xattr/ACL and temp-dir policies, and the output counters that rsync computes on
the generator side. An ❌ row is either rejected (`--stderr=client`, `--protocol`
with any value but the current one), an accepted no-op (`-s`/`--secluded-args`),
or impossible (`-N`/`--crtimes`). The counts are derived from the rows below;
update them together with the table.
tested but diverges in at least one documented way. An ❌ row is either
rejected (`--protocol` with any value but the current one, `--inc-recursive`),
an accepted no-op (`-s`/`--secluded-args`, `--protect-args`, `--old-args`),
deliberately non-rsync and non-interoperable (the FastSync daemon config/auth,
the batch container, `--fake-super`'s xattr format, `--copy-as` credential
switching), or impossible (`-N`/`--crtimes`). The counts are derived from the
rows below; update them together with the table.
**Recently closed parity gaps (protocol 2.23.0).** The rsync-parity wave wired up
**Option wave (protocol 2.26.0 → 2.27.0).** A differential pass against rsync
3.4.1 over the remaining option caveats. `--bwlimit` now parses rsync's units
exactly and paces like rsync's leaky bucket; `--ignore-errors` reproduces
rsync's default (an I/O error skips deletion unless the flag is set, while the
readable tree still transfers and the run exits 23); the `--info` categories
that map to a FastSync event (`name`, `flist`, `del`, `remove`, `nonreg`,
`progress`) now emit rsync's line format, including real-run `deleting PATH` /
`*deleting` lines carried over a new `report_deletes` wire bool; and two
genuinely non-interoperable residuals are reclassified divergent (`-M` over a
daemon/TCP connection, which FastSync's binary config handshake has no argv
channel for, and receiver-side `protect`/`risk` re-derivation for
destination-only entries, which would need a receiver filter engine; the
latter was later implemented in parity track 4a below, so only `-M` over a
daemon remains divergent). After the
combined `fix/parity-stats` + `fix/parity-options` + `fix/parity-fs` passes (and
the later `fix/parity-review` correction that moved `--delete-delay` to ⚠️) the
matrix is **111 ✅ / 13 ⚠️ / 33 ❌ = 157**.
**Parity track 1 (no-wire, on `feat/parity-2.28`).** Three residual burn-downs that required no protocol change: (1) `-n --delete` now propagates the filter-excluded and size-pruned protected prefixes (and the synchronized-directory scope) into the dry-run keep-set manifest, so its would-delete report matches rsync for source-derived protections and no longer lists the file merely being updated (the destination-only exclude/filter residual was closed by track 4a below; readdir ordering remains, leaving the row ⚠️); (2) `--delete-delay` now charges `--max-delete` on actual removals and re-scans a queued directory at commit to remove content created after the plan, with an independent cap bounding the deferred list and the row's prior caveat closed (ordering of partial removals remains, leaving the row ⚠️); and (3) `--info=name2` now emits `NAME is uptodate` and `--info=name` emits the leading `./` root name line (only the root-line trigger condition and the receiver-side `skip` wording remain, leaving the row ⚠️). The matrix is now **111 ✅ / 14 ⚠️ / 32 ❌ = 157**.
**Parity track 2b (no-wire, on `feat/parity-2.28`).** An opt-in, paths-only metadata pre-count for `--progress`/`-P`/`--info=progress` (only when not `--quiet`) now gives the `to-chk` denominator rsync's full file-list total (every regular file, directory, symlink and special plus the transfer root) and emits the per-directory/symlink/special name lines, in both the sequential and `--threads` paths; `--delete-during`/`--delete-delay` reuse their keep-set pre-scan so no second walk happens, and non-progress runs are unaffected. A fresh multi-directory differential against rsync 3.4.1 matches the name set and the `to-chk` denominator, while a single-file transfer stays byte-identical; the remaining caveats are emission order (rsync's sorted depth-first vs FastSync's readdir/BFS stream, so the `to-chk` numerator and interleaving differ) and re-run/receiver-state-driven over-naming (unconditional `./`, ancestor dirs emitted with a transferred child, and no quick-check for symlinks/empty dirs), keeping the row ⚠️. The matrix is unchanged at **111 ✅ / 14 ⚠️ / 32 ❌ = 157**.
**Parity track 3a (no-wire, on `feat/parity-2.28`).** Three codec refinements, none of which change the wire layout (`PROTOCOL_VERSION` stays 2.28.0): (1) an omitted `--compress-level` now resolves to rsync 3.4.1's per-codec default (zstd 3, zlib/zlibx 6, lz4 ignored) and an explicit level is clamped per codec (zstd 1-22, zlib/zlibx 1-9), verified against `rsync --debug=NSTR1`; (2) `auto` now honors `RSYNC_COMPRESS_LIST`/`RSYNC_CHECKSUM_LIST` (rsync's whitespace-separated preference syntax, unknown names skipped, first supported wins, all-unknown is exit 4) before the compiled-in order, with an explicit `--zc`/`--cc` still winning; and (3) `--compress-choice=zlibx` is reclassified out of the caveat list — FastSync's zlib stream already carries only the literal/delta bytes (rsync's zlibx semantics) and is observably identical to `--zc=zlib`, so the zlib/zlibx aliasing is only an implementation detail. The matrix is now **113 ✅ / 12 ⚠️ / 32 ❌ = 157**.
**Parity track 3b (no-wire, on `feat/parity-2.28`).** `--checksum-choice`/`--cc` is reclassified out of the caveat list: its only documented residual was the delta BLOCK strong checksum (rsync applies the negotiated algorithm there; FastSync keeps a fixed xxHash32, `DeltaBlockSig`). A pre-seeded delta-transfer differential against rsync 3.4.1 (probe: `--no-whole-file -B8192 --stats --out-format=%c|%C %n` for rsync vs `--incremental --delta` for FastSync) shows the choice is **not observable** in the surface the parity gate compares — across `xxh64`/`xxh128`/`xxh3`/`md5`/`md4`/`sha1` and both two-name orders the destination tree is byte-identical, the `Matched data`/`Literal data`/`Total transferred file size` counters are unchanged (and, with the block size pinned, equal to rsync's), the `%c` block-checksum token is invariant, and the exit code is 0. The negotiated algorithm is only visible in `%C`, which renders the whole-file transfer digest and is already byte-identical to rsync. No wire field is added (`PROTOCOL_VERSION` stays 2.28.0). The matrix is now **114 ✅ / 11 ⚠️ / 32 ❌ = 157**.
**Parity track 4a (wire, on `feat/parity-2.28`; `PROTOCOL_VERSION` stays 2.28.0).** The receiver now has a filter engine for deletion: the sender compiles its root-level selection rules exactly as the scanner does (`filter_base_build`, covering `--filter`/`-f`, `--exclude`/`--include`, `-C` and the `protect`/`risk`/`hide`/`show` words) and streams them as one bounded, self-describing block appended to the config frame (per rule: action, sides, anchored, dir-only, negate, owner, pattern; strictly validated with a bounded rule count and total pattern+owner bytes, and an unknown action/sides is a protocol error). The receiver reconstructs `protect_rules` and evaluates them first-match-wins against each extraneous destination path in every delete timing — the whole-tree commit walker (plain `--delete`/`--delete-before`/`--delete-after`), the `--delete-during`/`--delete-delay` per-directory plans, and the `-n` would-delete enumeration — so a `protect`/`P` rule now shields a DESTINATION-ONLY entry that never appeared on the sender, with `risk`/`R` cancelling. The existing sender-derived protected-prefix behavior is preserved when no rules are sent (and for source-derived protections), and `--delete-excluded` semantics are unchanged. Per-directory merge (`:`/`.`) receiver-side re-derivation is the remaining residual: those rules are still enforced only through the sender-derived protected prefixes, so a destination-only entry matching ONLY a per-directory merge rule is not yet shielded (the `-F` row keeps this documented). Differential-tested against rsync 3.4.1: `filter_protect`, `filter_protect_during`, `filter_protect_delay` (`-a --delete[-during|-delay] --filter='P *.log'` over seeded destination-only `.log` extras) plus a FastSync regression test for the dry-run would-delete enumeration (`TestFilterProtect`). The `--filter=RULE` row moves ❌ → ✅. The matrix is now **115 ✅ / 11 ⚠️ / 31 ❌ = 157**.
**Parity track 5a (wire, on `feat/parity-2.28`; `PROTOCOL_VERSION` stays 2.28.0 by project decision).** The three basis-dir options (`--compare-dest`/`--copy-dest`/`--link-dest`) now default to rsync's metadata quick-check instead of FastSync's historical xxHash64 content equality: a basis hit is accepted on equal size plus equal mtime (or size alone under `--size-only`; `-I` disables matching), so a same-size/different-content basis is trusted exactly as rsync trusts it. A new FastSync-only, long-only `--verify-basis` flag restores the stricter whole-file content equality, and its bool is appended to the basis block of the config frame (the golden wire frame grew by one int to 886 bytes; still 2.28.0). `--verify-basis` hashes the basis by streaming its confined descriptor, so an arbitrarily large basis is verified without buffering. Basis materialization is also no longer capped at the 256 MiB whole-file payload bound: a `--copy-dest` hit streams the basis through a bounded buffer, a `--link-dest` copy fallback streams from the basis, and a hit of any size is materialized (a basis MISS still falls back to the normal transfer, which keeps its own bound). A `--copy-dest` hit re-applies the SOURCE attributes (the sender now transmits the source metadata with the basis check frame), matching rsync's "copy then fix attributes"; a `--link-dest` success keeps the shared inode's own attributes exactly as before (writing through the shared inode would mutate the basis). The `--compare-dest`/`--copy-dest`/`--link-dest` rows move ❌ → ⚠️ (residuals: the relative-DIR resolution base and the over-limit MISS refusal). The matrix is now **116 ✅ / 13 ⚠️ / 28 ❌ = 157**.
**Parity track 5b (no-wire, on `feat/parity-2.28`; `PROTOCOL_VERSION` stays 2.28.0 by project decision).** `-y`/`--fuzzy` is reclassified ❌ → ⚠️: the receiver-side similar-file basis is an internal bandwidth optimization (the config `fuzzy` bool over the existing receiver-driven delta handshake, unchanged since 2.9.0), and the transferred tree is byte-exact by design regardless of which basis — or no basis — is chosen. A probe against real rsync 3.4.1 showed the remaining difference is not the name rule (FastSync already ports `util1.c fuzzy_distance`/`find_filename_suffix` plus the exact size+mtime pass) but candidate ELIGIBILITY: rsync will pick a fuzzy basis whose size ratio to the source is unrestricted (empirically from 0.25× to 10000×, and for files as small as 300 B), while FastSync's `delta_should_attempt` gate caps the ratio at 10× and requires both files ≥ 16 KiB, so an out-of-window sibling is declined and the file is sent whole. The choice is observable only as bandwidth (`Matched data`/`Literal data`/`Total transferred file size` in `--stats`); the destination tree and exit code are identical either way. A new differential case (`fuzzy_basis`: same-suffix sibling one name-edit away, content identical, block size pinned to 8192) asserts tree **and** normalized `--stats` parity where the two tools' choices coincide; `TestFuzzy` pins the window boundary on both sides (a >10× and a <16 KiB sibling are declined by FastSync while rsync uses them, both trees byte-identical). No wire field changed. The matrix is now **116 ✅ / 14 ⚠️ / 27 ❌ = 157**.
**Lockstep track 6 (delete default; `PROTOCOL_VERSION` stays 2.28.0).** Plain `--delete` with no explicit timing flag now defaults to rsync's delete-during (`--del`) timing: the client normalizes it onto the existing `delete_during` wire bool in `cli_finalize_config`, so no config-frame field was added, and a tight destination no longer has to hold the whole old+new tree at once (the old atomic commit could hit `ENOSPC`). The old late whole-tree commit is opt-in via `--delete-after` or the FastSync-only long spelling `--delete-commit`, which selects the identical `delete_after` timing (documented equivalence). Precedence is unchanged and order-independent: each timing flag implies `--delete`, at most one timing flag may be given, and a timing flag with `--no-delete` is rejected. `-d/--dirs` still falls back to the end commit; `--delay-updates` still deletes genuine extras before publication (the per-directory skip list protects the staging dir); `--files-from`/`-R` scope is unchanged. The per-directory `STATUS_DELETE_PLAN` frame gained a one-int `apply` flag (still 2.28.0): the one-shot per-run config block (protected prefixes, size-pruned mirrors, `--delete-missing-args` exact paths) is now always sent first on a config-only carrier with `apply=false`, fixing a latent bug where a `--delete-missing-args` run whose `--files-from` list synchronized no directory never transmitted its exact deletions. Differential evidence: `delete` (plain, vs rsync's default), `delete_commit` (FastSync `--delete-commit` vs rsync `--delete-after`), and `filter_protect_after` (whole-tree protect) cases; `TestDeleteTimingFinalStateParity` compares plain `--delete`/`--delete-commit` against rsync on completed runs, and `TestDeleteTimingFailure` proves plain `--delete` removes reached extras on a mid-transfer abort while `--delete-commit` removes nothing. The matrix is unchanged at **116 ✅ / 14 ⚠️ / 27 ❌ = 157** (the `--delete`/`--delete-during` rows stay ⚠️ for the abort boundary; `--delete-after` stays ✅).
**Parity completion wave (protocol 2.23.0 → 2.26.0).** This wave closed the
remaining gaps the rsync-parity wave left open (delete timing, wire counters and
output, codec breadth, general `-R`/`-d`, the full filter grammar, receiver-side
name resolution, absolute basis dirs, and the remaining client quick wins) and
reclassified the inherently non-rsync rows as **divergent** (native daemon
config/auth, the non-interoperable batch container, `--fake-super`'s xattr
format, `-X`'s privileged namespaces, and the safe-subset device/privilege
flags). It moved `PROTOCOL_VERSION` three times (`2.23.0 → 2.24.0` delete
timing, `2.24.0 → 2.25.0` wire stats, `2.25.0 → 2.26.0` codecs). See the
**Parity Completion Wave (protocol 2.26.0)** section near the end for the full
list and the remaining limitations.
**Previous wave — rsync-parity (protocol 2.23.0).** That wave wired up
the short options `-r`, `-b`, `-L`, `-B`; rsync short-option clustering
(`-av`, `-aAX`, `-rlpt`) and attached/inline values (`--opt=value`, `-B1000`,
`-essh`, `-MOPT`); `-c` now implies the checksum quick-check; `--checksum-choice`
accepts `xxh64`/`xxhash`/`xxh3`/`xxh128`/`md5`/`auto` and rejects `md4`/`sha1`/
`none` by name; `--compress-choice` accepts `zstd`/`none`/`auto`; `--checksum-seed=0`
is randomized per transfer; `--skip-compress` uses rsync's default suffix list;
`--timeout`/`--contimeout` match rsync's defaults; deletion gained
`--max-delete` partial semantics with exit 25; symlinks are stored verbatim; and
`--specials` recreates sockets. Every one of those still has an entry below with
its remaining caveats. See the **Rsync-Parity Wave (protocol 2.23.0)** section
near the end for the full list and the known limitations.
`-essh`, `-MOPT`); `-c` now implies the checksum quick-check; and the codec
and choice limits it introduced were broadened by the completion wave.
Every one of those has an entry below with its remaining caveats.
---
@@ -44,11 +83,12 @@ near the end for the full list and the known limitations.
| `-a`, `--archive` | Archive mode is -rlptgoD (rsync includes owner/group) | ✅ Parity | Phase 7 Wave A: real rsync archive. `-a`/`--archive` now implies `--links` + the four per-attribute preserve flags (perms/times/owner/group) + `--devices` + `--specials`, i.e. **`-rlptgoD`**. Owner/group **are** implied, but their application stays privilege-gated exactly like rsync: a receiver that cannot `chown` logs a warning and skips it (see the preserve-attribute split note below). FastSync is always recursive, so no `-r` is needed. It no longer implies compression or multithreading (those moved to `-z`/`-j`). The short-option namespace is now rsync-parity (see the Phase 7 note) |
| `-v`, `--verbose` | Increase verbosity | ✅ Parity | Sets `log_level=DEBUG` |
| `-q`, `--quiet` | Suppress non-error messages | ✅ Parity | Suppresses client output while preserving errors |
| `--help` | Show help | ✅ Parity | Prints usage and exits; `-h` is not accepted |
| `--help` | Show help | ✅ Parity | Prints usage and exits. A lone `-h` with no other transfer arguments also prints help (protocol 2.26.0), matching the rsync idiom; `-h` alongside a transfer keeps its rsync meaning of `--human-readable` (see that row) |
| `-V`, `--version` | Print version | ✅ Parity | |
| `--info=FLAGS` | Fine-grained info verbosity | ⚠️ Caveat | Supports `copy`, `misc`, `skip`, `stats`, `all`, and `none`; explicit flags override `--verbose`, and `none` suppresses info output; unsupported names are rejected |
| `--debug=FLAGS` | Fine-grained debug verbosity | ⚠️ Caveat | `io`, `proto`, `pack`, and `util` are supported; `--debug=help` lists flags; other rsync categories are rejected |
| `--info=FLAGS` | Fine-grained info verbosity | ⚠️ Caveat | Accepts rsync 3.4.1's full `--info` vocabulary — `backup`, `copy`, `del`, `flist`, `misc`, `mount`, `name`, `nonreg`, `progress`, `remove`, `skip`, `stats`, `symsafe`, `all`, `none` — with optional level suffixes (`--info=stats2`), so a valid rsync invocation is never rejected up front. Protocol 2.27.0 wires the categories that map to a real FastSync event, matching rsync's line format: `name` prints the updated entry names (with the ` -> target` link suffix), `flist` prints `sending incremental file list`, `del` prints `deleting PATH` (or `*deleting PATH` under `-i`/`--out-format`) for both dry-run would-delete and real deletions (real runs carry the removed paths over the new `report_deletes` wire bool), `remove` prints `sender removed PATH`, `nonreg` prints `skipping non-regular file "NAME"`, `progress` drives the per-file progress output, and `copy`/`misc`/`skip`/`stats` keep their existing channels. `none` suppresses info output, explicit flags override `--verbose`, and a genuinely unknown name is still rejected by name (matching rsync). **Fixed (no-wire):** `--info=name2` (and higher) also prints rsync's `NAME is uptodate` lines for entries the receiver already has, and `--info=name` emits the leading transfer-root `./` name line before the first transferred entry (the marker rides in the existing `info_level` bitset; differential tests vs rsync 3.4.1). **Caveat:** the root `./` line is emitted before the first transferred name rather than keyed off rsync's root-attribute-change decision, so a pre-existing root that rsync leaves untouched can differ; the categories with no client-observable event stay accepted-but-silent — `symsafe`, `mount`, and `backup` (the backup happens on the receiver, which FastSync's protocol does not echo back); and `skip` maps to FastSync's sender-side skip logging rather than rsync's receiver-side "not creating new file" lines |
| `--debug=FLAGS` | Fine-grained debug verbosity | ⚠️ Caveat | Protocol 2.26.0 accepts rsync 3.4.1's full `--debug` vocabulary with optional level suffixes. FastSync emits for its own channels (`io`, `proto`, `pack`, `util`, plus the aliases `hl`/`owner`); the rsync-only categories (`acl`, `filter`, `send`, ...) are accepted silently. `--debug=help` lists the flags; a genuinely unknown name is rejected by name. **Caveat:** most accepted rsync categories produce no output (e.g. `acl`, `filter`, `send`, `flist`, `del`, `deltasum`, `hash`, `recv`, `time`), so they are accepted for CLI compatibility only |
| `--stderr=MODE` | Change stderr output mode | ❌ Divergent | `errors` (default) and `all` are supported; `client` is rejected with a clear error (`--stderr=client is not supported`) because FastSync has no rsync client-message channel — the rejection itself is the documented behavior (Phase 7 Wave B decision). The modes that exist work; the missing rsync channel cannot be emulated without a wire change |
| `--msgs2stderr`, `--no-msgs2stderr` | Deprecated `--stderr` aliases | ⚠️ Caveat | `--msgs2stderr` maps to `--stderr=all` (supported, matching rsync). `--no-msgs2stderr` is rsync's spelling of `--stderr=client`, which FastSync has no client-message channel for, so it maps to the errors-only default instead of reproducing rsync's client mode. See `--stderr=MODE` |
| `--no-motd` | Suppress daemon MOTD | ✅ Parity | Client-only display switch (Wave C): the daemon still sends the configured `motd file` on a `host::module/path` connection; the client reads and discards the frame without showing it. Without the flag the MOTD is printed to stdout after the config/auth handshake and escaped so control bytes cannot inject terminal sequences |
| `--exclude=PATTERN` | Exclude files matching pattern | ✅ Parity | Glob matching in scanner |
| `--include=PATTERN` | Include files matching pattern | ✅ Parity | Glob matching in scanner |
@@ -58,14 +98,14 @@ near the end for the full list and the known limitations.
| Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------|
| `--stats` | Give transfer stats | ⚠️ Caveat | Prints file/byte counts. **Divergence:** the receiver-only counters rsync derives during its generator pass (matched/unchanged data, file-list bytes, deleted-entry count) are reported as **0** by FastSync, and the byte total counts source bytes actually sent rather than the post-delta/post-compression wire volume. Counts that FastSync can observe locally (files, bytes, timing) are accurate |
| `-h`, `--human-readable` | Human-readable numbers | ✅ Parity | Formats transfer byte and rate counts using rsync's **decimal** (base-1000) units, matching rsync `-h` (e.g. `1.23M`), not binary units |
| `--stats` | Give transfer stats | ⚠️ Caveat | Prints transfer statistics. Protocol 2.25.0 populates the receiver-only counters the sender cannot observe (`Matched data`, `Number of deleted files`) from the receiver's `STATUS_STATS` report; the sender tracks the scanned file list per type so `Number of files` carries rsync's `(reg: X, dir: Y, link: Z, special: W)` breakdown (directories come from the scanner's captured directory list, present for `-a`/`-t`/`-p`), `Number of regular files transferred` excludes symlinks/specials and up-to-date files, `Total file size` includes symlink target lengths, and `Total transferred file size` counts only transferred files. **Protocol 2.28.0 extends `STATUS_STATS`** with receiver-observed `literal_bytes` and the four `created_*` counters: `Number of created files` now carries rsync's `(reg/dir/link/special)` breakdown (the receiver reports which destination entries it newly created, including implicitly-created parent directories below the transfer root) and `Literal data` is exact for a delta transfer (the receiver counts the literal fragments it stored, not the whole source size) — all differential-tested in the sequential and `--threads` paths against rsync 3.4.1 for fresh-create, update and delta shapes. **Remaining divergences:** a recursive scan that preserves no directory attribute (`-r` without `-t`/`-p`) captures no directory entries, so the `dir:` category is then omitted from `Number of files`; rsync's per-type breakdown on `Number of deleted files` is not reproduced; and `Total bytes sent`/`received` are FastSync wire bytes framed differently from rsync's, so they are not numerically comparable |
| `-h`, `--human-readable` | Human-readable numbers | ✅ Parity | Formats transfer byte and rate counts using rsync's **decimal** (base-1000) units, matching rsync `-h` (e.g. `1.23M`), not binary units. **A lone `-h` with no transfer arguments prints help instead** (protocol 2.26.0), matching the rsync idiom; `-h` alongside a transfer remains human-readable |
| `-i`, `--itemize-changes` | Per-file change summary | ✅ Parity | Prints rsync-style `>f+++++++++` lines to stdout only for files actually sent (also under `-j`/`--threads`); unchanged files print nothing, matching single-`-i` behavior |
| `--progress` | Show progress | ⚠️ Caveat | Prints a periodic **aggregate** transfer line (bytes sent and current rate), not rsync's per-file progress block. With `-P` the partial-file retention behavior is fully implemented; only the progress presentation differs |
| `-P` | Same as --partial --progress | ⚠️ Caveat | Phase 7 Wave B: `-P` parses to `--partial` + `--progress`. On a failed/interrupted write the receiver now retains the already-written temp file at the destination path (best-effort rename instead of unlink when configured), so a later `--append`/`--append-verify` run can resume it; `--partial-dir` still stages completed files under the confined partial dir and installs them atomically. The retention never runs when `--partial` is off, when no data was actually written, or under `--ignore-existing`/`--existing` (the destination is not ours to overwrite), and it only ever renames the already-written temp (never a corrupt blend; a failed rename falls back to the normal unlink). See the `-S`/`--sparse` interplay note (a retained sparse temp has full logical size) |
| `--out-format=FORMAT` | Custom output format | ⚠️ Caveat | Per-transfer template on stdout; tokens `%f` `%n` `%l` `%b` `%M` `%%` (`%b` is the source length, always `== %l`; post-compression/delta wire bytes are not counted); unknown escapes preserved |
| `--progress` | Show progress | ⚠️ Caveat | Protocol 2.25.0 prints rsync-style per-file progress blocks (percent, transferred/total bytes, rate, elapsed, `(xfr#N, to-chk=M/T)`) fed by the receiver's `STATUS_STATS`, in both the sequential and `--threads` send paths. FastSync also prints rsync's leading `./` transfer-root line and, when progress is requested (`--progress`/`-P`/`--info=progress`) and not `--quiet`, runs a **paths-only metadata pre-scan** (no file reads, no hashing) that supplies rsync's file-list total `T` for the `to-chk` denominator and the directory names; `--delete-during`/`--delete-delay` reuse their existing keep-set pre-scan instead of walking twice, and non-progress runs are untouched. Per-directory name lines are emitted (trailing `/`), and symlink (` -> target`) and special entries are named too, so a **fresh multi-directory tree's name set and `to-chk` denominator match rsync 3.4.1** (differential test, sequential and `--threads`) and a **single-file transfer's name lines and deterministic frames remain byte-identical** to rsync. **Remaining divergences:** rsync emits entries in sorted depth-first order while FastSync streams them in the scanner's readdir/BFS order, so the interleaving and the `to-chk` numerator differ (the denominator matches); the leading `./` root line is emitted unconditionally rather than keyed off rsync's root-attribute-change decision, and an ancestor directory line is emitted whenever a child transfers (rsync suppresses it when the directory itself is unchanged); on a re-run, entries without a quick-check (symlinks, empty directories) are still named where rsync stays silent; and the rate/ETA are wall-clock dependent |
| `-P` | Same as --partial --progress | ✅ Parity | Parses to `--partial` + `--progress`. The independent `--partial` retention semantics are rsync parity: an interrupted write retains the already-written temp at the destination (best-effort) so a later `--append`/`--append-verify` can resume. Progress presentation is owned by the `--progress` row; there is no separate `-P` divergence |
| `--out-format=FORMAT` | Custom output format | ❌ Divergent | Per-transfer template on stdout; tokens `%f` `%n` `%l` `%b` `%c` `%C` `%i` `%M` `%o` `%U` `%G` `%t` `%%`. `%C` now uses the negotiated transfer algorithm (`--checksum-choice`, default `xxh128`, seed 0) and renders every algorithm exactly like rsync — xxh128 high-then-low, xxh64/xxh3 big-endian, md5/md4/sha1 standard hex, `none` a blank 2-char column — differential-tested across all algorithms. `%f`/`%n`/`%l`/`%i`/`%M`/`%U`/`%G`/`%B` also match. **Reclassified because `%b`/`%c` are protocol-specific and cannot match:** a differential against rsync 3.4.1 shows whole-file `%c = 16` for both, but rsync whole-file `%b = filesize + 27 + transfer-digest-bytes` (39 for a 0-byte file; 43/35/47 for xxh128/xxh64/sha1 on a 12-byte file) while FastSync `%b` counts its own framing; in delta mode rsync `%c = 16 + 6·ceil(filesize/block_size)` (verified at block sizes 512/700/1024/2048) while FastSync counts its own signature handshake, and rsync `%b` is its token stream. FastSync's wire bytes are a different quantity, so exact `%b`/delta-`%c` equality is impossible |
| `--log-file=FILE` | Log to file | ✅ Parity | `log_file` config field |
| `--log-file-format=FMT` | Log format | ✅ Parity | Requires `--log-file`; writes one template line per transferred file using the same token set as `--out-format` (including `%b` `==` source length) |
| `--log-file-format=FMT` | Log format | ✅ Parity | Requires `--log-file`; writes one template line per transferred file using the same token set as `--out-format` (including `%b` as the wire byte count) |
| `--8-bit-output`, `-8` | Leave high-bit chars unescaped | ✅ Parity | Applies to displayed paths and protocol debug output |
| `--list-only` | List files instead of copying | ✅ Parity | `ls -l`-style listing of files that would be transferred; scans the source only, contacts no server, writes nothing; also works with `-n` |
@@ -75,29 +115,30 @@ near the end for the full list and the known limitations.
|------|-------------------|-----------------|-------|
| `--exclude-from=FILE` | Read exclude patterns from file | ✅ Parity | Reads patterns from file |
| `--include-from=FILE` | Read include patterns from file | ✅ Parity | Reads patterns from file |
| `--filter=RULE` | Add file-filtering rule | ⚠️ Caveat | Long option only: rsync's short `-f` conflicts with FastSync sendfile (see FastSync-specific list), so `-f` is not reassigned. Supported subset: `+`/`-` include/exclude, implicit-exclude patterns, `include`/`exclude` word forms, a leading `/` anchor (to the transfer root, or to a `.rsync-filter` file's directory), and a trailing `/` for dir-only rules; first match wins with a default of include inside the filter layer. Filters are an independent layer from `--exclude`/`--include` (an entry must pass both). Rejected with a clear error (no silent no-ops): `merge`/`dir-merge`/`hide`/`show`/`protect`/`risk`/`clear` words, rules that begin with `:`/`.`/`!` (merge/dir-merge/list-clear shorthands), and include/exclude modifiers other than `/` (`! C s r p x`) |
| `--files-from=FILE` | Read source file list from file | ⚠️ Caveat | Entries are paths relative to the source root (leading `./` stripped, `..`/absolute entries rejected at parse time, blank lines ignored; NUL-delimited with `-0`). A listed regular file is transferred; a listed directory transfers its whole subtree (FastSync recursion is always on, unlike rsync's non-recursive default). Non-listed paths and their subtrees are pruned by the scanner. A listed entry that does not exist under the source (and an empty list) is a hard error reported before any transfer, unless `--ignore-missing-args` / `--delete-missing-args` is given (see the Safety & Security rows): those flags downgrade the listed-but-missing case to a skip and, for `--delete-missing-args`, a destination deletion; an empty list stays a hard error in every mode. Listing `.` (whole tree) and empty listed directories are fine. Scalability note: `file_list_affects` is O(list size) per scanned entry, so a very large `--files-from` list against a huge tree is quadratic; lists are typically small enough that this is acceptable, but it is the documented bound. Delete scoping (protocol 2.23.0): the manifest carries the set of synchronized directories, and the extras walk only visits those subtrees, so `--delete` with a `--files-from` subset no longer removes destination paths outside the listed directory subtrees (a data-loss fix matching rsync) |
| `--filter=RULE` | Add file-filtering rule | ✅ Parity | The short `-f` **is** bound to `--filter` (the old FastSync sendfile conflict is gone; sendfile is long-only `--sendfile`), and `-f RULE`, `-f=RULE`, `--filter=RULE` and the two-argument form all parse. Protocol 2.26.0 implements rsync's filter grammar: `+`/`-`, `include`/`exclude`, a leading `/` anchor (to the transfer root or a `.rsync-filter` file's directory), a trailing `/` dir-only rule, and the `merge`/`.`, `dir-merge`/`:`, `hide`/`H`, `show`/`S`, `protect`/`P`, `risk`/`R` and `clear`/`!` words, including the `:`/`.` modifiers. First match wins; the filter layer is independent of `--exclude`/`--include`. **Track 4a (protocol 2.28.0) adds the receiver filter engine:** the sender compiles its root-level rules exactly as the scanner does (`filter_base_build`) and streams them as one bounded, self-describing config-frame block; the receiver reconstructs them and re-applies first-match-wins to every extraneous destination path during deletion, so a `P *.log` rule protects a destination-only `extra.log` (differential `filter_protect`/`filter_protect_during`/`filter_protect_delay` vs rsync 3.4.1, plus the `-n` would-delete enumeration) — matching rsync's dual-sided engine for the command-line rule set. **Remaining residual:** per-directory merge (`:`/`.`, and therefore `-F`) is not yet re-derived on the receiver; a destination-only entry that matches ONLY a per-directory merge rule is still protected only through the sender-derived source-mirror prefixes, not by the received base rule list |
| `--files-from=FILE` | Read source file list from file | ✅ Parity | Entries are paths relative to the source root (leading `./` stripped, `..`/absolute rejected at parse time, blank lines ignored; NUL-delimited with `-0`). A listed regular file is transferred; a listed directory transfers its whole subtree (FastSync recursion is always on). Non-listed paths are pruned by the scanner; the delete manifest is scoped to the listed directory subtrees. A listed entry that does not exist is a hard error unless `--ignore-missing-args`/`--delete-missing-args` is given. **An empty list is a zero-transfer success (exit 0), matching rsync 3.4.1** — the earlier claim that rsync reports "no source files specified" was wrong. Scalability note: `file_list_affects` is O(list size) per scanned entry, so a very large list against a huge tree is quadratic (the documented bound) |
| `-0`, `--from0` | Delimit *-from files with NULs | ✅ Parity | `--files-from` entries become NUL-delimited; the flag may appear before or after `--files-from` on the command line. NUL mode preserves entry bytes exactly (trailing CR/LF are part of the name; only newline mode trims them) |
| `--max-size=SIZE` | Skip files larger than SIZE | ✅ Parity | `max_size` in scanner |
| `--min-size=SIZE` | Skip files smaller than SIZE | ✅ Parity | `min_size` in scanner |
| `-I`, `--ignore-times` | Don't skip files matching size+time | ✅ Parity | `ignore_times` config field (crosses the wire). Disables the size+mtime quick-check in the `--incremental` per-file handshake and the basis-dir quick-match, forcing the file to be transferred rather than skipped as unchanged. Receiver-side policy: `match_by_metadata` (file_receive.c) is bypassed, so the receiver never replies `STATUS_OK` for a matching size+mtime. Requires `--incremental` to have the handshake to act on (rsync does its quick check by default; FastSync's `-I`/`--size-only`/`--modify-window` only take effect under `--incremental`, exactly like they take effect through the basis check) |
| `--size-only` | Skip based on size only | ✅ Parity | With `--incremental`, ignores mtime |
| `--size-only` | Skip based on size only | ✅ Parity | With `--incremental`, ignores mtime. Applies identically to the basis-dir quick-check (track 5a): a same-size basis is a hit on size alone, and under the FastSync-only `--verify-basis` the content digest is still required (size-only drops the mtime leg, never the explicit verify) |
| `-@`, `--modify-window=NUM` | Mod-time comparison accuracy | ✅ Parity | Whole-second tolerance with nanosecond-aware comparisons |
| `--existing` | Skip creating new files on receiver | ✅ Parity | Existing destination files continue through normal update handling |
| `--ignore-existing` | Skip updating existing files | ⚠️ Caveat | `ignore_existing` config field (crosses the wire; receiver-side policy). For a destination entry that already exists, the receiver skips the write: in the regular-file path, existing/delay-updates-staged, hardlink-sibling, and special/device handlers all return `FILE_SAVE_SKIPPED` without overwriting (passed as `no_replace` to the write engine), and `--backup` is disabled for skipped files. Note: it is applied at write time, so an existing dest whose size+mtime differ still has its data (or delta) transmitted before the write is discarded — functionally correct, bandwidth-suboptimal vs rsync, which short-circuits earlier. Like rsync, it does not apply to directories/symlinks (those return before the block). Combines with `-j`/`--threads` and `--delay-updates`. See Phase-4/— notes below |
| `--existing` | Skip creating new files on receiver | ✅ Parity | Existing destination files continue through normal update handling. The rsync man-page alias `--ignore-non-existing` sets the same flag |
| `--ignore-existing` | Skip updating existing files | ✅ Parity | `ignore_existing` config field (crosses the wire; receiver-side policy). Protocol 2.26.0 short-circuits in the per-file check **before any payload**: when the destination entry already exists, the receiver answers the skip during the incremental handshake instead of letting the sender stream data that would be discarded, so an existing 4 MiB destination costs only the config/check frames (verified with a counting proxy, matching rsync). The write-time paths (regular, delay-updates-staged, hardlink-sibling, special/device) still return `FILE_SAVE_SKIPPED` without overwriting, and `--backup` is disabled for skipped files. Like rsync, it does not apply to directories/symlinks. Combines with `-j`/`--threads` and `--delay-updates` |
| `--remove-source-files` | Sender removes regular files after confirmed transfer | ✅ Parity | |
| `-x`, `--one-file-system` | Do not cross filesystem boundaries | ✅ Parity | Sender scanner captures the root device and does not descend into mount-point crossings (`st_dev` differs). **Protocol 2.23.0 matches rsync's entry emission:** the mount-point directory itself is emitted as a payload-less directory entry (so the destination gets an empty directory) while its contents are skipped; previously the crossing subdirectory was dropped entirely |
| `-F` | Add the default `.rsync-filter` rules | ⚠️ Caveat | Reads one filter rule per line from each directory's `.rsync-filter` file during traversal and applies it to that directory's subtree; the current directory's rules are evaluated before its ancestors', so deeper files override shallower ones and per-directory files override the command-line `--filter`/`-C` base by default (matching rsync's first-match-wins precedence); `.rsync-filter` files are never transferred. The rsync `-FF` behavior (also `.cvsignore`) is out of scope; unsupported rule types inside the file abort with a clear error |
| `-F` | Add the default `.rsync-filter` rules | ✅ Parity | Reads one filter rule per line from each directory's `.rsync-filter` file during traversal and applies it to that directory's subtree; the current directory's rules are evaluated before its ancestors', so deeper files override shallower ones and per-directory files override the command-line `--filter`/`-C` base by default (first match wins). **A single `-F` transfers the `.rsync-filter` files themselves, matching rsync; a repeated `-FF` additionally excludes them** (rsync 3.4.1's `-F`/`-FF` are exactly these two rules, with no `.cvsignore` branch). Unsupported/unparseable rules inside a per-directory file fail the scan with a clear error. **Residual (track 4a):** per-directory rules are still enforced receiver-side only through the sender-derived source-mirror protected prefixes; the base-rule receiver filter engine does not carry per-directory rules, so a destination-only entry matching ONLY a `.rsync-filter` rule is not yet shielded from `--delete` |
## 4. Directory Options
| Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------|
| `-r`, `--recursive` | Recurse into directories | ✅ Parity | Default behavior |
| `-R`, `--relative` | Use relative path names | ⚠️ Caveat | Meaningful together with `--files-from` (FastSync's default full-tree scan always mirrors the full source argument path below the destination root, so -R does not change it). With `-R` + `--files-from` each listed entry is transmitted under its bare relative destination path: an entry `sub/x.txt` lands at `<dest>/sub/x.txt` (its leading components preserved) instead of under the `<dest>/<full source path>` mirror. Only the path sent on the wire changes; the client still reads the absolute source path, and the delete manifest derives from the sent (relative) paths so `--delete` and `--remove-source-files` stay consistent in both layouts. Works single-threaded and under `-j`/`--threads` (including chunk serialization) |
| `--no-implied-dirs` | Don't send implied dirs with -R | ⚠️ Caveat | Client-side, meaningful only with `-R` + `--files-from`. rsync would normally create the ancestor directories implied by a listed file so it can be written; with `--no-implied-dirs` a listed file whose parent directory is not itself (or via an ancestor) explicitly listed cannot be placed, and FastSync fails the whole run up front with a clear error (`--no-implied-dirs: cannot place file '...': parent directory '...' is not explicitly listed`). Listing the directory (or an ancestor of it, or the whole tree `.`) permits the file. In every other mode the option has no effect. FastSync has no per-entry skip channel, so the rsync "omit the file" case is surfaced as a hard pre-transfer error |
| `-d`, `--dirs`, `--old-dirs`, `--old-d` | Transfer dirs without recursing | ⚠️ Caveat | `-d <dir>` transmits an explicit directory entry for the source-root directory, so the destination mirror is created empty and nothing is descended into. With `--files-from` exactly the listed items are transferred: a listed directory is created empty (no descent) and a listed file is transferred with its content; the dest layout follows the same -R rules as plain files. A new wire frame (`STATUS_MKDIR`) carries each directory entry — the path and, when `--preserve`/`-a` (metadata mode) is negotiated, the directory's metadata; the receiver creates it with the same confined mkdir-parent semantics as regular writes, in single-threaded and `-j`/`--threads` receivers (chunk serialization carries a per-entry type marker). Directory entries appear in the delete manifest so `--delete` prunes correctly. Directory TIMES are transmitted (the `STATUS_DIR_TIMES` frame carries every traversed source directory's captured times, including `--dirs` entries) and applied by the receiver at the END of the transfer, after all children and the delete/publication phases, so a later child write cannot clobber a directory's mtime (`-O`/`--omit-dir-times` skips this application). FastSync divergences: directory modes/ownership are still not applied (only times are), and empty directories are still never created (a `STATUS_DIR_TIMES` entry is record-only), filter/`--exclude` rules are not re-applied to the listed dirs mode (there is no descent during which they would apply), and `-d` never creates the intermediate directories between the destination root and a listed file beyond the usual on-demand parent creation. Under `--delay-updates` only regular files are staged: directory entries are created immediately, so a delayed run that fails part way can leave the already-created empty directories behind (matching rsync, which also creates directories as it processes the file list and only delays regular-file data) |
| `-R`, `--relative` | Use relative path names | ✅ Parity | Protocol 2.26.0 implements rsync's general `-R` path semantics: without a cut the source argument is mirrored in full below the destination root; a `/./` cut in the source argument (`src/./foo`) makes everything after the cut the destination prefix, so the layout matches rsync's relative reconstruction; and `--files-from` entries land under their bare relative path. The delete manifest derives from the sent (relative) paths and is scoped to the transferred prefix subtree, so `--delete` cannot remove destination content outside that prefix (a blocker fix). Works single-threaded and under `-j`/`--threads` |
| `--no-implied-dirs` | Don't send implied dirs with -R | ✅ Parity | With `-R`, rsync creates the ancestor directories implied by a listed path and, with `--no-implied-dirs`, omits their attributes from the transfer so they keep the destination's own state (or are created with default attributes when absent). Protocol 2.26.0 matches this: without `--files-from` the implied-dir walk applies per-attribute metadata only to explicitly transferred directories, and with `-R --files-from` a listed file whose parent is not itself listed is placed normally — the missing implied parent is created with default attributes (not the source's) and the file transfers with `rc 0`, exactly like rsync 3.4.1 (a differential test verifies the modes and mtimes with and without the flag). Works single-threaded and under `-j`/`--threads` |
| `-d`, `--dirs`, `--old-dirs`, `--old-d` | Transfer dirs without recursing | ✅ Parity | Protocol 2.26.0 implements rsync's one-level `-d` listing for `dir`, `dir/` and `.`: the source's immediate contents are transferred (files with content, directories as explicit entries), matching rsync's destination tree in a differential test. `--dirs --files-from` transfers exactly the listed items — a listed directory is created empty and a listed file with content — under the same `-R` layout rules. A plain recursive scan also recreates empty source directories now: the scanner emits a payload-less directory entry (with metadata) for every traversed directory that produced no transferred or descended child, unless `-m/--prune-empty-dirs` suppresses it or the run is `--files-from`/`--list-only` (a directory emptied by filtering is recreated too, matching rsync). Directory entries cross as `STATUS_MKDIR` and appear in the delete manifest, so `--delete` prunes correctly and an empty listed directory survives; an incoming directory replaces a destination regular file (rsync removes the non-directory and creates the directory), verified differentially. Directory times are applied at the end of the transfer; modes/ownership follow the per-attribute policy. Under `--delay-updates` directories are created immediately while only regular files are staged, exactly as rsync does |
| `--mkpath` | Create missing path components | ✅ Parity | Wire option (client → server). At connection start the server creates the client's destination root directory (and any missing leading components below its own authorized root) when `--mkpath` is set, failing the connection cleanly if it cannot. Without `--mkpath` a destination root that does not exist yet is rejected up front (rsync semantics), so the flag is the only way to transfer into a not-yet-created destination directory. Creation is confined by the same secure mkdir walk as file writes (`O_NOFOLLOW`, no `..`) |
| `--inc-recursive`, `--no-inc-recursive` | Incremental recursion mode | ❌ Divergent | rsync's man-page-only scanning-mode switch (and its short aliases). FastSync always performs a single full recursive scan, so both spellings are rejected as unknown options rather than accepted as a no-op; there is no incremental-recursion engine to toggle. A genuine implementation would be a scan-architecture change with no benefit for FastSync's push model |
## 5. Transfer Modifications
@@ -105,36 +146,39 @@ near the end for the full list and the known limitations.
|------|-------------------|-----------------|-------|
| `-u`, `--update` | Skip files newer on receiver | ✅ Parity | `update` config field (crosses the wire; receiver-side policy, implies metadata transmission). Before writing a regular file, the receiver checks `file_destination_is_newer_secure()` (via `stat_is_newer`, second-then-nanosecond strict `>` on the existing destination) and skips the write when the destination is newer than the source (`FILE_SAVE_SKIPPED`); equal-or-older destination (or a newer source) is transferred normally. Applied at write time on the regular-file, delay-updates-staged, hardlink-sibling, and special/device paths. Only regular destinations can be guarded (the newer-check requires `S_ISREG`), and like the other write-time policies it does not short-circuit the data transfer for a differing-size dest. `--remove-source-files` correctly respects the receiver's skip outcome so a skipped source is not removed |
| `--inplace` | Update files in-place | ✅ Parity | Direct write mode |
| `--append` | Append data to shorter files | ⚠️ Caveat | Tail-only resume. When an existing destination file is SHORTER than the source, the receiver negotiates a resume offset with the sender and only the tail is transferred; the receiver rebuilds the full file (retained prefix + tail) and installs it through the normal atomic store path, so the result is byte-identical to the source whenever the retained prefix matches. Plain `--append` does NOT content-verify that prefix (rsync parity): a destination whose prefix differs from the source is resumed anyway, so the result (wrong prefix + correct tail) is NOT byte-identical and the file is effectively left corrupt — the documented rsync-parity risk (use `--append-verify` when the prefix cannot be trusted). Non-content attributes (permissions/ownership/mtime, via `-M`) are still applied. Requires the per-file `STATUS_CHECK` handshake, so it implies `--incremental`; it takes precedence over block delta for a growing file and falls back to delta/full when the destination is not shorter. Incompatible with `-s` (chunk serialization) and `--whole-file` (both rejected up front so the mode never silently degrades to a full transfer). Combines with `--inplace`, `--partial`/`--partial-dir`, and `--delay-updates` (the reconstructed full file flows through those paths unchanged). Divergence: rsync appends in place; FastSync reconstructs and atomically installs, so an interrupted or failed resume never leaves a half-written file at the destination (no corruption window), and `--append` is thus safe to use with the normal atomic path — not only with in-place writes |
| `--append-verify` | Append with old-data checksum | ⚠️ Caveat | Like `--append`, but the retained prefix IS verified before resuming: the sender transmits the source prefix checksum and the receiver compares it to the xxHash64 of the retained destination prefix; on a match only the tail is transferred, on a MISMATCH the run falls back to a clean full transfer so the result is always a byte-identical source copy (never a corrupt prefix+tail blend). Wire/protocol: the append handshake adds `STATUS_APPEND` / `STATUS_APPEND_SIG` / `STATUS_APPEND_OK` / `STATUS_APPEND_DATA` frames and `PROTOCOL_VERSION` was bumped **2.9.0 → 2.10.0** (peers must match, and both must be 2.10.0 or the run fails the version check). Same implications/incompatibilities as `--append`; when both spellings are given `--append-verify` wins (the safer semantics). See the Phase-3 append notes below |
| `--append` | Append data to shorter files | ✅ Parity | Tail-only resume: when an existing destination file is shorter than the source, the receiver negotiates a resume offset and only the tail is transferred; the receiver rebuilds the full file (retained prefix + tail) and installs it atomically. Differential tests confirm the result is byte-identical to rsync both when the retained prefix matches and (for plain `--append`, which does not verify the prefix) when it differs. FastSync reconstructs and atomically installs rather than appending in place — a crash-safety superset (an interrupted resume never leaves a half-written file) with identical normal-run behavior |
| `--append-verify` | Append with old-data checksum | ✅ Parity | Like `--append`, but the retained prefix is verified: the sender transmits the source prefix checksum, the receiver compares it to the xxHash64 of the retained destination prefix, and on a mismatch the run falls back to a clean full transfer (always byte-identical to the source). Differential tests match rsync for both a matching and a mismatching prefix. Same atomic-install crash-safety superset as `--append` |
| `-W`, `--whole-file` | Copy whole file (no delta) | ✅ Parity | `whole_file` config field. Forces a full (whole-file) copy, disabling the block-level delta machinery: the sender only sends `STATUS_NEXT` + full data (client_send.c) and the receiver never requests a delta signature/reconstruction — the receiver's `try_delta = use_delta && !whole_file && ...` short-circuits. `whole_file` crosses the wire folded into `use_delta` (the wire carries `use_delta && !whole_file`), so no separate field/bump is needed. Delta is opt-in (`--delta` needs `--incremental`); `-W` additionally makes `--fuzzy` inert (no similar-file delta basis). `--append`/`--append-verify` are incompatible with `-W` and rejected up front (both sides). See the delta/append notes below |
| `--no-whole-file` | Negate `-W`/`--whole-file` | ✅ Parity | rsync spelling that clears `--whole-file`, re-enabling the delta path where `--delta`/`--incremental` are active. Accepted as a boolean negation of `-W` |
| `--block-size=SIZE` | Force checksum block-size | ✅ Parity | Phase 7 Wave B: `--block-size` is an alias for `--delta-block`; both set `config->delta_block_size` (default `DELTA_BLOCK_SIZE_DEFAULT`, bounds `DELTA_BLOCK_SIZE_MIN..MAX`, out-of-range values are rejected with the default kept). The value is genuinely honored by the delta engine end-to-end: `delta_signature_create_seeded(old, size, config->delta_block_size, seed)` on the sender and receiver, `delta_apply(old, ...)` with the same size, so a non-default block size changes the block count of every signature the harnesses exchange (verified by unit + integration tests) |
## 6. Destination Handling
| Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------|
| `-n`, `--dry-run` | Trial run with no changes | ⚠️ Caveat | Server-contacting since protocol 2.21.0. The final routing predicate is `dry_run_targets_server()` in `src/client/client_send.c`: any target a real run would reach over the wire selects the server-contacting path — an SSH transport, a daemon `host::module` destination, an explicit `--server-host` or `--server-port`/`--port`, TLS, or a source-bind `--address` — and the client handshakes with the receiver, which runs the normal read-only per-file check and answers `STATUS_DRY_RUN_TRANSFER`/`STATUS_OK` without mutating anything. A plain local destination (none of those) keeps the original client-side manifest that never dials the default `127.0.0.1:8080`. Would-delete reporting for `--delete*` is deferred (dry-run never deletes). |
| `-n`, `--dry-run` | Trial run with no changes | ⚠️ Caveat | Server-contacting since protocol 2.21.0. The routing predicate `dry_run_targets_server()` selects the server-contacting path for any target a real run would reach over the wire (SSH, daemon `host::module`, explicit `--server-host`/`--server-port`, TLS, source-bind `--address`); the client handshakes with the receiver, which runs the normal read-only per-file check and answers `STATUS_DRY_RUN_TRANSFER`/`STATUS_OK` without mutating anything. Protocol 2.25.0 also reports would-delete lines: with `--delete` the receiver's `STATUS_STATS` carries the extras it would have removed and the client prints rsync-style `*deleting` lines (sequential and `--threads`; control bytes escaped). Dry-run never deletes. **Fixed (no-wire):** the dry-run keep-set manifest now carries the same filter-excluded and size-pruned protected prefixes and synchronized-directory scope a real run sends, and the would-be-transferred entries are in the keep-set, so `-n --delete` lists exactly rsync's extras for source-derived protections — the file merely being updated is kept and an excluded/pruned source entry is protected (`test_dry_run_delete_lines_match_rsync`, differential vs rsync 3.4.1). **Track 4a (protocol 2.28.0):** the received receiver-side `protect`/`risk` rules are also applied to the would-delete enumeration, so a destination-only entry matching a `P` rule is no longer reported (or removed in a real run) — matching rsync (`TestFilterProtect::test_protect_dest_only_dry_run_enumeration`). **Remaining caveat:** the would-delete line ordering follows the destination readdir order rather than rsync's reverse-sorted walk (the differential compares the sorted set) |
| `-b`, `--backup` | Make backups of overwritten files | ✅ Parity | Backup before overwrite |
| `--backup-dir=DIR` | Backup directory hierarchy | ✅ Parity | `backup_dir` config field |
| `--suffix=SUFFIX` | Backup suffix (default ~) | ✅ Parity | `suffix` config field |
| `--delay-updates` | Put updated files in place at end | ⚠️ Caveat | Successfully received files are staged under a private 0700 `.fastsync-stage` dir inside the receive root and atomically renamed into their final destinations only after the whole transfer (manifest/delete handling included) succeeds, just before the success/outcome frame is sent. The delete walker deliberately skips the staging dir at the receive root, so `--delete` removes genuine extras but never the staged files (deletion runs before publication; rsync's delete-after ordering is not implemented). `--existing`/`--ignore-existing`/`--update` decide against the final destination path at stage time; `--backup` moves the old file aside at publication, and **`--force` is honored at publication** (protocol 2.23.0): a staged regular file or symlink may replace a destination directory that blocks it. Incompatible with `--inplace` and with `--backup-dir=.fastsync-stage` (the internal staging name is reserved; both are rejected). The staging dir name is fixed, so two simultaneous delayed transfers to the same destination root are serialized with an exclusive advisory lock held for the whole transfer: the second session fails cleanly instead of corrupting the first. Aborting or failing before publication installs nothing and removes the staging tree; a crash between stage and publish leaves staged leftovers that the next delayed run wipes at start (process death releases the lock). A stage→publish failure aborts the transfer (best-effort cleanup of the not-yet-published staged files; already-published files are not rolled back). Works in single-threaded and `-j`/`--threads` modes |
| `-T`, `--temp-dir=DIR` | Create temporary files in DIR | ⚠️ Caveat | `--temp-dir` with the rsync short `-T` (the timeout alias moved to long-only `--timeout`). **Protocol 2.23.0 receiver policy: the scratch dir is confined to the receive root — a relative dir is resolved below it, and an absolute path or one containing `..` is rejected by the receiver** (an absolute/foreign-filesystem scratch dir was the divergence; rsync's standalone mode would follow an absolute `--temp-dir`, while its daemon also confines). Temp copies use a unique name there and are atomically renamed into place. **On `EXDEV` (scratch dir and destination on different filesystems) the receiver falls back to a non-atomic copy instead of aborting the transfer**, matching rsync. `--inplace` and `--partial-dir` writes bypass the scratch dir |
| `--delay-updates` | Put updated files in place at end | ❌ Divergent | Successfully received files are staged under a private 0700 `.fastsync-stage` dir inside the receive root and atomically renamed into their final destinations only after the whole transfer (manifest/delete handling included) succeeds, just before the success/outcome frame is sent. The delete walker deliberately skips the staging dir at the receive root, so `--delete` removes genuine extras but never the staged files (deletion runs before publication; rsync's delete-after ordering is not implemented). `--existing`/`--ignore-existing`/`--update` decide against the final destination path at stage time; `--backup` moves the old file aside at publication, and **`--force` is honored at publication** (protocol 2.23.0): a staged regular file or symlink may replace a destination directory that blocks it. Incompatible with `--inplace` and with `--backup-dir=.fastsync-stage` (the internal staging name is reserved; both are rejected). The staging dir name is fixed, so two simultaneous delayed transfers to the same destination root are serialized with an exclusive advisory lock held for the whole transfer: the second session fails cleanly instead of corrupting the first. Aborting or failing before publication installs nothing and removes the staging tree; a crash between stage and publish leaves staged leftovers that the next delayed run wipes at start (process death releases the lock). A stage→publish failure aborts the transfer (best-effort cleanup of the not-yet-published staged files; already-published files are not rolled back). **Reclassified Divergent (differential evidence):** the staging name is fixed and a delayed run wipes a pre-existing destination tree of that name at start even without `--delete`, whereas rsync uses its own internal temp name and leaves a genuine destination entry named `.fastsync-stage` untouched (`test_delay_updates_staging_name_collision_residual`); deletion also runs before publication while rsync's `--delay-updates` implies `--delete-after`. Works in single-threaded and `-j`/`--threads` modes |
| `-T`, `--temp-dir=DIR` | Create temporary files in DIR | ❌ Divergent | `--temp-dir` with the rsync short `-T` (the timeout alias moved to long-only `--timeout`). A **relative** dir matches rsync exactly: it is resolved below the receive/destination root and must already exist (differentially verified: `rsync -a --temp-dir=scratch src/ dst/` and FastSync produce identical trees and an empty scratch dir). **Reclassified as a deliberate divergence because an absolute `--temp-dir` is rejected by the receiver** — it is resolved verbatim by rsync standalone (which will use `/tmp` or any other absolute directory, including one outside the destination), but FastSync's security-reviewed receiver confines the scratch dir to the authorized receive root and rejects any absolute path or one containing `..`. A differential test confirms rsync exits 0 using an absolute scratch dir while FastSync refuses before writing anything into it (the scratch dir stays empty). Its daemon mode also confines relative to the module, but standalone rsync's absolute-temp-dir behavior is not reproduced because it would let a client place receiver scratch files outside the sandbox. Temp copies use a unique name in the scratch dir and are atomically renamed into place; **on `EXDEV` (scratch dir and destination on different filesystems, reachable via a confined relative symlink) the receiver falls back to a non-atomic copy instead of aborting**, matching rsync. `--inplace` and `--partial-dir` writes bypass the scratch dir |
| `--partial` | Keep partially transferred files | ✅ Parity | On a failed/interrupted write the already-written temp file is retained at the destination path (best-effort rename instead of unlink) so a later `--append`/`--append-verify` run can resume it. Retention never runs when no data was actually written or under `--ignore-existing`/`--existing` (the destination is not ours to overwrite), and it only ever renames the already-written temp. A failed rename falls back to the normal unlink |
| `--partial-dir=DIR` | Keep partial files in DIR | ✅ Parity | With `--partial`, the working file is written under the confined partial directory (a relative dir below the receive root) and atomically renamed into place once complete, so an interrupted transfer leaves a resumable copy there and completed transfers do not linger under it. `--inplace` bypasses the partial dir (rsync parity). Requires `--partial` |
## 7. Deletion
| Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------|
| `--delete` | Delete extraneous files from dest | ⚠️ Caveat | `use_delete` config field. Deletion is always derived from the transmitted keep-set manifest of the paths the sender sent/keeps (never from unchecked input), runs through the symlink-safe walker bounded by `MAX_SERVER_DELETE_COUNT`, and skips the `.fastsync-stage` staging dir under `--delay-updates`. FastSync's default timing when no timing flag is given is **delete-after** (extras are removed only once the whole transfer succeeded) — intentionally NOT rsync's `--del`/delete-during default, to preserve FastSync's commit-style safety. By default the destination mirror of a path the source scan pruned (filter/exclude/size rules) is **protected** from deletion — matching rsync, which does not delete excluded files under `--delete`; `--delete-excluded` opts back into deleting them (see below). Deletion is scoped to the **synchronized directories** sent in the manifest (protocol 2.23.0), so a `--files-from` subset no longer deletes untransmitted paths outside the listed directory subtrees. The walk is bounded: a client `--max-delete=NUM` (or the 100000-entry server bound) makes it **partial** — entries up to the bound are removed, the rest are skipped, and the client exits **25** (`RERR_PARTIAL`), matching rsync, rather than failing the transfer. Extraneous destination symlinks are unlinked by name (never followed); a directory still holding a kept/protected entry is left behind rather than failing |
| `--delete-before` | Delete before transfer | ⚠️ Caveat | Implies `--delete`. The sender runs a full source pre-scan (paths only) and transmits the keep-set manifest BEFORE any file data; the receiver validates it, removes every destination entry not listed (bounded walk, staging-dir skip, protected prefixes honored), then acks `STATUS_OK`. The sender only starts streaming after the deletion committed, or aborts if the receiver reported a deletion error. By definition the deletions already happened when a later transfer phase fails — rsync's delete-before is destructive the same way; a subsequent failure does not restore the removed files. Divergence: the keep-set is the pre-scan snapshot, so a file that appears on the source between the pre-scan and the data pass is still transferred but was not protected from deletion |
| `--del`, `--delete-during` | Delete during transfer | ⚠️ Caveat | Both spellings accepted; imply `--delete`. FastSync streams the source in a single directory scan and has no per-directory generator pass, so deletions cannot be interleaved per-directory the way rsync's delete-during does. `--delete-during` therefore selects the same early engine mode as `--delete-before` (manifest transmitted before any data, extras removed and acknowledged before data is applied); observable success/failure behaviour equals `--delete-before`. That is the documented divergence from rsync, where `--del` is the default meaning of `--delete` |
| `--delete-delay` | Find deletions during, delete after | ⚠️ Caveat | Implies `--delete`. Commit-mode timing: extras are removed only after the whole transfer succeeded. rsync's delete-delay records the deletion list during its scan and applies it at the end; FastSync never snapshots the destination while data flows (the keep-set is the transmitted manifest and the destination is listed only at deletion time), so `--delete-delay` is implemented as the same end-of-transfer commit as `--delete-after` with identical safety. That is the documented divergence |
| `--delete-after` | Delete after transfer | ✅ Parity | Implies `--delete`. The delete-after timing is also what plain `--delete` does: the keep-set manifest closes the data stream and the receiver commits the bounded deletion only after the terminal `STATUS_FINISHED` proves the whole transfer (every data frame received and stored) succeeded. A failed or aborted transfer removes nothing |
| `--delete-excluded` | Also delete excluded files | ⚠️ Caveat | `delete_excluded` config field. Under `--delete` FastSync protects (rsync's default) the destination mirror of paths the sender's source scan pruned by the user-selection rules — the `--filter`/`-F`/`-C` layer and the legacy `--exclude`/`--include` layer. The sender transmits those concrete pruned paths as **protected prefixes** in the delete-manifest frame (see the Phase-3 notes below); the walker never descends into or removes them. `--delete-excluded` opts back in: the sender sends an empty protected list, so the excluded destination mirrors become ordinary extras and are removed. **`--max-size`/`--min-size` pruned mirrors are a separate, always-on protection** (protocol 2.23.0, rsync parity): size-pruned source mirrors survive `--delete` even with `--delete-excluded`. Divergences (documented): protection is derived only from what the source scan actually pruned — a stray destination-only file that happens to match an exclude rule is not protected (FastSync never re-applies rules to the destination, keeping deletion sender-derived) |
| `--delete` | Delete extraneous files from dest | ⚠️ Caveat | `use_delete` config field. Deletion is always derived from the keep-set the sender actually transmitted (the per-directory `STATUS_DELETE_PLAN` set by default, or the whole-tree manifest for the late timings — never from unchecked input), runs through the symlink-safe walker bounded by `MAX_SERVER_DELETE_COUNT`, and skips the `.fastsync-stage` staging dir under `--delay-updates`. **Lockstep track 6 (protocol 2.28.0): plain `--delete` with no explicit timing flag now defaults to `--delete-during`**, exactly like rsync's `--del` (the client normalizes it to the existing `delete_during` wire bool; no new wire field). This frees destination space progressively during the transfer and avoids the whole-old+new-tree peak that could `ENOSPC` a tight destination. The old late whole-tree commit is opt-in via `--delete-after` or the FastSync-only long spelling `--delete-commit`. **Caveat (shared with `--delete-during`):** the exact mid-transfer abort boundary can differ from rsync's generator (rsync removes all extras ahead of its throttled sender; FastSync removes only the directories it has reached), `-d/--dirs` falls back to the end-of-transfer commit, and the surviving-set ordering under a partial `--max-delete` can differ — on a completed run the trees agree. By default the destination mirror of a path the source scan pruned (filter/exclude/size rules) is **protected** from deletion — matching rsync, which does not delete excluded files under `--delete`; `--delete-excluded` opts back into deleting them (see below). Deletion is scoped to the **synchronized directories** sent on the wire (protocol 2.23.0), so a `--files-from` subset no longer deletes untransmitted paths outside the listed directory subtrees. The walk is bounded: a client `--max-delete=NUM` (or the 100000-entry server bound) makes it **partial** — entries up to the bound are removed, the rest are skipped, and the client exits **25** (`RERR_PARTIAL`), matching rsync, rather than failing the transfer. Extraneous destination symlinks are unlinked by name (never followed); a directory still holding a kept/protected entry is left behind rather than failing |
| `--delete-before` | Delete before transfer | ⚠️ Caveat | Implies `--delete`. The sender runs a full source pre-scan (paths only) and transmits the keep-set manifest BEFORE any file data; the receiver validates it, removes every destination entry not listed (bounded walk, staging-dir skip, protected prefixes honored), then acks `STATUS_OK`. The sender only starts streaming after the deletion committed, or aborts if the receiver reported a deletion error. By definition the deletions already happened when a later transfer phase fails — rsync's delete-before is destructive the same way; a subsequent failure does not restore the removed files. **Phase-0 divergence (sharpened):** rsync builds the full file list first, so a source file created after that scan is NOT transferred and its destination extra is deleted; FastSync's single-threaded data pass re-scans the source, so the late file IS transferred (a safe superset), while FastSync `--threads` pipelines the scan and matches rsync |
| `--del`, `--delete-during` | Delete during transfer | ⚠️ Caveat | Both spellings accepted; imply `--delete`, and since lockstep track 6 this is also the default timing of a plain `--delete`. **Protocol 2.24.0 implements per-directory delete plans:** as the sender reaches each source directory it streams a `STATUS_DELETE_PLAN` for that directory and the receiver removes that directory's extras (verified with a byte-slicing proxy). The one-shot per-run config block (protected prefixes, size-pruned mirrors, `--delete-missing-args` exact paths) rides a dedicated config-only carrier frame with an `apply=false` flag, so it reaches the receiver even when the scope allows no directory plan at all (a `--files-from` list of bare files synchronizes no directory). **Phase-0 divergence (sharpened):** on a mid-transfer abort rsync's generator (which runs ahead of its throttled sender) has already removed ALL the extras it planned, whereas FastSync has removed only the directories it actually reached; on a completed run both agree. Also `-d`/`--dirs` (no descent) falls back to the end-of-transfer whole-tree commit, and the surviving-set ordering under a partial `--max-delete` can differ. `-R` plans are scoped to the transferred prefix subtree |
| `--delete-delay` | Find deletions during, delete after | ⚠️ Caveat | Implies `--delete`. **Protocol 2.24.0 implements rsync's delete-delay timing:** the sender records each directory's delete plan while scanning and the receiver commits those removals only after the whole transfer succeeds (per plan), so an extra created in the destination after its directory's plan survives while `--delete-after` re-scans and removes it, and a failed transfer removes nothing. The **reported** deleted count advances only on an actual removal. **Fixed (no-wire):** the `--max-delete` budget is now charged on ACTUAL removals (an unlink/rmdir that succeeded), not at plan/snapshot time, and a queued directory is re-scanned at commit and removed recursively (content created after the plan included), matching rsync: a snapshotted entry that fails or is skipped consumes no budget, so a later extra rsync would delete is still deleted. The deferred snapshot list keeps an independent hard cap (`DELETE_PLAN_SERVER_LIMIT`) so it cannot grow without bound now that the budget is no longer charged while scanning. A `--max-delete=2` partial delete reports exactly 2 and exits 25 in both tools, and the refilled-directory differential (late content removed, directory removed, budget shared) now matches rsync 3.4.1 on both sides (`test_delete_delay_budget_parity.py`, `test_delete_timing_parity.py`). Unit tests cover recursive removal, actual-removal charging, and the bounded deferred list. **Caveat:** the exact ordering of which extras are removed first under a partial `--max-delete` can still differ from rsync's generator (the survivor set is compared by count, not identity) |
| `--delete-after` | Delete after transfer | ✅ Parity | Implies `--delete`. Selects the late whole-tree commit: the keep-set manifest closes the data stream and the receiver commits the bounded deletion only after the terminal `STATUS_FINISHED` proves the whole transfer (every data frame received and stored) succeeded. A failed or aborted transfer removes nothing. Since lockstep track 6 a plain `--delete` defaults to delete-during (rsync's `--del`); `--delete-after` — or the FastSync-only `--delete-commit` spelling, which selects the identical timing — is the explicit way to keep the old commit-style behavior |
| `--delete-excluded` | Also delete excluded files | ✅ Parity | `delete_excluded` config field. Under `--delete` FastSync protects (rsync's default) the destination mirror of paths the sender's source scan pruned by the user-selection rules — the `--filter`/`-F`/`-C` layer and the legacy `--exclude`/`--include` layer. The sender transmits those concrete pruned paths as **protected prefixes** in the delete-manifest frame (see the Phase-3 notes below); the walker never descends into or removes them. `--delete-excluded` opts back in: the sender sends an empty protected list, so the excluded destination mirrors become ordinary extras and are removed. **`--max-size`/`--min-size` pruned mirrors are a separate, always-on protection** (protocol 2.23.0, rsync parity): size-pruned source mirrors survive `--delete` even with `--delete-excluded`. Track 4a (protocol 2.28.0) additionally re-applies the received `protect`/`risk` rules on the receiver, so a destination-only entry matching an exclude rule is protected (or left at risk) exactly like rsync; the remaining sender-derived `--delete-excluded` behavior (an unqualified rule becomes sender-only, so its source mirror and matching destination-only extras are deleted) is unchanged |
| `--max-delete=NUM` | Max files to delete | ✅ Parity | `max_delete` config field (default -1 = no client limit; 0 = delete nothing). **Protocol 2.23.0 matches rsync's partial semantics:** the receiver deletes up to NUM entries (regular files, symlinks and empty directories; each directory removal counts as one) and then **stops deleting, skips the rest, and reports the run as partial**. The client prints a "deletions stopped due to `--max-delete` limit" message and exits **25** (rsync's `RERR_PARTIAL`), not a hard failure — the transfer itself succeeded. NUM only applies together with `--delete` (it is inert otherwise, matching rsync). A client NUM below the server hard bound `MAX_SERVER_DELETE_COUNT` (100000) replaces it; a NUM above it never raises that cap. Deleting an entire destination with no limit is still bounded by the server's 100000-entry ceiling. `--delete-missing-args` exact-path deletions and the ordinary extras walk draw from the same budget, matching rsync |
| `--ignore-errors` | Delete even with I/O errors | ⚠️ Caveat | Sender-side, client-only config field. rsync suppresses `--delete` when the transfer had I/O errors; FastSync's equivalent is a source-scan I/O error (an unreadable directory, e.g. EACCES): by default the scan aborts the run so no deletion happens. With `--ignore-errors` the scan continues past the unreadable directory, the readable tree is transferred and the deletion still runs (the mirror of the unreadable directory is treated as an extra). The run still exits non-zero (the error is reported, matching rsync's error status). Divergence: without the flag FastSync aborts the whole run on the scan error, whereas rsync transfers the rest of the tree and merely skips the deletion; both leave the deletion undone |
| `--force` | Force deletion of non-empty dirs | ⚠️ Caveat | `force_delete` receiver config field (crosses the wire). rsync's `--force` lets an incoming non-directory replace a destination directory; FastSync implements exactly that: when a regular file (or symlink) is written to a path that is currently a (possibly non-empty) destination directory, `--force` removes that directory tree first — confined to the receive root and symlink-safe (O_NOFOLLOW fd walk, symlinks removed by name, never followed) — so the install can place the file. **Protocol 2.23.0 honors `--force` on the `--delay-updates` publication path too**, not only the immediate-install path. Without `--force` such a write fails and the run aborts. Gated by the server `--allow-delete` policy (a client cannot use `--force` to remove a destination tree on a server that forbids deletion) |
| `-m`, `--prune-empty-dirs` | Prune empty dir chains | ✅ Parity | `-m`/`--prune-empty-dirs` (Phase 7 Wave A freed the rsync short `-m`; FastSync multithreading is now `-j`/`--threads`). FastSync's recursive transfer records directory times but never CREATES an empty directory (a `STATUS_DIR_TIMES` entry is record-only, and `--dirs` empty entries are pruned by this flag), so empty directories are inherently never transferred (which is rsync's `-m` behavior) and truly-empty destination directory chains are removed by `--delete` regardless of this flag. The flag's additional real effect is on the `--dirs` explicit directory-entry generator: a plain `-d <empty-dir>` run omits the empty source directory's entry, so nothing is created at the destination (no `STATUS_MKDIR`, no `-i`/`--out-format` change line, and an existing empty mirror becomes an extra that `--delete` prunes). Explicitly `--files-from`-listed directories always pass through (documented `--files-from` behavior). A directory that still holds an excluded-but-protected file survives, matching the `--delete-excluded` default |
| `--ignore-errors` | Delete even with I/O errors | ✅ Parity | Sender-side, client-only config field. Matches rsync's semantics exactly: an unreadable source subdirectory is always skipped so the readable tree transfers (the transfer root itself stays fatal), and the run reports rsync's partial-transfer exit **23**. Deletion policy follows rsync: by default an I/O error suppresses deletion (`IO error encountered -- skipping file deletion`), while `--ignore-errors` lets the deletion commit. The decision applies to every timing (`--delete`, `--delete-before`, `--delete-during`, `--delete-delay`, `--delete-after`) in both the sequential and `--threads` send paths. Differential-tested against rsync 3.4.1 with both tools run as an unprivileged user (mode-000 source directory); the reference build's root-only gate still excludes the EACCES differential, but the setpriv differential test exercises it. The piece that stays FastSync-specific is documented under the recursive-empty-directory residual: FastSync never emits an unreadable (or empty) directory entry, so that mirror is an extra that a run with `--ignore-errors` removes, where rsync emits the directory and keeps its mirror |
| `--force` | Force deletion of non-empty dirs | ✅ Parity | `force_delete` receiver config field (crosses the wire). rsync's `--force` lets an incoming non-directory replace a destination directory; FastSync implements exactly that: when a regular file (or symlink) is written to a path that is currently a (possibly non-empty) destination directory, `--force` removes that directory tree first — confined to the receive root and symlink-safe (O_NOFOLLOW fd walk, symlinks removed by name, never followed) — so the install can place the file. **Protocol 2.23.0 honors `--force` on the `--delay-updates` publication path too**, not only the immediate-install path. Without `--force` such a write fails and the run aborts. Gated by the server `--allow-delete` policy (a client cannot use `--force` to remove a destination tree on a server that forbids deletion) |
| `-m`, `--prune-empty-dirs` | Prune empty dir chains | ✅ Parity | `-m`/`--prune-empty-dirs` (Phase 7 Wave A freed the rsync short `-m`; FastSync multithreading is now `-j`/`--threads`). A recursive transfer now recreates empty source directories by default (rsync parity); this flag suppresses that emission, so an empty directory (physically empty, or emptied by filtering) is not created and a true empty-directory chain is removed by `--delete`, matching rsync's `-m`. It also affects the `--dirs` explicit directory-entry generator: a plain `-d <empty-dir>` run omits the empty source directory's entry, so nothing is created at the destination (no `STATUS_MKDIR`, no `-i`/`--out-format` change line, and an existing empty mirror becomes an extra that `--delete` prunes). Explicitly `--files-from`-listed directories always pass through (documented `--files-from` behavior; `--files-from` runs never emit implicit empty directories). A directory that still holds an excluded-but-protected file survives, matching the `--delete-excluded` default |
**Deletion-timing implementation notes (Phase 3):** the delete flags above are
real. Two new config booleans (`delete_during`, `delete_delay`) join the already
@@ -144,8 +188,8 @@ The `STATUS_MANIFEST` frame is count-delimited and position-independent: the
receiver commits the deletion either when the manifest arrives (early modes:
`--delete-before`/`--delete-during`, which additionally acknowledge with
`STATUS_OK` before data flows) or after the terminal `STATUS_FINISHED` proves
the whole transfer succeeded (commit modes: plain `--delete`/`--delete-after`/
`--delete-delay`). Timing is chosen purely from the config, so server policy
the whole transfer succeeded (commit modes: `--delete-after`/`--delete-delay`;
plain `--delete` joined the per-directory plans in track 6). Timing is chosen purely from the config, so server policy
(`--allow-delete` off) still disables deletion without deadlocking the early
manifest ack. `--delete-delay` and `--delete-during` are each implemented as
the closest safe approximation their engine mode allows; the divergences are
@@ -227,7 +271,10 @@ Flag-conflict policy: unlike rsync's last-one-wins behaviour, every deletion
timing flag implies `--delete`, and combining a timing flag with `--no-delete`
(in either argument order) — or more than one timing flag — is rejected as a
configuration error rather than silently resolved. Note the check is
order-independent because it runs over the fully parsed config. The deletion
order-independent because it runs over the fully parsed config. The
FastSync-only `--delete-commit` is a late-timing spelling (it maps onto
`--delete-after`), so it conflicts with any different timing flag exactly like
`--delete-after` does. The deletion
POLICY flags (`--delete-excluded`, `--max-delete`, `--ignore-errors`, `--force`)
do NOT imply `--delete`; without `--delete` they are inert (matching rsync).
@@ -268,26 +315,26 @@ why plain `--append` works on the normal atomic path, not only with `--inplace`.
| `-t`, `--times` | Preserve modification times | ✅ Parity | Real per-attribute flag (`preserve_times`): apply the source mtime independently of the other attributes. `-O/--omit-dir-times` suppresses directories only and `-J/--omit-link-times` suppresses symlinks only; `-U`/`-N` do not imply it. `--preserve`/`-a` imply it, and `--incremental`/`--delta` auto-enable it unless `--no-times`/`--no-preserve` |
| `-E`, `--executability` | Preserve executability | ✅ Parity | Preserves executable permission bits (implies metadata preservation) |
| `--chmod=CHMOD` | Affect file permissions | ✅ Parity | Faithful port of rsync 3.4.1's `parse_chmod`/`tweak_mode`: numeric octal and symbolic `ugo`/`rwx` changes, `D`/`F` directory/file selectors, `X` (execute only on directories or already-executable files), `s`/`t` setuid/setgid/sticky, and append semantics — repeated clauses and repeated `--chmod` options accumulate in order (joined with commas). The changes are applied to the new mode **without sanitization** (matching rsync) and `--chmod` does **not** imply `-p` (rsync parity). Applied to files and directories on the receiver |
| `-A`, `--acls` | Preserve ACLs | ⚠️ Caveat | Implemented on Linux via the POSIX-ACL xattr representation: the sender captures the `system.posix_acl_access` / `system.posix_acl_default` xattrs into the same bounded whitelisted set as `-X`, transmits them per-file, and the receiver re-applies them fd-relative. Setting an ACL the receiver is not permitted to set (non-root on a file it does not own, unsupported filesystem) is logged and skipped, never fatal. libacl is **not** required. Only the `system.posix_acl_*` namespaces plus `user.*` are ever applied; privileged namespaces are never applied (see the Phase-4 xattr/ACL notes below). Implies metadata transmission |
| `-X`, `--xattrs` | Preserve extended attributes | ⚠️ Caveat | Preserves unprivileged `user.*` extended attributes (Linux `listxattr`/`getxattr` on capture, `fsetxattr` on the written destination fd). Both capture (sender) and application (receiver) are restricted to the `user.*` namespace and the two POSIX ACL xattrs, so a client can **never** force a `security.*`/`trusted.*`/privileged attribute onto the destination; the receiver independently re-validates every incoming name against this whitelist and rejects anything else. Payloads are bounded (per-name ≤255B, per-value ≤1MiB, per-file count ≤256 total bytes ≤4MiB) on both ends, and an oversized/malformed frame is a clean protocol rejection (no OOM). Applied fd-relative to the exact written file. Implies metadata transmission. Incompatible with `-s` (chunk serialization), rejected up front (see the notes); a `--link-dest`/`-H` hard-link copy fallback re-applies the attributes so they are not dropped when a link is refused |
| `-A`, `--acls` | Preserve ACLs | ✅ Parity | Implemented on Linux via the POSIX-ACL xattr representation: the sender captures the `system.posix_acl_access` / `system.posix_acl_default` xattrs and the receiver re-applies them fd-relative. A differential test with `setfacl` confirms the complete access and default ACL sets (including `mask`) are identical to rsync's on a directory. libacl is not required; a `fsetxattr` an unprivileged receiver may not perform is logged and skipped, never fatal. Only the `system.posix_acl_*` namespaces plus `user.*` are ever applied; privileged namespaces are never applied. Implies metadata transmission |
| `-X`, `--xattrs` | Preserve extended attributes | ❌ Divergent | Deliberately restricted to unprivileged `user.*` extended attributes plus the two POSIX ACL xattrs; `security.*` (SELinux, capabilities, ...) and `trusted.*` are **never** captured or applied — a client can never force a privileged attribute onto the destination, and the receiver independently re-validates every incoming name against the whitelist. This is a security-policy divergence from rsync, which can preserve the privileged namespaces with the needed privilege; implementing them would defeat FastSync's privilege-escalation guard. `user.*` capture/apply matches rsync in a differential test. Payloads are bounded on both ends. Incompatible with `-s` |
| `-H`, `--hard-links` | Preserve hard links | ✅ Parity | Files on the source that share an inode (`st_dev`+`st_ino`, e.g. a `cp -al` tree) are re-created as hard links to one another on the destination, so duplicate links stay deduplicated and only the first member's data is sent (later members are transmitted as payload-less `STATUS_HARDLINK` frames). The receiver links each sibling to the first member's installed file with an atomic link + rename; on `link()` failure it falls back to a byte-identical local copy of the first member, never a partial/corrupt file. Requires the sequential scan for ordering (the first member is always emitted and installed before any sibling is linked). Works single-threaded and under `-j`/`--threads`, `--inplace`, `--delay-updates` (links staged and published by rename) and `--partial`. Crosses the wire (`preserve_hard_links` bool; `PROTOCOL_VERSION` bumped **2.11.0 → 2.12.0**, peers must match). Incompatible with `-s` (chunk serialization) and `--append`/`--append-verify`, rejected up front with a distinct error. See the Phase-4 hard-links notes below |
| `-D` | Same as --devices --specials | ✅ Parity | Implies `--devices --specials`. `-D` was unassigned in FastSync (verified: no collision), so it is free to imply both device-node and special-file preservation. As of protocol 2.23.0 `--specials` genuinely covers **both FIFOs and unix sockets**, so `-D` covers the full rsync set. See the `--devices`/`--specials` rows and the Phase-4 devices notes below |
| `--devices` | Preserve device files | ⚠️ Caveat | Recreates char/block device nodes on the destination via `mknod` instead of transferring content. Type + rdev are validated strictly (S_IFMT from the transmitted mode; major/minor range-checked, non-negative), and creation is **privilege-gated**: `mknod` needs `CAP_MKNOD`, so a non-root receiver (CI runs via setpriv as non-root) logs a warning and **skips the device entry safely** — the whole transfer never aborts just because the node could not be made. The node is created fd-relative below the receive root (`mknodat` on the confined secure parent), so it can never be placed outside the authorized root, never follows a symlink, and never replaces an existing directory. Only a char/block mode is honored. Crosses the wire (a `STATUS_SPECIAL` frame carries the path + metadata mode + rdev). Divergence: per-entry skip (not a hard error) when the receiver lacks `CAP_MKNOD`, documented in the Phase-4 devices notes |
| `--devices` | Preserve device files | ❌ Divergent | Recreates char/block device nodes with `mknodat` (type + rdev strictly validated, confined fd-relative below the receive root), but only when the receiver has `CAP_MKNOD`: a non-root receiver logs a warning and skips the entry instead of erroring, so a transfer with devices never aborts. Deliberate privilege-model divergence from rsync, which errors when it cannot create the node. `--specials` (FIFOs and unix sockets) is unprivileged and remains parity |
| `--specials` | Preserve special files | ✅ Parity | **FIFO and unix-socket recreation work** (protocol 2.23.0): FIFOs are recreated with `mkfifoat`, and sockets with `mknodat(..., S_IFSOCK)` — the latter is unprivileged on Linux because it materializes the socket *node*, not a live bound socket, so it is a real, assertable behavior under CI (it matches rsync, which also recreates a socket by `mknod`). Node creation is confined below the receive root (fd-relative parent; no `..`, no symlink follow) and type/rdev are validated strictly; a matching existing node is left in place and an unrelated entry is never replaced. Crosses the wire like `--devices` (the `STATUS_SPECIAL` frame). See the Phase-4 devices notes |
| `--copy-devices` | Copy device contents as file | ⚠️ Caveat | Copy a device's CONTENT into an ordinary regular file on the destination instead of recreating the node — non-privileged and safe. FastSync scans a device/FIFO as a regular file: its reported size (`st_size`, typically 0 for char devices and FIFOs) is copied, so a FIFO or a non-readable device becomes an empty (or size-bounded) regular file. The default data path is size-bounded and never blocks (it sends exactly `st_size` bytes, never an unbounded pseudo-device stream); with `--sendfile`, a non-regular source (FIFO/device) is detected from its `stat` mode and falls back to that same buffered read, so `--copy-devices --sendfile` cannot hang either. The run always succeeds and never crashes on such input. **Deliberate, safe divergence from rsync's dd-like unbounded device read.** See the Phase-4 devices notes |
| `--write-devices` | Write to devices as files | ⚠️ Caveat | Write the received data directly into an **existing** device node on the destination instead of creating a regular file. Restricted and best-effort: the destination must already exist and be a char/block device (opened only under the confined receive root, with `O_NOFOLLOW` + `O_NONBLOCK`); a missing, symlinked, FIFO-with-no-reader (`ENXIO`), non-device destination, or any write failure is **skipped with a warning** rather than allowed, so a run can never clobber the system, never blocks on a special-file target, and never aborts on an unusable target. See the Phase-4 devices notes |
| `--copy-devices` | Copy device contents as file | ❌ Divergent | Copies a device/FIFO's reported `st_size` into an ordinary regular file and never reads an unbounded pseudo-device, so `--sendfile` cannot hang and the run always succeeds. Deliberate safe divergence from rsync's dd-like unbounded device read, which can block; the dangerous behavior will not be implemented |
| `--write-devices` | Write to devices as files | ❌ Divergent | Writes only into an existing char/block node under the confined receive root (`O_NOFOLLOW` + `O_NONBLOCK`); a missing, symlinked, FIFO-with-no-reader, non-device, or otherwise unusable destination is skipped with a warning rather than allowed or aborted. Deliberate confinement divergence from rsync's more permissive behavior |
| `-U`, `--atimes` | Preserve access times | ✅ Parity | Captures the source access time (from the scanner's pre-read stat, so it is not clobbered by reading the file for transfer) and transmits it over the wire; the receiver restores it together with the mtime via `futimens`/`utimensat`. Implies metadata transmission (the times travel inside the shared metadata payload), but does not enable ownership application (that stays opt-in via the identity flags). Wire: `atime` fields on the metadata frame + a `preserve_atimes` config boolean; `PROTOCOL_VERSION` bumped **2.11.0 → 2.12.0** |
| `-N`, `--crtimes` | Preserve create times | ❌ Divergent | Birth-times cannot be set by any portable filesystem call (`utimensat`/`futimens` only set atime/mtime), so this row is an explicit **Divergent** entry (Phase 7 Wave B). Capture + transmit stays: `statx(STATX_BTIME)` on Linux records the source birth time as a wire field; the receiver logs a debug note that it cannot be applied and continues — never failing the transfer and never pretending it worked. On platforms without `statx` it parses as a documented no-op (flag accepted; nothing is captured). Implies metadata transmission. Wire: new `crtime` fields + a `preserve_crtimes` config boolean; `PROTOCOL_VERSION` bumped **2.11.0 → 2.12.0** (see the Phase-4 metadata-time notes) |
| `-O`, `--omit-dir-times` | Omit dirs from --times | ✅ Parity | Real modifier now that FastSync preserves directory times. With metadata on, the scanner captures every traversed source directory's mtime (and atime under `-U`) and the sender transmits them in trailing `STATUS_DIR_TIMES` frame(s) **after all file data and the optional delete manifest** (chunked at the receiver's `MAX_MANIFEST_ENTRIES` per-frame cap); a dir-time entry only RECORDS metadata and never creates the directory, so empty source directories stay untransferred. The receiver defers applying them until its delete / `--delay-updates` publication phases have committed, so writing or removing a child never clobbers a parent directory's mtime (rsync applies directory times at the end for exactly this reason). When `-O` is set (the boolean crosses the wire) the receiver does not apply any of them; without `-O` an `-a`/`--preserve` transfer now restores directory times (reversing the old "never preserves dir times" divergence). Wire change: the terminal `STATUS_DIR_TIMES` frame; `PROTOCOL_VERSION` bumped **2.16.0 → 2.17.0** |
| `-O`, `--omit-dir-times` | Omit dirs from --times | ✅ Parity | Real modifier now that FastSync preserves directory times. With metadata on, the scanner captures every traversed source directory's mtime (and atime under `-U`) and the sender transmits them in trailing `STATUS_DIR_TIMES` frame(s) **after all file data and the optional delete manifest** (chunked at the receiver's `MAX_MANIFEST_ENTRIES` per-frame cap); a dir-time entry only RECORDS metadata and never creates the directory (an empty source directory is created by the separate `STATUS_MKDIR` entry the scanner now emits, and `-m/--prune-empty-dirs` suppresses that; the trailing dir-time simply re-applies the metadata). The receiver defers applying them until its delete / `--delay-updates` publication phases have committed, so writing or removing a child never clobbers a parent directory's mtime (rsync applies directory times at the end for exactly this reason). When `-O` is set (the boolean crosses the wire) the receiver does not apply any of them; without `-O` an `-a`/`--preserve` transfer now restores directory times (reversing the old "never preserves dir times" divergence). Wire change: the terminal `STATUS_DIR_TIMES` frame; `PROTOCOL_VERSION` bumped **2.16.0 → 2.17.0** |
| `-J`, `--omit-link-times` | Omit symlinks from --times | ✅ Parity | Real modifier now that FastSync preserves symlink times. Symlink entries already carried their metadata on `STATUS_SYMLINK`; the receiver now applies it with **no-follow primitives only** (`utimensat(..., AT_SYMLINK_NOFOLLOW)`, plus best-effort `fchmodat(..., AT_SYMLINK_NOFOLLOW)` and policy-gated `fchownat(..., AT_SYMLINK_NOFOLLOW)`), so the link itself is stamped without ever dereferencing it, confined fd-relative below the authorized receive root. A symlink has no children, so the times are applied immediately at creation. When `-J` is set (the boolean crosses the wire) the receiver skips the timestamps (mode/ownership are unaffected); without `-J` an `-a`/`-l` transfer restores symlink mtimes. Wire change alongside `-O`: the shared `STATUS_DIR_TIMES` frame; `PROTOCOL_VERSION` bumped **2.16.0 → 2.17.0** |
| `--super` | Receiver attempts super-user activities | ⚠️ Caveat | Phase 7 Wave E: receiver-side **safe-subset + clear-refusal** privilege model, tri-state `super_mode` (auto/on/off). `--super` **permits** the receiver to attempt super-user activities — ownership application and char/block device-node creation — that are already confined fd-relative below the authorized receive root; `--no-super` **forbids** them even when the receiver is root; the default (`auto`) preserves the pre-existing **best-effort** behavior of *attempting* them (not only when already root: an unprivileged attempt is refused by the kernel and skipped per entry, matching FastSync's history). The server additionally accepts an operator-level `--no-super` veto that forces `OFF` for every connection it accepts (so it also refuses any client `--copy-as`/`--super`); a **privileged (root) standalone TCP listener now also defaults to `OFF`** unless the operator opts in with the new server-only `--allow-super` flag (the flag is **rejected with `--stdio`**, whose remote argv is composed by the client and must never defeat the secure default; operators exposing `fastsync-server --stdio` over SSH need a forced command if the default must hold. An unprivileged receiver is unchanged, since the kernel refuses the confined attempts anyway; the `--daemon` path keeps its per-module `client owner = yes` opt-in); the `--fake-super` owner replay and the `--write-devices` write path are gated by the same policy. **FastSync never elevates**: no `setuid`/`seteuid`/`setgid` is ever called, and `--super` never bypasses the confinement floor (`file_open_secure_parent`, `O_NOFOLLOW`, root checks) — it only permits an attempt that is already confined. `--super` does **not** imply `--numeric-ids` and never enables client-chosen ownership on its own: ownership is applied only when an explicit identity policy (`--usermap`/`--groupmap`/`--chown`/`--numeric-ids`/`--copy-as`) or a preserve-source request (`-o`/`-g`, or `-a`/`--archive`) is also given. A non-root receiver given `--super` logs exactly one warning at activation and each confined attempt is then refused by the kernel and skipped per entry (never aborts); `--no-super` suppresses ownership, char/block `mknod`, `--write-devices` and the fake-super owner replay, while unprivileged FIFO creation is unaffected. Wire: one trailing `super_mode` int on the config frame (validated 0..2), sent **before** the `--copy-as` block (fixed order: super int, then copy-as presence int + ids); `PROTOCOL_VERSION` bumped **2.17.0 → 2.18.0**. **Documented divergence from rsync:** rsync's `--super` runs the receiver with elevated privilege; FastSync only permits a confined attempt and never elevates |
| `--fake-super` | Store/recover privileged attrs via xattrs | ⚠️ Caveat | Full record **and replay** (protocol 2.23.0 parity update). The receiver writes the resolved `uid:gid:mode:mtime_sec:mtime_nsec` into a reserved `user.fastsync.stat` xattr on each written file (best-effort, fd-relative), then immediately re-applies the mode and times via `fake_super_restore_fd` (`fchmod` + `futimens`; absent/malformed records are a silent no-op, never fatal). **`--fake-super` never performs a real `chown`**: when an explicit ownership mapping (`--chown`/`--usermap`/`--groupmap`/`--copy-as`) is active the receiver records the *resolved* id, otherwise the source's own id, but the owner leg is always suppressed so recording can never defeat the flag; the record is retained for a later privileged restore. The replayed mode goes through the shared `metadata_mode_for_policy` helper, so under `-p` it is copied exactly (including group/other-write and special bits — strict rsync parity, no masking) and under `-E` it follows the rsync executability rule. Directory ownership and directory xattrs/ACLs are preserved alongside file entries (mode/owner are applied to directories under the same per-attribute policy and `-A`/`-X` carry the directory ACL/xattr block). Implies metadata transmission so the source uid/gid/mode/mtime are available. The recording format diverges from rsync's `user.rsync.%stat%`; no cross-tool conversion is attempted. Both it and `-X`/`-A` are incompatible with `-s` (chunk serialization), rejected up front |
| `--super` | Receiver attempts super-user activities | ❌ Divergent | Safe-subset privilege model. `--super` permits the receiver to attempt already-confined super-user activities (ownership application, char/block device-node creation, `--write-devices`); `--no-super` forbids them even for root; `auto` keeps the historical best-effort attempt. **FastSync never elevates** — no `setuid`/`seteuid`/`setgid` — and `--super` never bypasses the confinement floor, so it diverges from rsync's real elevation. A server `--no-super` veto forces it off for every connection; a privileged standalone listener defaults off without `--allow-super`; daemon modules opt in with `client owner = yes` |
| `--fake-super` | Store/recover privileged attrs via xattrs | ❌ Divergent | Records the resolved `uid:gid:mode:mtime_sec:mtime_nsec` in a reserved `user.fastsync.stat` xattr and immediately replays mode/times fd-relative, but **never performs a real `chown`** (the owner is recorded for a later privileged restore). The on-disk key and format are FastSync-native, not rsync's `user.rsync.%stat%`, so recordings are not interoperable with rsync — the same class as the native auth and batch formats. Implies metadata transmission; incompatible with `-s` |
| `--open-noatime` | Avoid changing access time when opening files | ✅ Parity | Sender-side policy: the sender opens source files with `O_NOATIME` (Linux) when reading them for transfer, so the open/read does NOT bump the source's on-disk access time. Degrades safely when `O_NOATIME` is unavailable (not defined) or refused (`EPERM`, since it needs `CAP_FOWNER` or file ownership): the code falls back to a normal open, so the data always transfers — only the atime-bump is skipped. It does not itself capture/preserve atime; it only avoids modifying it. **Client-only, never crosses the wire.** Exposed as `file_open_for_read()` and applied to both the buffered data path and the sendfile path |
| `--numeric-ids` | Do not map uid/gid by name | ✅ Parity | **A mapping modifier only:** when ownership is being applied it uses the transmitted numeric uid/gid directly, skipping the name lookup. It does **not** request ownership application on its own — combine it with `-o`/`-g`, `-a`, or an explicit map (`--chown`/`--usermap`/`--groupmap`) — and it does not need any metadata flag merely to parse. Ownership is only applied when metadata (hence the source uid/gid) is actually transmitted (see the Phase-4 identity notes) |
| `--usermap=STRING` | Map usernames | ⚠️ Caveat | Opt-in ownership application. rsync subset implemented (protocol 2.23.0): comma-separated `FROM:TO` rules evaluated in order, first match wins. `FROM` accepts a user name (resolved on the SOURCE machine at parse time), an `@N`/bare `N` numeric id, an inclusive `LOW-HIGH` **id range**, `*` (matches any id), or an **empty** field (matches ids with no name on the source). `TO` accepts a name (resolved on the **receiver**), an `@N`/bare `N` id, or `*` (the receiving process's current euid). Rules are carried over the wire as resolved numeric id pairs; the receiver applies a matching rule (else falls back to `--chown`, `--numeric-ids`, then a best-effort name lookup) via an fd-relative `fchown`, including directory entries. Malformed/unresolvable specs are rejected with a clear error, never a silent no-op. Implies metadata preservation so the source uid/gid travel. Only effective when the receiver can actually change ownership (root or membership); otherwise it warns and continues |
| `--groupmap=STRING` | Map group names | ⚠️ Caveat | Same rsync subset and semantics as `--usermap` (names, `@N`/bare `N`, inclusive ranges, `*`, empty-FROM for unnamed ids, receiver-resolved `TO` names) but for the group (gid) side and the group databases. See the Phase-4 identity notes |
| `--chown=USER:GROUP` | Map owner and group | ⚠️ Caveat | Opt-in ownership override applied receiver-side. Forms: `USER:GROUP`, `USER` (owner only), `:GROUP` (group only); a `*` for USER/GROUP means the current/root user or group as appropriate; an `@N`/bare `N` numeric id is accepted. A `:` inside a name may be escaped as `\:`. Equivalent to a trailing `*:*` usermap+groupmap rule (so an explicit `--usermap`/`--groupmap` match wins). **Protocol 2.23.0 makes `--chown` and `--usermap`/`--groupmap` mutually exclusive on the same side: combining them (in either order) is a clear configuration error** (`--usermap conflicts with prior --chown`), matching rsync and never an order-dependent silent winner. Malformed or unresolvable specs are clear parse errors. Implies metadata preservation. Only effective when the receiver has permission to chown; otherwise it warns and continues (rsync parity) |
| `--copy-as=USER[:GROUP]` | Perform the copy as another user/group | ⚠️ Caveat | Safe-subset implementation, an explicit divergence from rsync's **real identity switching**. rsync makes the receiving process actually assume USER/GROUP (setuid/setgid); FastSync's receiver is multithreaded, so a real credential drop would be unsafe and is never attempted — FastSync never calls `setuid`/`seteuid`/`setgid`. Instead the receiver FORCES the ownership of every entry it writes to `copy_as_uid`/`copy_as_gid` through the existing confined, fd-relative identity path (the same `fchown`/`fchownat` mechanism as `--chown`/`--usermap`/`--groupmap`; symlinks use `fchownat(..., AT_SYMLINK_NOFOLLOW)`, and directories — including intermediate parents created implicitly while writing a nested file — and char/block/FIFO nodes are owned no-follow too, so a directory never keeps the receiver's owner while its children get the target owner), with `--copy-as` at the **highest priority** — it beats usermap/groupmap/`--chown`/`--numeric-ids` and the best-effort name lookup. This REQUIRES a privileged (root) receiver: an unprivileged receiver REFUSES the whole transfer up front at the config handshake (`server_module_gate`, running inside `config_receive_with_validate` before the `STATUS_OK` ack) with a clear error and no file data exchanged — never a silent wrong-ownership result. A server running with an operator `--no-super` veto also refuses it; a privileged (root) standalone TCP listener refuses it by default too and only honors it after the operator passes `--allow-super` (the flag is rejected with `--stdio`, where the client-composed remote argv could otherwise defeat the default; a forced command is required if the default must hold), and a **daemon** refuses `--copy-as`, like every other client-chosen-ownership request (`--numeric-ids`/`--chown`/`--usermap`/`--groupmap`/`--fake-super`/explicit `--super`), unless the selected module opts in with `client owner = yes`; without that per-module opt-in a daemon must not honor an arbitrary client-selected owner (a root standalone listener honors these for its single operator-authorized root only when started with `--allow-super`). `--fake-super` interaction: `--copy-as` is authoritative, so the recorded source owner is never replayed over the forced target owner. If the ownership apply still fails with EPERM/EACCES (capability-restricted root, root-squash, read-only mount) the failure is logged at ERROR and the **entry is reported as failed** rather than written with the wrong owner, which fails the transfer (fail-fast) so overall success is never reported with the wrong owner. USER is resolved on the client against the user database (a name, an `@N`/bare `N` numeric id, or `*` meaning the client's current euid); when `:GROUP` is present it is resolved against the group database (`*` meaning the client's egid). **Group-default rule:** when the group is omitted FastSync uses the user's primary gid (`getpwuid(uid)->pw_gid`); a numeric id with no local passwd entry has no primary gid to look up, so `gid` falls back to `uid` (documented divergence). Malformed/empty/unresolvable specs are clear parse errors, never a silent no-op. Never elevates privileges and never bypasses the confined receive root. Implies metadata preservation (the source uid/gid must be transmitted). Wire: a new trailing config-frame block **sent after** the `--super` int (presence int, then the two int32 ids, both validated `>= 0` on receive; the ids are also rejected if they do not fit int32 at CLI parse time); `PROTOCOL_VERSION` bumped **2.17.0 → 2.18.0** |
| `--usermap=STRING` | Map usernames | ✅ Parity | Opt-in ownership application. Comma-separated `FROM:TO` rules evaluated in order, first match wins. `FROM` accepts a source-resolved user name, an `@N`/bare `N` numeric id, an inclusive `LOW-HIGH` id range, `*`, or an empty field (ids with no source name). `TO` accepts a receiver-resolved **name** (protocol 2.26.0 resolves it on the receiving side against the receiver's account database, matching rsync), an `@N`/bare `N` id, or `*` (the receiving process's euid). Rules travel as resolved numeric pairs plus an optional TO name; the receiver applies a matching rule, else falls back to `--chown`, `--numeric-ids`, then a best-effort name lookup, via fd-relative `fchown`. Malformed specs are clear errors. Implies metadata; only effective where the receiver can chown (otherwise a warning) |
| `--groupmap=STRING` | Map group names | ✅ Parity | Same rules and receiver-side `TO`-name resolution as `--usermap`, applied to the group (gid) side |
| `--chown=USER:GROUP` | Map owner and group | ✅ Parity | Opt-in ownership override. Forms `USER:GROUP`, `USER`, `:GROUP`; `*` means the current user/group as appropriate; `@N`/bare `N` ids; a name may escape `:` as `\:`. A name that resolves on the sender is sent as an id; an unresolvable name is carried as a receiver-resolved `TO` name (protocol 2.26.0), matching rsync's receiver-side resolution. Equivalent to a trailing `*:*` usermap+groupmap rule (an explicit map match wins). Conflicts with `--usermap`/`--groupmap` on the same side are a clear configuration error. Implies metadata; a non-root receiver warns and continues (rsync parity) |
| `--copy-as=USER[:GROUP]` | Perform the copy as another user/group | ❌ Divergent | Close-refusal safe subset. FastSync never switches process credentials (its receiver is multithreaded, so a real `setuid`/`setgid` would be unsafe); instead the receiver forces the ownership of every entry it writes to the client-resolved ids through the confined fd-relative identity path. A privileged (root) receiver is required: an unprivileged receiver refuses the whole transfer at the config handshake, before any data, rather than produce wrong ownership. Deliberate divergence from rsync's real identity switching; a daemon refuses it unless the module sets `client owner = yes` |
**Phase-4 metadata-time notes:** `-U/--atimes`, `-N/--crtimes`,
`-O/--omit-dir-times`, `-J/--omit-link-times`, and `--open-noatime` are new.
@@ -527,9 +574,9 @@ warning + skip, never a system-clobbering write or an abort.
| Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------|
| `-l`, `--links` | Copy symlinks as symlinks | ⚠️ Caveat | A symlink is transmitted as a real symlink: its target string crosses the wire (`STATUS_SYMLINK` / chunk entry type) and the receiver creates it with `symlinkat` beneath the receive root, never following the target. **Targets are stored verbatim (protocol 2.23.0), matching rsync `-l`: an absolute target or one containing `..` is copied exactly, and the receiver no longer enforces a containment predicate by default.** `--safe-links` is the sender-side opt-in that drops unsafe targets before transmission; `--trust-sender` does **not** affect symlink targets (it only relaxes the receiver's path-list re-validation). The *placement* path is still hard-confined (`has_path_traversal`, O_NOFOLLOW fd walk), and the link's own mode/times are applied with no-follow primitives. See the Phase-4 symlink-trust notes and the residual-risk note below |
| `-L`, `--copy-links` | Transform symlink to referent | ⚠️ Caveat | Sender-side: every symlink is replaced by its referent's content (`copy_links` config field). A referent that cannot be read, including a broken symlink, is treated as a non-error and the run exits 0 — where rsync exits 23 (`RERR_PARTIAL`). This is the documented status-code divergence |
| `--copy-unsafe-links` | Transform unsafe symlinks | ⚠️ Caveat | Sender-side: only symlinks whose target is unsafe (absolute or escaping via `..`, matching rsync's `unsafe_symlink()` semantics) are dereferenced into their referent; safe links stay symlinks. Same broken-referent exit-0 caveat as `-L` (`copy_unsafe_links` config field) |
| `-l`, `--links` | Copy symlinks as symlinks | ✅ Parity | A symlink is transmitted as a real symlink: its target string crosses the wire (`STATUS_SYMLINK` / chunk entry type) and the receiver creates it with `symlinkat` beneath the receive root, never following the target. **Targets are stored verbatim (protocol 2.23.0), matching rsync `-l`: an absolute target or one containing `..` is copied exactly, and the receiver no longer enforces a containment predicate by default.** `--safe-links` is the sender-side opt-in that drops unsafe targets before transmission; `--trust-sender` does **not** affect symlink targets (it only relaxes the receiver's path-list re-validation). The *placement* path is still hard-confined (`has_path_traversal`, O_NOFOLLOW fd walk), and the link's own mode/times are applied with no-follow primitives. See the Phase-4 symlink-trust notes and the residual-risk note below |
| `-L`, `--copy-links` | Transform symlink to referent | ✅ Parity | Sender-side: every symlink is replaced by its referent's content. A referent that cannot be read, including a broken symlink, makes the run exit 23 (`RERR_PARTIAL`) like rsync while the rest of the tree still transfers, in both the sequential and `--threads` paths (differential test). The transferred tree matches rsync |
| `--copy-unsafe-links` | Transform unsafe symlinks | ✅ Parity | Sender-side: only symlinks whose target is unsafe (absolute or escaping via `..`, matching rsync's `unsafe_symlink()` semantics) are dereferenced into their referent; safe links stay symlinks. A broken unsafe referent makes the run exit 23 like rsync (differential test), while a safe broken symlink is not dereferenced and exits 0 |
| `--safe-links` | Ignore symlinks outside tree | ✅ Parity | Sender-side: a symlink whose target is unsafe is not transmitted at all (skipped), matching rsync's `--safe-links`. Because FastSync applies this while scanning the source, the receiver does not need to repeat it (`safe_links` config field) |
| `--munge-links` | Munge symlinks for safety | ✅ Parity | Sender rewrites each transmitted symlink target with rsync's `/rsyncd-munged/` prefix; the receiver strips the marker (only when the negotiated `munge_links` policy is on, so a source link that genuinely begins with the marker round-trips verbatim) and restores the exact real target. Unlike rsync, FastSync prefixes on the *sender* and un-munges on the receiver, but the wire result and the stored marker match rsync. See the Phase-4 symlink-trust notes |
| `-k`, `--copy-dirlinks` | Transform symlink to dir | ✅ Parity | A symlink whose referent is a directory is dereferenced and recursed as a real directory; a symlink to a regular file stays a symlink. Sender-side only. See the Phase-4 symlink-trust notes |
@@ -608,8 +655,8 @@ targets verbatim, matching rsync.
| Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------|
| `-S`, `--sparse` | Sparse block handling | ✅ Parity | Phase 7 Wave B: real hole preservation with no wire change. The receiver's sparse-aware writer (`write_all_sparse`, next to `write_all` in `src/shared/file.c` and `src/shared/file_store.c`) walks the in-memory file image and emits any all-zero run ≥ 4096 bytes as a hole via `lseek(SEEK_CUR)` (the pre-size `ftruncate` guarantees the offset bookkeeping and logical size), `ftruncate(size)` after the last run pins the final size even with a hole tail. Wired into both the atomic temp+rename store and `--inplace` when `sparse` is set; the non-sparse path is byte-identical to before. **Sparse wins over `--preallocate`** (posix_fallocate is skipped when sparse is set, so the holes are not re-allocated). Interplay note: under `--partial` a retained sparse temp already has the full logical size (trailing content is holes), so `--append`'s "shorter destination" resume does not re-run; the retained file is still valid and a normal re-transfer (or `-W`/delta) repairs it — documented so the combination is never surprising |
| `--preallocate` | Allocate dest files before writing | ⚠️ Caveat | The receiver preallocates the destination file's full expected space before any data is written, so a transfer that would overflow disk fails fast at allocation time (a clean error, not a half-written file) and the file is laid out contiguously, avoiding fragmentation. Crosses the wire (the config frame carries a `preallocate` boolean; `PROTOCOL_VERSION` bumped **2.10.0 → 2.11.0**, peers must match) so the sender knows the receiver will preallocate and the receiver performs it. **Allocation approach:** `posix_fallocate()` is preferred because it reserves *real* disk blocks (true fail-fast on ENOSPC), falling back to plain `ftruncate()` only when the filesystem reports the allocation is unsupported (`EOPNOTSUPP`/`ENOSYS`); `ftruncate` still extends the logical size so the intent degrades gracefully. **Fallback/error semantics:** `EOPNOTSUPP`/`ENOSYS` → clean fallback to `ftruncate` (best-effort, preallocates the logical size and never fails a transfer on filesystems that lack `posix_fallocate`); a genuine allocation failure (`ENOSPC`/`EDQUOT`/`EFBIG`/…) aborts the file/receive with a distinct `preallocate failed ... transfer aborted` error — it does **not** fall back to a normal non-preallocated write, preserving the fail-fast purpose. **Size-known requirement:** preallocation only runs when the final size is already known up front (the normal regular-file case); unknown-length data is skipped (never failed). **Orthogonality:** applies uniformly across the atomic temp+rename store path, `--inplace`, `--partial`/`--partial-dir`, `--delay-updates` (the staged temp file is preallocated before data flows) and the `--link-dest` copy fallback; it neither implies nor conflicts with `-s`, `--append`, or delta. rsync-divergence: rsync signals that `--preallocate` is ignored with `--sparse`; FastSync gives **sparse precedence** — when both are set, `posix_fallocate` is skipped so the holes the sparse writer creates are not re-allocated (the `ftruncate` presize sizing stays), matching the intent of "sparse wins". See the Phase-4 preallocate notes below |
| `-S`, `--sparse` | Sparse block handling | ✅ Parity | Phase 7 Wave B: real hole preservation with no wire change. The receiver's sparse-aware writer (`write_all_sparse`, next to `write_all` in `src/shared/file.c` and `src/shared/file_store.c`) walks the in-memory file image and emits any all-zero run ≥ 4096 bytes as a hole via `lseek(SEEK_CUR)` (the pre-size `ftruncate` guarantees the offset bookkeeping and logical size), `ftruncate(size)` after the last run pins the final size even with a hole tail. Wired into both the atomic temp+rename store and `--inplace` when `sparse` is set; the non-sparse path is byte-identical to before. **`--preallocate` wins over `--sparse`** (protocol 2.26.0: the allocation still runs when both are set, so the sparse writer's seeks do not re-hole the reserved blocks, matching rsync's observed `st_blocks`). Interplay note: under `--partial` a retained sparse temp already has the full logical size (trailing content is holes), so `--append`'s "shorter destination" resume does not re-run; the retained file is still valid and a normal re-transfer (or `-W`/delta) repairs it — documented so the combination is never surprising |
| `--preallocate` | Allocate dest files before writing | ✅ Parity | The receiver preallocates the destination file's full expected space before any data is written, so a transfer that would overflow disk fails fast at allocation time (a clean error, not a half-written file) and the file is laid out contiguously, avoiding fragmentation. Crosses the wire (the config frame carries a `preallocate` boolean; `PROTOCOL_VERSION` bumped **2.10.0 → 2.11.0**, peers must match) so the sender knows the receiver will preallocate and the receiver performs it. **Allocation approach (protocol 2.26.0):** `fallocate(2)` is tried first (what rsync uses); `posix_fallocate()` is the fallback and also reserves *real* disk blocks (true fail-fast on ENOSPC); plain `ftruncate()` is the final fallback when the filesystem reports the allocation is unsupported, still extending the logical size so the intent degrades gracefully. **Fallback/error semantics:** `EOPNOTSUPP`/`ENOSYS` → clean fallback to `ftruncate` (best-effort, preallocates the logical size and never fails a transfer on filesystems that lack `posix_fallocate`); a genuine allocation failure (`ENOSPC`/`EDQUOT`/`EFBIG`/…) aborts the file/receive with a distinct `preallocate failed ... transfer aborted` error — it does **not** fall back to a normal non-preallocated write, preserving the fail-fast purpose. **Size-known requirement:** preallocation only runs when the final size is already known up front (the normal regular-file case); unknown-length data is skipped (never failed). **Orthogonality:** applies uniformly across the atomic temp+rename store path, `--inplace`, `--partial`/`--partial-dir`, `--delay-updates` (the staged temp file is preallocated before data flows) and the `--link-dest` copy fallback; it neither implies nor conflicts with `-s`, `--append`, or delta. Protocol 2.26.0 matches rsync: `--preallocate` wins over `--sparse` — when both are set the allocation still runs (its reserved blocks survive the sparse writer's seeks), and a differential test's `st_blocks` agrees for every flag combination. See the Phase-4 preallocate notes below |
**Preallocate notes (Phase 4, preallocate wave):** `--preallocate` is implemented as a real receiver-side allocation of the destination file's space before data is written. It is a plain boolean config flag that crosses the wire (serialized in the config frame's selection-options block, mirroring `--inplace`/`--append`/`--force`), so the run requires matching ends: `PROTOCOL_VERSION` was bumped **2.10.0 → 2.11.0** (peers must match or the version check fails). The allocation is performed on the exact destination fd, immediately after it is opened, before any bytes are streamed; `posix_fallocate` (and the `ftruncate` fallback) leave the fd's file offset untouched, so the subsequent data write at offset 0 is unaffected and complete. Because FastSync writes each file's byte payload in one in-memory batch, the "full expected size" is exactly the known `data_size`, which is what gets preallocated. Unknown-length/streamed payloads are skipped rather than failed. A failed allocation logs a distinct `preallocate failed` error and aborts the file (the atomic temp is unlinked, the inplace target is left untrimmed) so the run fails cleanly and never silently degrades to a non-preallocated write — preserving rsync's fail-fast intent on a full disk.
@@ -618,22 +665,22 @@ targets verbatim, matching rsync.
| Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------|
| `--checksum` | Skip based on checksum | ✅ Parity | `-c`/`--checksum` compares per-file whole-file content digests to skip unchanged files. **As of protocol 2.23.0 the short `-c` implies the checksum quick-check**, so a plain `-c` run verifies content rather than only affecting the `--incremental` handshake. The digest algorithm is `xxh64` by default and is selectable via `--checksum-choice`/`--cc` (`xxh64`/`xxhash`/`xxh3`/`xxh128`/`md5`/`auto`) and `--checksum-seed=NUM` (see those rows) |
| `--checksum-choice=STR`, `--cc=STR` | Choose checksum algorithm | ⚠️ Caveat | Real algorithm selection for the per-file whole-file digest used by the `--incremental`/`--checksum` handshake and by the basis-dir content verification. **Protocol 2.23.0 accepts `xxh64` (the default), `xxhash` (rsync's spelling of xxHash64), `xxh3`, `xxh128`, `md5`, and `auto` (which selects FastSync's default).** rsync choices FastSync does not implement — `md4`, `sha1`, `none`, and the two-name `transfer,pre-transfer` form — are **rejected by name** with a clear error at parse time, never a silent no-op. `--cc` is the alias (`--cc=ALG` and space forms both parse). The algorithm id and seed cross the wire with the config frame, so the receiver hashes its on-disk old file with the SAME algorithm+seed the sender used and both agree on a match; the sender's digest and the receiver's comparison live in the per-file `STATUS_CHECK` handshake, which carries a length-prefixed, bounded (1..16 byte) digest, and the receiver pins the received length to the negotiated algorithm's digest length (defense-in-depth: a mismatched/malicious length only forces a safe re-transfer). Digest lengths: `xxh64`/`xxh3` = 8 bytes, `xxh128`/`md5` = 16. Note: `md5` is a FIPS-non-approved algorithm, so under an OpenSSL build with FIPS mode enabled `--checksum-choice=md5` fails loudly rather than silently falling back. `PROTOCOL_VERSION` has moved well past the original 2.10.0 digest-frame bump. Like rsync, the choice only takes effect where a whole-file digest is actually computed (`--checksum` on, or a basis-dir flag). Closely-related divergence: the delta BLOCK strong checksum stays xxHash32 — `--checksum-choice` selects only the whole-file digest, matching rsync where the per-block checksum is independent of the whole-file choice |
| `--compare-dest=DIR` | Compare dest files relative to DIR | ⚠️ Caveat | DIR is a receiver-side basis relative to the destination root (confined below it; absolute/`..`/`.` rejected, `//` collapsed and trailing `/` dropped). On the receiver's per-file check (implies `--incremental`) an exact match = same size + mtime (unless `--size-only`; `-I` disables matching) **and** equal xxHash64 of the sender's file; a match suppresses the data transfer. compare-dest never copies: it only skips a file the destination does **not** already hold (sparse destination, rsync parity), and is consulted before the normal delta/full paths. Repeatable; searched in command-line order, first match wins. Divergences: when the destination already holds a *different* version rsync deletes it but FastSync instead transfers the data (keeps the mirror complete; never deletes without `--delete`); attribute-only differences on a match are not re-applied (data is skipped so the sender never sends metadata); content is verified by xxHash64, stricter than rsync's default quick check. Sizing: FastSync's whole-file payload limit is 256 MiB on **every** transfer path (not basis-specific); rsync applies basis dirs to arbitrary sizes, so FastSync refuses a basis run whose source contains a larger file up front with a clear error before any transfer. Wire: a basis-count field is always present on the config frame (protocol 2.9.0, so clients and servers must both be 2.9.0) |
| `--copy-dest=DIR` | Include copies of unchanged files | ⚠️ Caveat | Same basis rules as `--compare-dest`, but an exact match materializes a **local copy** of the DIR file into the destination (via the normal atomic temp+rename store path, so `--existing`/`--ignore-existing`/`--update`/`--backup`/`--delay-updates` all still apply) instead of transferring data. Repeatable; command-line order = priority. Content is xxHash64-verified before the copy. Divergences: a basis-hit destination keeps the basis file's own mode/uid/gid and mtime (the sender sends no metadata on a skip), so with `--size-only` its mtime can differ from the source and attribute-only differences are copied with the basis attributes rather than rsync's "copy + fix attributes". Requires `--incremental` (implied); incompatible with `-s`. Wire: protocol 2.9.0 |
| `--link-dest=DIR` | Hardlink to files when unchanged | ⚠️ Caveat | Same basis rules as `--copy-dest`, but an exact match installs an atomic **hard link** to the DIR file (temp hard link + rename) so no data or disk space is used; where the link is impossible (basis on another filesystem, filesystem refuses links) it falls back cleanly to a byte-identical local copy, never a corrupt/partial file. `--delay-updates` stages the link and publishes by rename, so the final entry stays a real hard link. Repeatable (searched in command-line order, first match wins). Content is xxHash64-verified before linking. Divergences and caveats: an already up-to-date destination file is not re-linked to a basis file (only files that would otherwise be written are linked); a link keeps the basis inode's own mode/uid/gid and mtime — metadata is never written through the shared inode (that would mutate the basis file), so a later `--inplace` run that rewrites such a destination path **will mutate the basis snapshot** through the shared inode (use `--copy-dest` when the destination must stay independently writable); with `--size-only` the linked mtime can differ from the source; a `--remove-source-files` source satisfied by a basis dir is treated as skipped and therefore **retained** (never removed); basis dirs are excluded from `--delete`. Requires `--incremental` (implied); incompatible with `-s`. Wire: protocol 2.9.0 |
| `-y`, `--fuzzy`, `--no-fuzzy` | Find similar file for basis | ⚠️ Caveat | `-y/--fuzzy` is a pure bandwidth optimization on the existing receiver-driven delta path: when a file must be transferred and the destination holds no usable content at the exact path (file absent, or the destination file is outside the delta engine's size bounds), the receiver searches the SAME destination directory for an existing regular file whose basename is similar to the incoming name and uses it as the delta basis, so the sender transmits only the differences instead of the whole file. The output is always byte-exact regardless of which (or whether any) basis is chosen. Decision location: the receiver performs the candidate search inside `receive_incremental_check` and sends the normal `STATUS_DELTA_SIGNATURE`; the sender never learns the basis was a different file, so no new frame type or sender logic was needed — only the config frame grew a `fuzzy` boolean, so `PROTOCOL_VERSION` was bumped **2.8.0 → 2.9.0** (peers must match). Similarity heuristic (deterministic, simpler than rsync's deliberately-fuzzy matching, and documented precisely): candidates are the target's sibling entries in its destination directory, opened `O_NOFOLLOW`/`AT_SYMLINK_NOFOLLOW` under the confined root (symlinks never followed; nothing outside the destination root is ever read or hashed); dotfiles, directories, the target's own name, and the `.fastsync-stage`/temp scratch names are excluded; like the ordinary delta path, the block signature the receiver transmits is derived from on-disk content it may not otherwise send, so a negotiated `--fuzzy` run exposes the destination's sibling files (at block granularity) to the sender as a known-plaintext oracle — the same information class as the normal delta handshake over the file being replaced; the size gate is the delta engine's own bounds (both files ≥ 16 KiB, ≤ `--delta-max`, ratio ≤ 10×) rather than rsync's ~1.5× size window; the name gate is a Levenshtein edit distance between the basenames accepted only when ≤ half the length of the longer basename; the single best candidate (smallest distance, tie-break size closest to the incoming file then lexicographically smaller basename) is read; the directory scan is capped at 4096 entries so a pathological directory cannot stall a transfer. When fuzzy applies: only to files the receiver would otherwise send whole — the destination's own file is always preferred as the delta basis when it exists and fits the delta size bounds, so fuzzy does NOT replace an existing-but-different destination basis; FastSync's 10× delta size-ratio bound means an existing destination file that is too far away in size still lets the fuzzy search run. When no similar candidate exists the transfer falls back to the normal whole-file transfer. rsync-divergence note: rsync's own matching uses a fuzzy name/size rule set; FastSync implements the closest safe deterministic approximation above. Because FastSync's delta machinery is off by default (rsync's is on), `--fuzzy` implies `--incremental` + `--delta` (unless `--whole-file`/`-W` or an explicit `--no-delta` switched delta off, in which case fuzzy is inert — matching rsync where `--whole-file` makes fuzzy irrelevant). Unlike the basis-dir options, `--fuzzy` honors an explicit `--no-incremental` (it does not force the handshake back on); an explicit `--no-incremental` also suppresses the delta implication so no invalid `--delta requires --incremental` config results. `--no-fuzzy` negates it. All surrounding semantics are untouched: a fuzzy-reconstructed file is stored as a normal file, so `--remove-source-files`, itemize/`-i`, `--stats`, `--backup`, `--delay-updates`, `--existing`/`--ignore-existing`/`--update` behave exactly as for a whole-file transfer (the fuzzy delta does not skip the file) |
| `--checksum` | Skip based on checksum | ✅ Parity | `-c`/`--checksum` compares per-file whole-file content digests to skip unchanged files. **As of protocol 2.23.0 the short `-c` implies the checksum quick-check**, so a plain `-c` run verifies content rather than only affecting the `--incremental` handshake. The digest algorithm is `xxh128` by default (protocol 2.26.0's negotiated default) and is selectable via `--checksum-choice`/`--cc` (`xxh128`/`xxh3`/`xxh64`/`xxhash`/`md5`/`md4`/`sha1`/`none`/`auto`, plus rsync's two-name form) and `--checksum-seed=NUM` (see those rows) |
| `--checksum-choice=STR`, `--cc=STR` | Choose checksum algorithm | ✅ Parity | Real algorithm selection for the per-file whole-file digest used by the `--incremental`/`--checksum` handshake and basis-dir verification. **Protocol 2.26.0 accepts rsync 3.4.1's full set** — `xxh128` (the negotiated default), `xxh3`, `xxh64`, `xxhash`, `md5`, `md4`, `sha1`, `none`, `auto`, and the two-name `transfer,pre-transfer` form — with rsync's exit-4 rejection of an unknown name and of `none` on the transfer side when `--checksum` is on. `--cc=ALG` and space forms both parse. The algorithm id and seed cross the wire; the receiver hashes its old file with the same algorithm+seed and the per-file `STATUS_CHECK` handshake carries a bounded digest pinned to the negotiated length. `checksum_digest_file` now streams **every** supported algorithm (md4 via the self-contained RFC 1320 code, sha1/md5 via EVP, none as an empty digest), so the streaming path matches its contract, and `--out-format %C` uses the selected **transfer** half of a two-name choice and renders each algorithm byte-for-byte like rsync (xxh128 high-then-low, xxh64/xxh3 big-endian, md5/md4/sha1 standard hex, none a blank 2-char column) — differential-tested across all algorithms. **Track-3b finding — the block-checksum residual is not observable.** rsync applies the choice to the block checksum on its wire too, while FastSync selects only the whole-file comparison digest and keeps the delta BLOCK strong checksum fixed at xxHash32 (`DeltaBlockSig`, `delta_signature_create_seeded`). Because FastSync does not interoperate with rsync on the wire, only the compared surface matters, and a pre-seeded delta differential against rsync 3.4.1 (`--no-whole-file -B8192 --stats --out-format=%c|%C %n` vs `--incremental --delta`) shows the choice does not move it: across `xxh64`, `xxh128`, `xxh3`, `md5`, `md4`, `sha1` and both two-name orders the destination tree is byte-identical, `Matched data`/`Literal data`/`Total transferred file size` are unchanged (and equal to rsync's with the block size pinned), the `%c` block-checksum token is invariant (rsync `16 + 6·ceil(size/block)`, already documented under `--out-format`; FastSync its own basis-read counter), and the exit code is 0. The negotiated algorithm is visible only in `%C`, which applies it to the whole-file transfer digest and matches rsync byte-for-byte. A false block match would require the 4-byte adler32 AND the 4-byte xxHash32 to collide; at the 256 MiB maximum with the 1 KiB minimum block size the expected false matches are ≤2⁻¹⁸, and the choice cannot change this because FastSync's block strong sum is fixed. `auto` now consults `RSYNC_CHECKSUM_LIST` (rsync's whitespace-separated preference list; unknown names skipped, first supported wins, all-unknown is exit 4) before the compiled-in order; because both peers run the identical build this deterministic resolution needs no rsync peer probe, and an explicit `--cc` still wins. The list is differential-tested through `--out-format %C` (byte-identical digests to rsync for md5/sha1/xxh3) |
| `--compare-dest=DIR` | Compare dest files relative to DIR | ⚠️ Caveat | DIR is a receiver-side basis; protocol 2.26.0 uses an absolute path verbatim (rsync semantics) and resolves a relative path below the destination root (`..` components are rejected, `//` collapsed and trailing `/` dropped) — note rsync resolves a relative DIR against the destination directory while FastSync resolves it below the receive root and appends the mirrored source path, so the same relative spelling addresses a different tree (use an absolute DIR for exact parity). On the receiver's per-file check (implies `--incremental`) an exact match is rsync's metadata quick-check: same size and mtime (unless `--size-only`; `-I` disables matching), with NO content digest required by default (track 5a). A match suppresses the data transfer. The FastSync-only `--verify-basis` restores the stricter whole-file content equality. compare-dest never copies: it only skips a file the destination does **not** already hold (sparse destination, rsync parity), and is consulted before the normal delta/full paths. Repeatable; searched in command-line order, first match wins. Differential-tested against rsync 3.4.1 (`compare_dest`, and `test_verify_basis_restores_strict_content`). Residual: a basis MISS above the 256 MiB whole-file payload bound is refused up front (FastSync's general whole-file limit, not basis-specific); rsync applies basis dirs to arbitrary sizes. Wire: a basis-count field plus the `verify_basis` bool are present on the config frame (protocol 2.9.0/2.28.0) |
| `--copy-dest=DIR` | Include copies of unchanged files | ⚠️ Caveat | Same basis rules as `--compare-dest`, but an exact match materializes a **local copy** of the DIR file into the destination (via the atomic temp+rename store path, so `--existing`/`--ignore-existing`/`--update`/`--backup`/`--delay-updates` all still apply) instead of transferring data. Track 5a re-applies the SOURCE attributes on the copy (rsync's "copy then fix attributes"): the sender transmits the source metadata with the basis check frame, so the copy's mode/uid/gid/mtime match the source rather than the basis inode (differential `copy_dest` compares modes). The copy streams the basis file through a bounded buffer, so a basis larger than the whole-file payload bound still materializes. Repeatable; command-line order = priority. Requires `--incremental` (implied); incompatible with `-s`. Wire: protocol 2.9.0 |
| `--link-dest=DIR` | Hardlink to files when unchanged | ⚠️ Caveat | Same basis rules as `--copy-dest`, but an exact match installs an atomic **hard link** to the DIR file (temp hard link + rename) so no data or disk space is used; where the link is impossible (basis on another filesystem, filesystem refuses links) it falls back cleanly to a byte-identical local copy (streamed from the basis, so an over-limit basis still works), never a corrupt/partial file. `--delay-updates` stages the link and publishes by rename, so the final entry stays a real hard link. Repeatable (searched in command-line order, first match wins). Differential-tested against rsync 3.4.1 (`link_dest`). Inherent shared-inode semantics (identical to rsync): a link keeps the basis inode's own mode/uid/gid and mtime — metadata is never written through the shared inode (that would mutate the basis file), so a later `--inplace` run that rewrites such a destination path **will mutate the basis snapshot** through the shared inode (use `--copy-dest` when the destination must stay independently writable); protocol 2.26.0 re-links an already up-to-date destination file to the basis; a `--remove-source-files` source satisfied by a basis dir is treated as skipped and therefore **retained** (never removed); basis dirs are excluded from `--delete`. Residual: a basis MISS above the 256 MiB whole-file payload bound is refused (FastSync's general whole-file limit). Requires `--incremental` (implied); incompatible with `-s`. Wire: protocol 2.9.0 |
| `-y`, `--fuzzy`, `--no-fuzzy` | Find similar file for basis | ⚠️ Caveat | `-y/--fuzzy` is a pure bandwidth optimization on the existing receiver-driven delta path: when a file must be transferred and the destination holds no usable content at the exact path (file absent, or the destination file is outside the delta engine's size bounds), the receiver searches the SAME destination directory for an existing regular file whose basename is similar to the incoming name and uses it as the delta basis, so the sender transmits only the differences instead of the whole file. The output is always byte-exact regardless of which (or whether any) basis is chosen. Decision location: the receiver performs the candidate search inside `receive_incremental_check` and sends the normal `STATUS_DELTA_SIGNATURE`; the sender never learns the basis was a different file, so no new frame type or sender logic was needed — only the config frame grew a `fuzzy` boolean, so `PROTOCOL_VERSION` was bumped **2.8.0 → 2.9.0** (peers must match). Similarity heuristic (a deterministic port of rsync 3.4.1's matcher — `util1.c` `fuzzy_distance`/`find_filename_suffix` plus `generator.c find_fuzzy`'s exact size+mtime pass — documented precisely): candidates are the target's sibling entries in its destination directory, opened `O_NOFOLLOW`/`AT_SYMLINK_NOFOLLOW` under the confined root (symlinks never followed; nothing outside the destination root is ever read or hashed); dotfiles, directories, the target's own name, and the `.fastsync-stage`/temp scratch names are excluded; like the ordinary delta path, the block signature the receiver transmits is derived from on-disk content it may not otherwise send, so a negotiated `--fuzzy` run exposes the destination's sibling files (at block granularity) to the sender as a known-plaintext oracle — the same information class as the normal delta handshake over the file being replaced; the size gate is the delta engine's own bounds (both files ≥ 16 KiB, ≤ `--delta-max`, ratio ≤ 10×); rsync's fuzzy matcher is not tied to a delta size bound and empirically reuses a basis well outside FastSync's window (a 64 KiB source against a repeated-content sibling from 0.25× to 10000×, and files as small as 300 B), so candidate ELIGIBILITY — and hence the chosen basis — can differ even though the name heuristic is the same; protocol 2.26.0 uses rsync's weighted-Levenshtein name/suffix distance plus an exact size+mtime pass and reads a single best candidate; the tie-break (smallest size gap, then lexical name) is deterministic where rsync leaves equal distances to its file-list order; the directory scan is capped at 4096 entries so a pathological directory cannot stall a transfer. When fuzzy applies: only to files the receiver would otherwise send whole — the destination's own file is always preferred as the delta basis when it exists and fits the delta size bounds, so fuzzy does NOT replace an existing-but-different destination basis; FastSync's 10× delta size-ratio bound means an existing destination file that is too far away in size still lets the fuzzy search run. When no similar candidate exists the transfer falls back to the normal whole-file transfer. rsync-divergence note: the name matching is rsync's own rule; the residual is eligibility bounded by FastSync's delta engine, so no name-matcher port can widen it. Because FastSync's delta machinery is off by default (rsync's is on), `--fuzzy` implies `--incremental` + `--delta` (unless `--whole-file`/`-W` or an explicit `--no-delta` switched delta off, in which case fuzzy is inert — matching rsync where `--whole-file` makes fuzzy irrelevant). Unlike the basis-dir options, `--fuzzy` honors an explicit `--no-incremental` (it does not force the handshake back on); an explicit `--no-incremental` also suppresses the delta implication so no invalid `--delta requires --incremental` config results. `--no-fuzzy` negates it. All surrounding semantics are untouched: a fuzzy-reconstructed file is stored as a normal file, so `--remove-source-files`, itemize/`-i`, `--stats`, `--backup`, `--delay-updates`, `--existing`/`--ignore-existing`/`--update` behave exactly as for a whole-file transfer (the fuzzy delta does not skip the file). **Reclassified Caveat (track 5b):** the output is always byte-exact regardless of the basis, and the name rule is rsync's, so the residual is candidate ELIGIBILITY: FastSync's 10× ratio / 16 KiB delta gates make its size window strictly narrower than rsync's, and when eligibility differs the chosen basis — and therefore the `--stats` `Matched data`/`Literal data`/`Total transferred file size` counters — can differ even though the tree cannot. Where the two tools' choices coincide and the block size is pinned, both the tree and the counters match rsync (differential `fuzzy_basis`); `TestFuzzy` pins the window boundary on both sides (a >10× sibling and a <16 KiB sibling are declined, with a byte-exact whole-file fallbaLine truncated
## 12. Compression
| Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------|
| `-z`, `--compress` | Compress file data | ⚠️ Caveat | Streaming zstd (rsync supports multiple algorithms — a documented divergence, selectable via `--compress-choice`). `-z` is the compression short form; `-c` is rsync's `--checksum`. `--skip-compress` applies rsync 3.4.1's default suffix list when no list is given |
| `--compress-choice=STR`, `--zc=STR` | Choose compression algorithm | ⚠️ Caveat | FastSync supports `zstd` (default), `none`, and `auto`. rsync's other compiled-in choices (`lz4`, `zlib`, `zlibx`) are **rejected by name** at parse time with a clear error, never silently ignored. `--zc` is the alias |
| `--compress-level=NUM`, `--zl=NUM` | Set compression level | ✅ Parity | 1-22, default 5 |
| `-z`, `--compress` | Compress file data | ✅ Parity | Streaming compression. **Protocol 2.26.0 implements rsync 3.4.1's codec set** (`zstd` default, `lz4`, `zlib`, `zlibx`, `none`), selectable via `--compress-choice`/`--zc` and negotiated with `auto`. `-z` is the compression short form; `-c` is rsync's `--checksum`. `--skip-compress` applies rsync 3.4.1's default suffix list when no list is given. **Track 3a closes the codec caveats:** `zlibx` is no longer a divergence — FastSync's zlib stream already carries only the delta/token (literal) bytes, which is exactly rsync's zlibx semantics, so `--zc=zlib` and `--zc=zlibx` land the same tree/stdout/exit (differential `test_compress_codec_matches_rsync_bytes`) and the zlib/zlibx aliasing is only an implementation detail. Each codec now uses rsync's own default `--compress-level` (zstd 3, zlib/zlibx 6, lz4 ignored) and `auto` consults `RSYNC_COMPRESS_LIST` before the compiled-in order; the deterministic same-build resolution needs no peer probe |
| `--compress-choice=STR`, `--zc=STR` | Choose compression algorithm | ✅ Parity | Protocol 2.26.0 accepts rsync 3.4.1's compiled-in choices — `zstd` (default), `lz4`, `zlib`, `zlibx`, `none`, `auto` — and rejects an unknown name with exit 4 like rsync. The negotiated codec id crosses the wire (`compression_algo`), so the receiver decodes with the sender's codec. `--zc` is the alias. `auto` now resolves through `RSYNC_COMPRESS_LIST` (whitespace-separated; unknown names skipped, first supported wins, all-unknown is exit 4) and then the compiled-in order, and an explicit `--zc` wins; the deterministic same-build resolution needs no peer probe. `zlib`/`zlibx` share FastSync's literal-only zlib path, which is rsync's zlibx behavior and is observably identical for both, so `zlibx` is not a divergence (the aliasing is an implementation detail) |
| `--compress-level=NUM`, `--zl=NUM` | Set compression level | ✅ Parity | Accepted range 1-22. When omitted, rsync 3.4.1's **per-codec default** applies: zstd 3 (`ZSTD_CLEVEL_DEFAULT`), zlib/zlibx 6 (`Z_DEFAULT_COMPRESSION` resolved), lz4 ignored (no tunable level; FastSync keeps a positive gate value and `lz4_compress` ignores it, so the bytes match rsync). An explicit level is clamped per codec like rsync's `init_compression_level()`: zstd 1-22, zlib/zlibx 1-9, lz4 ignored. Verified against `rsync --debug=NSTR1`, which reports the same effective level per codec |
| `--compress-threads=NUM` | Set compression threads | ✅ Parity | `compression_threads` config field (client-only; does not cross the wire). Sets the number of worker threads used by the zstd compression pool to NUM (1..64; 0/garbage/oversized rejected up front). Accepted in both `--compress-threads=NUM` and two-argument `--compress-threads NUM` forms. Composes with `-z`/compression; under the `-j`/`--threads` multithreaded pipeline it parallelizes compressed chunk encoding. See test_tcp.py `-z --compress-threads=2` and test_client_cli.c |
| `--skip-compress=LIST` | Skip compress for suffixes | ⚠️ Caveat | Comma-separated (or `/`-separated, as in rsync) case-insensitive suffix list; a leading dot is optional; an empty list skips none. **When the option is omitted, rsync 3.4.1's built-in default suffix list applies** (`3g2 3gp 7z aac … zip zst`); an explicit list replaces that default entirely, matching rsync. A user-supplied list is a client-side compression choice; incompatible with FastSync chunk serialization (`-s`) |
| `--skip-compress=LIST` | Skip compress for suffixes | ✅ Parity | Comma-separated (or `/`-separated, as in rsync) case-insensitive suffix list; a leading dot is optional; an empty list skips none. **When the option is omitted, rsync 3.4.1's built-in default suffix list applies** (`3g2 3gp 7z aac … zip zst`); an explicit list replaces that default entirely, matching rsync. A user-supplied list is a client-side compression choice; incompatible with FastSync chunk serialization (`-s`) |
## 13. Connectivity
@@ -649,19 +696,20 @@ targets verbatim, matching rsync.
| `--address=ADDRESS` | Bind address for outgoing socket | ✅ Parity | Binds the outgoing client socket to a local source address before `connect()` (resolved with the same `-4`/`-6` family hints as the destination). Local socket concern: never crosses the wire |
| `-4`, `--ipv4` | Prefer IPv4 | ✅ Parity | Forces `AF_INET` in the `getaddrinfo` hints for client destination/source resolution and the server bind (see the Phase 5, Wave B note). Mutually exclusive with `-6` |
| `-6`, `--ipv6` | Prefer IPv6 | ✅ Parity | Forces `AF_INET6` in the `getaddrinfo` hints for client destination/source resolution and the server bind. Mutually exclusive with `-4` |
| `--remote-option=OPT`, `-M` | Send an option only to the remote side | ⚠️ Caveat | Each value is appended to the remote server invocation over SSH as an individually single-quote-escaped shell word in `ssh_build_remote_command()`. Values are validated (non-empty, no control characters) and shell metacharacters cannot break out of the quoting (`;`, `&`, `\|`, <code>`</code>, `$`, `(`, `)`, quotes are neutralized), so a value cannot inject an arbitrary remote command and a subsequent `--` on the client line cannot be turned into one. The short `-M` form (`-M OPT`, `-M=OPT`, and rsync-style attached `-MOPT`) is available, matching rsync; metadata mode moved to long-only `--preserve`. **Divergence:** `-M` is only meaningful for the SSH transport (`user@host:path`); a daemon (`host::module/path`) or local TCP destination **rejects** it (there is no remote command line to append to), whereas rsync applies it to its own remote process on every transport. The options never cross the binary config frame |
| `--remote-option=OPT`, `-M` | Send an option only to the remote side | ❌ Divergent | Each value is appended to the remote server invocation over SSH as an individually single-quote-escaped shell word in `ssh_build_remote_command()`. Values are validated (non-empty, no control characters) and shell metacharacters cannot break out of the quoting (`;`, `&`, `\|`, <code>`</code>, `$`, `(`, `)`, quotes are neutralized), so a value cannot inject an arbitrary remote command and a subsequent `--` on the client line cannot be turned into one. The short `-M` form (`-M OPT`, `-M=OPT`, and rsync-style attached `-MOPT`) is available, matching rsync; metadata mode moved to long-only `--preserve`. **Reclassified because the daemon/TCP case cannot be reproduced:** `-M` is only meaningful for the SSH transport (`user@host:path`); a daemon (`host::module/path`) or local TCP destination **rejects** it, whereas rsync forwards it to its own remote process on every transport. A differential test starts a real rsync daemon and shows `-M--totally-bogus` reaching the remote parser (`unknown option`) while a valid `-M--safe-links` is accepted. FastSync's daemon handshake is a fixed binary config frame with no per-connection argv channel; adding one would let a client set arbitrary server-side options (the same class of divergence as the native daemon config/auth), so the safe subset stays SSH-only |
| `--bwlimit=RATE` | Limit I/O bandwidth | ✅ Parity | A faithful port of rsync 3.4.1's `parse_size_arg(bwlimit_arg, 'K', "bwlimit", 512, -1, True)`: a bare value is KiB/s, `K`/`M`/`G`/`T`/`P` are binary suffixes, `KB`/`MB` are decimal, `KiB`/`MiB` are binary, decimals are accepted and quantized to whole KiB exactly like rsync's `(size + 512) / 1024`, `0` (or an empty value) means "no limit", and any other value below the 512-byte floor is rejected. The token bucket's burst capacity is ~100 ms of bandwidth, matching the point at which rsync's leaky bucket starts sleeping, so a throttled transfer paces like rsync (4 MiB at `--bwlimit=1024`/`2048` matches rsync within ~4%). Differential-tested: the accept/reject matrix and the wall-clock rate both match rsync 3.4.1. The limit is a local I/O concern and is not negotiated on the wire |
## 14. Daemon Mode
| Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------|
| `--daemon` | Run as rsync daemon | ⚠️ Caveat | Wave A: a real persistent listener. `fastsync-server --daemon --config FILE` (plus `--no-detach` to stay foreground; without it the listener detaches to the background after binding) reads a FastSync-native module config file and serves each connection confined to the requested module's `path` root (never a client-chosen root; every client-chosen-ownership/super-user request (`--numeric-ids`/`--chown`/`--usermap`/`--groupmap`/`--fake-super`/`--copy-as`/explicit `--super`) is refused unless the module opts in with `client owner = yes`, and the operator `--no-super` veto is honored). TCP/TLS via the existing `--tls` stack; plaintext still requires `--allow-unauthenticated` (same secure default as the standalone server). Client destinations use rsync's `host::module/path` form. Wire/protocol: the config frame gained a trailing daemon-module string and `PROTOCOL_VERSION` was bumped **2.14.0 → 2.15.0** (see the Daemon Mode notes below). Daemon mode is built in FastSync's own protocol/config grammar, not rsync's SMB/daemon option encoding |
| `--config=FILE` | Alternate rsyncd.conf file | ⚠️ Caveat | Wave A: selects the daemon config file. Default when omitted (in `--daemon` mode): `~/.config/fastsync/fastsyncd.conf` if it exists, else `/etc/fastsyncd.conf`. The grammar is FastSync-native (documented in the Daemon Mode notes below) and strictly rejects unknown keys so a typo can never silently change what a module serves; requires `--daemon` |
| `--dparam=OVERRIDE` | Override global daemon config | ⚠️ Caveat | Wave A: overrides one global scalar from the command line (`--dparam port=8734` and `--dparam=KEY=VALUE` both work). Limited to the global keys the grammar defines (`port`, `motd file`, `address`, `max connections`, `max connections per host`, `auth failure delay`, `auth lockout threshold`, `auth lockout duration`, `hosts allow`, `hosts deny`); keys are case-insensitive and unknown keys/invalid values are rejected. Requires `--daemon` |
| `--daemon` | Run as rsync daemon | ❌ Divergent | Wave A: a real persistent listener. `fastsync-server --daemon --config FILE` (plus `--no-detach` to stay foreground; without it the listener detaches to the background after binding) reads a FastSync-native module config file and serves each connection confined to the requested module's `path` root (never a client-chosen root; every client-chosen-ownership/super-user request (`--numeric-ids`/`--chown`/`--usermap`/`--groupmap`/`--fake-super`/`--copy-as`/explicit `--super`) is refused unless the module opts in with `client owner = yes`, and the operator `--no-super` veto is honored). TCP/TLS via the existing `--tls` stack; plaintext still requires `--allow-unauthenticated` (same secure default as the standalone server). Client destinations use rsync's `host::module/path` form. Wire/protocol: the config frame gained a trailing daemon-module string and `PROTOCOL_VERSION` was bumped **2.14.0 → 2.15.0** (see the Daemon Mode notes below). Daemon mode is built in FastSync's own protocol/config grammar, not rsync's SMB/daemon option encoding |
| `--config=FILE` | Alternate rsyncd.conf file | ❌ Divergent | Wave A: selects the daemon config file. Default when omitted (in `--daemon` mode): `~/.config/fastsync/fastsyncd.conf` if it exists, else `/etc/fastsyncd.conf`. The grammar is FastSync-native (documented in the Daemon Mode notes below) and strictly rejects unknown keys so a typo can never silently change what a module serves; requires `--daemon` |
| `--dparam=OVERRIDE` | Override global daemon config | ❌ Divergent | Wave A: overrides one global scalar from the command line (`--dparam port=8734` and `--dparam=KEY=VALUE` both work). Limited to the global keys the grammar defines (`port`, `motd file`, `address`, `max connections`, `max connections per host`, `auth failure delay`, `auth lockout threshold`, `auth lockout duration`, `hosts allow`, `hosts deny`); keys are case-insensitive and unknown keys/invalid values are rejected. Requires `--daemon` |
| `--no-detach` | Don't detach from parent | ✅ Parity | Wave A: with `--daemon`, keeps the listener in the foreground (what integration tests use). Without it the daemonizes (fork/setsid, stdio redirected to /dev/null) after the listening socket is bound. Requires `--daemon` |
| `--password-file=FILE` | Read daemon password from file | ⚠️ Caveat | A7 daemon auth. Client: `--password-file` supplies `user:password` for a `host::module/path` destination (the username is taken from this file, so `user@host::module` stays rejected); the literal password is held client-side only for the SCRAM handshake and wiped at teardown. Server (`fastsync-server --daemon --password-file FILE`): the salted-PBKDF2 verifier store that modules with `auth users` are verified against. **Neither the password nor any replayable bearer value crosses the wire or is stored server-side** — the store holds a per-user salt plus derived keys, and the daemon proves the secret with a per-connection nonce challenge. The file must be private to its owner: both the client and server verify the exact inode they read (open-then-`fstat`, so the check cannot be raced) and refuse a `--password-file`/`--early-input` that is not owned by the current user or grants any group/other permission bit (mode 0600), mirroring the TLS private-key check. A process-substitution pipe (`--early-input <(vault ...)`) is still accepted when it satisfies those checks. See the Daemon Mode notes below for the file formats and the plaintext/TLS caveat |
| `--early-input=FILE` | Use FILE for daemon early exec | ⚠️ Caveat | Server-only (requires `--daemon`): a second credential-store file, same new-format grammar as `--password-file`, read before the listener accepts connections (a secrets-manager / process-substitution source). Its entries layer over `--password-file`: byte-identical verifiers dedupe, a conflicting verifier for the same user is a startup error. A daemon whose modules declare `auth users` must be given at least one of the two, or it refuses to start (fail closed) |
| `--hash-credentials=FILE`, `--iterations N` | Hash a plaintext credential file | ⚠️ Caveat | Server-only offline tool (A7): reads the `user:password` lines of FILE (same owner-only 0600 check) and prints one new-format store line per entry to stdout, then exits. `--iterations` sets the PBKDF2 work factor (default 600000, range 100000–10000000). Dependency-free and does not run a listener. Use its output as `--password-file` for `--daemon`. There is no auto-upgrade: a legacy store line is hard-rejected by the loader and must be regenerated |
| `--password-file=FILE` | Read daemon password from file | ❌ Divergent | A7 daemon auth. Client: `--password-file` supplies `user:password` for a `host::module/path` destination (the username is taken from this file, so `user@host::module` stays rejected); the literal password is held client-side only for the SCRAM handshake and wiped at teardown. Server (`fastsync-server --daemon --password-file FILE`): the salted-PBKDF2 verifier store that modules with `auth users` are verified against. **Neither the password nor any replayable bearer value crosses the wire or is stored server-side** — the store holds a per-user salt plus derived keys, and the daemon proves the secret with a per-connection nonce challenge. The file must be private to its owner: both the client and server verify the exact inode they read (open-then-`fstat`, so the check cannot be raced) and refuse a `--password-file`/`--early-input` that is not owned by the current user or grants any group/other permission bit (mode 0600), mirroring the TLS private-key check. A process-substitution pipe (`--early-input <(vault ...)`) is still accepted when it satisfies those checks. See the Daemon Mode notes below for the file formats and the plaintext/TLS caveat |
| `--early-input=FILE` | Use FILE for daemon early exec | ❌ Divergent | Server-only (requires `--daemon`): a second credential-store file, same new-format grammar as `--password-file`, read before the listener accepts connections (a secrets-manager / process-substitution source). Its entries layer over `--password-file`: byte-identical verifiers dedupe, a conflicting verifier for the same user is a startup error. A daemon whose modules declare `auth users` must be given at least one of the two, or it refuses to start (fail closed) |
| `--hash-credentials=FILE`, `--iterations N` | Hash a plaintext credential file | ❌ Divergent | Server-only offline tool (A7): reads the `user:password` lines of FILE (same owner-only 0600 check) and prints one new-format store line per entry to stdout, then exits. `--iterations` sets the PBKDF2 work factor (default 600000, range 100000–10000000). Dependency-free and does not run a listener. Use its output as `--password-file` for `--daemon`. There is no auto-upgrade: a legacy store line is hard-rejected by the loader and must be regenerated |
**Daemon Mode notes (Wave A protocol 2.15.0; A7 auth protocol 2.19.0; MOTD no bump):** FastSync daemon mode is supported in FastSync's own protocol/config grammar, not rsync's SMB/daemon option encoding.
@@ -690,30 +738,43 @@ targets verbatim, matching rsync.
| Max data/string/chunk sizes | Prevent OOM attacks | ✅ Parity | Per-message limits |
| Per-connection memory limit | Cap memory per connection | ✅ Parity | `MAX_CONNECTION_MEMORY` is **256 MiB per connection** (256 * 1024 * 1024 bytes), charged across protocol reservations and decompression/chunk allocations. This is a FastSync-internal bound with no direct rsync analogue |
| `--max-alloc=SIZE` | Limit a single memory allocation | ✅ Parity | Caps the largest single allocation; binary units, default 1G |
| `--trust-sender` | Trust remote sender's file list | ⚠️ Caveat | Long-form-only, receiver-local policy that never crosses the wire. The receiver skips its redundant up-front re-validation of the incoming file list (empty/`..` path rejection), trusting the sender instead of double-checking (fewer checks, faster, potentially unsafe, matching rsync). Off by default. **It no longer affects symlink targets** (protocol 2.23.0): targets are stored verbatim under `-l` regardless of `--trust-sender`; the flag only relaxes the receiver's path-list checks. The low-level fd-relative confinement primitives (`file_open_secure_parent`, the O_NOFOLLOW parent walk, leaf/destination confinement) are deliberately KEPT even under `--trust-sender`, so a hostile sender still cannot write or link outside the authorized root (see Phase-5 notes below) |
| `--old-args` | Disable modern arg protection | ⚠️ Caveat | SSH-only; accepted for CLI compatibility but is now a **documented no-op**: FastSync always single-quote-escapes the remote server path and each `--remote-option` value (`ssh_build_remote_command`), so a metacharacter-bearing `--rsync-path` can never be interpreted by the remote shell. The flag no longer disables that quoting (the old raw-construction behavior was an injection foot-gun and is removed); the safety-relevant behavior is identical either way |
| `--ignore-missing-args` | Ignore missing source args | ⚠️ Caveat | FastSync has a single source-root argument (which always exists), so the "explicitly requested source arguments" are the `--files-from` entries and the flags only ever apply there (inert without `--files-from`, like `-R`). Without the flag a listed-but-missing entry stays a hard pre-transfer error (nothing is transferred). With it each missing entry is skipped: nothing is sent for it, it never enters the keep-set, and the run succeeds for the rest — an all-missing non-empty list succeeds transferring nothing, matching rsync. `--dirs` + `--files-from` missing entries are skipped the same way. Every skipped entry is logged and a per-run warning names the count, so the handling is never a silent no-op. Divergences: an EMPTY `--files-from` file stays a hard error in every mode (no argument was requested at all; rsync likewise reports "no source files specified"); missing-arg skipping only applies to the pre-transfer list validation, so an entry that is present at preflight and vanishes mid-transfer still fails (matching rsync, whose flag "does not affect subsequent vanished-file errors"); `--no-ignore-missing-args` is not a supported negation |
| `--trust-sender` | Trust remote sender's file list | ✅ Parity | Long-form-only, receiver-local policy that never crosses the wire. The receiver skips its redundant up-front re-validation of the incoming file list (empty/`..` path rejection), trusting the sender instead of double-checking (fewer checks, faster, potentially unsafe, matching rsync). Off by default. **It no longer affects symlink targets** (protocol 2.23.0): targets are stored verbatim under `-l` regardless of `--trust-sender`; the flag only relaxes the receiver's path-list checks. The low-level fd-relative confinement primitives (`file_open_secure_parent`, the O_NOFOLLOW parent walk, leaf/destination confinement) are deliberately KEPT even under `--trust-sender`, so a hostile sender still cannot write or link outside the authorized root (see Phase-5 notes below) |
| `--old-args` | Disable modern arg protection | ❌ Divergent | SSH-only; accepted for CLI compatibility but is now a **documented no-op**: FastSync always single-quote-escapes the remote server path and each `--remote-option` value (`ssh_build_remote_command`), so a metacharacter-bearing `--rsync-path` can never be interpreted by the remote shell. The flag no longer disables that quoting (the old raw-construction behavior was an injection foot-gun and is removed); the safety-relevant behavior is identical either way |
| `--ignore-missing-args` | Ignore missing source args | ✅ Parity | The flags apply to the `--files-from` entries (the single source root always exists; inert without `--files-from`). Without the flag a listed-but-missing entry is a hard pre-transfer error. With it each missing entry is skipped: nothing is sent for it, it never enters the keep-set, and the run succeeds for the rest (an all-missing list transfers nothing). Every skipped entry is logged and a per-run warning names the count. **An empty `--files-from` list is now a zero-transfer success with or without this flag (exit 0), matching rsync 3.4.1.** `--no-ignore-missing-args` is rejected exactly as rsync 3.4.1 rejects it, rather than being accepted as a negation |
| `--delete-missing-args` | Delete missing source args | ✅ Parity | Implies `--ignore-missing-args` (order-independent) and additionally removes each missing entry's destination mirror receiver-side. The mirror is computed exactly like a present sibling's wire path: the bare relative entry under `-R`, otherwise the full source-mirror path below the destination root. rsync parity, verified against the man page: it does **not** imply `--delete` generally and is "independent of any other type of delete processing" — unrelated destination extras are untouched unless `--delete` is also present. Composition with `--delete` + timing: the exact-path deletions commit with the manifest, early for `--delete-before`/`--delete-during`, else only after a fully-successful transfer (delete-after/commit). A non-empty directory mirror is removed only when `--force` or `--delete` is in effect (otherwise it is left with a warning and the run continues, like rsync); an absent mirror is a no-op. `--force` is deletion authority and is therefore gated by the server `--allow-delete` policy exactly like `--delete`/`--delete-missing-args`: without it the receiver clears the flag, so a client cannot use `--force` to recursively replace or remove a destination directory tree. An explicitly listed missing arg is a user request, not an excluded file: its deletion is never blocked by the filter-exclusion protection of excluded destination mirrors (a mirror sitting inside a filter-excluded directory is still removed). Safety/policy: gated by the server `--allow-delete` policy like `--delete`; the request paths cross the wire only in the delete-manifest frame and are confined by the same receiver validation as the keep-set (non-empty, relative, traversal-free, bounded by the per-section/per-frame manifest caps); the `--delay-updates` staging directory and basis snapshots are protected exactly as in the extras walker. Protocol 2.23.0 parity: the missing-args exact-path removals and the ordinary extras walk **draw from one shared `--max-delete` budget**, so a capped run stops part-way and exits 25 exactly like rsync. See the Phase-3 wire note below for the `PROTOCOL_VERSION` bump |
**Setuid/setgid/sticky bits under `-p` (security note).** As with upstream
rsync, `-p`/`--perms` reproduces the source's special bits as well as the rwx
bits: setuid, setgid, and sticky are applied whenever the receiver can apply
them (the receiving user owns the file and the mount permits it), and a refused
chmod is logged rather than silently masked (`tests/test_metadata.c:575`). This
is a deliberate change from earlier FastSync releases, which always masked
special bits. Deployments that do not trust the sender should rely on the
existing mitigations rather than on that masking: keep the daemon module default
`client owner = no`, run the daemon unprivileged, and use
`--munge-links`/`--safe-links` so a hostile source cannot weaponize preserved
modes or links.
## 16. Batch Operations
| Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------|
| `--write-batch=FILE` | Write batched update to file | ⚠️ Caveat | Phase-6 residual-batch (client-only): runs the normal live transfer AND additionally emits a self-contained single-file batch of the whole source tree. The batch is a magic/format-version header followed by length-prefixed `chunk_serialize` blobs (full file images), replayable byte-identically by `--read-batch` on another machine with no source/server. `--write-batch` drives the single-threaded transfer path (the multithreaded path consumes the config before the separate batch scan pass). See the Phase-6 batch note below |
| `--only-write-batch=FILE` | Write batch without updating dest | ⚠️ Caveat | Phase-6 residual-batch: emits the self-contained batch FILE only — NO destination update, NO server connection. Requires a source (scans it and serializes the full tree to FILE). Same single-file format as `--write-batch`, so the file is re-appliable via `--read-batch=FILE DEST`. See the Phase-6 batch note below |
| `--read-batch=FILE` | Read batched update from file | ⚠️ Caveat | Phase-6 residual-batch: applies a previously written batch FILE locally to the destination. NO source and NO server — positional args are the destination only. Reads the magic/version header, then length-prefixed records, `chunk_deserialize`, and applies each via the confined `file_save_to_disk_full` path (same O_NOFOLLOW / `..`-rejection / root-confinement as the network receiver, so an attacker-controlled batch cannot escape the destination root). Malformed/truncated/oversized/traversal records are rejected cleanly. See the Phase-6 batch note below |
| `--write-batch=FILE` | Write batched update to file | ❌ Divergent | Phase-6 residual-batch (client-only): runs the normal live transfer AND additionally emits a self-contained single-file batch of the whole source tree. The batch is a magic/format-version header followed by length-prefixed `chunk_serialize` blobs (full file images), replayable byte-identically by `--read-batch` on another machine with no source/server. `--write-batch` drives the single-threaded transfer path (the multithreaded path consumes the config before the separate batch scan pass). The FastSync container is deliberately not interoperable with rsync batch files. See the Phase-6 batch note below |
| `--only-write-batch=FILE` | Write batch without updating dest | ❌ Divergent | Phase-6 residual-batch: emits the self-contained batch FILE only — NO destination update, NO server connection. Requires a source (scans it and serializes the full tree to FILE). Same single-file format as `--write-batch`, so the file is re-appliable via `--read-batch=FILE DEST`. The FastSync container is deliberately not interoperable with rsync batch files. See the Phase-6 batch note below |
| `--read-batch=FILE` | Read batched update from file | ❌ Divergent | Phase-6 residual-batch: applies a previously written batch FILE locally to the destination. NO source and NO server — positional args are the destination only. Reads the magic/version header, then length-prefixed records, `chunk_deserialize`, and applies each via the confined `file_save_to_disk_full` path (same O_NOFOLLOW / `..`-rejection / root-confinement as the network receiver, so an attacker-controlled batch cannot escape the destination root). Malformed/truncated/oversized/traversal records are rejected cleanly. The FastSync container is deliberately not interoperable with rsync batch files. See the Phase-6 batch note below |
## 17. Advanced
| Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------|
| `--stop-after=MINS` | Stop after N minutes | ✅ Parity | Client-only sender stop deadline (Phase 6): computing `--stop-after=MINS` (a positive minute count; 0/negative/garbage rejected) and `--stop-at=TIME` (`HH:MM`, `HH:MM:SS`, or `now+N[smhd]`; a past time stops immediately). The transfer stops ELEGANTLY at the next chunk boundary: everything already fully sent is kept and applied, the run returns 0, and --delete (late/delete-after timing) does NOT wipe the destination — when the scan is cut short the partial keep-set manifest is suppressed with a warning (the delete walk is skipped rather than acting on an incomplete keep-set, so unscanned source mirrors survive). `--delete-before`/`--delete-during` still run their complete pre-scan (which ignores the deadline). Local client-only fields: never serialized into the wire config frame, so no PROTOCOL_VERSION bump. `--stop-after` uses CLOCK_MONOTONIC; `--stop-at` uses the wall clock. Works single-threaded and under `-j`/`--threads` (multithreaded). Divergence: rsync computes `--stop-after` from the run start; FastSync likewise. When both are given, the earlier of the two deadlines wins (checked per iteration). See the Phase-6 stop notes below |
| `--stop-at=TIME` | Stop at specified time | ⚠️ Caveat | Same feature as `--stop-after` (deadline transfer stop), absolute wall-clock form (`HH:MM[:SS]` or `now+N[smhd]`). See the row above and the Phase-6 stop notes |
| `--stop-at=TIME` | Stop at specified time | ✅ Parity | Deadline transfer stop (client-only, never serialized). Protocol 2.26.0 accepts rsync's full date/time grammar (`2030-12-31T23:59`, `2030/12/31T23:59`, `2030-12-31`, `12-31`, `14:00`, `:59`, `1`) in addition to FastSync's `HH:MM[:SS]` and `now+N[smhd]`; a past time stops immediately. Everything already transferred is kept and an early stop suppresses the late `--delete` keep-set so unscanned source mirrors survive. Works single-threaded and under `-j`/`--threads` |
| `--fsync` | Fsync every written file before publication | ✅ Parity | |
| `--protocol=NUM` | Force older protocol version | ❌ Divergent | Forces the wire protocol version for this transfer. FastSync has exactly ONE wire format (`PROTOCOL_VERSION`, currently 2.23.0) with no downgrade/backward-compat code paths, so `--protocol=2.23.0` is accepted (it sets the version claim the client sends, which the server already requires to match exactly) and **every other value is rejected up front** with a clear error before any connection — it does not and cannot speak an older or virtual wire format. Divergence from rsync (which negotiates a range and downgrades to an integer 0..31): FastSync's honest contract is force-to-the-one-supported-value; a genuine downgrade would require a per-version compatibility layer that does not exist. Client-only; the server-side exact-match check is unchanged. `--protocol=2.21.0`/`2.20.0`/`2.19.0`/`2.18.0`/`2.18`/`2.17.0`/`2.16.0`/`2.15.0`/`216`/`31`/garbage are all rejected. See the Phase-6 protocol note below |
| `--iconv=CONVERT_SPEC` | Charset conversion | ⚠️ Caveat | Charset conversion of FILE NAMES (not content) at the protocol boundary via iconv(3): `--iconv=LOCAL[,REMOTE]` — the sender converts each local filename LOCAL→REMOTE before transmitting, and the receiver converts each wire filename REMOTE→LOCAL before creating/writing. The full CONVERT_SPEC is serialized into the config frame as a new trailing string field so the peer knows the wire charset; **PROTOCOL_VERSION bumped 2.15.0 → 2.16.0**. `LOCAL[,REMOTE]` parse: single charset ⇒ LOCAL==REMOTE (identity both ways); garbage rejected up front. Validation probes BOTH directions (a spec that only opens one way is refused, as is a NUL-emitting target charset like utf-16/utf-32/ucs-2, since filenames cannot contain NUL). An unrepresentable name (EILSEQ/EINVAL) fails that path cleanly with a logged `--iconv: cannot convert file name ...` and is never written mangled/truncated. Conversion is applied at EVERY wire-path site (regular/MKDIR/hardlink path+target/symlink path+target/SPECIAL, the delete manifest, the incremental-check path, and the `-s`/`chunk_serialize` embedded blob path), on both client and server (`--iconv` is also a server/daemon option). Zero overhead when unset. See the Phase-6 iconv notes below |
| `--protocol=NUM` | Force older protocol version | ❌ Divergent | Forces the wire protocol version for this transfer. FastSync has exactly ONE wire format (`PROTOCOL_VERSION`, currently 2.28.0) with no downgrade/backward-compat code paths, so `--protocol=2.28.0` is accepted (it sets the version claim the client sends, which the server already requires to match exactly) and **every other value is rejected up front** with a clear error before any connection — it does not and cannot speak an older or virtual wire format. Divergence from rsync (which negotiates a range and downgrades to an integer 0..31): FastSync's honest contract is force-to-the-one-supported-value; a genuine downgrade would require a per-version compatibility layer that does not exist. Client-only; the server-side exact-match check is unchanged. `--protocol=2.27.0`/`2.26.0`/`2.25.0`/`2.24.0`/`2.23.0`/`2.22.0`/`2.21.0`/`2.20.0`/`2.19.0`/`2.18.0`/`2.17.0`/`2.16.0`/`2.15.0`/`216`/`31`/garbage are all rejected. See the Phase-6 protocol note below |
| `--iconv=CONVERT_SPEC` | Charset conversion | ✅ Parity | Charset conversion of FILE NAMES (not content) at the protocol boundary via iconv(3): `--iconv=LOCAL[,REMOTE]` — the sender converts each local filename LOCAL→REMOTE before transmitting, matching rsync's rule that the spec "stays the same whether you're pushing or pulling": on a PUSH the destination end's charset is the spec's REMOTE half, so the default receiver writes the wire bytes verbatim, and only a server started with its own `--iconv` (the daemon `charset` analog) declares a different destination charset and converts REMOTE→that LOCAL (rsync push parity, differential-tested with and without a server `--iconv`). The full CONVERT_SPEC is serialized into the config frame as a new trailing string field so the peer knows the wire charset; **PROTOCOL_VERSION bumped 2.15.0 → 2.16.0**. `LOCAL[,REMOTE]` parse: single charset ⇒ LOCAL==REMOTE (identity both ways); garbage rejected up front; protocol 2.26.0 additionally accepts `--iconv=.` (the locale's default charset for both directions), `--iconv=-` and `--no-iconv` (disable conversion). Validation probes BOTH directions (a spec that only opens one way is refused, as is a NUL-emitting target charset like utf-16/utf-32/ucs-2, since filenames cannot contain NUL). An unrepresentable name (EILSEQ/EINVAL) fails that path cleanly with a logged `--iconv: cannot convert file name ...` and is never written mangled/truncated. Conversion is applied at EVERY wire-path site (regular/MKDIR/hardlink path+target/symlink path+target/SPECIAL, the delete manifest, the incremental-check path, and the `-s`/`chunk_serialize` embedded blob path), on both client and server (`--iconv` is also a server/daemon option). Zero overhead when unset. See the Phase-6 iconv notes below |
| `--checksum-seed=NUM` | Set checksum seed | ✅ Parity | Sets the seed for FastSync's whole-file xxHash digest (full 64-bit seed) and for the delta path's per-block xxHash32 strong checksum (low 32 bits of the seed). **As of protocol 2.23.0 a seed of `0` — the default when the flag is unset — is randomized per transfer and the chosen seed is sent to the receiver**, exactly like rsync, so two runs against different content do not share a predictable seed; an explicit non-zero seed is used verbatim, so an explicit seed deterministically reproduces every computed digest on BOTH endpoints (the seed crosses in the config frame). `--checksum-choice=md5` has no seed and ignores it (documented). The value is a strict decimal 0..2⁶⁴-1 (blank, signed, or non-numeric values are rejected). Like rsync, a seed only matters where a digest is actually computed (`--checksum` or a basis-dir run, or a delta transfer); it does not by itself enable `--checksum`/`--delta` |
| `--secluded-args`, `-s` | Use protocol to send args | ❌ Divergent | Accepted for CLI compatibility (including the rsync short `-s`, Phase 7 Wave A) but a documented **no-op / divergence**. rsync's `-s` protects arguments from shell expansion by shipping them over the protocol; FastSync never passes remote arguments through a shell expansion boundary in the first place — its SSH transport builds the remote argv as **single-quote-escaped shell words** (`ssh_build_remote_command`), so the injection/leak that `-s` guards against does not exist and there is nothing to "seclude". Implementing a true arg-send protocol would mean replacing the argv-based SSH launch with an in-band argument channel, a large redesign of the transport that buys no security here. Chunk serialization remains the long-only `--chunk-serialization`. |
| `--protect-args` | Old name of --secluded-args | ❌ Divergent | Accepted for CLI compatibility as a documented no-op; the same rationale as `--secluded-args`/`-s` (FastSync's remote SSH argv is already built injection-safe, so there is no argument-leak to close) |
| `--no-OPTION` | Turn off implied option | ✅ Parity | Supported boolean FastSync options and archive-implied options; unsafe or value-taking options are rejected. |
---
@@ -824,9 +885,9 @@ These are the hardest compatibility items because they require durable formats o
**Phase 6, Wave A (stop deadline) shipping note:** `--stop-after=MINS` and `--stop-at=TIME` are client-only sender stop deadlines. `--stop-after` takes a positive minute count (0/negative/garbage rejected); `--stop-at` takes `HH:MM`, `HH:MM:SS`, or `now+N[smhd]` (a past time stops immediately, a garbage spec is rejected at parse time). The deadline is computed once at the start of the transfer (CLOCK_MONOTONIC for `--stop-after`, wall clock via `time()` for `--stop-at`) and checked at every chunk boundary in both the single-threaded `send_files` loop and the multithreaded `send_chunks_multithreaded` path, and inside the scanner loops so a busy scan itself stops. When it fires, the transfer stops ELEGANTLY: the in-flight chunk completes, the existing completion tail runs (summary, `disconnect`), and the run returns 0 — exactly like rsync's clean early stop. Because the deadline is client-only and never crosses the wire config frame, no PROTOCOL_VERSION bump is required. The safety-critical interaction is with `--delete`: FastSync streams while scanning, so a deadline can cut the source scan short and yield a PARTIAL keep-set manifest; committing that would make the receiver delete destination mirrors of source files not yet scanned. So the sender tracks `scan_stopped_early` and, when it is true on the late/delete-after (`--delete`/`--delete-after`/`--delete-delay`) path, SUPPRESSES the keep-set manifest (logs a warning) so no deletion happens from an incomplete set — this is the safe direction (preserves data; the delete simply does not run). `--delete-before`/`--delete-during` are unaffected: their complete pre-scan runs before any data and ignores the deadline (a stop can be exceeded by that pre-scan). Under `-j`/`--threads` the stop is symmetric and the scanner thread's still-in-progress manifest appends can never race the tail because the tail does not read the manifest on the early-stop path.
**Phase 6, Wave B (iconv) shipping note (PROTOCOL 2.15.0 → 2.16.0):** `--iconv=LOCAL[,REMOTE]` converts file NAMES at the wire boundary (never content). The full CONVERT_SPEC is serialized into the config frame as a new trailing string field (empty→NULL canonicalized), so both ends share the same wire charset interpretation; this required the PROTOCOL bump because the frame is a strict ordered sequence and a peer that does not parse the new trailing field would desynchronize. Each end derives LOCAL (its own charset) and REMOTE (the wire charset): the sender opens LOCAL→REMOTE and converts every transmitted filename; the receiver opens REMOTE→LOCAL and converts every received filename before creating/writing. Conversion is applied at every wire-path site (regular/MKDIR/hardlink path+target/symlink path+target/SPECIAL, the delete manifest keep/protected/missing entries, the incremental-check path, and the embedded `-s`/chunk-blob path). A name it cannot convert (EILSEQ/EINVAL) is failed cleanly with a logged `--iconv: cannot convert file name ...` and is never written truncated/mangled. Validation probes both directions up front (both the sender local→remote and the receiver remote→local, and, for a server/daemon with its own `--iconv`, the client-REMOTE→server-LOCAL pair) so an unusable spec is rejected before the connection rather than mid-transfer, and NUL-emitting target charsets (utf-16/utf-32/ucs-2) are refused because filenames cannot contain NUL. Divergence documented upstream: the receiver does NOT half-swap; the wire charset always comes from the sender's REMOTE half, so a server whose local charset differs from the client's LOCAL must declare it with its own `--iconv`. Conversion is process-global and runs on a single thread per process (sender thread / receiver-loop thread), initialized before worker threads start and freed after they join.
**Phase 6, Wave B (iconv) shipping note (PROTOCOL 2.15.0 → 2.16.0):** `--iconv=LOCAL[,REMOTE]` converts file NAMES at the wire boundary (never content). The full CONVERT_SPEC is serialized into the config frame as a new trailing string field (empty→NULL canonicalized), so both ends share the same wire charset interpretation; this required the PROTOCOL bump because the frame is a strict ordered sequence and a peer that does not parse the new trailing field would desynchronize. Each end derives its charset and the wire charset: the sender opens LOCAL→REMOTE and converts every transmitted filename; on a push the receiver's destination charset is the spec's REMOTE half, so it writes the wire bytes verbatim, unless the server was started with its own `--iconv` naming a different LOCAL charset (then it opens REMOTE→that LOCAL). Conversion is applied at every wire-path site (regular/MKDIR/hardlink path+target/symlink path+target/SPECIAL, the delete manifest keep/protected/missing entries, the incremental-check path, and the embedded `-s`/chunk-blob path). A name it cannot convert (EILSEQ/EINVAL) is failed cleanly with a logged `--iconv: cannot convert file name ...` and is never written truncated/mangled. Validation probes both directions up front (both the sender local→remote and the receiver remote→destination, and, for a server/daemon with its own `--iconv`, the client-REMOTE→server-LOCAL pair) so an unusable spec is rejected before the connection rather than mid-transfer, and NUL-emitting target charsets (utf-16/utf-32/ucs-2) are refused because filenames cannot contain NUL. The wire charset always comes from the sender's REMOTE half; a server whose local charset differs from the client's REMOTE must declare it with its own `--iconv` (the daemon `charset` analog). Conversion is process-global and runs on a single thread per process (sender thread / receiver-loop thread), initialized before worker threads start and freed after they join.
**Phase 6, Wave C (protocol-version) shipping note (no PROTOCOL_VERSION change):** `--protocol=NUM` lets the client force the wire protocol version for a transfer. FastSync's protocol is a single lockstep format: the config frame is a strict ordered sequence and the server requires the client's version string to equal `PROTOCOL_VERSION` exactly (`config_receive_with_validate`, src/shared/config.c) — there are no older-format code paths and no downgrade/negotiation machinery, so a lower/higher/virtual version can never be spoken. The honest contract is therefore: `--protocol=2.23.0` (the current `PROTOCOL_VERSION`, as of the rsync-parity wave) is accepted and stored into the client's `version` claim (which `config_send` already transmits), and every other value — `2.22.0`, `2.21.0`, `2.20.0`, `2.19.0`, `2.18.0`, `2.18`, `2.17.0`, `2.16.0`, `2.15.0`, `3.0.0`, rsync-integer spellings like `216`/`31`, garbage, empty — is rejected up front in `validate_config()` before any connection, with a clear error that FastSync supports only its current wire protocol and cannot speak an older or virtual one. Implementation is client-only: a server-side `--protocol` is intentionally not added because the server has no negotiation (it only enforces exact match), and it could only ever be the current version. This preserves (and slightly tightens) existing validation: the client now also refuses to launch with a version it cannot actually speak, rather than only the server rejecting it later. A genuine downgrade would require a per-version compatibility layer for every frame/feature added since (append 2.10, preallocate 2.11, hardlinks 2.12, devices/specials/symlink-trust/xattr 2.13, remote-option 2.14, daemon module/auth 2.15, iconv 2.16, dir/symlink times 2.17, privilege flags --super/--copy-as 2.18, SCRAM daemon auth 2.19, packed metadata 2.20, error-detail/dry-run 2.21, preserve-attribute split 2.22, rsync-parity wave 2.23) and is intentionally out of scope — documented divergences from rsync's integer-negotiated downgrade remain.
**Phase 6, Wave C (protocol-version) shipping note (no PROTOCOL_VERSION change):** `--protocol=NUM` lets the client force the wire protocol version for a transfer. FastSync's protocol is a single lockstep format: the config frame is a strict ordered sequence and the server requires the client's version string to equal `PROTOCOL_VERSION` exactly (`config_receive_with_validate`, src/shared/config.c) — there are no older-format code paths and no downgrade/negotiation machinery, so a lower/higher/virtual version can never be spoken. The honest contract is therefore: the current `PROTOCOL_VERSION` (2.28.0 as of the parity 2.28.0 cycle) is accepted and stored into the client's `version` claim (which `config_send` already transmits), and every other value — `2.25.0`, `2.24.0`, `2.23.0`, `2.22.0`, `2.21.0`, `2.20.0`, `2.19.0`, `2.18.0`, `2.18`, `2.17.0`, `2.16.0`, `2.15.0`, `3.0.0`, rsync-integer spellings like `216`/`31`, garbage, empty — is rejected up front in `validate_config()` before any connection, with a clear error that FastSync supports only its current wire protocol and cannot speak an older or virtual one. Implementation is client-only: a server-side `--protocol` is intentionally not added because the server has no negotiation (it only enforces exact match), and it could only ever be the current version. This preserves (and slightly tightens) existing validation: the client now also refuses to launch with a version it cannot actually speak, rather than only the server rejecting it later. A genuine downgrade would require a per-version compatibility layer for every frame/feature added since (append 2.10, preallocate 2.11, hardlinks 2.12, devices/specials/symlink-trust/xattr 2.13, remote-option 2.14, daemon module/auth 2.15, iconv 2.16, dir/symlink times 2.17, privilege flags --super/--copy-as 2.18, SCRAM daemon auth 2.19, packed metadata 2.20, error-detail/dry-run 2.21, preserve-attribute split 2.22, rsync-parity wave 2.23) and is intentionally out of scope — documented divergences from rsync's integer-negotiated downgrade remain.
**Phase-1/2 selection-and-update status correction (docs):** `-I/--ignore-times`, `--size-only`, `-@/--modify-window`, `--existing`, `--ignore-existing`, `-u/--update`, `-W/--whole-file`, and `--compress-threads` were previously listed as not-implemented in this document but are in fact fully implemented and tested on `dev`. This pass corrects the matrix to match the code. The realistic model of these is that FastSync is a *sender-driven* whole-tree copy, so the size+mtime quick-check and all three receiver-policy skips (`--existing`, `--ignore-existing`, `-u`) are evaluated against the **destination** on the receiver side, and their booleans cross the wire in the config frame. `-I`/`--size-only`/`--modify-window` modify the `--incremental` per-file `STATUS_CHECK` handshake's match predicate (`-I` disables the mtime leg and forces transfer; `--size-only` drops only the mtime leg; `--modify-window` adds tolerance to `metadata_mtime_matches`); they require `--incremental` (or a basis dir) to have a handshake to affect, mirroring how they only matter where a quick-check exists in rsync. `--existing`/`--ignore-existing`/`-u` are receiver write-time policies (skipping the write / newer-destination guard) applied across the regular-file, `--delay-updates`-staged, hardlink-sibling, and special/device paths; `-u` implies `-M` metadata and uses a second-then-nanosecond strict `>` newer check; both correctly influence `--remove-source-files` (a skipped source is not removed). `-W/--whole-file` disables block-level delta (opt-in via `--delta`), folded into the wire `use_delta` so no protocol bump was needed, and makes `--fuzzy` inert; `--append`/`--append-verify` are rejected with `-W`. `--compress-threads=NUM` (1..64, client-only, never crosses the wire) sizes the zstd compression worker pool. No code was changed by this correction; the implementation had landed in earlier merge waves (feat/ignore-times, feat/ignore-existing via the newer `file_to_disk_secure_no_replace`/`linkat EEXIST` path, feat/size-only, feat/modify-window, feat/whole-file, feat/update, compression-threads).
@@ -849,13 +910,13 @@ These are the last compatibility items and the closing phase toward rsync flag p
| `-T` / `--timeout` | `-T` = `--temp-dir` | → `--timeout` (long-only) |
| `-a` / `--archive` (= `-c -m -M`) | `-a` = `-rlptD` | → becomes **real rsync `-a`** after the renames |
**Wave B — Output & filesystem completion (✅ implemented).** `-S`/`--sparse` (`⚠️→✅`): real hole preservation — a sparse-aware writer (`write_all_sparse`) skips all-zero runs ≥ 4096 bytes with `lseek(SEEK_CUR)` and `ftruncate`s the final size, wired into both the atomic temp+rename store and `--inplace` receiver-side with **no wire change** (the full file image is already in memory; the ftruncate presize is kept). `-P` (`⚠️→✅`): interrupted-write retention — on a save failure after data reached the temp fd, `--partial` now renames the already-written temp to the destination path (best-effort; falls through to the normal unlink on failure, never retains when `--partial` is off) so a later `--append`/`--append-verify` run can resume. `--block-size=SIZE` (`⚠️→✅`): promoted after verification — `--block-size` is now an alias for `--delta-block`, both set `config->delta_block_size`, which the delta engine already honored end-to-end (`delta_signature_create_seeded` + `delta_apply`); out-of-range values keep the default. `--fake-super` (`⚠️→✅`): added `fake_super_restore_fd` to parse and re-apply the recorded `user.fastsync.stat` record fd-relative (mode/time only — protocol 2.23.0: **never a real chown**; the resolved owner is recorded for a later privileged restore); a save under `--fake-super` now re-applies the recorded attrs instead of only recording them, with the recording format unchanged. `--stderr=client` (`⚠️→❌ Divergent`): FastSync has no rsync client-message channel, and `client` is rejected at CLI parse — the rejection is the documented behavior (unit-tested). `-N`/`--crtimes` (`⚠️→❌ Divergent`): birth-times cannot be set by any portable fs call (`utimensat` sets only atime/mtime); capture/transmit stays, setting is impossible, the flag is accepted and safely inert. Review-hardening (post-eval): fake-super replay applies the mode through the shared `metadata_mode_for_policy` helper (protocol 2.23.0: exactly the source mode under `-p`, with no masking); `--sparse` takes precedence over `--preallocate` (posix_fallocate skipped so holes survive); `--partial` retention is disabled for `--no_replace` (ignore/existing) and only marks a write-attempt after the actual write begins; `--block-size=SIZE`/`--delta-block=SIZE` inline forms are accepted.
**Wave B — Output & filesystem completion (✅ implemented).** `-S`/`--sparse` (`⚠️→✅`): real hole preservation — a sparse-aware writer (`write_all_sparse`) skips all-zero runs ≥ 4096 bytes with `lseek(SEEK_CUR)` and `ftruncate`s the final size, wired into both the atomic temp+rename store and `--inplace` receiver-side with **no wire change** (the full file image is already in memory; the ftruncate presize is kept). `-P` (`⚠️→✅`): interrupted-write retention — on a save failure after data reached the temp fd, `--partial` now renames the already-written temp to the destination path (best-effort; falls through to the normal unlink on failure, never retains when `--partial` is off) so a later `--append`/`--append-verify` run can resume. `--block-size=SIZE` (`⚠️→✅`): promoted after verification — `--block-size` is now an alias for `--delta-block`, both set `config->delta_block_size`, which the delta engine already honored end-to-end (`delta_signature_create_seeded` + `delta_apply`); out-of-range values keep the default. `--fake-super` (`⚠️→✅`): added `fake_super_restore_fd` to parse and re-apply the recorded `user.fastsync.stat` record fd-relative (mode/time only — protocol 2.23.0: **never a real chown**; the resolved owner is recorded for a later privileged restore); a save under `--fake-super` now re-applies the recorded attrs instead of only recording them, with the recording format unchanged. `--stderr=client` (`⚠️→❌ Divergent`): FastSync has no rsync client-message channel, and `client` is rejected at CLI parse — the rejection is the documented behavior (unit-tested). `-N`/`--crtimes` (`⚠️→❌ Divergent`): birth-times cannot be set by any portable fs call (`utimensat` sets only atime/mtime); capture/transmit stays, setting is impossible, the flag is accepted and safely inert. Review-hardening (post-eval): fake-super replay applies the mode through the shared `metadata_mode_for_policy` helper (protocol 2.23.0: exactly the source mode under `-p`, with no masking); `--sparse` takes precedence over `--preallocate` (posix_fallocate skipped so holes survive) — **reversed by the parity-completion wave: `--preallocate` now wins, matching rsync**; `--partial` retention is disabled for `--no_replace` (ignore/existing) and only marks a write-attempt after the actual write begins; `--block-size=SIZE`/`--delta-block=SIZE` inline forms are accepted.
**Wave C — Devices & special files (finalize statuses + tests) (✅ implemented).** The four special-file rows are finalized with coverage tests. `--devices`, `--copy-devices`, and `--write-devices` are **✅ Implemented**, each with a documented, safety-driven divergence: device-node creation is privilege-gated, so a receiver without `CAP_MKNOD` skips that entry with a warning (a per-entry skip, never a transfer failure); `--copy-devices` copies a device/FIFO's reported size into an ordinary regular file (a size-bounded safe divergence from rsync's unbounded dd-like read); `--write-devices` writes only into an existing char/block node under the confined receive root and skips every unusable target rather than clobbering or aborting. `--specials` reclassified from **⛔ Impossible/Divergence** to **✅ Parity** in protocol 2.23.0: **FIFO recreation works** (unprivileged `mkfifo`) **and unix sockets are recreated** with `mknod(S_IFSOCK)`, which Linux permits unprivileged (the flag previously assumed sockets were impossible — see the `--specials` row). Tests assert FIFO recreation, socket recreation, the regular-file result of `--copy-devices`, the skipped/missing and non-device `--write-devices` targets, and (root-gated) real device-node creation; a root runner additionally drops the receiver to an unprivileged user to assert the `CAP_MKNOD` skip is graceful.
**Wave C — Devices & special files (finalize statuses + tests) (✅ implemented).** The four special-file rows are finalized with coverage tests. `--devices`, `--copy-devices`, and `--write-devices` are **✅ Implemented**, each with a documented, safety-driven divergence: device-node creation is privilege-gated, so a receiver without `CAP_MKNOD` skips that entry with a warning (a per-entry skip, never a transfer failure); `--copy-devices` copies a device/FIFO's reported size into an ordinary regular file (a size-bounded safe divergence from rsync's unbounded dd-like read); `--write-devices` writes only into an existing char/block node under the confined receive root and skips every unusable target rather than clobbering or aborting. `--specials` reclassified from **⛔ Impossible/Divergence** to **✅ Parity** in protocol 2.23.0: **FIFO recreation works** (unprivileged `mkfifo`) **and unix sockets are recreated** with `mknod(S_IFSOCK)`, which Linux permits unprivileged (the flag previously assumed sockets were impossible — see the `--specials` row). Tests assert FIFO recreation, socket recreation, the regular-file result of `--copy-devices`, the skipped/missing and non-device `--write-devices` targets, and (root-gated) real device-node creation; a root runner additionally drops the receiver to an unprivileged user to assert the `CAP_MKNOD` skip is graceful. (The parity-completion wave later reclassified `--devices`, `--copy-devices`, and `--write-devices` as explicit **❌ Divergent** rows, because their safe subsets are deliberately not rsync's behavior; the implementation itself is unchanged.)
**Wave D — Times superstructure & arg-protection no-ops (✅ implemented, `--secluded-args` ❌).** `-O`/`--omit-dir-times` and `-J`/`--omit-link-times` are now **real modifiers** (both `🔄 → ✅ Implemented`), reversing the old "never preserves directory/symlink times" divergence:
- **Directory times.** The recursive scanner captures every traversed source directory's metadata (mtime, plus atime under `-U`) into a per-transfer list — two paths are covered: the sequential `DirectoryScanner` captures each opened directory (including the transfer root), and the parallel scanner captures both the root in `parallel_scanner_create_with_options` and each worker's subdirectories in `open_next_directory` (appends are guarded by a mutex shared with the sender's pipeline context). The sender transmits them in trailing `STATUS_DIR_TIMES` frames (each: int count + count × (wire path, metadata) pairs) sent **after all file data and after the optional delete manifest**, just before `STATUS_FINISHED`. A tree larger than `MAX_MANIFEST_ENTRIES` (1 048 576) directories is chunked into repeated frames, each within the receiver's per-frame bound. A dir-time entry is RECORD-ONLY (`file->dir_time_only`): `file_save_to_disk_full` returns `FILE_SAVE_SKIPPED` without creating anything, so a source directory that was empty (or pruned by `-m/--prune-empty-dirs`) is never resurrected. The receiver accumulates received directory metadata in a `DirTimeList` and applies it only at the very end — after the entire stream, after the commit-style `--delete` deletion, and after `--delay-updates` publication — because creating or removing a child bumps the parent's mtime. Application is fd-relative/walk-confined (`file_open_secure_parent` + `utimensat(..., AT_SYMLINK_NOFOLLOW)`) and best-effort per entry: an absent path (an intentionally uncreated empty dir) is skipped QUIETLY and only a real existing directory is stamped. `-O` (config boolean, already on the wire) makes the receiver skip the whole set. The single-threaded sink applies in `receiver_send_success_frame`; the `-j`/`--threads` sink accumulates in `write_thread` and server.c applies after both threads join and the deletion commits.
- **Directory times.** The recursive scanner captures every traversed source directory's metadata (mtime, plus atime under `-U`) into a per-transfer list — two paths are covered: the sequential `DirectoryScanner` captures each opened directory (including the transfer root), and the parallel scanner captures both the root in `parallel_scanner_create_with_options` and each worker's subdirectories in `open_next_directory` (appends are guarded by a mutex shared with the sender's pipeline context). The sender transmits them in trailing `STATUS_DIR_TIMES` frames (each: int count + count × (wire path, metadata) pairs) sent **after all file data and after the optional delete manifest**, just before `STATUS_FINISHED`. A tree larger than `MAX_MANIFEST_ENTRIES` (1 048 576) directories is chunked into repeated frames, each within the receiver's per-frame bound. A dir-time entry is RECORD-ONLY (`file->dir_time_only`): `file_save_to_disk_full` returns `FILE_SAVE_SKIPPED` without creating anything (the directory's creation, when it is empty, is now carried by a separate `STATUS_MKDIR` entry the scanner emits for every directory that produced no transferred child, and `-m/--prune-empty-dirs` suppresses that). The receiver accumulates received directory metadata in a `DirTimeList` and applies it only at the very end — after the entire stream, after the commit-style `--delete` deletion, and after `--delay-updates` publication — because creating or removing a child bumps the parent's mtime. Application is fd-relative/walk-confined (`file_open_secure_parent` + `utimensat(..., AT_SYMLINK_NOFOLLOW)`) and best-effort per entry: an absent path (an intentionally uncreated empty dir) is skipped QUIETLY and only a real existing directory is stamped. `-O` (config boolean, already on the wire) makes the receiver skip the whole set. The single-threaded sink applies in `receiver_send_success_frame`; the `-j`/`--threads` sink accumulates in `write_thread` and server.c applies after both threads join and the deletion commits.
- **Symlink times/owner/mode.** `STATUS_SYMLINK` already carried metadata; the receiver now applies it with no-follow primitives only: `utimensat(..., AT_SYMLINK_NOFOLLOW)`, best-effort `fchmodat(..., AT_SYMLINK_NOFOLLOW)` (honest no-op where unsupported, e.g. Linux), and policy-gated `fchownat(..., AT_SYMLINK_NOFOLLOW)` via a new `identity_apply_ownership_link` that shares the identity resolver with the fd path. `-J` suppresses only the timestamps; ownership stays governed by the identity opt-in (`--numeric-ids`/`--usermap`/`--groupmap`/`--chown`) exactly like regular files. A symlink has no children, so this is applied immediately at creation.
- **Wire:** the shared `STATUS_DIR_TIMES` frame (and metadata on `STATUS_MKDIR` for `--dirs` entries) is a frame-sequence change, so `PROTOCOL_VERSION` was bumped **2.16.0 → 2.17.0**; every version-sensitive test (`--protocol` accepted/rejected values) was updated. The config-frame layout itself is unchanged (the omit booleans already crossed). Non-metadata and `--no-preserve` transfers send no `STATUS_DIR_TIMES` frame and no directory metadata, keeping them byte-identical.
@@ -869,7 +930,10 @@ These are the last compatibility items and the closing phase toward rsync flag p
**Wire:** two trailing config-frame blocks after the `--iconv` spec, in fixed order — `send_privilege_options`/`receive_privilege_options` (one `super_mode` int, validated `0..2`), then `send_copy_as_options`/`receive_copy_as_options` (presence int + two int32 ids, validated `>= 0`, with `copy_as_set ⇒ use_metadata`). `PROTOCOL_VERSION` bumped **2.17.0 → 2.18.0**. **Divergences from rsync:** rsync's `--super` elevates the receiver and `--copy-as` actually switches its credentials; FastSync never elevates and only permits/forwards confined attempts, and `--copy-as` forces ownership rather than switching identity.
**Current honest status (protocol 2.23.0).** ✅ Parity 83 / ⚠️ Caveat 63 / ❌ Divergent 4 = 150 rows. Earlier revisions of this document reported "143 ✅ / 0 divergence / 0 partial"; that conflated "parsed and tested" with "rsync parity", because many rows carried documented behavioral differences and some short options were not parsed at all. The reclassification makes the differences explicit and the rsync-parity wave closed the genuine gaps (short options, clustering, checksum/compression choices, seed randomization, timeout defaults, delete scoping and partial limits, verbatim symlink storage, socket recreation, `--chmod`, and more — see the next section). The four ❌ rows are `--stderr=client` (no rsync client-message channel), `-N/--crtimes` (no portable setter), `--protocol=NUM` (only the current wire version is accepted), and `-s/--secluded-args` (accepted no-op). `--specials` is now ✅ because sockets are recreated with `mknod(S_IFSOCK)`. **No `❌ Not Implemented` rows remain.**
**Honest status after the parity 2.28.0 cycle (protocol 2.28.0), updated by the rsync-parity-stats, rsync-parity-options, rsync-parity-fs, parity-review, no-wire parity-track-1/2b and wire parity-track-4a/5a passes.** ✅ Parity 116 / ⚠️ Caveat 14 / ❌ Divergent 27 = 157 rows. Earlier revisions of this document reported "143 ✅ / 0 divergence / 0 partial"; that conflated "parsed and tested" with "rsync parity", because many rows carried documented behavioral differences and some short options were not parsed at all. This reclassification makes every difference explicit. The completion wave closed 23 previously-caveated rows (9 that triage showed were already parity, plus 14 genuine fixes) and turned the 17 inherently non-rsync rows — native daemon config/auth, the FastSync batch container, the safe-subset device/privilege flags, `-X`'s privileged namespaces, `--fake-super`'s native xattr format, and the `--old-args` no-op — into explicit ❌ divergences. The stats pass flipped `--delete-delay` to ✅ (actual-removal accounting), but the parity-review pass moved it back to ⚠️ because FastSync charged the `--max-delete` budget at plan/snapshot time and left a refilled snapshotted directory in place, whereas rsync charges on actual removals and recursively removes a queued directory (including content created after its plan). The no-wire parity-track-1 pass fixed both (actual-removal charging plus recursive deferred removal with an independent deferred-list cap), narrowing the caveat to the partial-delete ordering. The stats pass also reclassified `--out-format` to ❌ (protocol-specific `%b`/delta-`%c`), and sharpened the `--stats`/`--progress`/`--checksum-choice` residuals. The options pass flipped `--bwlimit` and `--ignore-errors` to ✅ (rsync-exact size parsing and ~100 ms leaky-bucket throttling, and rsync's skip-unreadable-subdir plus IO-error-suppressed deletion with exit 23) and emits rsync-format `--info=name/flist/del/remove/nonreg/progress` lines (real-run `deleting`/`*deleting` carried over a new trailing `report_deletes` wire bool, `PROTOCOL_VERSION` 2.26.0 → 2.27.0), while reclassifying `-M` over daemon/TCP
and receiver-side `protect`/`risk` re-derivation to ❌ (no argv channel /
receiver filter engine); the wire parity-track-4a pass later added that
receiver filter engine, flipping `--filter=RULE` back to ✅ (see above). The fs pass flips `-d/--dirs` and `--iconv` to ✅ — recursive transfers now recreate empty source directories (and replace a blocking destination non-directory with an incoming directory); `-R --no-implied-dirs --files-from` places a listed file under a missing implied parent with default attributes instead of refusing; and `--iconv` now reproduces rsync's push direction (destination charset = the spec's REMOTE half) — and reclassified six rows to ❌ after reproducing their exact residual with differential tests: `--temp-dir` (the receiver confines the scratch dir to the receive root, so an absolute temp dir is deliberately rejected although standalone rsync follows it), the three basis-dir options (FastSync xxHash-verifies a basis hit while rsync's `--size-only` quick check installs the wrong basis content), `--delay-updates` (fixed staging name wipes an unrelated destination entry of that name), and `--dry-run` (would-delete report over-reports). `--fuzzy` was also reclassified to ❌ (deterministic heuristic with a 10× size window, not rsync's matcher), but its residual is the candidate-selection heuristic itself: the final tree is byte-exact by design, so no destination differential can expose it and the row is pinned by the `TestFuzzy` threshold suite rather than a byte-level rsync differential. (Track 5b later showed the name heuristic is in fact rsync's own and moved the row ❌ → ⚠️, leaving only the narrower delta size window as the residual; see the track 5b paragraph above.) The remaining ⚠️ rows are the ones with a documented residual (see the row notes and the **Parity Completion Wave (protocol 2.26.0)** section below).
**Preserve-attribute split (protocol 2.21.0 → 2.22.0) — ✅ implemented.** FastSync splits the former single metadata bundle into four independent, rsync-compatible per-attribute flags — `-p/--perms`, `-t/--times`, `-o/--owner`, `-g/--group` — each with a negation (`--no-perms`/`--no-times`/`--no-owner`/`--no-group`, short `--no-p`/`--no-t`/`--no-o`/`--no-g`), plus `--no-preserve` clearing all four. `-a/--archive` is now full rsync `-rlptgoD` (owner and group included, though their application stays privilege-gated), `-A/--acls` implies `-p`, `-X/--xattrs` does not, `-E/--executability` sets only executability, and `-U`/`-N` do not imply `-t`. `--incremental`/`--delta` still auto-preserve perms+times unless the user explicitly negated them. Wire: the binary config frame gains four appended booleans (`preserve_perms`/`preserve_times`/`preserve_owner`/`preserve_group`) after `omit_link_times`, so `PROTOCOL_VERSION` is bumped **2.21.0 → 2.22.0**; the fixed-width `FileMetadata` layout is unchanged and the receiver gates the metadata frame on a derived `use_metadata`. Receiver behavior: each attribute is applied independently, directory modes are applied under `-p` (at the end of the transfer, alongside dir times), symlink mode under `-p`, and `-O/--omit-dir-times` suppresses directory times only. Documented divergences as of 2.22.0, **all but (d)/(e) removed by the rsync-parity wave (protocol 2.23.0)**: (a) the mode-masking divergence is **gone** — under `-p` the source mode is now copied exactly, including `S_IWGRP`/`S_IWOTH` and setuid/setgid/sticky; (b) a brand-new file without `-p` still gets `source_mode & ~umask` when metadata is present (else the historical fixed `0644`), and a new *directory* without `-p` still uses FastSync's `0755` default; (c) the `--chmod`-implies-`-p` divergence is **gone** — `--chmod` no longer implies `-p` (rsync parity); (d) `-o`/`-g` map by name on the receiver with a raw-numeric fallback (only numeric ids cross the wire); (e) a daemon module without `client owner = yes` does not refuse a plain `-a`/`-o`/`-g` — it forces super off, applies no ownership, and logs a warning, while explicit `--chown`/`--usermap`/`--groupmap`/`--numeric-ids`/`--copy-as`/`--super` are still refused.
@@ -1011,6 +1075,141 @@ These remain after the wave; they are the reasons a row above is ⚠️.
`--protocol` accepts only the current version and `-s`/`--secluded-args` is an
accepted no-op.
## Parity Completion Wave (protocol 2.26.0)
This wave closed the remaining rsync-parity gaps left by the rsync-parity wave
and reclassified the inherently non-rsync rows as **divergent**. It moved the
wire protocol three times (full rationale in `src/shared/config.h`):
- **2.23.0 → 2.24.0 (delete timing):** the sender streams one delete plan per
source directory (`STATUS_DELETE_PLAN`) so `--delete-during`/`--delete-delay`
reproduce rsync's per-directory deletion timing.
- **2.24.0 → 2.25.0 (wire stats):** the config frame gains `report_stats` and
the receiver emits a `STATUS_STATS` frame carrying the receiver-only counters
(matched data, deleted-file count) and, for `-n --delete`, the would-delete
paths.
- **2.25.0 → 2.26.0 (codecs):** the config frame gains the negotiated
`compression_algo` int, and the checksum codec accepts `md4`/`sha1`/`none`
(default `xxh128`, negotiated with `auto`).
### Deletion timing (2.24.0)
- **`--delete-during`/`--del`** streams a per-directory plan as each directory
is scanned, so its extras are removed before the next directory's data; a
mid-transfer abort has already deleted the reached directories' extras.
- **`--delete-delay`** records each directory's plan while scanning and commits
the removals only after the whole transfer succeeds, so an extra created
mid-transfer after its directory's plan survives, while `--delete-after`'s
fresh end scan removes it.
- Per-directory plans are scoped to `-R`'s transferred prefix and bounded by the
shared manifest caps; `-d`/`--dirs` (no descent) falls back to the end commit.
- Empty in-scope source directories survive the per-directory delete. Dry-run
never deletes; `-n --delete` prints the would-delete lines (see below).
### Receiver stats and output (2.25.0)
- **`STATUS_STATS`** is sent immediately before the terminal success status and
carries `Matched data`, the deleted-file count, and the dry-run would-delete
path list. The client reads it before the `--remove-source-files` acks so the
counters are always populated, in both the sequential and `--threads` paths.
- **`--stats`** prints octets/rates/file counts from the sender plus the
receiver counters above; the protocol-independent lines match rsync exactly.
- **`--progress`/`-P`** print rsync-style per-file blocks (the first frame is
byte-identical) from the wire counters.
- **`--out-format`** gains `%b` (FastSync wire bytes), `%c` (block-sum bytes)
and `%C` (whole-file digest). `%C` is protocol-independent and matches rsync
for a whole-file transfer.
- **`-n --delete`** prints escaped `*deleting` lines from the receiver's
would-delete list.
### Codec breadth and negotiation (2.26.0)
- **Compression:** `zstd` (default), `lz4`, `zlib`, `zlibx`, `none`, `auto`.
The resolved codec id crosses the wire and the receiver validates it against
its own set (rsync's "no common choice is an error").
- **Checksums:** `xxh128` (default), `xxh3`, `xxh64`/`xxhash`, `md5`, `md4`,
`sha1`, `none`, `auto`, plus the two-name `transfer,pre-transfer` form.
Unknown names and `none` on the transfer side under `--checksum` exit 4 like
rsync.
- **Codec residuals:** the transfer checksum is not independently selectable
(only the whole-file comparison digest is); `zlib`/`zlibx` share FastSync's
literal-only zlib path (rsync's zlibx behavior, observably identical for both,
so `zlibx` is not a divergence). Per-codec compression-level defaults and
`RSYNC_COMPRESS_LIST`/`RSYNC_CHECKSUM_LIST` are implemented (track 3a).
### Selection, paths, and filters
- **General `-R`/`--relative`** implements the `/./` cut and prefix-scoped
deletion; **`--no-implied-dirs`** stops implied-parent attribute application.
- **`-d`/`--dirs`** implements rsync's one-level listing for `dir`, `dir/` and
`.`, with `STATUS_MKDIR` directory entries in the delete manifest.
- **Filter grammar:** `merge`/`.`, `dir-merge`/`:`, `hide`/`H`, `show`/`S`,
`protect`/`P`, `risk`/`R`, `clear`/`!`, include/exclude and the `:`/`.`
modifiers; `-f` is bound to `--filter`; a single `-F` transfers
`.rsync-filter` and `-FF` excludes it.
- **Absolute basis directories** are used verbatim (rsync semantics) and
**`--link-dest`** relinks an already up-to-date destination.
### Client quick wins and aliases
- `--iconv=.`/`-`/`--no-iconv`; a lone `-h` prints help; an empty
`--files-from` succeeds (exit 0); a broken referent under `-L`/
`--copy-unsafe-links` exits 23; the full `--info`/`--debug` vocabularies; and
the aliases `--ignore-non-existing`, `--protect-args`, `--msgs2stderr`.
- Receiver-side `--chown`/`--usermap`/`--groupmap` TO-name resolution; a
receiver-side `--ignore-existing` short-circuit before any payload; and
`--preallocate` now wins over `--sparse` via `fallocate(2)`.
### Residuals and intentional divergences
These remain after the wave; the individual rows carry the precise wording.
- **`--stats`** now reproduces rsync's `(reg/dir/link/special)` breakdown on
`Number of files`, the regular-transferred count and the size totals, but
`Number of created files` is the transferred-regular count without a type
breakdown and wire-byte totals differ; **`--progress`** now prints the leading
`./` line and includes the root in `to-chk` (single-file output is
byte-identical), but a multi-directory `to-chk` denominator and per-directory
name lines still differ; **`--out-format`** `%C` matches for every algorithm,
but `%b`/`%c` count FastSync wire bytes (protocol-specific, hence ❌).
- **`-n --delete`** ordering can differ from rsync's delete-during walk (the
differential compares the sorted would-delete set).
- **Delete timing:** plain `--delete` now defaults to rsync's delete-during
(lockstep track 6); the residual is the mid-transfer abort boundary, where
rsync's generator removes all extras ahead of its throttled sender while
FastSync removes only the reached directories (completed runs agree).
`--delete-before` keeps its pre-scan snapshot race; and while
base-rule `protect`/`risk` rules are now re-applied on the receiver (track 4a),
per-directory merge (`.rsync-filter`) protection is still sender-derived, so a
destination-only entry matching only a per-directory rule is not re-derived.
`--ignore-errors` exits 23 but its EACCES differential is not
exercised in CI.
- **`--delay-updates`** uses a fixed staging name with an advisory lock and
deletes before publication; **`--temp-dir`** rejects absolute/foreign paths
(deliberately confined, see the row); **`--remote-option`** is SSH-only.
**`--iconv`** now matches rsync's push direction (destination charset = the
spec's REMOTE half; a server `--iconv` overrides it).
- **Basis dirs** now use rsync's metadata quick-check by default (track 5a) and
stream a hit of any size; the FastSync-only `--verify-basis` restores the
stricter content equality. Remaining residuals: the relative-DIR resolution
base and the over-limit basis-MISS refusal. **`--fuzzy`** uses rsync's
name heuristic, but its candidate eligibility is bounded by the delta
engine (both files ≥ 16 KiB, size ratio ≤ 10×), a narrower window than
rsync's, so the selected basis — and the `--stats` bandwidth counters —
can differ while the tree stays byte-exact; and **`--bwlimit`** rejects rsync's
`0`/decimal/suffixed rates.
- **`--inc-recursive`/`--no-inc-recursive`** are not implemented (rejected).
### Intentional divergences (explicit ❌ rows)
Native daemon config/auth (`--daemon`, `--config`, `--dparam`,
`--password-file`, `--early-input`, `--hash-credentials`/`--iterations`), the
non-interoperable batch container (`--write-batch`/`--only-write-batch`/
`--read-batch`), `--fake-super`'s native xattr format, `-X`'s privileged
namespaces, `--devices`/`--copy-devices`/`--write-devices`'s safe subsets,
`--super`/`--copy-as`'s refusal to elevate or switch credentials, and the
`-s`/`--secluded-args`/`--protect-args`/`--old-args` accepted no-ops.
## Packed Metadata Frame (protocol 2.20.0)
A file's metadata used to cross the wire as up to 12 separate per-field framed
@@ -1072,5 +1271,7 @@ Ranked by user demand, implementation complexity, and interoperability impact (_
| `--tls` | TLS encryption (mutual auth) |
| `--fastsync-server-path` | Path to fastsync-server binary |
| `--server-host` / `--server-port` | Direct TCP connection |
| `--verify-basis` | FastSync-only (long form, not in rsync): require a `--compare-dest`/`--copy-dest`/`--link-dest` basis hit to match the source by whole-file digest instead of trusting rsync's size+mtime (or `--size-only`) quick-check. Off by default (the default matches rsync). Crosses the wire (protocol 2.28.0) |
| `--delete-commit` | FastSync-only (long form, not in rsync): restore the late whole-tree delete commit — the keep-set manifest is committed only after the entire transfer succeeded. Identical timing to `--delete-after`, and implemented as the same `delete_after` wire bool (no new field); it exists because plain `--delete` now defaults to `--delete-during` (lockstep track 6). Implies `--delete` and conflicts with any different timing flag |
| Incremental sync | Skip unchanged files (size+mtime) |
| Delta transfer | Block-level delta for changed files |
+3
View File
@@ -8,3 +8,6 @@ markers =
daemon_detach: real double-fork backgrounding path (--daemon without
--no-detach); slower/fragile, so it runs in the full suite but not the
fast PR gate
parity: differential rsync-parity case (full set; runs on push to
dev/main)
parity_ci: fast differential rsync-parity subset (runs on the PR gate)
+2
View File
@@ -38,6 +38,8 @@ pkgs.mkShell {
buildInputs = with pkgs; [
zstd
zlib
lz4
openssl
];
+222 -15
View File
@@ -1,5 +1,8 @@
#include "change_list.h"
#include "checksum.h"
#include "log.h"
#include "utils.h"
#include <fcntl.h>
#include <limits.h>
#include <stdint.h>
#include <stdio.h>
@@ -68,7 +71,16 @@ static bool strbuf_append(StrBuf* buf, const char* text) {
bool change_list_enabled(const Config* config) {
return config != NULL && (config->itemize_changes || config->out_format != NULL ||
(config->log_file != NULL && config->log_file_format != NULL));
(config->log_file != NULL && config->log_file_format != NULL) ||
(config->info_level & LOG_INFO_NAME) != 0);
}
/* Emitted once, lazily, ahead of the first --info=name entry: rsync prints the
* transfer-root `./` name line when the root directory is (re)created. */
static bool name_root_printed = false;
void change_reset_name_root(void) {
name_root_printed = false;
}
/* ---- Itemize code ---- */
@@ -156,14 +168,6 @@ static void itemize_code(const Config* config, const ChangeEvent* event, char co
code[11] = '\0';
}
char* change_render_itemize_code(const Config* config, const ChangeEvent* event) {
if (event == NULL || event->decision != CHANGE_SENT)
return str_dup("");
char code[12];
itemize_code(config, event, code);
return str_dup(code);
}
/* rsync %n: the transfer-relative name, with a trailing slash for directories. */
static bool append_name(StrBuf* buf, const ChangeEvent* event) {
if (!strbuf_append(buf, event->name != NULL ? event->name : ""))
@@ -198,8 +202,121 @@ char* change_render_itemize(const Config* config, const ChangeEvent* event) {
return line.data;
}
/* rsync's `--info=name` line for an updated entry: the transfer-relative name
* (trailing slash for directories) plus the ` -> target` / ` => target` link
* suffix. `--info=name` does not alter an itemize/out-format run. */
static char* change_render_name(const ChangeEvent* event) {
StrBuf line = {0};
bool ok = append_name(&line, event) && append_link_suffix(&line, event);
if (!ok) {
strbuf_free(&line);
return NULL;
}
if (line.data == NULL) {
line.data = str_dup("");
if (!line.data)
return NULL;
}
return line.data;
}
/* rsync's `--info=name2` line for an unchanged entry: `NAME is uptodate`. */
static char* change_render_name_uptodate(const ChangeEvent* event) {
char* name = change_render_name(event);
if (name == NULL)
return NULL;
size_t length = strlen(name);
char* line = malloc(length + sizeof(" is uptodate"));
if (line == NULL) {
free(name);
return NULL;
}
memcpy(line, name, length);
memcpy(line + length, " is uptodate", sizeof(" is uptodate"));
free(name);
return line;
}
/* ---- --out-format / --log-file-format ---- */
/* rsync 3.4.1's `%C` uses the negotiated TRANSFER checksum (the first name of a
* two-name "transfer,pre-transfer" --checksum-choice), not the pre-transfer
* whole-file digest FastSync compares against on the wire. The default "auto"
* resolves to xxh128, so an explicit selection and the default both render the
* selected algorithm's digest. */
static ChecksumAlgo out_format_checksum_algo(const Config* config) {
return (ChecksumAlgo)config->checksum_transfer_algo;
}
/* Render a digest as rsync's sum_as_hex: xxh128 prints the HIGH 64-bit half
* before the low half, and xxh64/xxh3 print their 64-bit value big-endian; every
* other algorithm prints its bytes in order. */
static void digest_to_hex(ChecksumAlgo algo, const uint8_t* digest, size_t len, char* out) {
if (algo == CHECKSUM_ALGO_XXH128 && len == 16) {
uint64_t low = 0;
uint64_t high = 0;
memcpy(&low, digest, sizeof(low));
memcpy(&high, digest + 8, sizeof(high));
snprintf(out, len * 2 + 1, "%016llx%016llx", (unsigned long long)high, (unsigned long long)low);
return;
}
if ((algo == CHECKSUM_ALGO_XXH64 || algo == CHECKSUM_ALGO_XXH3) && len == 8) {
uint64_t value = 0;
memcpy(&value, digest, sizeof(value));
snprintf(out, len * 2 + 1, "%016llx", (unsigned long long)value);
return;
}
static const char hex[] = "0123456789abcdef";
for (size_t i = 0; i < len; i++) {
out[i * 2] = hex[(digest[i] >> 4) & 0xf];
out[i * 2 + 1] = hex[digest[i] & 0xf];
}
out[len * 2] = '\0';
}
static bool format_uses_checksum(const char* format) {
if (format == NULL)
return false;
for (const char* p = format; *p != '\0';) {
if (*p != '%') {
p++;
continue;
}
char token = p[1];
if (token == '\0')
break;
if (token == 'C')
return true;
p += 2;
}
return false;
}
/* Fill event->checksum/checksum_known for a transferred regular file. A
* non-regular entry (or a hard-link sibling) leaves checksum_known false, which
* renders as spaces like rsync. */
static void fill_event_checksum(const Config* config, const File* file, ChangeEvent* event) {
if (file == NULL || file->is_dir || file->is_symlink || file->is_special ||
(file->link_group != 0 && !file->link_first))
return;
if (!format_uses_checksum(config->out_format) && !format_uses_checksum(config->log_file_format))
return;
if (file->path == NULL)
return;
ChecksumAlgo algo = out_format_checksum_algo(config);
/* rsync renders `--checksum-choice=none` as a blank 2-character column. */
if (algo == CHECKSUM_ALGO_NONE)
return;
uint8_t digest[CHECKSUM_MAX_DIGEST_LEN];
size_t len = 0;
/* rsync's %C is the transfer checksum, which is always seeded with 0 (it is
* independent of --checksum-seed, as rsync 3.4.1 demonstrates). */
if (!checksum_digest_file(algo, 0, file->path, digest, sizeof(digest), &len))
return;
digest_to_hex(algo, digest, len, event->checksum);
event->checksum_known = true;
}
char* change_render_format(const char* format, const Config* config, const ChangeEvent* event) {
if (format == NULL || event == NULL)
return NULL;
@@ -221,6 +338,11 @@ char* change_render_format(const char* format, const Config* config, const Chang
ok = strbuf_append_char(&line, '%');
break;
case 'i': {
if (event->deleted) {
/* rsync's ITEM_DELETED itemize code: `*deleting ` (11 chars). */
ok = strbuf_append(&line, "*deleting ");
break;
}
char code[12];
itemize_code(config, event, code);
ok = strbuf_append(&line, code);
@@ -245,6 +367,23 @@ char* change_render_format(const char* format, const Config* config, const Chang
int written = snprintf(digits, sizeof(digits), "%llu", event->bytes_sent);
ok = written >= 0 && (size_t)written < sizeof(digits) && strbuf_append(&line, digits);
} break;
case 'c': {
char digits[32];
int written = snprintf(digits, sizeof(digits), "%llu", event->bytes_read);
ok = written >= 0 && (size_t)written < sizeof(digits) && strbuf_append(&line, digits);
} break;
case 'C': {
if (event->checksum_known) {
ok = strbuf_append(&line, event->checksum);
} else {
/* rsync pads a non-regular / untransferred / `none` entry with spaces;
`none` renders as a blank 2-character column. */
ChecksumAlgo algo = out_format_checksum_algo(config);
int width = algo == CHECKSUM_ALGO_NONE ? 2 : checksum_digest_len(algo) * 2;
for (int i = 0; i < width && ok; i++)
ok = strbuf_append_char(&line, ' ');
}
} break;
case 'M': {
char when[32];
if (format_rsync_datetime(event->mtime_sec, true, when, sizeof(when)))
@@ -347,10 +486,24 @@ static void print_escaped_line(FILE* stream, const char* line, bool eight_bit_ou
void change_emit(const Config* config, const ChangeEvent* event) {
if (event == NULL || !change_list_enabled(config))
return;
if (event->decision == CHANGE_UP_TO_DATE)
return;
bool to_stdout = config->itemize_changes || config->out_format != NULL;
bool to_log = config->log_file != NULL && config->log_file_format != NULL;
bool progress_active = config->show_progress || (config->info_level & LOG_INFO_PROGRESS);
if (event->decision == CHANGE_UP_TO_DATE) {
/* --info=name2 prints `NAME is uptodate` for entries the receiver already
had. An itemize/out-format run reports them through its own format (or
not at all), the progress stream has no frame for them, and neither the
itemize nor the log-file stream previously reported an up-to-date entry,
so nothing else here changes. */
if (!to_stdout && (config->info_level & LOG_INFO_NAME_UPTODATE) != 0 && !progress_active) {
char* line = change_render_name_uptodate(event);
if (line != NULL) {
print_escaped_line(stdout, line, config->eight_bit_output);
free(line);
}
}
return;
}
if (to_stdout) {
char* line = config->out_format != NULL
? change_render_format(config->out_format, config, event)
@@ -359,6 +512,20 @@ void change_emit(const Config* config, const ChangeEvent* event) {
print_escaped_line(stdout, line, config->eight_bit_output);
free(line);
}
} else if ((config->info_level & LOG_INFO_NAME) != 0 && !progress_active) {
/* --info=name without -i/--out-format: print the updated entry's name. The
--progress path owns the name line when progress output is active (it
emits the same names before the progress frames), so do not duplicate.
The transfer-root `./` line precedes the first such name. */
if (!name_root_printed) {
name_root_printed = true;
fputs("./\n", stdout);
}
char* line = change_render_name(event);
if (line != NULL) {
print_escaped_line(stdout, line, config->eight_bit_output);
free(line);
}
}
if (to_log) {
char* line = change_render_format(config->log_file_format, config, event);
@@ -464,7 +631,8 @@ static void fill_event_from_file(const Config* config, const File* file, ChangeE
}
}
void change_emit_file_sent(const Config* config, const File* file) {
void change_emit_file_sent_bytes(const Config* config, const File* file,
unsigned long long bytes_sent, unsigned long long bytes_read) {
if (file == NULL || !change_list_enabled(config))
return;
ChangeEvent event;
@@ -489,13 +657,52 @@ void change_emit_file_sent(const Config* config, const File* file) {
event.hardlink_target = file->hardlink_target;
event.bytes_sent = 0;
} else {
/* Literal payload bytes delivered; compressed/delta wire bytes are not
* separately counted. */
event.bytes_sent = event.size;
event.bytes_sent = bytes_sent;
/* rsync's %c is the block-checksum bytes received for the file. Even a
* whole-file transfer (no basis; --append/--inplace included) receives
* rsync's 16-byte sum header, so rsync reports 16; a dry run transfers
* nothing and reports 0. FastSync's whole-file path has no sum header, so
* report rsync's value for parity. With delta enabled the real received
* bytes are kept, but FastSync's signature framing differs from rsync's so
* those stay numerically divergent. */
bool delta_active = config->use_delta && !config->whole_file;
event.bytes_read = (!config->dry_run && !delta_active) ? 16 : bytes_read;
}
char* name = NULL;
char* path = NULL;
fill_event_from_file(config, file, &event, &name, &path);
if (name != NULL && path != NULL) {
fill_event_checksum(config, file, &event);
change_emit(config, &event);
}
free(name);
free(path);
}
void change_emit_file_sent(const Config* config, const File* file) {
if (file == NULL)
return;
unsigned long long payload = file->data != NULL ? file->data->size : 0;
change_emit_file_sent_bytes(config, file, payload, 0);
}
void change_emit_file_uptodate(const Config* config, const File* file) {
if (file == NULL || !change_list_enabled(config))
return;
ChangeEvent event;
memset(&event, 0, sizeof(event));
event.decision = CHANGE_UP_TO_DATE;
event.is_directory = false;
event.is_symlink = file->is_symlink;
event.is_special = file->is_special;
event.is_hardlink = file->link_group != 0 && !file->link_first;
event.symlink_target = file->symlink_target;
event.hardlink_target = file->hardlink_target;
event.size = file->data != NULL ? file->data->size : 0;
event.dest = file->dest_state;
char* name = NULL;
char* path = NULL;
fill_event_from_file(config, file, &event, &name, &path);
if (name != NULL && path != NULL)
change_emit(config, &event);
free(name);
+31 -6
View File
@@ -2,6 +2,7 @@
#define CHANGE_LIST_H
#include "config.h"
#include "checksum.h"
#include "file_types.h"
#include "format.h"
#include <stdbool.h>
@@ -34,10 +35,17 @@ typedef struct {
bool is_symlink;
bool is_special;
bool is_hardlink; /* a hard-link sibling (linked, no data sent) */
bool deleted; /* a would-delete report (-n --delete); no source file */
const char* symlink_target;
const char* hardlink_target;
unsigned long long size; /* source file length in bytes */
unsigned long long bytes_sent; /* literal data bytes actually transferred */
unsigned long long bytes_sent; /* wire bytes actually transferred (rsync %b) */
unsigned long long bytes_read; /* wire bytes read back for this file (rsync %c) */
/* rsync %C: whole-file checksum hex for a transferred regular file. Only
* filled when the active format uses %C (checksum_known == false otherwise,
* which renders as spaces like rsync for non-regular entries). */
bool checksum_known;
char checksum[CHECKSUM_MAX_DIGEST_LEN * 2 + 1];
time_t mtime_sec;
long mtime_nsec;
mode_t mode;
@@ -55,13 +63,13 @@ bool change_list_enabled(const Config* config);
* (`%i %n%L`): `>f+++++++++ sub/b.txt`. Caller frees the result. */
char* change_render_itemize(const Config* config, const ChangeEvent* event);
/* Render only the 11-character itemize code (rsync %i). Caller frees. */
char* change_render_itemize_code(const Config* config, const ChangeEvent* event);
/* Expand an --out-format/--log-file-format template. Supported tokens:
* %i itemize code %n transfer-relative name (dir: trailing /)
* %f long display path %l file length in bytes
* %b bytes actually sent %M mtime (YYYY/MM/DD-HH:MM:SS)
* %b wire bytes transferred %c block-checksum bytes received (rsync: 16
* for a whole-file transfer, 0 for a dry run)
* %C whole-file checksum hex (xxh128 by default; spaces for non-regular)
* %M mtime (YYYY/MM/DD-HH:MM:SS)
* %t current time %o operation ("send"/"del.")
* %p pid %B permission bits without the type char
* %U uid %G gid
@@ -80,10 +88,27 @@ char* change_render_list_line(const Config* config, const ChangeEvent* event);
* CHANGE_UP_TO_DATE events produce no output. */
void change_emit(const Config* config, const ChangeEvent* event);
/* Build and emit a CHANGE_SENT event for a file the client just sent. */
/* Build and emit a CHANGE_SENT event for a file the client just sent. `bytes_sent`
* is the process-wide wire-byte delta for this file (rsync's %b) and `bytes_read`
* the received bytes used for the delta handshake; pass 0 when unknown. For a
* whole-file transfer %c is pinned to rsync's 16-byte sum header regardless. */
void change_emit_file_sent_bytes(const Config* config, const File* file,
unsigned long long bytes_sent, unsigned long long bytes_read);
/* Build and emit a CHANGE_SENT event for a file the client just sent, deriving
* the wire byte counts from the source payload length. */
void change_emit_file_sent(const Config* config, const File* file);
/* Build and emit a CHANGE_SENT event for an explicit directory entry (-d). */
void change_emit_dir_sent(const Config* config, const File* file);
/* Build and emit a CHANGE_UP_TO_DATE event for a file the receiver already had.
* With --info=name2 it renders rsync's "NAME is uptodate" line (no output
* otherwise). */
void change_emit_file_uptodate(const Config* config, const File* file);
/* Reset the lazy transfer-root `./` line emitted ahead of the first
* --info=name entry. Call once at the start of a transfer. */
void change_reset_name_root(void);
#endif
+572 -83
View File
@@ -20,7 +20,10 @@
#include "utils.h"
#include <errno.h>
#include <fcntl.h>
#include <langinfo.h>
#include <limits.h>
#include <locale.h>
#include <math.h>
#include <time.h>
#include <signal.h>
#include <stdbool.h>
@@ -148,47 +151,111 @@ static int set_positive_int_option(int* dest, const char* value, const char* opt
}
/* Set and validate the compression algorithm selected by the client. rsync
* 3.4.1 can be built with zstd, none, lz4, zlibx, zlib and auto; FastSync only
* implements zstd (and no compression). "auto" is accepted as the default
* zstd choice; any other rsync choice is rejected by name instead of being
* silently accepted and ignored. */
* 3.4.1 can be built with zstd, none, lz4, zlibx, zlib and auto; all of those
* names are accepted and mapped to a real codec here. "auto" resolves through
* FastSync's compiled-in preference order (rsync 3.4.1's list). An unknown
* name is a hard error with rsync's exit code 4, never a silent no-op. */
static int set_compression_choice(Config* config, const char* value) {
/* rsync's "auto" is normalized to the canonical "zstd" at parse time (like
--checksum-choice=auto), so the value that crosses the wire is always one
the receiver accepts. */
const char* canonical = strcmp(value, "auto") == 0 ? "zstd" : value;
if (strcmp(canonical, "zstd") != 0 && strcmp(canonical, "none") != 0) {
log_message(LOG_LEVEL_ERROR,
"--compress-choice '%s' is not implemented; FastSync supports zstd, none or auto "
"(rsync's lz4/zlib/zlibx are rejected, never silently ignored)",
value);
if (!value) {
config->cli_exit_code = 4;
return -1;
}
int algo;
if (strcasecmp(value, "auto") == 0) {
algo = compression_choice_resolve();
if (algo < 0) {
log_message(LOG_LEVEL_ERROR, "RSYNC_COMPRESS_LIST names no supported compression algorithm");
config->cli_exit_code = 4;
return -1;
}
} else {
algo = compression_algo_from_name(value);
}
if (algo < 0) {
log_message(LOG_LEVEL_ERROR,
"--compress-choice '%s' is not a supported algorithm; FastSync supports zstd, "
"lz4, zlib, zlibx, none or auto",
value);
config->cli_exit_code = 4;
return -1;
}
const char* canonical = compression_algo_name((CompressionAlgo)algo);
if (set_string_option(&config->compress_choice, canonical, "--compress-choice") != 0)
return -1;
config->use_compression = strcmp(canonical, "none") != 0;
config->compression_algo = algo;
config->use_compression = (algo != (int)COMPRESSION_ALGO_NONE);
return 0;
}
/* Validate and store the --checksum-choice/--cc algorithm. Only the algorithms
* the engine genuinely supports are accepted (xxh64/xxhash, xxh3, xxh128, md5);
* rsync's compiled-in choices that FastSync does not implement (md4, sha1,
* none) and the two-name transfer/pre-transfer syntax are a clear error, never
* a silent no-op. "auto" (rsync's default automatic choice) selects FastSync's
* default algorithm. */
/* Store one algorithm name into *out. Returns 0 for a valid name, 1 for
* "auto" (caller resolves it), -1 for an unknown/too-long name. */
static int resolve_checksum_name(const char* name, size_t len, int* out) {
char buf[64];
if (len == 0 || len >= sizeof(buf))
return -1;
memcpy(buf, name, len);
buf[len] = '\0';
if (strcasecmp(buf, "auto") == 0)
return 1;
int algo = checksum_algo_from_name(buf);
if (algo < 0)
return -1;
*out = algo;
return 0;
}
/* Validate and store the --checksum-choice/--cc algorithm. rsync 3.4.1 accepts
* a single name (used for both the transfer and pre-transfer checksums) or the
* two-name "TRANSFER,PRE-TRANSFER" form (only one comma is significant). The
* pre-transfer half is FastSync's whole-file digest; the transfer half is
* validated for parity and, when "none", forces --whole-file like rsync. An
* unknown name (including an empty half or a second comma) is exit 4. "auto"
* resolves to FastSync's negotiated default (xxh128). */
static int set_checksum_choice(Config* config, const char* value) {
if (strcasecmp(value, "auto") == 0)
return 0;
int algo = checksum_algo_from_name(value);
if (algo < 0) {
log_message(LOG_LEVEL_ERROR,
"--checksum-choice '%s' is not implemented; FastSync supports xxh64 (or xxhash), "
"xxh3, xxh128, md5 or auto (rsync's md4/sha1/none and the two-name "
"transfer,pre-transfer form are rejected, never silently ignored)",
value);
if (!value) {
config->cli_exit_code = 4;
return -1;
}
config->checksum_algo = algo;
const char* comma = strchr(value, ',');
const char* name1 = value;
size_t len1 = comma ? (size_t)(comma - value) : strlen(value);
const char* name2 = comma ? comma + 1 : NULL;
size_t len2 = name2 ? strlen(name2) : 0;
int transfer = -1;
int pre = -1;
int rc1 = resolve_checksum_name(name1, len1, &transfer);
int rc2 = name2 ? resolve_checksum_name(name2, len2, &pre) : 1;
if (rc1 < 0 || rc2 < 0) {
log_message(LOG_LEVEL_ERROR,
"--checksum-choice '%s' is invalid; FastSync supports xxh64 (or xxhash), xxh128, "
"xxh3, md5, md4, sha1, none or auto, optionally as 'transfer,pre-transfer'",
value);
config->cli_exit_code = 4;
return -1;
}
int negotiated = -1;
if (rc1 == 1 || rc2 == 1) {
negotiated = checksum_choice_resolve();
if (negotiated < 0) {
log_message(LOG_LEVEL_ERROR, "RSYNC_CHECKSUM_LIST names no supported checksum algorithm");
config->cli_exit_code = 4;
return -1;
}
}
if (rc1 == 1)
transfer = negotiated;
if (!name2)
pre = transfer;
else if (rc2 == 1)
pre = negotiated;
config->checksum_algo = pre;
config->checksum_transfer_algo = transfer;
config->checksum_choice_set = true;
/* rsync: "none" for the transfer checksum forces --whole-file. */
if (transfer == (int)CHECKSUM_ALGO_NONE)
config->whole_file = true;
return 0;
}
@@ -327,10 +394,10 @@ static int config_add_remote_option(Config* config, const char* value, const cha
}
/* Validate and append one --compare-dest/--copy-dest/--link-dest directory.
* The path is interpreted on the receiver relative to the destination root,
* so it must be a non-empty relative path with no "." / ".." components (an
* absolute or escaping path is rejected up front instead of failing on the
* server). Returns 0 on success, -1 on error. */
* A relative path is interpreted on the receiver below the destination root; an
* absolute path is used verbatim on the receiver (matching rsync), still subject
* to the receiver's authorized-root confinement. Either way the path must be
* non-empty and traversal-free (no ".."). Returns 0 on success, -1 on error. */
static int set_basis_dest_option(Config* config, BasisDestType type, const char* value,
const char* option_name) {
if (!value || !value[0]) {
@@ -339,8 +406,9 @@ static int set_basis_dest_option(Config* config, BasisDestType type, const char*
}
if (config_basis_append(config, type, value) != 0) {
log_message(LOG_LEVEL_ERROR,
"%s requires a non-empty relative directory name with no '.', '..', or absolute "
"path (resolved below the destination root)",
"%s requires a non-empty directory name with no '..' component "
"(relative paths resolve below the destination root; absolute paths are used "
"verbatim)",
option_name);
return -1;
}
@@ -395,6 +463,72 @@ static void apply_output_buffering(const Config* config) {
static int read_patterns_from_file(const char* filepath, char*** patterns, int* count,
Config* config, char sign, const char* optname);
/* Split one --debug/--info item into its category name and an optional rsync
* verbosity level suffix (e.g. "io2", "all4", "none0"). The output `name` is
* NUL-terminated and `level` is >= 0 (0 silences the item). Returns false for
* an empty token or a token that is all digits. */
static bool split_flag_level(const char* token, char* name, size_t name_size, int* level) {
size_t len = strlen(token);
if (len == 0 || name_size == 0)
return false;
size_t end = len;
while (end > 0 && token[end - 1] >= '0' && token[end - 1] <= '9')
end--;
if (end == 0)
return false; /* all digits: not a category name */
size_t name_len = end < name_size - 1 ? end : name_size - 1;
for (size_t i = 0; i < name_len; i++) {
char c = token[i];
name[i] = (c >= 'A' && c <= 'Z') ? (char)(c - 'A' + 'a') : c;
}
name[name_len] = '\0';
int lvl = 1;
if (end < len) {
lvl = 0;
for (size_t i = end; i < len; i++) {
int digit = token[i] - '0';
if (lvl > (1000 - digit) / 10)
return false;
lvl = lvl * 10 + digit;
}
}
*level = lvl;
return true;
}
/* rsync --debug/--info categories that FastSync accepts for CLI parity but has
* no output wired to (yet). They must parse successfully so a valid rsync
* invocation is not rejected up front; only categories with a FastSync
* counterpart set a log flag. `pack`/`util` are FastSync-specific (packed
* metadata / general utility logging). `syms`, `hl`, and `owner` are aliases
* of rsync's `symsafe`, `hlink`, and `own`. */
static bool is_accepted_debug_category(const char* name) {
static const char* const categories[] = {
"acl", "backup", "bind", "chdir", "cmd", "connect", "del", "deltasum",
"dup", "exit", "filter", "flist", "fuzzy", "genr", "hash", "hl",
"hlink", "iconv", "nstr", "own", "owner", "recv", "send", "time",
};
for (size_t i = 0; i < sizeof(categories) / sizeof(categories[0]); i++) {
if (strcmp(name, categories[i]) == 0)
return true;
}
return false;
}
static bool is_accepted_info_category(const char* name) {
static const char* const categories[] = {
"backup",
"mount",
"syms",
"symsafe",
};
for (size_t i = 0; i < sizeof(categories) / sizeof(categories[0]); i++) {
if (strcmp(name, categories[i]) == 0)
return true;
}
return false;
}
static int parse_debug_flags(const char* value, Config* config) {
if (!value || value[0] == '\0' || value[0] == ',' || value[strlen(value) - 1] == ',' ||
strstr(value, ",,")) {
@@ -412,30 +546,42 @@ static int parse_debug_flags(const char* value, Config* config) {
for (char* token = strtok_r(flags, ",", &saveptr); token != NULL;
token = strtok_r(NULL, ",", &saveptr)) {
uint32_t flag = 0;
if (strcmp(token, "help") == 0) {
char name[32];
int level = 1;
if (!split_flag_level(token, name, sizeof(name), &level)) {
log_message(LOG_LEVEL_ERROR, "unsupported --debug flag: %s", token);
free(flags);
return -1;
}
if (strcmp(name, "help") == 0) {
print_debug_usage();
free(flags);
return 1;
} else if (strcmp(token, "all") == 0) {
parsed = LOG_DEBUG_ALL;
} else if (strcmp(name, "all") == 0) {
parsed = level == 0 ? 0 : LOG_DEBUG_ALL;
continue;
} else if (strcmp(token, "none") == 0) {
} else if (strcmp(name, "none") == 0) {
parsed = 0;
continue;
} else if (strcmp(token, "io") == 0) {
} else if (strcmp(name, "io") == 0) {
flag = LOG_DEBUG_IO;
} else if (strcmp(token, "proto") == 0) {
} else if (strcmp(name, "proto") == 0) {
flag = LOG_DEBUG_PROTO;
} else if (strcmp(token, "pack") == 0) {
} else if (strcmp(name, "pack") == 0) {
flag = LOG_DEBUG_PACK;
} else if (strcmp(token, "util") == 0) {
} else if (strcmp(name, "util") == 0) {
flag = LOG_DEBUG_UTIL;
} else if (is_accepted_debug_category(name)) {
continue;
} else {
log_message(LOG_LEVEL_ERROR, "unsupported --debug flag: %s", token);
free(flags);
return -1;
}
parsed |= flag;
if (level == 0)
parsed &= ~flag;
else
parsed |= flag;
}
free(flags);
config->debug_level = (int)parsed;
@@ -461,28 +607,67 @@ static int parse_info_flags(const char* value, Config* config) {
for (char* token = strtok_r(flags, ",", &saveptr); token != NULL;
token = strtok_r(NULL, ",", &saveptr)) {
uint32_t flag = 0;
if (strcmp(token, "all") == 0) {
parsed = LOG_INFO_ALL;
char name[32];
int level = 1;
if (!split_flag_level(token, name, sizeof(name), &level)) {
log_message(LOG_LEVEL_ERROR, "unsupported --info flag: %s", token);
free(flags);
return -1;
}
if (strcmp(name, "all") == 0) {
parsed = level == 0 ? 0 : LOG_INFO_ALL;
continue;
}
if (strcmp(token, "none") == 0) {
if (strcmp(name, "none") == 0) {
parsed = 0;
continue;
}
if (strcmp(token, "copy") == 0)
if (strcmp(name, "help") == 0) {
print_info_usage();
free(flags);
return 1;
}
if (strcmp(name, "copy") == 0)
flag = LOG_INFO_COPY;
else if (strcmp(token, "misc") == 0)
else if (strcmp(name, "name") == 0) {
/* name level 2 adds rsync's "is uptodate" lines. */
if (level == 0)
parsed &= ~(uint32_t)(LOG_INFO_NAME | LOG_INFO_NAME_UPTODATE);
else {
parsed |= LOG_INFO_NAME;
if (level >= 2)
parsed |= LOG_INFO_NAME_UPTODATE;
else
parsed &= ~(uint32_t)LOG_INFO_NAME_UPTODATE;
}
continue;
} else if (strcmp(name, "misc") == 0)
flag = LOG_INFO_MISC;
else if (strcmp(token, "skip") == 0)
else if (strcmp(name, "skip") == 0)
flag = LOG_INFO_SKIP;
else if (strcmp(token, "stats") == 0)
else if (strcmp(name, "stats") == 0)
flag = LOG_INFO_STATS;
else if (strcmp(name, "del") == 0)
flag = LOG_INFO_DEL;
else if (strcmp(name, "remove") == 0)
flag = LOG_INFO_REMOVE;
else if (strcmp(name, "flist") == 0)
flag = LOG_INFO_FLIST;
else if (strcmp(name, "nonreg") == 0)
flag = LOG_INFO_NONREG;
else if (strcmp(name, "progress") == 0)
flag = LOG_INFO_PROGRESS;
else if (is_accepted_info_category(name))
continue;
else {
log_message(LOG_LEVEL_ERROR, "unsupported --info flag: %s", token);
free(flags);
return -1;
}
parsed |= flag;
if (level == 0)
parsed &= ~flag;
else
parsed |= flag;
}
free(flags);
config->info_level = (int)parsed;
@@ -597,16 +782,25 @@ static int config_add_pattern(char*** patterns, int* count, const char* value,
/* Validate and append one --filter=RULE string. Returns 0 on success, -1 on error. */
static int config_add_filter(Config* config, const char* rule) {
char err[160];
FilterRule* parsed = filter_rule_parse(rule, err, sizeof(err));
if (!parsed) {
char err[256];
/* Validate through the full list parser so clear/merge/dir-merge and the rule
modifiers are accepted (and a merge file is readable) at parse time. */
FilterParseOptions opts = {.delete_excluded = config->delete_excluded,
.cvs_exclude = config->cvs_exclude};
FilterRuleList* probe = filter_rule_list_create();
if (!probe) {
log_message(LOG_LEVEL_ERROR, "memory allocation failed for --filter");
return -1;
}
bool ok = filter_rule_list_parse_append(probe, rule, &opts, NULL, err, sizeof(err));
filter_rule_list_free(probe);
if (!ok) {
char* escaped = output_escape(rule, log_get_8_bit_output());
log_message(LOG_LEVEL_ERROR, "invalid --filter rule '%s': %s",
escaped ? escaped : "<allocation failed>", err);
free(escaped);
return -1;
}
filter_rule_free(parsed);
if (!config->filters) {
config->filters = array_list_create(free);
if (!config->filters) {
@@ -734,6 +928,9 @@ static const OptionEntry OPTION_TABLE[] = {
{"--human-readable", "-h", OPT_FLAG, offsetof(Config, human_readable)},
{"--partial", NULL, OPT_FLAG, offsetof(Config, partial)},
{"--secluded-args", "-s", OPT_NOOP, 0},
/* rsync's pre-3.2.6 name for --secluded-args (--protect-args) is accepted
* as the same secure-argv no-op. */
{"--protect-args", NULL, OPT_NOOP, 0},
/* rsync -r/--recursive: FastSync is always recursive, so this is a
* faithful no-op (accepted silently, never consumes an argument). */
{"--recursive", "-r", OPT_NOOP, 0},
@@ -762,6 +959,8 @@ static const OptionEntry OPTION_TABLE[] = {
{"--out-format", NULL, OPT_STRING, offsetof(Config, out_format)},
{"--log-file-format", NULL, OPT_STRING, offsetof(Config, log_file_format)},
{"--existing", NULL, OPT_FLAG, offsetof(Config, existing)},
/* rsync's man-page alias for --existing (--ignore-non-existing). */
{"--ignore-non-existing", NULL, OPT_FLAG, offsetof(Config, existing)},
{"--ignore-existing", NULL, OPT_FLAG, offsetof(Config, ignore_existing)},
{"--delay-updates", NULL, OPT_FLAG, offsetof(Config, delay_updates)},
{"--chmod", NULL, OPT_STRING, offsetof(Config, chmod_spec)},
@@ -798,6 +997,12 @@ static const OptionEntry OPTION_TABLE[] = {
{"--delete-during", "--del", OPT_FLAG, offsetof(Config, delete_during)},
{"--delete-delay", NULL, OPT_FLAG, offsetof(Config, delete_delay)},
{"--delete-after", NULL, OPT_FLAG, offsetof(Config, delete_after)},
/* FastSync-only long spelling of the late whole-tree commit, which selects
the same timing as rsync's --delete-after in FastSync (the whole-tree
keep-set manifest is committed only after the entire transfer succeeded).
Plain --delete now defaults to delete-during, so this restores the old
FastSync behavior; it maps onto the same delete_after wire field. */
{"--delete-commit", NULL, OPT_FLAG, offsetof(Config, delete_after)},
{"--delete-excluded", NULL, OPT_FLAG, offsetof(Config, delete_excluded)},
{"--max-delete", NULL, OPT_SIGNED_INT, offsetof(Config, max_delete)},
{"--ignore-errors", NULL, OPT_FLAG, offsetof(Config, ignore_errors)},
@@ -855,6 +1060,11 @@ static const OptionEntry OPTION_TABLE[] = {
* --remote-option is parsed. --trust-sender is a local receiver policy and
* never travels to the remote peer. */
{"--trust-sender", NULL, OPT_FLAG, offsetof(Config, trust_sender)},
/* FastSync-only (not an rsync option): require a basis-hit's content to
* match the source by whole-file digest instead of trusting rsync's
* size+mtime quick-check. Long-only; crosses the wire so the receiver
* performs the extra read/hash. */
{"--verify-basis", NULL, OPT_FLAG, offsetof(Config, verify_basis)},
};
/* Only boolean options with no required argument are safe to negate. */
@@ -897,6 +1107,7 @@ static const NegatableOption NEGATABLE_OPTIONS[] = {
{"xattrs", "X", offsetof(Config, preserve_xattrs)},
{"acls", "A", offsetof(Config, preserve_acls)},
{"fake-super", NULL, offsetof(Config, fake_super)},
{"verify-basis", NULL, offsetof(Config, verify_basis)},
};
static bool opt_is(const char* arg, const char* name, const char* alias) {
@@ -976,6 +1187,26 @@ static int apply_negation(Config* config, const char* arg) {
return 0;
}
/* rsync's --iconv accepted extra spellings beyond explicit charset pairs:
* "." selects the locale's default charset for both directions, and "-" (or
* --no-iconv) disables conversion entirely. Normalize both here so the rest
* of the pipeline only ever sees a real charset spec or NULL. */
static int set_iconv_option(char** field, const char* value) {
if (value && strcmp(value, "-") == 0) {
free(*field);
*field = NULL;
return 0;
}
if (value && strcmp(value, ".") == 0) {
setlocale(LC_ALL, "");
const char* codeset = nl_langinfo(CODESET);
if (!codeset || codeset[0] == '\0')
codeset = "UTF-8";
return set_string_option(field, codeset, "--iconv");
}
return set_string_option(field, value, "--iconv");
}
static int apply_table_option(Config* config, const OptionEntry* entry, const char* value) {
if (entry->kind == OPT_NOOP)
return 0;
@@ -989,6 +1220,8 @@ static int apply_table_option(Config* config, const OptionEntry* entry, const ch
case OPT_STRING:
if (entry->offset == offsetof(Config, chmod_spec))
return append_chmod_spec((char**)field, value);
if (entry->offset == offsetof(Config, iconv_spec))
return set_iconv_option((char**)field, value);
return set_string_option((char**)field, value, entry->name);
case OPT_POS_INT:
return set_positive_int_option((int*)field, value, entry->name);
@@ -1027,17 +1260,22 @@ typedef struct {
} CliParseCtx;
/* Apply output controls before processing other options so their order is
* irrelevant. Returns 0 on success, -1 on error. */
* irrelevant. Returns 0 on success, a positive code for a help request
* (parse_args returns it verbatim), or -1 on error. */
static int cli_apply_output_controls(Config* config, int argc, char* argv[]) {
for (int i = 1; i < argc; i++) {
if (strcmp(argv[i], "-v") == 0 || strcmp(argv[i], "--verbose") == 0) {
set_log_level(LOG_LEVEL_DEBUG);
} else if (strncmp(argv[i], "--info=", 7) == 0) {
if (parse_info_flags(argv[i] + 7, config) != 0)
return -1;
int ret = parse_info_flags(argv[i] + 7, config);
if (ret != 0)
return ret;
} else if (strcmp(argv[i], "--info") == 0) {
if (i + 1 >= argc || parse_info_flags(argv[++i], config) != 0)
if (i + 1 >= argc)
return -1;
int ret = parse_info_flags(argv[++i], config);
if (ret != 0)
return ret;
}
}
return 0;
@@ -1062,6 +1300,19 @@ static bool cli_handle_pre_negation(CliParseCtx* ctx) {
config->no_implied_dirs = true;
return true;
}
/* "--no-iconv" is a real rsync option name that turns charset conversion off
* (the negation of the argument-taking --iconv), so it is handled before the
* generic --no-* negation branch. */
if (strcmp(arg, "--no-iconv") == 0) {
free(config->iconv_spec);
config->iconv_spec = NULL;
return true;
}
/* "--no-msgs2stderr" is the deprecated spelling of --stderr=client (rsync
* 3.4.1). FastSync has no separate client message channel, so the closest
* supported mode is the errors-only default. */
if (strcmp(arg, "--no-msgs2stderr") == 0)
return set_stderr_mode("errors") == 0;
/* "--no-motd" is a real rsync option name (client-side daemon MOTD display
* suppression), not a negation of a "--motd" flag, so it is handled before
* the generic --no-* negation branch. */
@@ -1240,6 +1491,8 @@ static bool cli_handle_table_option(CliParseCtx* ctx) {
ctx->exit_code = -1;
return true;
}
if (entry->offset == offsetof(Config, compression_level))
config->compression_level_set = true;
if (entry->offset == offsetof(Config, chmod_spec)) {
mode_t ignored;
if (!chmod_apply(0, config->chmod_spec, &ignored)) {
@@ -1260,8 +1513,15 @@ static bool cli_handle_table_option(CliParseCtx* ctx) {
}
if (entry->offset == offsetof(Config, eight_bit_output))
protocol_set_8_bit_output(true);
/* -F is repeatable: rsync's single -F transfers .rsync-filter files, a
repeated -FF excludes them. Count the occurrences so the scanner can
distinguish the two. */
if (entry->offset == offsetof(Config, per_dir_filter) && config->per_dir_filter_count < INT_MAX)
config->per_dir_filter_count++;
/* A delete-timing flag selects when --delete removes extras, so it
implies --delete exactly like the rsync options do. */
implies --delete exactly like the rsync options do. --delete-commit (the
FastSync-only late-commit spelling) is mapped onto delete_after and so is
covered here too. */
if (entry->offset == offsetof(Config, delete_before) ||
entry->offset == offsetof(Config, delete_during) ||
entry->offset == offsetof(Config, delete_delay) ||
@@ -1519,6 +1779,7 @@ static bool cli_handle_transfer_flags(CliParseCtx* ctx) {
return true;
}
config->compression_level = (int)level;
config->compression_level_set = true;
log_info_message(LOG_INFO_MISC, "Set Compression level to %ld", level);
ctx->i++;
}
@@ -1616,22 +1877,130 @@ static int set_log_file_option(Config* config, const char* log_path) {
return 0;
}
/* Apply a --bwlimit value (kilobytes per second). Returns 0 on success, -1 on
* error. */
/* Faithful port of rsync 3.4.1's `parse_size_arg(bwlimit_arg, 'K', "bwlimit",
* 512, -1, True)`: a default KiB suffix, binary (1024) multipliers unless a
* `b`/`B` decimal suffix or explicit `iB` is given, an optional decimal
* fraction, the P/T/G/M/K suffixes, and the special rules that a value of 0
* means "no limit" while any other value below 512 bytes is rejected. The
* parsed byte count is then quantized to whole KiB exactly like rsync's
* `bwlimit = (size + 512) / 1024`. Returns 0 on success, -1 on a parse error. */
static int parse_bwlimit_value(const char* value, unsigned long long* bytes_per_sec_out) {
if (!value || !bytes_per_sec_out)
return -1;
const char* arg = value;
int reps;
long long mult;
while (*arg >= '0' && *arg <= '9')
arg++;
if (*arg != '\0' && (*arg == '.' || *arg == localeconv()->decimal_point[0]))
for (arg++; *arg >= '0' && *arg <= '9'; arg++) {
}
char suffix = *arg && *arg != '+' && *arg != '-' ? *arg++ : 'K';
switch (suffix) {
case 'b':
case 'B':
reps = 0;
break;
case 'k':
case 'K':
reps = 1;
break;
case 'm':
case 'M':
reps = 2;
break;
case 'g':
case 'G':
reps = 3;
break;
case 't':
case 'T':
reps = 4;
break;
case 'p':
case 'P':
reps = 5;
break;
default:
log_message(LOG_LEVEL_ERROR, "--bwlimit=%s is invalid", value);
return -1;
}
if (*arg == 'b' || *arg == 'B') {
mult = 1000;
arg++;
} else if (*arg == '\0' || *arg == '+' || *arg == '-') {
mult = 1024;
} else if ((arg[0] == 'i' || arg[0] == 'I') && (arg[1] == 'b' || arg[1] == 'B')) {
mult = 1024;
arg += 2;
} else {
log_message(LOG_LEVEL_ERROR, "--bwlimit=%s is invalid", value);
return -1;
}
long long base = 1;
for (int i = 0; i < reps; i++) {
if (base > LLONG_MAX / mult) {
log_message(LOG_LEVEL_ERROR, "--bwlimit=%s is too large", value);
return -1;
}
base *= mult;
}
/* rsync multiplies the numeric prefix (atof) by mult^reps in a signed
* ssize_t, which is undefined on overflow. Scale in double and range-check
* before converting, so a huge value is rejected as "too large" (where
* rsync's overflow happens to land on a negative result) without invoking
* signed-overflow UB. */
double scaled = (double)base * strtod(value, NULL);
/* (double)LLONG_MAX rounds up to 2^63, which is itself out of range for the
* cast, so reject at >= that bound; LLONG_MIN == -2^63 is exactly
* representable and thus castable, so the lower bound stays strict. */
if (!isfinite(scaled) || scaled >= (double)LLONG_MAX || scaled < (double)LLONG_MIN) {
log_message(LOG_LEVEL_ERROR, "--bwlimit=%s is too large", value);
return -1;
}
long long size = (long long)scaled;
if ((*arg == '+' || *arg == '-') && arg[1] == '1' && arg != value) {
/* The only form accepted here is "+1"/"-1" (a longer number leaves a
trailing byte and is rejected below), so apply the delta directly and
guard the one overflow direction. */
if (*arg == '+') {
if (size == LLONG_MAX) {
log_message(LOG_LEVEL_ERROR, "--bwlimit=%s is too large", value);
return -1;
}
size += 1;
} else {
size -= 1;
}
arg += 2;
}
if (*arg != '\0' || size < 0) {
log_message(LOG_LEVEL_ERROR, "--bwlimit=%s is %s", value, size < 0 ? "too large" : "invalid");
return -1;
}
if (size != 0 && size < 512) {
log_message(LOG_LEVEL_ERROR, "--bwlimit=%s is too small (min: 512 or 0 for unlimited)", value);
return -1;
}
long long kib = size == 0 ? 0 : (size + 512) / 1024;
if (kib > (long long)(ULLONG_MAX / 1024)) {
log_message(LOG_LEVEL_ERROR, "--bwlimit=%s is too large", value);
return -1;
}
*bytes_per_sec_out = (unsigned long long)kib * 1024;
return 0;
}
/* Apply a --bwlimit value using rsync 3.4.1's units/semantics. Returns 0 on
* success, -1 on error. */
static int set_bwlimit_option(const char* value) {
unsigned long long kbps;
if (parse_ull_arg(value, &kbps, "--bwlimit") != 0)
unsigned long long bytes_per_sec;
if (parse_bwlimit_value(value, &bytes_per_sec) != 0)
return -1;
if (kbps == 0) {
log_message(LOG_LEVEL_ERROR, "--bwlimit must be a positive integer");
return -1;
}
if (kbps > ULLONG_MAX / 1024) {
log_message(LOG_LEVEL_ERROR, "--bwlimit value too large");
return -1;
}
io_set_bwlimit(kbps * 1024);
log_info_message(LOG_INFO_MISC, "Set bandwidth limit to %llu KB/s", kbps);
io_set_bwlimit(bytes_per_sec);
log_info_message(LOG_INFO_MISC, "Set bandwidth limit to %llu KB/s", bytes_per_sec / 1024);
return 0;
}
@@ -1732,6 +2101,12 @@ static bool cli_handle_io_options(CliParseCtx* ctx) {
ctx->exit_code = -1;
return true;
}
/* rsync's deprecated spelling of --stderr=all. */
if (opt_is(arg, "--msgs2stderr", NULL)) {
if (set_stderr_mode("all") != 0)
ctx->exit_code = -1;
return true;
}
return false;
}
@@ -1842,13 +2217,19 @@ static bool cli_handle_logging_options(CliParseCtx* ctx) {
return true;
}
if (strncmp(arg, "--info=", 7) == 0) {
if (parse_info_flags(arg + 7, config) != 0)
ctx->exit_code = -1;
int info_ret = parse_info_flags(arg + 7, config);
if (info_ret != 0)
ctx->exit_code = info_ret;
return true;
}
if (opt_is(arg, "--info", NULL)) {
if (ctx->i + 1 >= ctx->argc || parse_info_flags(ctx->argv[++ctx->i], config) != 0)
if (ctx->i + 1 >= ctx->argc) {
ctx->exit_code = -1;
return true;
}
int info_ret = parse_info_flags(ctx->argv[++ctx->i], config);
if (info_ret != 0)
ctx->exit_code = info_ret;
return true;
}
if (strncmp(arg, "--skip-compress=", 16) == 0) {
@@ -2194,8 +2575,70 @@ static bool cli_handle_outbuf_option(CliParseCtx* ctx) {
* -1 on error. */
static int cli_finalize_config(Config* config, bool verbose, bool no_delta, bool no_incremental) {
set_log_level(config->quiet ? LOG_LEVEL_ERROR : (verbose ? LOG_LEVEL_DEBUG : LOG_LEVEL_WARNING));
if (config->compress_choice)
config->use_compression = strcmp(config->compress_choice, "none") != 0;
/* rsync's plain --delete defaults to delete-during (--del): each directory's
extras are removed as that directory is processed, so space is freed
progressively and a tight destination never has to hold the whole old+new
tree at once. The late whole-tree commit FastSync historically used is
still selected explicitly by --delete-after or by the FastSync-only long
spelling --delete-commit (an exact alias for --delete-after). Resolve the
default on the client, before validation and before the config crosses the
wire, so exactly one timing flag is ever set; an explicit timing (including
--delete-commit) always wins. */
if (config->use_delete && !config->delete_before && !config->delete_during &&
!config->delete_delay && !config->delete_after)
config->delete_during = true;
if (config->compress_choice) {
int algo = compression_algo_from_name(config->compress_choice);
if (algo >= 0) {
config->compression_algo = algo;
config->use_compression = (algo != (int)COMPRESSION_ALGO_NONE);
}
}
/* A bare -z (no --compress-choice) resolves like rsync's "auto": the
* RSYNC_COMPRESS_LIST preference list first, then the compiled-in order. A
* list that names no supported codec is rsync's failed negotiation (exit 4). */
if (config->use_compression && !config->compress_choice) {
int resolved = compression_choice_resolve();
if (resolved < 0) {
log_message(LOG_LEVEL_ERROR, "RSYNC_COMPRESS_LIST names no supported compression algorithm");
config->cli_exit_code = 4;
return -1;
}
config->compression_algo = resolved;
if (resolved == (int)COMPRESSION_ALGO_NONE)
config->use_compression = false;
}
/* Apply rsync's per-codec compression level: an explicit --compress-level is
* clamped to the codec's range, otherwise the codec's own default is used. */
if (config->use_compression) {
CompressionAlgo algo = (CompressionAlgo)config->compression_algo;
config->compression_level = config->compression_level_set
? compression_clamp_level(algo, config->compression_level)
: compression_default_level(algo);
log_debug_message(LOG_DEBUG_UTIL, "Client compression: %s (level %d)",
compression_algo_name(algo), config->compression_level);
}
/* The negotiated checksum is always resolved (rsync negotiates one for the
* delta strong sum even without --checksum): RSYNC_CHECKSUM_LIST first, then
* the compiled-in order. An explicit --checksum-choice already set it. */
if (!config->checksum_choice_set) {
int resolved = checksum_choice_resolve();
if (resolved < 0) {
log_message(LOG_LEVEL_ERROR, "RSYNC_CHECKSUM_LIST names no supported checksum algorithm");
config->cli_exit_code = 4;
return -1;
}
config->checksum_algo = resolved;
config->checksum_transfer_algo = resolved;
}
/* rsync parity: "none" as the pre-transfer checksum cannot be combined with
* --checksum (exit 4). The check runs here because --checksum may appear on
* either side of --checksum-choice. */
if (config->checksum && config->checksum_algo == (int)CHECKSUM_ALGO_NONE) {
log_message(LOG_LEVEL_ERROR, "Invalid checksum-choice for --checksum: none");
config->cli_exit_code = 4;
return -1;
}
/* rsync randomizes the checksum seed for every transfer when the user did not
* supply one (a seed of 0, including an explicit --checksum-seed=0), using
@@ -2280,6 +2723,15 @@ static int cli_finalize_config(Config* config, bool verbose, bool no_delta, bool
config->preserve_times = true;
}
/* --ignore-existing is a receiver-side existence policy: the receiver must
* answer "skip" BEFORE the sender transmits any payload, which only the
* per-file STATUS_CHECK handshake provides. Imply --incremental here (after
* the auto-preserve capture above, so a bare --ignore-existing does not gain
* -p/-t, which rsync likewise does not imply) so an existing destination is
* skipped on the wire instead of being streamed and discarded. */
if (config->ignore_existing)
config->use_incremental = true;
/* Derive the transport bit from the FINAL parsed flags. Every
* preservation/ownership option that needs the metadata frame (per-attribute
* perms/times/owner/group, atimes/crtimes, executability, xattrs/acls,
@@ -2297,6 +2749,28 @@ static int cli_finalize_config(Config* config, bool verbose, bool no_delta, bool
* check. This is a wire field. */
config->report_dest_info = config->itemize_changes || config->out_format != NULL ||
(config->log_file != NULL && config->log_file_format != NULL);
/* Wire-stats parity: --stats, --progress/-P, an --out-format token that needs
* a wire counter (%b/%c), or a dry-run --delete need the receiver's
* end-of-transfer STATUS_STATS report. This is a wire field (protocol
* 2.25.0). */
bool format_needs_wire = false;
if (config->out_format != NULL) {
for (const char* p = config->out_format; *p != '\0'; p++) {
if (p[0] == '%' && (p[1] == 'b' || p[1] == 'c')) {
format_needs_wire = true;
break;
}
}
}
/* --info=del on a real --delete run asks the receiver to report the paths it
actually removed; the report rides the STATUS_STATS path list, so the wire
stats frame must be negotiated too. */
config->report_deletes = config->use_delete && !config->dry_run &&
((config->info_level & LOG_INFO_DEL) != 0 || config->itemize_changes ||
config->out_format != NULL);
config->report_stats = config->stats || config->show_progress ||
(config->info_level & LOG_INFO_PROGRESS) || format_needs_wire ||
config->report_deletes || (config->dry_run && config->use_delete);
return 0;
}
@@ -2436,8 +2910,18 @@ int parse_args(Config* config, int argc, char* argv[], int* positional_args,
}
int result = -1;
if (cli_apply_output_controls(config, exp_argc, exp_argv) != 0)
/* rsync treats a lone -h as a help request (it only means human-readable
* when combined with a source/destination or other options). */
if (exp_argc == 2 && strcmp(exp_argv[1], "-h") == 0) {
print_usage();
result = 1;
goto done;
}
int output_ret = cli_apply_output_controls(config, exp_argc, exp_argv);
if (output_ret != 0) {
result = output_ret;
goto done;
}
CliParseCtx ctx = {
.config = config,
@@ -2620,7 +3104,7 @@ int main(int argc, char* argv[]) {
int parse_ret = parse_args(config, argc, argv, positional_args, &positional_count);
if (parse_ret != 0) {
if (parse_ret < 0)
exit_code = 1;
exit_code = config->cli_exit_code ? config->cli_exit_code : 1;
goto cleanup;
}
@@ -2710,6 +3194,11 @@ int main(int argc, char* argv[]) {
goto cleanup;
}
/* Install the negotiated codec for this process before any transfer thread
* is spawned; the compressed frames are self-describing, so the receiver's
* decompressor does not need this, but the sender compressor does. */
compression_set_algo((CompressionAlgo)config->compression_algo);
/* --iconv: install the sender-side local->wire conversion before any path is
scanned or serialized (the scanner and the chunk/data path read windows are
all driven from this process, so one global initialization covers every
+1269 -330
View File
File diff suppressed because it is too large. Load diff
+6
View File
@@ -23,6 +23,12 @@ void client_set_abort_armed(bool armed);
* config_delete() once the call returns). */
int send_files(Config* config);
int send_files_multithreaded(Config** config);
/* rsync's --ignore-errors deletion gate: with no I/O error during the scan the
* deletion phase always proceeds; with one it is suppressed unless
* `--ignore-errors` was given. Exposed so the decision can be unit-tested
* without a privileged (mode-000) source directory. See client_send.c. */
bool ignore_errors_allows_delete(const Config* config, bool had_io_error);
/* Phase 6 residual-batch (client-only). See client_send.c. */
int write_batch_from_source(const Config* config, const char* batch_path);
int apply_batch_to_dest(const Config* config, const char* batch_path, const char* dest_root);
+538 -91
View File
@@ -51,29 +51,63 @@ static FilterNode* filter_node_alloc(FilterNode* parent, FilterRuleList* own) {
return node;
}
/* Evaluate a rule chain for an entry inside the directory whose content
* context is `node`. rsync precedence, highest first: the innermost (current)
* directory's .rsync-filter rules, then each ancestor's, then the root's, and
* finally the command-line base rules (--filter/-C). A deeper per-directory
* file therefore overrides a shallower one, and per-directory files override
* the base rules by default. Returns FILTER_ACTION_NONE when nothing matched. */
static FilterAction chain_rules_apply(const FilterRuleList* base, const FilterNode* node,
const char* rel, const char* leaf, bool is_dir) {
if (node) {
FilterAction own_action = filter_rules_apply(node->own, rel, leaf, is_dir);
if (own_action != FILTER_ACTION_NONE)
return own_action;
return chain_rules_apply(base, node->parent, rel, leaf, is_dir);
/* Evaluate a rule chain for one entry. rsync precedence, highest first: the
* innermost (current) directory's .rsync-filter rules, then each ancestor's,
* then the root's, and finally the command-line base rules (--filter/-C). The
* sender-side verdict decides whether the entry is hidden from the transfer;
* the receiver-side verdict decides whether its destination mirror is protected
* from --delete. Each side takes the FIRST matching rule independently. */
typedef struct {
bool hide; /* sender-side exclude matched */
bool protect; /* receiver-side exclude matched */
} FilterOutcome;
static void chain_rules_outcome(const FilterRuleList* base, const FilterNode* node, const char* rel,
const char* leaf, bool is_dir, FilterOutcome* out) {
memset(out, 0, sizeof(*out));
bool sender_decided = false;
bool receiver_decided = false;
const FilterNode* n = node;
while (!sender_decided || !receiver_decided) {
const FilterRuleList* list = n ? n->own : base;
if (list) {
if (!sender_decided) {
FilterAction action = filter_rules_apply_side(list, rel, leaf, is_dir, FILTER_SIDE_SENDER);
if (action != FILTER_ACTION_NONE) {
out->hide = action == FILTER_ACTION_EXCLUDE;
sender_decided = true;
}
}
if (!receiver_decided) {
FilterAction action =
filter_rules_apply_side(list, rel, leaf, is_dir, FILTER_SIDE_RECEIVER);
if (action != FILTER_ACTION_NONE) {
out->protect = action == FILTER_ACTION_PROTECT;
receiver_decided = true;
}
}
}
if (!n)
break;
n = n->parent;
}
return base ? filter_rules_apply(base, rel, leaf, is_dir) : FILTER_ACTION_NONE;
}
static bool entry_allowed(const FilterRuleList* base, const FilterNode* node, const char* rel,
const char* leaf, bool is_dir, bool per_dir_filters) {
/* -F: per-directory .rsync-filter files are never transferred. */
if (per_dir_filters && !is_dir && strcmp(leaf, ".rsync-filter") == 0)
const char* leaf, bool is_dir, bool exclude_filter_files,
bool* protect_out) {
/* -FF: per-directory .rsync-filter files are never transferred (single -F
transfers them, matching rsync). */
if (exclude_filter_files && !is_dir && strcmp(leaf, ".rsync-filter") == 0) {
if (protect_out)
*protect_out = false;
return false;
return chain_rules_apply(base, node, rel, leaf, is_dir) != FILTER_ACTION_EXCLUDE;
}
FilterOutcome outcome;
chain_rules_outcome(base, node, rel, leaf, is_dir, &outcome);
if (protect_out)
*protect_out = outcome.protect;
return !outcome.hide;
}
static void dir_entry_destroy(void* item) {
@@ -163,6 +197,12 @@ typedef struct {
Size pruning protects the destination mirror even under --delete-excluded,
so it is recorded into a separate sink from `excluded`. */
bool size_excluded;
/* True when a symlink selected for dereferencing (-L/--copy-links or an
unsafe target under --copy-unsafe-links) had no usable referent (a broken
link or a stat() failure). rsync still reports this as a partial transfer
(exit 23) even though the entry is skipped, so the scanner records it as a
non-fatal I/O error. */
bool referent_error;
} ScannerEntry;
/* --one-file-system (-x) decision. Only directories can carry a different
@@ -214,6 +254,44 @@ char* scanner_path_relative(const char* root, const char* fs_path) {
return str_dup(fs_path + root_len + 1);
}
/* -R/--relative destination-relative prefix reconstructed from a source spec:
* everything after the first '.' path component (rsync's '/./' cut point),
* with leading/trailing slashes removed; or the whole spec (normalized) when
* there is no cut. Returns "" for the receive root. Exposed for tests. */
char* scanner_relative_prefix(const char* spec) {
if (!spec || spec[0] == '\0')
return NULL;
const char* after = spec;
if (spec[0] == '.' && spec[1] == '/') {
after = spec + 2;
} else {
const char* cut = strstr(spec, "/./");
if (cut)
after = cut + 3;
}
size_t cap = strlen(spec) + 1;
char* out = malloc(cap);
if (!out)
return NULL;
size_t len = 0;
for (const char* s = after; *s;) {
while (*s == '/')
s++;
const char* comp = s;
while (*s && *s != '/')
s++;
size_t clen = (size_t)(s - comp);
if (clen == 0 || (clen == 1 && comp[0] == '.'))
continue;
if (len)
out[len++] = '/';
memcpy(out + len, comp, clen);
len += clen;
}
out[len] = '\0';
return out;
}
/* Relative path of a child entry below the current directory. */
static char* child_rel_path(const char* parent_rel, const char* name) {
if (!parent_rel || parent_rel[0] == '\0')
@@ -221,14 +299,28 @@ static char* child_rel_path(const char* parent_rel, const char* name) {
return path_cat(parent_rel, name);
}
/* Apply the --files-from allow-set and the filter layer to one entry. */
/* Destination-relative wire path for an entry under an -R prefix. */
static char* scanner_prefix_send_path(const char* prefix, const char* rel) {
if (prefix[0] == '\0')
return str_dup(rel);
if (rel[0] == '\0')
return str_dup(prefix);
return path_cat(prefix, rel);
}
/* Apply the --files-from allow-set and the filter layer to one entry. On
* return `*protect_out` is true when a receiver-side rule protects the entry's
* destination mirror from deletion. */
static bool entry_passes_selection(const FileListSet* file_list, const FilterRuleList* base,
const FilterNode* node, const char* rel, const char* leaf,
bool is_dir, bool per_dir_filters) {
bool is_dir, bool per_dir_filters, bool exclude_filter_files,
bool* protect_out) {
if (protect_out)
*protect_out = false;
if (file_list && !file_list_affects(file_list, rel))
return false;
if (base || per_dir_filters)
return entry_allowed(base, node, rel, leaf, is_dir, per_dir_filters);
return entry_allowed(base, node, rel, leaf, is_dir, exclude_filter_files, protect_out);
return true;
}
@@ -340,6 +432,19 @@ static void scanner_record_protected(DirectoryScanner* scanner, const char* fs_p
scanner->failed = true;
}
/* rsync's `--info=nonreg` line for a non-regular entry that is not being
* preserved: `skipping non-regular file "NAME"`. The name is the path relative
* to the transfer root, so it matches rsync's displayed name. */
static void scanner_note_nonreg(const ScannerOptions* options, const char* fs_path) {
if (!options || !options->note_nonreg || !fs_path)
return;
const char* rel = utils_strip_transfer_root(fs_path, options->send_directory);
char* escaped = output_escape(rel, options->eight_bit_output);
printf("skipping non-regular file \"%s\"\n", escaped ? escaped : rel);
free(escaped);
fflush(stdout);
}
/* A user-selection exclusion (--filter/-C/per-dir or --exclude/--include). */
static void scanner_record_excluded(DirectoryScanner* scanner, const char* fs_path) {
scanner_record_protected(scanner, fs_path, scanner->options.excluded_paths);
@@ -357,40 +462,109 @@ static void scanner_record_size_skipped(DirectoryScanner* scanner, const char* f
Returns false on allocation failure. */
static bool scanner_record_synced_dir(const ScannerOptions* options, const char* fs_path,
const char* rel, bool relative_mode) {
if (!options->synced_dirs)
if (!options->synced_dirs && !options->plan_dirs)
return true;
if (!file_list_dir_in_scope(options->file_list, rel))
return true;
const char* dest = relative_mode ? rel : fs_path;
char* prefixed = NULL;
const char* dest;
if (relative_mode) {
dest = rel;
} else if (options->relative_prefix) {
prefixed = scanner_prefix_send_path(options->relative_prefix, rel);
if (!prefixed)
return false;
dest = prefixed;
} else {
dest = fs_path;
}
if (dest[0] == '/')
dest++;
if (dest[0] == '\0')
dest = ".";
return excluded_sink_append(options->synced_dirs, options->excluded_mutex, dest);
bool ok = true;
if (options->synced_dirs)
ok = excluded_sink_append(options->synced_dirs, options->excluded_mutex, dest);
/* The delete-plan keep set needs an entry for every traversed source
directory, including empty ones, so its destination mirror is kept rather
than deleted as an extra; the receive root (".") is implicit. */
if (ok && options->plan_dirs && strcmp(dest, ".") != 0)
ok = excluded_sink_append(options->plan_dirs, options->excluded_mutex, dest);
free(prefixed);
return ok;
}
/* Merge the open directory's own .rsync-filter rules into the inherited
* context, returning the context used for this directory's entries. On a parse
* error the scanner is marked failed. Returns 0 on success, -1 on failure. */
static int open_directory_filter_context(DirectoryScanner* scanner, const FilterNode* inherited) {
if (!scanner->options.per_dir_filters) {
scanner->current_node = (FilterNode*)inherited;
return 0;
}
char err[256];
bool exists = false;
FilterRuleList* own =
filter_file_read(scanner->current_path, scanner->current_rel ? scanner->current_rel : "",
&exists, err, sizeof(err));
/* Read every per-directory filter file that applies to `dir_path` (its
* .rsync-filter when -F is active, plus each registered "dir-merge NAME") into a
* fresh list. Returns NULL on allocation/parse failure (message in `err`);
* returns an empty list (and *any_exists=false) when no file exists. */
static FilterRuleList* read_dir_filters(const ScannerOptions* options, const char* dir_path,
const char* rel, bool* any_exists, char* err,
size_t err_size) {
if (err && err_size > 0)
err[0] = '\0';
const FilterRuleList* base = options->base_filters;
bool have_names = options->per_dir_filters || (base && base->dir_merge_count > 0);
if (any_exists)
*any_exists = false;
if (!have_names)
return NULL;
FilterRuleList* own = filter_rule_list_create();
if (!own) {
snprintf(err, err_size, "memory allocation failed");
return NULL;
}
FilterParseOptions opts = {.delete_excluded = options->delete_excluded, .cvs_exclude = false};
bool exists = false;
if (options->per_dir_filters) {
if (!filter_file_append(own, dir_path, ".rsync-filter", rel, &opts, &exists, err, err_size))
goto fail;
if (exists && any_exists)
*any_exists = true;
}
if (base) {
for (int i = 0; i < base->dir_merge_count; i++) {
if (!filter_file_append(own, dir_path, base->dir_merge_names[i], rel, &opts, &exists, err,
err_size))
goto fail;
if (exists && any_exists)
*any_exists = true;
}
}
return own;
fail:
filter_rule_list_free(own);
return NULL;
}
/* Merge the open directory's own per-directory filter files (the default
* .rsync-filter when -F is active, plus every "dir-merge NAME" registered on the
* base rule list) into the inherited context, returning the context used for
* this directory's entries. On a parse error the scanner is marked failed.
* Returns 0 on success, -1 on failure. */
static int open_directory_filter_context(DirectoryScanner* scanner, const FilterNode* inherited) {
char err[256];
bool any_exists = false;
FilterRuleList* own = read_dir_filters(&scanner->options, scanner->current_path,
scanner->current_rel ? scanner->current_rel : "",
&any_exists, err, sizeof(err));
if (!own) {
/* read_dir_filters() leaves `err` set on a parse/allocation failure even
when an earlier merge file in the same directory existed (any_exists true);
key off the error text rather than any_exists so an invalid per-directory
filter file can never be silently ignored. */
if (err[0] == '\0') {
scanner->current_node = (FilterNode*)inherited;
return 0;
}
char* escaped_path = output_escape(scanner->current_path, log_get_8_bit_output());
log_message(LOG_LEVEL_ERROR, "invalid .rsync-filter in %s: %s",
log_message(LOG_LEVEL_ERROR, "invalid per-directory filter in %s: %s",
escaped_path ? escaped_path : "<allocation failed>", err);
free(escaped_path);
scanner->failed = true;
return -1;
}
if (exists && own->count > 0) {
if (any_exists && (own->count > 0 || own->dir_merge_count > 0)) {
FilterNode* node = filter_node_alloc((FilterNode*)inherited, own);
if (!node || !array_list_add(scanner->filter_nodes, node)) {
filter_node_destroy(node);
@@ -412,6 +586,7 @@ static int scanner_inspect_entry(const ScannerOptions* options, const char* cont
const char* link_rel, const char* name, ScannerEntry* entry) {
entry->excluded = false;
entry->size_excluded = false;
entry->referent_error = false;
entry->is_symlink = false;
entry->link_target = NULL;
entry->path = path_cat(containing_dir, name);
@@ -438,12 +613,13 @@ static int scanner_inspect_entry(const ScannerOptions* options, const char* cont
goto skip;
case LINK_ACTION_DEREF:
if (stat(entry->path, &entry->stats) != 0) {
/* rsync reports "symlink has no referent" and continues (exit 23); we
surface the same condition rather than silently dropping the entry. */
/* rsync reports "symlink has no referent" and continues with a partial
transfer (exit 23); record the error so the run exits 23 too. */
char* escaped = output_escape(entry->path, log_get_8_bit_output());
log_message(LOG_LEVEL_WARNING, "symlink has no referent: %s",
escaped ? escaped : "<allocation failed>");
free(escaped);
entry->referent_error = true;
goto skip;
}
entry->is_directory = S_ISDIR(entry->stats.st_mode);
@@ -664,9 +840,11 @@ static Chunk* chunk_data_to_chunk(ArrayList* chunk_data) {
* non-directory path is silently skipped (the transfer is unaffected); an
* allocation failure is fatal and reported to the caller. */
static bool scanner_capture_dir_time(ArrayList* dir_entries, mtx_t* mutex, const char* root_path,
const char* fs_path, bool relative_mode, bool preserve_atimes,
const char* fs_path, bool relative_mode,
const char* relative_prefix, bool preserve_atimes,
bool preserve_crtimes, bool preserve_xattrs,
bool preserve_acls) {
bool preserve_acls, bool no_implied_dirs,
const FileListSet* file_list) {
if (!dir_entries || !root_path || !fs_path)
return true;
struct stat st;
@@ -675,20 +853,43 @@ static bool scanner_capture_dir_time(ArrayList* dir_entries, mtx_t* mutex, const
char* rel = scanner_path_relative(root_path, fs_path);
if (!rel)
return true;
/* --no-implied-dirs: an implied parent directory (not listed, and not under
a listed directory) keeps the destination's own/default attributes, so its
source metadata is not transmitted. */
if (no_implied_dirs && file_list && !file_list_dir_in_scope(file_list, rel)) {
free(rel);
return true;
}
if (relative_mode && rel[0] == '\0') {
/* -R + --files-from: the transfer root itself has no bare relative wire
path (matches the -R scan, which never emits the root). */
free(rel);
return true;
}
char* prefixed = NULL;
if (relative_prefix) {
prefixed = scanner_prefix_send_path(relative_prefix, rel);
if (!prefixed) {
free(rel);
return false;
}
if (prefixed[0] == '\0') {
/* -R with a cut at the receive root: the root itself has no wire path. */
free(prefixed);
free(rel);
return true;
}
}
File* file = file_create(fs_path);
if (!file) {
free(prefixed);
free(rel);
return false;
}
file->is_dir = true;
file->metadata = file_metadata_create(fs_path, &st, preserve_atimes, preserve_crtimes);
if (!file->metadata) {
free(prefixed);
free(rel);
file_destroy(file);
return false;
@@ -701,7 +902,11 @@ static bool scanner_capture_dir_time(ArrayList* dir_entries, mtx_t* mutex, const
if (relative_mode) {
file->send_path = rel;
rel = NULL;
} else if (prefixed) {
file->send_path = prefixed;
prefixed = NULL;
}
free(prefixed);
free(rel);
bool added;
if (mutex) {
@@ -718,6 +923,43 @@ static bool scanner_capture_dir_time(ArrayList* dir_entries, mtx_t* mutex, const
return true;
}
/* Recursive scan: emit a payload-less directory entry for the directory that
* just finished scanning. rsync creates every source directory at the
* destination; FastSync otherwise creates one only implicitly through a
* transferred child, so a directory emptied on the transfer side (physically
* empty, or all of its entries filtered out) would never appear. The transfer
* root is skipped (it maps to the receive root, which already exists), as are
* --files-from (only listed items and their implied parents transfer),
* --list-only (directory lines are emitted by the caller) and
* -m/--prune-empty-dirs. Returns false on allocation failure. */
static bool scanner_emit_empty_dir(DirectoryScanner* scanner, ArrayList* chunk_data) {
if (!scanner->current_path || !scanner->current_rel || scanner->current_rel[0] == '\0')
return true;
struct stat st;
if (lstat(scanner->current_path, &st) != 0 || !S_ISDIR(st.st_mode))
return true;
File* dir = scanner_build_dir_file(scanner->current_path, &st, &scanner->options);
if (!dir)
return false;
if (scanner->relative_mode) {
dir->send_path = str_dup(scanner->current_rel);
} else if (scanner->options.relative_prefix) {
dir->send_path =
scanner_prefix_send_path(scanner->options.relative_prefix, scanner->current_rel);
}
if ((scanner->relative_mode || scanner->options.relative_prefix) && !dir->send_path) {
file_destroy(dir);
return false;
}
if (scanner->options.preserve_xattrs || scanner->options.preserve_acls)
dir->xattrs = xattr_capture_path(scanner->current_path, scanner->options.preserve_acls);
if (!array_list_add(chunk_data, dir)) {
file_destroy(dir);
return false;
}
return true;
}
/* Open the next queued directory and set up its filter context. Returns 1 when
a directory is open, 0 when the queue is exhausted, and -1 on a fatal error.
A directory that cannot be opened is an I/O error: it is recorded on the
@@ -741,6 +983,7 @@ static int open_next_directory(DirectoryScanner* scanner) {
* the directory that enqueued them. */
const FilterNode* inherited = scanner->at_seed_dir ? scanner->seed_node : de->context;
scanner->at_seed_dir = false;
scanner->current_dir_produced = false;
free(de);
free(scanner->current_rel);
@@ -770,11 +1013,18 @@ static int open_next_directory(DirectoryScanner* scanner) {
scanner->current_rel = NULL;
free(scanner->current_path);
scanner->current_path = NULL;
if (!scanner->options.ignore_io_errors || is_root_seed) {
if (is_root_seed) {
/* The transfer ROOT being unreadable is always fatal: an empty keep-set
would delete the whole destination. Mark the scan as errored so the
client can report the partial-transfer exit code (rsync's 23). */
scanner->root_io_error = true;
scanner->failed = true;
return -1;
}
/* --ignore-errors: record the I/O error and keep scanning the rest. */
/* A subdirectory that cannot be opened is always skipped (rsync continues
with a partial transfer), whether or not --ignore-errors is set. The
error is recorded so the client exits 23; --ignore-errors only changes
what the deletion phase does with the recorded error. */
continue;
}
if (open_directory_filter_context(scanner, inherited) != 0) {
@@ -799,9 +1049,10 @@ static int open_next_directory(DirectoryScanner* scanner) {
if (scanner->options.capture_dir_times &&
!scanner_capture_dir_time(
scanner->options.dir_entries, scanner->options.dir_entries_mutex, scanner->root_path,
scanner->current_path, scanner->relative_mode, scanner->options.preserve_atimes,
scanner->options.preserve_crtimes, scanner->options.preserve_xattrs,
scanner->options.preserve_acls)) {
scanner->current_path, scanner->relative_mode, scanner->options.relative_prefix,
scanner->options.preserve_atimes, scanner->options.preserve_crtimes,
scanner->options.preserve_xattrs, scanner->options.preserve_acls,
scanner->options.no_implied_dirs, scanner->options.file_list)) {
closedir(scanner->current_dir);
scanner->current_dir = NULL;
free(scanner->current_path);
@@ -817,18 +1068,20 @@ static int open_next_directory(DirectoryScanner* scanner) {
/* ---- --dirs mode ----
With -d the scanner transfers directory entries and never recurses into
contents. A plain `-d <dir>` sends only the source-root directory mirror
(created empty at the destination). With -d + --files-from exactly the
listed items are sent: listed directories become empty directory entries and
listed regular files are transferred as files; nothing else is scanned, so
no descent into a listed directory can happen. */
(created empty at the destination); `-d dir/`, `-d dir/.` and `-d .` list
the directory's immediate contents instead (files plus empty directory
entries), matching rsync. With -d + --files-from exactly the listed items
are sent: listed directories become empty directory entries and listed
regular files are transferred as files; nothing else is scanned, so no
descent into a listed directory can happen. */
/* Directory entries carry no payload, so the dirs generator also bounds every
chunk by element count; chunk_deserialize refuses more than this many files
per chunk (see MAX_FILES_PER_CHUNK in chunk.c). */
#define DIRS_CHUNK_MAX_FILES 65536U
/* Build the File for the transfer root directory itself (the `-d <dir>` and
* "." cases). */
/* Build the File for the transfer root directory itself (the `-d <dir>`
* no-trailing-slash case). */
static File* dirs_root_dir_file(DirectoryScanner* scanner) {
struct stat st;
if (stat(scanner->root_path, &st) != 0 || !S_ISDIR(st.st_mode)) {
@@ -851,6 +1104,14 @@ static File* dirs_root_dir_file(DirectoryScanner* scanner) {
return NULL;
}
}
if (scanner->options.relative_prefix && scanner->options.relative_prefix[0] != '\0') {
file->send_path = str_dup(scanner->options.relative_prefix);
if (!file->send_path) {
file_destroy(file);
scanner->failed = true;
return NULL;
}
}
scanner_capture_xattrs(scanner, file);
return file;
}
@@ -955,6 +1216,13 @@ static File* dirs_file_for_entry(DirectoryScanner* scanner, const char* entry) {
scanner->failed = true;
return NULL;
}
} else if (scanner->options.relative_prefix) {
file->send_path = scanner_prefix_send_path(scanner->options.relative_prefix, entry);
if (!file->send_path) {
file_destroy(file);
scanner->failed = true;
return NULL;
}
}
if (scanner->options.use_metadata) {
file->metadata = file_metadata_create(file->path, &effective, scanner->options.preserve_atimes,
@@ -988,9 +1256,64 @@ static bool dirs_source_dir_is_empty(const char* path) {
return empty;
}
/* The next immediate child of the source root for a one-level --dirs listing
* (rsync: -d DIR/ lists DIR's immediate contents without recursing). */
static File* dirs_next_child(DirectoryScanner* scanner) {
if (!scanner->current_dir)
return NULL;
const struct dirent* entry;
while ((entry = readdir(scanner->current_dir)) != NULL) {
if (strcmp(entry->d_name, ".") == 0 || strcmp(entry->d_name, "..") == 0)
continue;
File* file = dirs_file_for_entry(scanner, entry->d_name);
if (scanner->failed)
return NULL;
if (file && !entry_passes_selection(scanner->options.file_list, scanner->options.base_filters,
NULL, entry->d_name, entry->d_name, file->is_dir,
scanner->options.per_dir_filters,
scanner->options.exclude_per_dir_filter_files, NULL)) {
file_destroy(file);
continue;
}
if (file && file->is_dir && scanner->options.prune_empty_dirs &&
dirs_source_dir_is_empty(file->path)) {
file_destroy(file);
continue;
}
if (file)
return file;
}
closedir(scanner->current_dir);
scanner->current_dir = NULL;
return NULL;
}
/* The next File from the --dirs generator, or NULL when exhausted. */
static File* dirs_next_file(DirectoryScanner* scanner) {
if (!scanner->options.file_list) {
const char* spec = scanner->root_path ? scanner->root_path : "";
size_t n = strlen(spec);
/* rsync: a trailing slash or "/." on the source argument lists the
directory's immediate contents (files and empty directory entries)
without recursing. A bare directory sends only its own entry. */
bool list_children =
(n == 1 && spec[0] == '.') ||
(n > 0 && (spec[n - 1] == '/' || (n >= 2 && spec[n - 1] == '.' && spec[n - 2] == '/')));
if (list_children) {
if (!scanner->dirs_root_emitted) {
scanner->dirs_root_emitted = true;
if (scanner->options.prune_empty_dirs && dirs_source_dir_is_empty(scanner->root_path))
return NULL;
scanner->current_dir = opendir(scanner->root_path);
if (!scanner->current_dir) {
scanner->io_error = true;
log_perror("Could not open directory");
scanner->failed = true;
return NULL;
}
}
return dirs_next_child(scanner);
}
if (scanner->dirs_root_emitted)
return NULL;
scanner->dirs_root_emitted = true;
@@ -1094,10 +1417,22 @@ Chunk* directory_scanner_next(DirectoryScanner* scanner) {
const struct dirent* entry = readdir(scanner->current_dir);
if (entry == NULL) {
/* The directory is exhausted: if nothing was transferred or descended
from it, recreate it at the destination as an explicit entry. */
if (scanner->options.emit_empty_dirs && !scanner->current_dir_produced &&
!scanner->options.prune_empty_dirs && !scanner->options.list_dirs &&
scanner->options.file_list == NULL) {
if (!scanner_emit_empty_dir(scanner, chunk_data))
scanner->failed = true;
}
closedir(scanner->current_dir);
scanner->current_dir = NULL;
free(scanner->current_path);
scanner->current_path = NULL;
if (scanner->failed) {
array_list_delete(chunk_data);
return NULL;
}
continue;
}
@@ -1118,6 +1453,10 @@ Chunk* directory_scanner_next(DirectoryScanner* scanner) {
break;
}
if (inspection == 0) {
/* A dereferenced symlink with no referent is a partial-transfer error
(rsync exit 23): record it as a non-fatal scan I/O error. */
if (inspected.referent_error)
scanner->io_error = true;
/* A user-selection exclude protects its destination mirror from --delete
unless --delete-excluded; a size prune is always protected. Other
skips (unreadable, symlink policy) protect nothing. Under -R +
@@ -1125,9 +1464,17 @@ Chunk* directory_scanner_next(DirectoryScanner* scanner) {
wire path, not its source path (which would not match the destination
layout and would leave the mirror deletable). */
if (inspected.excluded) {
char* protected_path = scanner->relative_mode
? child_rel_path(scanner->current_rel, entry->d_name)
: path_cat(scanner->current_path, entry->d_name);
char* protected_path;
if (scanner->relative_mode) {
protected_path = child_rel_path(scanner->current_rel, entry->d_name);
} else if (scanner->options.relative_prefix) {
char* relc = child_rel_path(scanner->current_rel, entry->d_name);
protected_path =
relc ? scanner_prefix_send_path(scanner->options.relative_prefix, relc) : NULL;
free(relc);
} else {
protected_path = path_cat(scanner->current_path, entry->d_name);
}
if (!protected_path) {
scanner->failed = true;
break;
@@ -1152,10 +1499,15 @@ Chunk* directory_scanner_next(DirectoryScanner* scanner) {
scanner->failed = true;
break;
}
bool protect = false;
bool passes_selection = entry_passes_selection(
scanner->options.file_list, scanner->options.base_filters, scanner->current_node, rel,
entry->d_name, is_dir, scanner->options.per_dir_filters);
if (!passes_selection) {
entry->d_name, is_dir, scanner->options.per_dir_filters,
scanner->options.exclude_per_dir_filter_files, &protect);
/* A sender-side hide leaves the entry out of the transfer; an independent
receiver-side protect rule keeps a transferred entry's destination mirror
from being deleted. Both are recorded in the same protection set. */
if (!passes_selection || protect) {
/* --files-from subset pruning is not a filter exclusion: its delete
semantics stay keep-set-only (an unlisted source path is treated as
absent, so its destination mirror is a deletable extra). A rule-based
@@ -1163,14 +1515,33 @@ Chunk* directory_scanner_next(DirectoryScanner* scanner) {
wire paths are never recorded (see ScannerOptions.excluded_paths). */
bool files_from_prune =
scanner->options.file_list && !file_list_affects(scanner->options.file_list, rel);
if (!files_from_prune && !scanner->relative_mode)
scanner_record_excluded(scanner, cur_path);
if (protect && scanner->relative_mode) {
/* -R + --files-from: the destination/wire path is the bare relative
name, so the protected mirror prefix must be `rel` (not the source
path) for the delete walker to match it. */
scanner_record_excluded(scanner, rel);
} else if (!files_from_prune && !scanner->relative_mode) {
if (scanner->options.relative_prefix) {
char* wrel = scanner_prefix_send_path(scanner->options.relative_prefix, rel);
if (!wrel) {
free(rel);
free(cur_path);
scanner->failed = true;
break;
}
scanner_record_excluded(scanner, wrel);
free(wrel);
} else {
scanner_record_excluded(scanner, cur_path);
}
}
}
/* With -R + --files-from the wire/destination path is the entry's bare
relative path; keep `rel` alive to attach it to a transferred file. */
char* rel_copy = scanner->relative_mode ? str_dup(rel) : NULL;
/* With -R the wire/destination path is a reconstructed relative path, not
the source path; keep `rel` alive to build it for a transferred file. */
bool needs_rel = scanner->relative_mode || scanner->options.relative_prefix != NULL;
char* rel_copy = needs_rel ? str_dup(rel) : NULL;
free(rel);
if (rel_copy == NULL && scanner->relative_mode) {
if (rel_copy == NULL && needs_rel) {
free(cur_path);
scanner->failed = true;
break;
@@ -1196,6 +1567,7 @@ Chunk* directory_scanner_next(DirectoryScanner* scanner) {
scanner->failed = true;
break;
}
scanner->current_dir_produced = true;
free(cur_path);
continue;
}
@@ -1210,6 +1582,7 @@ Chunk* directory_scanner_next(DirectoryScanner* scanner) {
break;
}
}
scanner->current_dir_produced = true;
int next_depth = scanner->current_depth + 1;
if (scanner->options.max_depth <= 0 || next_depth < scanner->options.max_depth) {
DirEntry* de = dir_entry_create(cur_path, next_depth, scanner->current_node);
@@ -1245,6 +1618,15 @@ Chunk* directory_scanner_next(DirectoryScanner* scanner) {
if (scanner->relative_mode) {
file->send_path = rel_copy;
rel_copy = NULL;
} else if (scanner->options.relative_prefix) {
file->send_path = scanner_prefix_send_path(scanner->options.relative_prefix, rel_copy);
free(rel_copy);
rel_copy = NULL;
if (!file->send_path) {
file_destroy(file);
scanner->failed = true;
break;
}
}
/* --devices/--specials: a device/FIFO/socket entry marked for preservation
becomes a node to recreate (is_special, no data, rdev captured); an
@@ -1253,6 +1635,7 @@ Chunk* directory_scanner_next(DirectoryScanner* scanner) {
scanner->options.preserve_specials,
scanner->options.copy_devices, file, &stats);
if (special == SCANNER_SPECIAL_SKIP) {
scanner_note_nonreg(&scanner->options, file->path);
free(rel_copy);
file_destroy(file);
continue;
@@ -1276,6 +1659,7 @@ Chunk* directory_scanner_next(DirectoryScanner* scanner) {
scanner->failed = true;
break;
}
scanner->current_dir_produced = true;
chunk_data_size += file->data->size;
if (chunk_data_size > scanner->options.chunk_size) {
free(rel_copy);
@@ -1303,7 +1687,7 @@ bool directory_scanner_failed(const DirectoryScanner* scanner) {
}
bool directory_scanner_had_io_error(const DirectoryScanner* scanner) {
return scanner != NULL && scanner->io_error;
return scanner != NULL && (scanner->io_error || scanner->root_io_error);
}
typedef struct {
@@ -1511,6 +1895,8 @@ static void scan_root_entry(const ScannerOptions* options, const FilterNode* roo
return;
}
if (inspection == 0) {
if (inspected.referent_error)
ps->io_error = true;
ArrayList* sink = NULL;
if (inspected.excluded)
sink = inspected.size_excluded ? options->size_skipped_paths : options->excluded_paths;
@@ -1522,6 +1908,15 @@ static void scan_root_entry(const ScannerOptions* options, const FilterNode* roo
if (options->relative && options->file_list != NULL) {
if (!excluded_sink_append(sink, options->excluded_mutex, entry->d_name))
ps->failed = true;
} else if (options->relative_prefix) {
char* wrel = scanner_prefix_send_path(options->relative_prefix, entry->d_name);
if (!wrel) {
ps->failed = true;
} else {
if (!excluded_sink_append(sink, options->excluded_mutex, wrel))
ps->failed = true;
free(wrel);
}
} else {
char* abs_path = path_cat(root_directory, entry->d_name);
if (!abs_path) {
@@ -1545,31 +1940,54 @@ static void scan_root_entry(const ScannerOptions* options, const FilterNode* roo
ps->failed = true;
return;
}
bool protect = false;
bool passes = entry_passes_selection(options->file_list, options->base_filters, root_node, rel,
entry->d_name, is_dir, options->per_dir_filters);
entry->d_name, is_dir, options->per_dir_filters,
options->exclude_per_dir_filter_files, &protect);
/* -R + --files-from: root-level files keep their bare relative send path. */
bool use_rel = options->relative && options->file_list != NULL;
if (!passes) {
if (!passes || protect) {
/* --files-from subset pruning is not a filter exclusion; -R bare-wire-path
exclusions are never recorded (see ScannerOptions.excluded_paths). */
bool files_from_prune = options->file_list && !file_list_affects(options->file_list, rel);
if (!files_from_prune && !use_rel && options->excluded_paths) {
const char* rel_path = *cur_path == '/' ? cur_path + 1 : cur_path;
if (!excluded_sink_append(options->excluded_paths, options->excluded_mutex, rel_path))
if ((!files_from_prune && !use_rel) || protect) {
const char* rel_path;
char* prefixed = NULL;
if (use_rel) {
/* -R + --files-from: the destination/wire path is the bare relative
name, not the source path. */
rel_path = rel;
} else if (options->relative_prefix) {
prefixed = scanner_prefix_send_path(options->relative_prefix, entry->d_name);
if (!prefixed) {
free(rel);
free(cur_path);
ps->failed = true;
return;
}
rel_path = prefixed;
} else {
rel_path = *cur_path == '/' ? cur_path + 1 : cur_path;
}
if (options->excluded_paths &&
!excluded_sink_append(options->excluded_paths, options->excluded_mutex, rel_path))
ps->failed = true;
free(prefixed);
}
if (!passes) {
free(rel);
free(cur_path);
return;
}
free(rel);
free(cur_path);
return;
}
if (is_dir) {
free(rel);
if (!scanner_same_filesystem(options->one_file_system, root_dev, st.st_dev)) {
/* -x/--one-file-system: emit the mount-point directory entry (empty) but
do not descend into it (see the sequential scanner for the same rule). */
File* mount = file_create(cur_path);
free(cur_path);
if (mount == NULL) {
free(rel);
ps->failed = true;
return;
}
@@ -1578,17 +1996,29 @@ static void scan_root_entry(const ScannerOptions* options, const FilterNode* roo
mount->metadata = file_metadata_create(mount->path, &st, options->preserve_atimes,
options->preserve_crtimes);
if (!mount->metadata) {
free(rel);
file_destroy(mount);
ps->failed = true;
return;
}
}
if (options->relative_prefix) {
mount->send_path = scanner_prefix_send_path(options->relative_prefix, rel);
if (!mount->send_path) {
free(rel);
file_destroy(mount);
ps->failed = true;
return;
}
}
free(rel);
if (!array_list_add(root_files, mount)) {
file_destroy(mount);
ps->failed = true;
}
return;
}
free(rel);
if (!array_list_add(subdirs, cur_path)) {
free(cur_path);
ps->failed = true;
@@ -1614,10 +2044,20 @@ static void scan_root_entry(const ScannerOptions* options, const FilterNode* roo
if (use_rel) {
file->send_path = rel;
rel = NULL;
} else if (options->relative_prefix) {
file->send_path = scanner_prefix_send_path(options->relative_prefix, rel);
free(rel);
rel = NULL;
if (!file->send_path) {
file_destroy(file);
ps->failed = true;
return;
}
}
ScannerSpecial special = scanner_prepare_special(
options->preserve_devices, options->preserve_specials, options->copy_devices, file, &st);
if (special == SCANNER_SPECIAL_SKIP) {
scanner_note_nonreg(ps->options, file->path);
free(rel);
file_destroy(file);
return;
@@ -1778,6 +2218,7 @@ ParallelScanner* parallel_scanner_create_with_options(const char* root_directory
return NULL;
}
ps->allocation_session = allocation_session;
ps->options = options;
ArrayList* root_files = array_list_create(file_destroy);
ArrayList* subdirs = array_list_create(free);
@@ -1801,22 +2242,27 @@ ParallelScanner* parallel_scanner_create_with_options(const char* root_directory
root_dev = root_stats.st_dev;
}
/* Build the root directory's .rsync-filter context once; workers seed their
* scanners with it so per-dir rules behave identically to the sequential
/* Build the root directory's per-directory filter context once; workers seed
* their scanners with it so per-dir rules behave identically to the sequential
* scanner. */
FilterNode* root_node = NULL;
if (options->per_dir_filters) {
{
char err[256];
bool exists = false;
FilterRuleList* own = filter_file_read(root_directory, "", &exists, err, sizeof(err));
bool any_exists = false;
FilterRuleList* own =
read_dir_filters(options, root_directory, "", &any_exists, err, sizeof(err));
if (!own) {
log_message(LOG_LEVEL_ERROR, "invalid .rsync-filter in %s: %s", root_directory, err);
array_list_delete(root_files);
array_list_delete(subdirs);
parallel_scanner_destroy(ps);
return NULL;
}
if (exists && own->count > 0) {
/* A parse/allocation failure must fail the scan even when an earlier
merge file in the same directory existed (see the sequential scanner). */
if (err[0] != '\0') {
log_message(LOG_LEVEL_ERROR, "invalid per-directory filter in %s: %s", root_directory, err);
array_list_delete(root_files);
array_list_delete(subdirs);
parallel_scanner_destroy(ps);
return NULL;
}
/* no files exist: leave root_node NULL */
} else if (any_exists && (own->count > 0 || own->dir_merge_count > 0)) {
root_node = filter_node_alloc(NULL, own);
if (!root_node) {
filter_rule_list_free(own);
@@ -1841,10 +2287,11 @@ ParallelScanner* parallel_scanner_create_with_options(const char* root_directory
transfer root itself (it hands the root's immediate subdirectories to
workers), so capture the root's directory time here. */
if (options->capture_dir_times &&
!scanner_capture_dir_time(options->dir_entries, options->dir_entries_mutex, root_directory,
root_directory, options->relative && options->file_list != NULL,
options->preserve_atimes, options->preserve_crtimes,
options->preserve_xattrs, options->preserve_acls)) {
!scanner_capture_dir_time(
options->dir_entries, options->dir_entries_mutex, root_directory, root_directory,
options->relative && options->file_list != NULL, options->relative_prefix,
options->preserve_atimes, options->preserve_crtimes, options->preserve_xattrs,
options->preserve_acls, options->no_implied_dirs, options->file_list)) {
array_list_delete(root_files);
array_list_delete(subdirs);
parallel_scanner_destroy(ps);
+60 -5
View File
@@ -63,8 +63,19 @@ typedef struct {
const FileListSet* file_list; /* --files-from allow-set, or NULL */
const FilterRuleList* base_filters; /* command-line + -C rules, or NULL */
bool per_dir_filters; /* -F: read .rsync-filter per directory */
bool dirs; /* -d/--dirs: transfer dir entries, no recursion */
bool relative; /* -R/--relative (dest rel paths, with --files-from) */
/* --delete-excluded: per-directory plain rules become sender-only, so they no
longer protect the receiver from deletion. */
bool delete_excluded;
/* -FF: also exclude the per-directory filter files themselves from the
transfer (single -F transfers them). */
bool exclude_per_dir_filter_files;
bool dirs; /* -d/--dirs: transfer dir entries, no recursion */
bool relative; /* -R/--relative (dest rel paths, with --files-from) */
/* -R/--relative outside --files-from: the destination-relative path prefix
* reconstructed from the source spec (rsync's '/./' cut point), or NULL when
* -R is off or --files-from is in use (the bare-relative path then comes from
* the listed entry). Borrowed read-only; owned by client_send. */
const char* relative_prefix;
/* --list-only: emit an is_dir File for every traversed directory (the listing
* includes directory entries, matching rsync). Client-only; never set on a
* real transfer, which relies on implicit parent creation. */
@@ -103,9 +114,25 @@ typedef struct {
* directories, exactly like rsync; the receive root is the "." sentinel.
* Guarded by `excluded_mutex`. */
ArrayList* synced_dirs;
/* --ignore-errors: an unreadable directory during the scan is recorded as an
* I/O error and skipped instead of aborting the scan. Client-only. */
/* Delete-plan directory sink (optional): when non-NULL the scanner appends
* the destination-relative path of every directory it traverses (except the
* receive root). The per-directory --delete-during/--delete-delay plan
* builder uses this to keep an empty in-scope source directory (rsync keeps
* it) and to emit its plan after the data stream, when no file frame would
* otherwise trigger it. Guarded by `excluded_mutex`. */
ArrayList* plan_dirs;
/* --ignore-errors: an unreadable subdirectory no longer aborts the scan (it
* is always skipped so the rest of the tree transfers); this flag is kept so
* the client can distinguish the option state when deciding deletion policy.
* Client-only. */
bool ignore_io_errors;
/* --info=nonreg: print rsync's `skipping non-regular file "NAME"` line for a
* non-regular entry that is not being preserved. Client-only. */
bool note_nonreg;
/* Source root and 8-bit-output policy used to render a `--info=nonreg` name
* relative to the transfer root. Borrowed read-only. */
const char* send_directory;
bool eight_bit_output;
/* --ignore-missing-args (implied by --delete-missing-args): an explicitly
* --files-from-listed entry that does not exist under the source is skipped
* instead of failing (the --dirs generator is the only scanner path that
@@ -133,6 +160,17 @@ typedef struct {
bool capture_dir_times;
ArrayList* dir_entries;
mtx_t* dir_entries_mutex;
/* Recreate empty source directories on a recursive transfer: emit a
* payload-less directory entry for every traversed directory that produced
* no transferred/descended child. Off by default so low-level scanner users
* (unit helpers, --list-only) see only the historical file list; the real
* sender sets it in prepare_scanner. */
bool emit_empty_dirs;
/* --no-implied-dirs with -R + --files-from: a directory that is only an
* implied parent of a listed entry (not itself listed, nor below a listed
* directory) must not carry source metadata; it is created with default
* attributes at the destination, matching rsync. */
bool no_implied_dirs;
} ScannerOptions;
/* Internal per-scanner filter state. FilterNode chains represent the ordered
@@ -150,6 +188,11 @@ typedef struct {
int current_depth;
dev_t root_dev;
bool failed;
/* Recursive scan: whether the open directory yielded any transferred or
descended entry. When it did not, closing it emits a directory entry so
the empty source directory is recreated at the destination (rsync
parity). */
bool current_dir_produced;
/* Phase 2 (files-from / filter layer). */
char* root_path; /* transfer root (fs path) for rel computation */
char* current_rel; /* rel path of the open directory ("" == root) */
@@ -168,6 +211,10 @@ typedef struct {
--ignore-errors the scan continues past it and the caller decides what to
do; `failed` is reserved for fatal errors that always abort the scan. */
bool io_error;
/* The transfer ROOT could not be opened. It is always fatal, even under
--ignore-errors, but the client still maps it to rsync's partial-transfer
exit (23) rather than a generic failure. */
bool root_io_error;
} DirectoryScanner;
typedef struct {
@@ -187,7 +234,8 @@ typedef struct {
int completed;
Chunk* initial_chunk;
ProtocolSession* allocation_session;
FilterNode* root_filter_node; /* root .rsync-filter context (owned by ps) */
FilterNode* root_filter_node; /* root .rsync-filter context (owned by ps) */
const ScannerOptions* options; /* borrowed scan options (--info=nonreg output) */
} ParallelScanner;
DirectoryScanner* directory_scanner_create(const char* root_directory, bool use_metadata,
@@ -213,6 +261,13 @@ bool scanner_same_filesystem(bool one_file_system, dev_t root_device, dev_t entr
* "/". Exposed so tests can exercise the mapping directly. */
char* scanner_path_relative(const char* root, const char* fs_path);
/* -R/--relative destination-relative prefix reconstructed from a source spec:
* the path after rsync's first '.' path component (the '/./' cut point), with
* leading/trailing slashes removed, or the whole spec (normalized) when there
* is no cut. Returns "" for the receive root, or NULL when `spec` is NULL or
* allocation fails. Exposed so tests can exercise the mapping directly. */
char* scanner_relative_prefix(const char* spec);
ParallelScanner* parallel_scanner_create_with_options(const char* root_directory,
const ScannerOptions* options,
ProtocolSession* allocation_session);
+56 -32
View File
@@ -62,23 +62,24 @@ void print_usage(void) {
printf(" NOTE: the FastSync batch format is NOT interoperable with rsync's batch\n");
printf(" files (different container format); do not mix the two tools.\n");
printf(" --delete Delete files on receiver not in source\n");
printf(" (default timing: delete only after the whole\n");
printf(" transfer has succeeded)\n");
printf(" (default timing: delete-during, like rsync --del)\n");
printf(" --delete-before Delete extras before the transfer starts\n");
printf(" (implies --delete)\n");
printf(" --delete-during Delete extras once the keep-set manifest is known,\n");
printf(" before the data is applied (implies --delete)\n");
printf(" --delete-during Delete a directory's extras as that directory is\n");
printf(" processed (implies --delete)\n");
printf(" --del Alias for --delete-during\n");
printf(" --delete-delay Delete extras only after a successful transfer\n");
printf(" (implies --delete)\n");
printf(" --delete-delay Record the extras during the scan but remove them\n");
printf(" only after a successful transfer (implies --delete)\n");
printf(" --delete-after Delete only after the whole transfer succeeded\n");
printf(" (the default --delete timing; implies --delete)\n");
printf(" (implies --delete)\n");
printf(" --delete-commit FastSync-only: restore the late whole-tree commit\n");
printf(" (identical to --delete-after; implies --delete)\n");
printf(" --delete-excluded Also delete destination files that were excluded on\n");
printf(" the source (default protects them, matching rsync)\n");
printf(" --max-delete=NUM Never delete more than NUM destination entries per run;\n");
printf(" if the extras would exceed NUM, nothing is deleted and\n");
printf(" the run fails with a clear error (implies --delete only\n");
printf(" when used with it)\n");
printf(" --max-delete=NUM Delete at most NUM destination entries per run; if the\n");
printf(" extras exceed NUM, the rest are skipped and the run is\n");
printf(" reported as partial (exit 25, matching rsync). Only\n");
printf(" applies together with --delete\n");
printf(" --ignore-errors Continue (and still delete) when a source directory is\n");
printf(" unreadable during the scan, instead of aborting with no\n");
printf(" deletion\n");
@@ -90,10 +91,11 @@ void print_usage(void) {
printf(" entry's destination mirror receiver-side. Independent of\n");
printf(" --delete (it does not imply --delete; a non-empty directory\n");
printf(" mirror is removed only with --force or --delete)\n");
printf(" -m, --prune-empty-dirs Do not transfer empty directory entries (--dirs mode);\n");
printf(" recursive transfers never send empty dirs\n");
printf(" -m, --prune-empty-dirs Do not create empty directories (a recursive transfer\n");
printf(" otherwise recreates them, like rsync)\n");
printf(" Note: each timing flag implies --delete. Combining a timing flag with\n");
printf(" --no-delete (in either order) is rejected as a config error.\n");
printf(" --no-delete (in either order) is rejected as a config error, as is more\n");
printf(" than one timing flag.\n");
printf(" --ignore-existing Skip files that already exist on receiver\n");
printf(" --delay-updates Put updated files into place only at the end of transfer\n");
printf(" --dirs, -d, --old-dirs, --old-d Transfer the named directory entries without\n");
@@ -103,8 +105,9 @@ void print_usage(void) {
printf(" -R, --relative With --files-from, preserve each listed entry's relative path\n");
printf(" below the destination root instead of mirroring the full\n");
printf(" source path (no effect without --files-from)\n");
printf(" --no-implied-dirs With -R --files-from, refuse to place a listed file whose\n");
printf(" parent directory is not itself listed\n");
printf(" --no-implied-dirs With -R, do not apply the source metadata of a listed file's\n");
printf(" implied parent directories (they are still created with\n");
printf(" default attributes)\n");
printf(" --mkpath Create the destination root directory on the server when it\n");
printf(" does not exist yet\n");
printf(" --exclude <pattern>, --exclude=<pattern> Exclude files matching pattern\n");
@@ -114,10 +117,12 @@ void print_usage(void) {
printf(" --files-from <file> Read the source file list from FILE (paths relative to the "
"source root)\n");
printf(" -0, --from0 Entries in --files-from are NUL-delimited\n");
printf(" -f, --filter=RULE rsync-style filter rule (+/- include/exclude; repeatable;\n");
printf(" both --filter=RULE and the -f RULE / -f=RULE short forms work)\n");
printf(" -f, --filter=RULE rsync-style filter rule: exclude/- include/+ hide/H show/S\n");
printf(" protect/P risk/R merge/. dir-merge/: clear/! with modifiers\n");
printf(" (repeatable; --filter=RULE and -f RULE / -f=RULE both work)\n");
printf(" -C, --cvs-exclude Auto-ignore common CVS/SCM files (.git/, .svn/, *.o, *~, ...)\n");
printf(" -F Apply per-directory .rsync-filter files during the scan\n");
printf(" -F Apply per-directory .rsync-filter files; repeated -FF also\n");
printf(" excludes the .rsync-filter files themselves\n");
printf(" --max-size <n> Skip files larger than n bytes\n");
printf(" --min-size <n> Skip files smaller than n bytes\n");
printf(" --max-alloc <SIZE> Maximum single allocation (default: 1G; 0 = no limit,\n");
@@ -135,11 +140,14 @@ void print_usage(void) {
printf(" into the destination instead of transferring its data\n");
printf(" --link-dest <dir> Like --copy-dest, but hard-links the unchanged file from DIR\n");
printf(" into the destination (repeatable; earlier DIRs win)\n");
printf(" --verify-basis FastSync-only: require a basis hit's content to match the\n");
printf(" source by whole-file digest instead of trusting rsync's\n");
printf(" size+mtime (or --size-only) quick-check\n");
printf(" --checksum-choice, --cc <alg> Whole-file checksum algorithm for --incremental/\n");
printf(" --checksum compares. Accepted: xxh64 (aka xxhash), xxh3,\n");
printf(" xxh128, md5, or auto (default xxh64). rsync choices FastSync\n");
printf(" does not implement (md4, sha1, none) and the two-name\n");
printf(" transfer,pre-transfer form are rejected by name\n");
printf(" --checksum compares. Accepted: xxh128 (default), xxh3, xxh64\n");
printf(" (aka xxhash), md5, md4, sha1, or none. A two-name\n");
printf(" 'transfer,pre-transfer' form is accepted like rsync; 'none' as\n");
printf(" the pre-transfer algorithm is rejected with --checksum\n");
printf(" --checksum-seed <num> Seed for the whole-file xxHash digest (and the delta\n");
printf(" block strong hash, low 32 bits); md5 ignores the seed. A seed\n");
printf(" of 0 (the default) is randomized per transfer, exactly like\n");
@@ -166,13 +174,14 @@ void print_usage(void) {
printf(" SSH argv is already built injection-safe)\n");
printf(" --sendfile Enable sendfile zero-copy (TCP only; long form only;\n");
printf(" -f is bound to --filter, not --sendfile)\n");
printf(" --compress-choice <alg> Compression algorithm (default: zstd)\n");
printf(" --compress-choice <alg> Compression algorithm: zstd (default), lz4, zlib,\n");
printf(" zlibx, none, or auto\n");
printf(" --zc <alg> Alias for --compress-choice\n");
printf(" -v, --verbose Enable debug logging\n");
printf(" -q, --quiet Suppress non-error output\n");
printf(" --debug=FLAGS Fine-grained debug logging (use --debug=help for flags)\n");
printf(" --info=FLAGS Fine-grained info: copy,misc,skip,stats,all,none\n");
printf(" none suppresses info even with --verbose\n");
printf(" --info=FLAGS Fine-grained info: copy,name,misc,skip,stats,all,none\n");
printf(" (use --info=help for flags; none suppresses --verbose)\n");
printf(" --preserve Preserve permissions and times (= -pt; long form only)\n");
printf(" --no-perms Negate -p/--perms\n");
printf(" --no-times Negate -t/--times\n");
@@ -240,7 +249,9 @@ void print_usage(void) {
printf(" reusable digest is sent (keep the file mode 0600)\n");
printf(" --no-motd Suppress display of the daemon's MOTD (the server\n");
printf(" still sends it; the client just does not show it)\n");
printf(" --bwlimit <KB/s> Bandwidth limit in kilobytes per second\n");
printf(" --bwlimit=RATE Limit socket I/O bandwidth (default unit KiB/s,\n");
printf(" rsync-style: 0 = no limit; K/M/G/T/P suffixes are\n");
printf(" binary, KB/MB decimal, KiB/MiB binary; decimals allowed)\n");
printf(" --tls Enable TLS encryption\n");
printf(" --cert <path> TLS certificate file (PEM)\n");
printf(" --key <path> TLS private key file (PEM)\n");
@@ -268,7 +279,7 @@ void print_usage(void) {
printf(" --suffix <str> Backup suffix (default: ~)\n");
printf(" --stats Print transfer statistics at end\n");
printf(" -i, --itemize-changes Print an rsync-style per-file change line\n");
printf(" --out-format=FORMAT Output format for changed files (%%f %%n %%l %%b %%M %%%%)\n");
printf(" --out-format=FORMAT Output format (%%f %%n %%l %%b %%c %%C %%i %%M %%%%)\n");
printf(" --list-only List source files instead of transferring\n");
printf(" --log-file-format=FORMAT Per-file log line format (needs --log-file)\n");
printf(" -h, --human-readable Print byte sizes in human-readable form\n");
@@ -279,8 +290,8 @@ void print_usage(void) {
printf(" --partial Keep partial files on interrupted transfer\n");
printf(" --partial-dir <dir> Directory for partial files\n");
printf(" -T, --temp-dir <dir> Scratch dir for temp files before atomic install.\n");
printf(" Relative dirs resolve below the destination root; absolute\n");
printf(" dirs are used as-is (rsync semantics). The dir must\n");
printf(" Confined to the receive root: a relative dir resolves below\n");
printf(" it and an absolute/traversal dir is rejected. The dir must\n");
printf(" already exist; a different filesystem falls back to a\n");
printf(" non-atomic copy instead of aborting\n");
printf(" --fastsync-server-path <path>\n");
@@ -338,7 +349,20 @@ void print_usage(void) {
}
void print_debug_usage(void) {
printf("Supported debug flags: IO,PROTO,PACK,UTIL,ALL,NONE\n");
printf("Emitting debug flags: IO,PROTO,PACK,UTIL,ALL,NONE\n");
printf("Also accepted for rsync CLI parity (silent): ACL,BACKUP,BIND,CHDIR,\n");
printf("CONNECT,CMD,DEL,DELTASUM,DUP,EXIT,FILTER,FLIST,FUZZY,GENR,HASH,HLINK,\n");
printf("ICONV,NSTR,OWN,RECV,SEND,TIME.\n");
printf("Flags may be comma-separated, for example: --debug=io,proto\n");
printf("Other rsync debug flags are unsupported and rejected.\n");
printf("An optional level suffix is accepted (e.g. --debug=io2); level 0\n");
printf("silences that item. Unknown names are rejected.\n");
}
void print_info_usage(void) {
printf("Emitting info flags: COPY,NAME,MISC,SKIP,STATS,ALL,NONE\n");
printf("Also accepted for rsync CLI parity (silent): BACKUP,DEL,FLIST,MOUNT,\n");
printf("NONREG,PROGRESS,REMOVE,SYMSAFE.\n");
printf("Flags may be comma-separated, for example: --info=name,stats\n");
printf("An optional level suffix is accepted (e.g. --info=stats2); level 0\n");
printf("silences that item. Unknown names are rejected.\n");
}
+1
View File
@@ -3,5 +3,6 @@
void print_usage(void);
void print_debug_usage(void);
void print_info_usage(void);
#endif
+202 -24
View File
@@ -3,6 +3,7 @@
#include "charset.h"
#include "chunk.h"
#include "config.h"
#include "delete_plan.h"
#include "delay_updates.h"
#include "file.h"
#include "file_receive.h"
@@ -11,6 +12,7 @@
#include "protocol.h"
#include "utils.h"
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <time.h>
@@ -58,6 +60,60 @@ bool receiver_send_final_success(int fd, const Config* config, const ReceiverOut
return send_status(fd, final_status);
}
bool receiver_send_stats_frame(int fd, const Config* config, const ReceiverStats* stats,
const struct ArrayList* would_delete,
const struct ArrayList* deleted_paths) {
if (!config->report_stats)
return true;
ReceiverStats local;
memset(&local, 0, sizeof(local));
const ReceiverStats* out = stats ? stats : &local;
/* The path list carries the dry-run would-delete set for a -n run and the
actually-removed set for a real --info=del run. */
const struct ArrayList* paths =
config->dry_run ? would_delete : (config->report_deletes ? deleted_paths : NULL);
size_t count = paths ? (size_t)paths->size : 0;
if (count > (size_t)MAX_MANIFEST_ENTRIES)
count = MAX_MANIFEST_ENTRIES;
ReceiverStats record = *out;
record.would_delete_count = count;
if (!send_status(fd, STATUS_STATS) || !format_stats_send(fd, &record) ||
!send_int(fd, (int)count))
return false;
for (size_t i = 0; i < count; i++) {
const char* path = (const char*)paths->items[i];
if (!send_wire_str(fd, path ? path : ""))
return false;
}
return true;
}
/* Add a delete commit's tally to the sink's end-of-transfer wire counters (when
the sink reports them). Runs on the receiving thread, so no locking. */
static void receiver_tally_deleted(const ReceiverSink* sink, size_t deleted) {
if (sink && sink->stats && deleted > 0)
sink->stats->deleted_files += deleted;
}
/* Observer for --info=del: record each truly-removed destination-relative path
in the ArrayList passed as the observer context, so the terminal STATUS_STATS
frame can list it. A failed append is best-effort (the deletion already
happened; output is cosmetic). Shared by the single-threaded receiver and
the -m pipeline's deferred commit. */
void receiver_record_deleted_path(void* context, const char* rel_path) {
ArrayList* paths = context;
if (!paths || !rel_path)
return;
/* Bound the retained list like the keep-set manifest: only MAX_MANIFEST_ENTRIES
paths are ever transmitted in the terminal STATUS_STATS frame, so recording
more only grows memory. A hostile/huge deletion set is therefore capped. */
if ((size_t)paths->size >= (size_t)MAX_MANIFEST_ENTRIES)
return;
char* copy = str_dup(rel_path);
if (copy && !array_list_add(paths, copy))
free(copy);
}
static bool receiver_process_chunk(Chunk* chunk, const ReceiverSink* sink) {
if (!chunk || !sink || !sink->store_file)
return false;
@@ -244,21 +300,23 @@ static bool receiver_note_status(const struct timespec* session_start,
}
int receiver_process(Config* config, int file_descriptor, const ReceiverSink* sink) {
return receiver_process_pending(config, file_descriptor, sink, NULL);
return receiver_process_pending(config, file_descriptor, sink, NULL, NULL);
}
/* Runs the whole receive loop. The delete manifest may legitimately arrive
either FIRST (--delete-before / --delete-during: the sender transmits the
validated keep-set before any file data) or LAST (plain --delete /
--delete-after / --delete-delay: the manifest closes the data stream). In
validated keep-set before any file data) or LAST (--delete-after /
--delete-commit / --delete-delay: the manifest closes the data stream). In
the early modes the receiver deletes as soon as the manifest has been read
and acknowledges with STATUS_OK so the sender only starts streaming once the
deletion has committed (or failed); in the late modes the manifest is held
and the deletion is committed only after the terminal STATUS_FINISHED proves
the whole transfer succeeded. See receiver_process_pending() for how the -m
receiver defers that commit until its disk writer has drained. */
the whole transfer succeeded. A plain --delete defaults to the per-directory
delete-during plan mode (no manifest at all). See
receiver_process_pending() for how the -m receiver defers that commit until
its disk writer has drained. */
int receiver_process_pending(Config* config, int file_descriptor, const ReceiverSink* sink,
DeleteManifest** pending_manifest) {
DeleteManifest** pending_manifest, DeletePlanSession** pending_plans) {
Status status;
if (!receive_status(file_descriptor, &status))
return -1;
@@ -272,14 +330,22 @@ int receiver_process_pending(Config* config, int file_descriptor, const Receiver
if (!receiver_note_status(&session_start, &last_progress, status, file_descriptor, sink))
return -1;
bool early_delete = config_delete_timing_early(config);
bool per_dir_delete = config_delete_timing_per_dir(config);
/* Parked keep-set for the late/commit timing. Every exit path below frees it
exactly once; the only exception is the successful FINISHED handoff, which
transfers ownership to *pending_manifest (used by the -m receiver). */
DeleteManifest* deferred_manifest = NULL;
/* Per-directory delete session for --delete-during/--delete-delay. During the
loop it applies plans inline (during) or snapshots their extras (delay); on
a successful FINISHED it is either committed here or handed to
*pending_plans so the -m caller commits after its disk writer drained. */
DeletePlanSession* plan_session = NULL;
bool delete_limit_noted = false;
while (status == STATUS_NEXT || status == STATUS_CHUNK || status == STATUS_CHECK ||
status == STATUS_KEEPALIVE || status == STATUS_ABORT || status == STATUS_CHECK_BATCH ||
status == STATUS_MKDIR || status == STATUS_MANIFEST || status == STATUS_HARDLINK ||
status == STATUS_SYMLINK || status == STATUS_SPECIAL || status == STATUS_DIR_TIMES) {
status == STATUS_SYMLINK || status == STATUS_SPECIAL || status == STATUS_DIR_TIMES ||
status == STATUS_DELETE_PLAN) {
if (status == STATUS_KEEPALIVE) {
if (!send_status(file_descriptor, STATUS_KEEPALIVE))
goto fail;
@@ -337,23 +403,35 @@ int receiver_process_pending(Config* config, int file_descriptor, const Receiver
if (config->dry_run) {
/* Server-contacting --dry-run mutates nothing, so a keep-set manifest
is consumed and discarded. The early-delete mode still needs its ACK
so a sender blocked on the delete handshake is not left hanging. */
so a sender blocked on the delete handshake is not left hanging.
When would-delete reporting is armed, enumerate (read-only) the
destination extras so the terminal STATUS_STATS frame can list them. */
if (config->use_delete && sink->would_delete) {
size_t count = 0;
if (!manifest_would_delete_list(config, manifest, sink->would_delete, &count))
log_message(LOG_LEVEL_WARNING, "dry-run: could not enumerate would-delete paths");
}
delete_manifest_free(manifest);
if (early_delete && !send_status(file_descriptor, STATUS_OK))
goto fail;
goto next_status;
}
if (early_delete) {
/* --delete-before / --delete-during: the manifest is authoritative the
moment it arrives, before any file data. Delete now and acknowledge
so the sender only starts streaming once the deletion committed (or
failed). This is the rsync delete-before/delete-during window: a
later transfer failure does not restore these deletions. A
/* --delete-before: the whole-tree manifest is authoritative the moment
it arrives, before any file data. Delete now and acknowledge so the
sender only starts streaming once the deletion committed (or failed).
A later transfer failure does not restore these deletions. A
--max-delete-capped commit still succeeds and the transfer proceeds;
the terminal success frame reports the cap. */
DeleteCommitResult deletion = (config->use_delete || config->delete_missing_args)
? manifest_delete_all(config, manifest)
: DELETE_COMMIT_OK;
size_t deleted = 0;
DeletePathObserver observer =
(config->report_deletes && sink->deleted_paths) ? receiver_record_deleted_path : NULL;
DeleteCommitResult deletion =
(config->use_delete || config->delete_missing_args)
? manifest_delete_all_observed(config, manifest, &deleted, observer,
(void*)sink->deleted_paths)
: DELETE_COMMIT_OK;
receiver_tally_deleted(sink, deleted);
delete_manifest_free(manifest);
if (deletion == DELETE_COMMIT_ERROR) {
send_status(file_descriptor, STATUS_ERROR);
@@ -364,9 +442,9 @@ int receiver_process_pending(Config* config, int file_descriptor, const Receiver
if (!send_status(file_descriptor, STATUS_OK))
goto fail;
} else if (config->use_delete || config->delete_missing_args) {
/* Plain --delete / --delete-after / --delete-delay and the
--delete-missing-args exact-path deletions: hold the manifest and
commit it only after STATUS_FINISHED. */
/* Plain --delete / --delete-after and the --delete-missing-args
exact-path deletions: hold the manifest and commit it only after
STATUS_FINISHED. The per-directory modes never send this frame. */
if (deferred_manifest) {
log_message(LOG_LEVEL_ERROR, "Received a second delete manifest");
delete_manifest_free(deferred_manifest);
@@ -380,6 +458,27 @@ int receiver_process_pending(Config* config, int file_descriptor, const Receiver
delete_manifest_free(manifest);
}
goto next_status;
} else if (status == STATUS_DELETE_PLAN) {
if (!per_dir_delete) {
log_message(LOG_LEVEL_ERROR, "Received a per-directory delete plan without a per-dir "
"delete timing");
send_status(file_descriptor, STATUS_ERROR);
goto fail;
}
if (!plan_session) {
plan_session = delete_plan_session_create(config);
if (plan_session && config->report_deletes && sink->deleted_paths)
delete_plan_session_set_delete_observer(plan_session, receiver_record_deleted_path,
(void*)sink->deleted_paths);
}
if (!plan_session || delete_plan_session_receive(plan_session, config, file_descriptor) != 0)
goto fail;
if (delete_plan_session_limit_reached(plan_session) && !delete_limit_noted &&
sink->note_delete_limit) {
sink->note_delete_limit(sink->context);
delete_limit_noted = true;
}
goto next_status;
} else {
File* file = file_receive(config, file_descriptor);
if (!file) {
@@ -413,7 +512,12 @@ int receiver_process_pending(Config* config, int file_descriptor, const Receiver
*pending_manifest = deferred_manifest;
deferred_manifest = NULL;
} else {
DeleteCommitResult deletion = manifest_delete_all(config, deferred_manifest);
size_t deleted = 0;
DeletePathObserver observer =
(config->report_deletes && sink->deleted_paths) ? receiver_record_deleted_path : NULL;
DeleteCommitResult deletion = manifest_delete_all_observed(
config, deferred_manifest, &deleted, observer, (void*)sink->deleted_paths);
receiver_tally_deleted(sink, deleted);
delete_manifest_free(deferred_manifest);
deferred_manifest = NULL;
if (deletion == DELETE_COMMIT_ERROR) {
@@ -424,6 +528,36 @@ int receiver_process_pending(Config* config, int file_descriptor, const Receiver
sink->note_delete_limit(sink->context);
}
}
/* Per-directory deletion: --delete-during already applied each plan inline, so
this only finishes the missing-args deletions; --delete-delay committed
nothing yet and applies its decompressed snapshot here. The -m receiver
hands the session to its caller instead, which commits after the disk
writer drained. */
if (plan_session) {
if (config->report_deletes && sink->deleted_paths)
delete_plan_session_set_delete_observer(plan_session, receiver_record_deleted_path,
(void*)sink->deleted_paths);
if (pending_plans) {
*pending_plans = plan_session;
plan_session = NULL;
} else if (config->dry_run) {
/* Central dry-run no-op: never commit a deletion for a -n run. */
delete_plan_session_destroy(plan_session);
plan_session = NULL;
} else {
DeleteCommitResult deletion = delete_plan_session_commit(plan_session, config);
bool limit = delete_plan_session_limit_reached(plan_session);
receiver_tally_deleted(sink, delete_plan_session_deleted(plan_session));
delete_plan_session_destroy(plan_session);
plan_session = NULL;
if (deletion == DELETE_COMMIT_ERROR) {
send_status(file_descriptor, STATUS_ERROR);
goto fail;
}
if (limit && !delete_limit_noted && sink->note_delete_limit)
sink->note_delete_limit(sink->context);
}
}
if (sink->send_success) {
if (sink->send_success_frame) {
if (!sink->send_success_frame(file_descriptor, sink->context))
@@ -436,11 +570,14 @@ int receiver_process_pending(Config* config, int file_descriptor, const Receiver
fail:
/* Failure exits that must not (or already did) report a STATUS_ERROR. The
parked keep-set is dropped: never commit a deletion for a failed stream. */
parked keep-set/session is dropped: never commit a deletion for a failed
stream. */
if (deferred_manifest) {
delete_manifest_free(deferred_manifest);
deferred_manifest = NULL;
}
if (plan_session)
delete_plan_session_destroy(plan_session);
return -1;
receive_error:
@@ -448,6 +585,8 @@ receive_error:
delete_manifest_free(deferred_manifest);
deferred_manifest = NULL;
}
if (plan_session)
delete_plan_session_destroy(plan_session);
if (sink->send_error)
send_status(file_descriptor, STATUS_ERROR);
return -1;
@@ -465,11 +604,19 @@ typedef struct {
/* Set when a --max-delete commit was capped; the terminal frame then carries
STATUS_DELETE_LIMIT so the sender exits 25 like rsync. */
bool delete_limit_reached;
/* End-of-transfer wire counters (protocol 2.25.0) and the -n/--dry-run
--delete would-delete path list collected while processing the manifest. */
ReceiverStats stats;
ArrayList* would_delete;
/* --info=del: actually-removed paths collected during the delete commit. */
ArrayList* deleted_paths;
} ReceiverSaveContext;
static bool receiver_save_file(File* file, void* context_pointer) {
ReceiverSaveContext* context = context_pointer;
FileSaveResult result = FILE_SAVE_ERROR;
bool created = false;
unsigned created_dirs = 0;
if (context->config->dry_run) {
/* Defense in depth: a dry-run receiver mutates nothing even if a data
frame reaches the sink (the sender is not supposed to send one). */
@@ -479,8 +626,17 @@ static bool receiver_save_file(File* file, void* context_pointer) {
--remove-source-files sender keeps its source. */
result = FILE_SAVE_SKIPPED;
} else {
result = file_save_to_disk_full(context->config->receive_root_directory, file, context->config);
result = file_save_to_disk_full_ex(context->config->receive_root_directory, file,
context->config, &created, &created_dirs);
}
/* Wire-stats tally: bytes reconstructed from the basis file (delta matches)
count as matched data in the end-of-transfer report. */
if (result != FILE_SAVE_ERROR && file->matched_bytes > 0)
context->stats.matched_data += file->matched_bytes;
/* Protocol 2.28.0: receiver-observed literal bytes and the created-entry
breakdown (regular/dir/link/special) for the `--stats` report. */
if (result == FILE_SAVE_WRITTEN)
receiver_stats_note_saved(&context->stats, file, created, created_dirs);
/* A directory's metadata is deferred, never applied inline: collect it now
and apply it at the end. -O/--omit-dir-times and --preserve_perms/-times
are honored by dir_metadata_list_apply's caller (see
@@ -513,6 +669,9 @@ static void receiver_note_delete_limit(void* context_pointer) {
static bool receiver_send_success_frame(int fd, void* context_pointer) {
ReceiverSaveContext* context = context_pointer;
Status final_status = context->delete_limit_reached ? STATUS_DELETE_LIMIT : STATUS_OK;
if (!receiver_send_stats_frame(fd, context->config, &context->stats, context->would_delete,
context->deleted_paths))
return false;
/* Server-contacting --dry-run: nothing was staged or written, so there is
nothing to publish and no directory times to stamp. */
if (context->config->dry_run)
@@ -540,12 +699,31 @@ static bool receiver_send_success_frame(int fd, void* context_pointer) {
int receiver_receive_files(Config* config, int file_descriptor) {
ReceiverSaveContext context = {.config = config, .outcomes = {0}};
dir_time_list_init(&context.dir_times);
ReceiverSink sink = {receiver_save_file, &context, true, true, receiver_send_success_frame,
receiver_note_delete_limit};
context.would_delete = array_list_create(free);
/* report_deletes (--info=del / -i / --out-format under --delete) is the only
reason to retain the actually-removed paths; a plain --delete must not
str_dup every removal. NULL is handled by every consumer. */
context.deleted_paths = config->report_deletes ? array_list_create(free) : NULL;
if (!context.would_delete || (config->report_deletes && !context.deleted_paths)) {
array_list_delete(context.would_delete);
array_list_delete(context.deleted_paths);
return -1;
}
ReceiverSink sink = {receiver_save_file,
&context,
true,
true,
receiver_send_success_frame,
receiver_note_delete_limit,
&context.stats,
context.would_delete,
context.deleted_paths};
int ret = receiver_process(config, file_descriptor, &sink);
if (ret != 0 && config->delay_updates && config->delay_context)
delay_updates_cleanup(config->delay_context);
receiver_outcomes_destroy(&context.outcomes);
dir_time_list_free(&context.dir_times);
array_list_delete(context.would_delete);
array_list_delete(context.deleted_paths);
return ret;
}
+29 -3
View File
@@ -2,6 +2,7 @@
#define RECEIVER_H
#include "config.h"
#include "delete_plan.h"
#include "file.h"
#include "file_receive.h"
#include "protocol.h"
@@ -39,24 +40,49 @@ typedef struct {
ReceiverSuccessFrame send_success_frame;
/* Optional; may be NULL when the sink has no --max-delete handling. */
ReceiverNoteDeleteLimit note_delete_limit;
/* Optional end-of-transfer wire counters (protocol 2.25.0). When non-NULL
and the wire config carries report_stats, the success frame is preceded by
a STATUS_STATS record; `would_delete` (optional, receiver-owned strings)
carries the -n/--dry-run --delete path list. */
ReceiverStats* stats;
struct ArrayList* would_delete;
/* When --info=del requested it, receiver-owned strings for every path the
deletion commit ACTUALLY removed, sent in the terminal STATUS_STATS frame's
path list so the sender can print rsync's `deleting PATH` lines. */
struct ArrayList* deleted_paths;
} ReceiverSink;
bool receiver_outcomes_append(ReceiverOutcomes* outcomes, unsigned char code);
void receiver_outcomes_destroy(ReceiverOutcomes* outcomes);
/* DeletePathObserver implementation for --info=del: `context` is an ArrayList*
that receives owned copies of every truly-removed destination-relative path.
Shared by the single-threaded receiver and the -m pipeline's deferred commit. */
void receiver_record_deleted_path(void* context, const char* rel_path);
/* Send the terminal success frame. `final_status` is usually STATUS_OK, or
STATUS_DELETE_LIMIT when a --max-delete commit was capped. */
bool receiver_send_final_success(int fd, const Config* config, const ReceiverOutcomes* outcomes,
Status final_status);
/* Emit STATUS_STATS (a fixed ReceiverStats record plus, when `would_delete` is
non-NULL, a count and that many wire strings) when the wire config requested
report_stats. A no-op otherwise. */
bool receiver_send_stats_frame(int fd, const Config* config, const ReceiverStats* stats,
const struct ArrayList* would_delete,
const struct ArrayList* deleted_paths);
int receiver_process(Config* config, int file_descriptor, const ReceiverSink* sink);
/* receiver_process with an escape hatch for the commit-style (late) deletion:
when `pending_manifest` is non-NULL the receiver does NOT delete at
STATUS_FINISHED itself; instead it stores the owned keep-set manifest there
(leaving *pending_manifest untouched on early modes/errors) so the caller can
commit the deletion only after its disk writer has fully drained. Pass NULL
to keep the default behaviour (delete before the success frame). */
commit the deletion only after its disk writer has fully drained. Likewise,
when `pending_plans` is non-NULL the --delete-delay per-directory session is
handed to the caller instead of being committed at STATUS_FINISHED. Pass NULL
for either to keep the default behaviour (delete before the success frame). */
int receiver_process_pending(Config* config, int file_descriptor, const ReceiverSink* sink,
DeleteManifest** pending_manifest);
DeleteManifest** pending_manifest, DeletePlanSession** pending_plans);
int receiver_receive_files(Config* config, int file_descriptor);
/* ---- Connection time bounds (anti-slowloris) ----
+56 -5
View File
@@ -27,7 +27,11 @@ PipelineContextReceiver* pipeline_context_receiver_create(Config* config, Queue*
context->queued_bytes = 0;
context->max_queue_bytes = 0;
context->deferred_manifest = NULL;
context->deferred_plans = NULL;
context->delete_limit_reached = false;
memset(&context->stats, 0, sizeof(context->stats));
context->would_delete = NULL;
context->deleted_paths = NULL;
atomic_init(&context->cancelled, false);
int init = 0;
if (mtx_init(&context->mutex, mtx_plain) != thrd_success)
@@ -40,6 +44,18 @@ PipelineContextReceiver* pipeline_context_receiver_create(Config* config, Queue*
goto fail;
// cppcheck-suppress unreadVariable
init++;
context->would_delete = array_list_create(free);
if (!context->would_delete)
goto fail;
/* The actually-removed path list is only needed to render rsync's
`deleting PATH` lines, which the client requests via report_deletes
(--info=del / -i / --out-format under --delete). A plain --delete run must
not allocate it or observe every removal. */
if (config->report_deletes) {
context->deleted_paths = array_list_create(free);
if (!context->deleted_paths)
goto fail;
}
return context;
fail:
@@ -50,6 +66,12 @@ fail:
cnd_destroy(&context->condition_not_full);
if (init >= 1)
mtx_destroy(&context->mutex);
/* Free every list that was already created before the failing allocation:
`context` itself is freed below, so they would otherwise leak. */
if (context->would_delete)
array_list_delete(context->would_delete);
if (context->deleted_paths)
array_list_delete(context->deleted_paths);
free(context);
return NULL;
}
@@ -58,9 +80,15 @@ void pipeline_context_receiver_destroy(PipelineContextReceiver* context) {
config_delete(context->config);
if (context->deferred_manifest)
delete_manifest_free(context->deferred_manifest);
if (context->deferred_plans)
delete_plan_session_destroy(context->deferred_plans);
queue_destroy(context->queue);
receiver_outcomes_destroy(&context->outcomes);
dir_time_list_free(&context->dir_times);
if (context->would_delete)
array_list_delete(context->would_delete);
if (context->deleted_paths)
array_list_delete(context->deleted_paths);
mtx_destroy(&context->mutex);
cnd_destroy(&context->condition_not_full);
cnd_destroy(&context->condition_not_empty);
@@ -133,6 +161,11 @@ bool pipeline_context_receiver_enqueue_file(PipelineContextReceiver* context, Fi
static bool receiver_enqueue_file(File* file, void* context_pointer) {
PipelineContextReceiver* context = (PipelineContextReceiver*)context_pointer;
if (file && file->matched_bytes > 0) {
mtx_lock(&context->mutex);
context->stats.matched_data += file->matched_bytes;
mtx_unlock(&context->mutex);
}
return pipeline_context_receiver_enqueue_file(context, file);
}
@@ -162,10 +195,17 @@ int receive_thread(void* pipeline_context) {
const Config* config = context->config;
mtx_unlock(&context->mutex);
ReceiverSink sink = {
receiver_enqueue_file, context, false, false, NULL, receiver_pipeline_note_delete_limit};
if (receiver_process_pending((Config*)config, file_descriptor, &sink,
&context->deferred_manifest) != 0) {
ReceiverSink sink = {receiver_enqueue_file,
context,
false,
false,
NULL,
receiver_pipeline_note_delete_limit,
&context->stats,
context->would_delete,
context->deleted_paths};
if (receiver_process_pending((Config*)config, file_descriptor, &sink, &context->deferred_manifest,
&context->deferred_plans) != 0) {
receiver_thread_fail(context);
protocol_session_unbind();
return thrd_error;
@@ -207,12 +247,23 @@ int write_thread(void* pipeline_context) {
}
size_t file_bytes = file->data ? file->data->size : 0;
FileSaveResult result = FILE_SAVE_SKIPPED;
bool created = false;
unsigned created_dirs = 0;
/* Server-contacting --dry-run: never write. The receiver thread does not
enqueue anything on the dry-run path, but this keeps the writer thread
provably mutation-free if a data frame ever reached it. */
bool dry_run = context->config->dry_run;
if (save_to_disk && !dry_run) {
result = file_save_to_disk_full(root_directory, file, context->config);
result =
file_save_to_disk_full_ex(root_directory, file, context->config, &created, &created_dirs);
if (result == FILE_SAVE_WRITTEN) {
/* Protocol 2.28.0: fold the receiver-observed literal bytes and the
created-entry type into the shared stats block under its mutex (the
receive thread also writes stats.matched_data). */
mtx_lock(&context->mutex);
receiver_stats_note_saved(&context->stats, file, created, created_dirs);
mtx_unlock(&context->mutex);
}
if (result == FILE_SAVE_ERROR) {
file_destroy(file);
pipeline_context_receiver_note_bytes_released(context, file_bytes);
+15
View File
@@ -41,6 +41,11 @@ typedef struct PipelineContextReceiver {
transfer truly succeeded. NULL in the early delete modes (which delete at
the manifest). */
DeleteManifest* deferred_manifest;
/* Per-directory delete session for --delete-delay: receive_thread snapshots
each plan's extras as it arrives and hands the session here instead of
committing while the disk writer may still be draining; server.c commits it
after both threads joined. NULL for every other timing. */
DeletePlanSession* deferred_plans;
/* Set by server.c when the deferred delete commit hit the --max-delete
budget; the terminal success frame then carries STATUS_DELETE_LIMIT
(rsync exit 25) while the transfer itself still succeeds. */
@@ -49,6 +54,16 @@ typedef struct PipelineContextReceiver {
directory entries. Only write_thread mutates it (before it joins); the
caller (server.c) applies it after the delete/delay-updates phase. */
DirTimeList dir_times;
/* End-of-transfer wire counters (protocol 2.25.0). receive_thread accumulates
matched_data under `mutex`; server.c adds the delete-commit tallies after
both threads join and emits the STATUS_STATS frame. */
ReceiverStats stats;
/* -n/--dry-run --delete would-delete path list, collected by receive_thread
and reported in the STATUS_STATS frame. */
struct ArrayList* would_delete;
/* --info=del actually-removed path list, collected by the deferred delete
commit in server.c and reported in the STATUS_STATS frame. */
struct ArrayList* deleted_paths;
} PipelineContextReceiver;
PipelineContextReceiver* pipeline_context_receiver_create(Config* config, Queue* queue_receiver,
+35 -2
View File
@@ -741,6 +741,10 @@ void handler(int file_descriptor) {
* received config. */
if (gate_ctx.super_mode_override != -1)
config->super_mode = (SuperMode)gate_ctx.super_mode_override;
/* Install the codec this connection negotiated before the receiver/writer
* threads start (the server forks per connection, so the process-global
* codec is private to this session). */
compression_set_algo((CompressionAlgo)config->compression_algo);
/* If the client requested ownership but the effective super mode forbids it
* (operator --no-super, a privileged standalone receiver's secure default, or
* a daemon module without `client owner = yes`), say so ONCE per connection so
@@ -951,7 +955,11 @@ void handler(int file_descriptor) {
--delay-updates run; the walker skips the staging directory. A
server-contacting --dry-run deletes nothing (no manifest is sent). */
if (context->deferred_manifest) {
DeleteCommitResult deletion = manifest_delete_all(config, context->deferred_manifest);
size_t deleted = 0;
DeletePathObserver observer = config->report_deletes ? receiver_record_deleted_path : NULL;
DeleteCommitResult deletion = manifest_delete_all_observed(
config, context->deferred_manifest, &deleted, observer, (void*)context->deleted_paths);
context->stats.deleted_files += deleted;
if (deletion == DELETE_COMMIT_ERROR) {
transfer_ok = false;
} else if (deletion == DELETE_COMMIT_LIMIT_REACHED) {
@@ -962,6 +970,27 @@ void handler(int file_descriptor) {
delete_manifest_free(context->deferred_manifest);
context->deferred_manifest = NULL;
}
/* --delete-delay: receive_thread snapshotted each plan's extras as it
arrived; with the disk writer drained, commit the deferred removals.
--delete-during already applied its plans on the receive thread. */
if (context->deferred_plans) {
/* Defence in depth (the enclosing block already excludes dry-run): a
-n run never commits a deletion. */
if (config->report_deletes)
delete_plan_session_set_delete_observer(
context->deferred_plans, receiver_record_deleted_path, (void*)context->deleted_paths);
DeleteCommitResult deletion =
config->dry_run ? DELETE_COMMIT_OK
: delete_plan_session_commit(context->deferred_plans, config);
context->stats.deleted_files += delete_plan_session_deleted(context->deferred_plans);
if (deletion == DELETE_COMMIT_ERROR) {
transfer_ok = false;
} else if (deletion == DELETE_COMMIT_LIMIT_REACHED) {
context->delete_limit_reached = true;
}
delete_plan_session_destroy(context->deferred_plans);
context->deferred_plans = NULL;
}
}
if (transfer_ok && !config->dry_run) {
/* --delay-updates: receive_thread has finished the whole protocol stream
@@ -982,7 +1011,11 @@ void handler(int file_descriptor) {
}
if (transfer_ok) {
Status final_status = context->delete_limit_reached ? STATUS_DELETE_LIMIT : STATUS_OK;
if (!receiver_send_final_success(file_descriptor, config, &context->outcomes, final_status))
/* Emit the optional wire-stats record first (protocol 2.25.0), then the
success/outcome frame, exactly like the single-threaded receiver. */
if (!receiver_send_stats_frame(file_descriptor, config, &context->stats,
context->would_delete, context->deleted_paths) ||
!receiver_send_final_success(file_descriptor, config, &context->outcomes, final_status))
transfer_ok = false;
} else {
send_error_detail(file_descriptor, "transfer failed on receiver");
+15 -10
View File
@@ -168,9 +168,14 @@ bool charset_spec_valid_direction(const char* from_charset, const char* to_chars
return direction_probe_valid(from_charset, to_charset);
}
/* The receiver's real conversion is wire(client REMOTE) -> server-local (the
* server's own --iconv LOCAL half, or the client's LOCAL half when the server
* has no --iconv). A dedicated pre-ack check so an impossible direction is
/* The receiver's conversion is wire charset -> destination charset. rsync's
* CONVERT_SPEC is LOCAL,REMOTE and "stays the same whether you're pushing or
* pulling", so for a PUSH (FastSync's only direction) the destination end's
* charset is the spec's REMOTE half: the client converts LOCAL -> REMOTE on the
* sender and the receiver writes the wire bytes verbatim. Only a server that
* declares its OWN --iconv (the daemon "charset" analog) has a different local
* charset, and then it is that spec's LOCAL half and the receiver converts
* wire -> server-local. A dedicated pre-ack check so an impossible direction is
* rejected before the connection instead of refusing mid-transfer. */
bool charset_wire_receiver_spec_valid(const char* spec, const char* server_spec) {
if (!spec)
@@ -180,7 +185,7 @@ bool charset_wire_receiver_spec_valid(const char* spec, const char* server_spec)
if (charset_spec_parse(spec, &local, &remote) != 0)
return false;
const char* wire = remote;
const char* target_local = local;
const char* target_local = remote;
char* server_local = NULL;
char* server_remote = NULL;
if (server_spec) {
@@ -302,13 +307,13 @@ bool charset_wire_init_receiver(const char* spec, const char* server_spec) {
char* remote;
if (charset_spec_parse(spec, &local, &remote) != 0)
return false;
/* The wire charset is the client spec's REMOTE half; the local charset is
* the client spec's LOCAL half unless the server was itself started with
* --iconv naming a different local charset (the server halves above never
* travel, so the server's own flag is the only way its local charset can
* differ from what the client assumed). */
/* The wire charset is the client spec's REMOTE half (rsync's LOCAL,REMOTE
* spec stays the same push or pull, so on a push the destination end's
* charset is REMOTE and the receiver writes the wire bytes verbatim). Only a
* server started with its own --iconv declares a different local charset (the
* server halves above never travel), and then it is that spec's LOCAL half. */
const char* wire = remote;
const char* target_local = local;
const char* target_local = remote;
char* server_local = NULL;
char* server_remote = NULL;
if (server_spec) {
+6 -5
View File
@@ -57,8 +57,9 @@ void charset_conversion_close(void* conversion);
/* Process-wide wire conversion. charset_wire_init_sender (client side) opens
* LOCAL->REMOTE; charset_wire_init_receiver (server side) opens
* wire(REMOTE)->server-local. server_spec is the server's own --iconv, whose
* LOCAL half may override the local charset the client assumed; NULL reuses
* the client spec's LOCAL half. Both return false on an unsupported spec.
* LOCAL half overrides the destination charset; NULL means the destination
* charset is the client spec's REMOTE half (rsync's push semantics: the wire
* bytes are written verbatim). Both return false on an unsupported spec.
* The state is freed with charset_wire_free. */
bool charset_wire_init_sender(const char* spec);
bool charset_wire_init_receiver(const char* spec, const char* server_spec);
@@ -66,9 +67,9 @@ void charset_wire_free(void);
bool charset_wire_active(void);
/* Pre-ack receiver-direction sanity (see charset_wire_init_receiver): true
* when the exact wire->server-local conversion the receiver will use (client
* spec's REMOTE half into the server's own LOCAL half, or the client's LOCAL
* half when the server has no --iconv) opens and produces NUL-free output. */
* when the exact wire->destination conversion the receiver will use (client
* spec's REMOTE half into the server's own LOCAL half, or REMOTE->REMOTE when
* the server has no --iconv) opens and produces NUL-free output. */
bool charset_wire_receiver_spec_valid(const char* spec, const char* server_spec);
/* Convert a path across the wire in the process direction. Returns a malloc'd
+349 -22
View File
@@ -1,12 +1,171 @@
#include "checksum.h"
#include "utils.h"
#include <fcntl.h>
#include <openssl/evp.h>
#include <string.h>
#include <strings.h>
#include <unistd.h>
/* delta.c owns the single XXH_IMPLEMENTATION that provides the xxHash symbols
* for the whole binary; this TU only needs the declarations. */
* for the whole binary; this TU only needs the declarations. The streaming
* state structs and XXH3_update are exposed only with XXH_STATIC_LINKING_ONLY. */
#define XXH_STATIC_LINKING_ONLY
#include <xxhash.h>
/* ---------------------------------------------------------------------------
* Self-contained MD4 (RFC 1320). OpenSSL's MD4 lives in the legacy provider
* and is not guaranteed present, so FastSync carries its own implementation to
* keep --checksum-choice=md4 working on every build.
* ------------------------------------------------------------------------- */
typedef struct {
uint32_t state[4];
uint64_t bit_count;
uint8_t buffer[64];
size_t buffer_len;
} Md4Ctx;
static uint32_t md4_rotl(uint32_t x, int n) {
return (x << n) | (x >> (32 - n));
}
static void md4_transform(uint32_t state[4], const uint8_t block[64]) {
uint32_t x[16];
for (int i = 0; i < 16; i++)
x[i] = (uint32_t)block[i * 4] | ((uint32_t)block[i * 4 + 1] << 8) |
((uint32_t)block[i * 4 + 2] << 16) | ((uint32_t)block[i * 4 + 3] << 24);
uint32_t a = state[0], b = state[1], c = state[2], d = state[3];
#define F(x, y, z) (((x) & (y)) | (~(x) & (z)))
#define G(x, y, z) (((x) & (y)) | ((x) & (z)) | ((y) & (z)))
#define H(x, y, z) ((x) ^ (y) ^ (z))
#define ROUND1(a, b, c, d, k, s) a = md4_rotl(a + F(b, c, d) + x[k], s)
#define ROUND2(a, b, c, d, k, s) a = md4_rotl(a + G(b, c, d) + x[k] + 0x5a827999u, s)
#define ROUND3(a, b, c, d, k, s) a = md4_rotl(a + H(b, c, d) + x[k] + 0x6ed9eba1u, s)
ROUND1(a, b, c, d, 0, 3);
ROUND1(d, a, b, c, 1, 7);
ROUND1(c, d, a, b, 2, 11);
ROUND1(b, c, d, a, 3, 19);
ROUND1(a, b, c, d, 4, 3);
ROUND1(d, a, b, c, 5, 7);
ROUND1(c, d, a, b, 6, 11);
ROUND1(b, c, d, a, 7, 19);
ROUND1(a, b, c, d, 8, 3);
ROUND1(d, a, b, c, 9, 7);
ROUND1(c, d, a, b, 10, 11);
ROUND1(b, c, d, a, 11, 19);
ROUND1(a, b, c, d, 12, 3);
ROUND1(d, a, b, c, 13, 7);
ROUND1(c, d, a, b, 14, 11);
ROUND1(b, c, d, a, 15, 19);
ROUND2(a, b, c, d, 0, 3);
ROUND2(d, a, b, c, 4, 5);
ROUND2(c, d, a, b, 8, 9);
ROUND2(b, c, d, a, 12, 13);
ROUND2(a, b, c, d, 1, 3);
ROUND2(d, a, b, c, 5, 5);
ROUND2(c, d, a, b, 9, 9);
ROUND2(b, c, d, a, 13, 13);
ROUND2(a, b, c, d, 2, 3);
ROUND2(d, a, b, c, 6, 5);
ROUND2(c, d, a, b, 10, 9);
ROUND2(b, c, d, a, 14, 13);
ROUND2(a, b, c, d, 3, 3);
ROUND2(d, a, b, c, 7, 5);
ROUND2(c, d, a, b, 11, 9);
ROUND2(b, c, d, a, 15, 13);
ROUND3(a, b, c, d, 0, 3);
ROUND3(d, a, b, c, 8, 9);
ROUND3(c, d, a, b, 4, 11);
ROUND3(b, c, d, a, 12, 15);
ROUND3(a, b, c, d, 2, 3);
ROUND3(d, a, b, c, 10, 9);
ROUND3(c, d, a, b, 6, 11);
ROUND3(b, c, d, a, 14, 15);
ROUND3(a, b, c, d, 1, 3);
ROUND3(d, a, b, c, 9, 9);
ROUND3(c, d, a, b, 5, 11);
ROUND3(b, c, d, a, 13, 15);
ROUND3(a, b, c, d, 3, 3);
ROUND3(d, a, b, c, 11, 9);
ROUND3(c, d, a, b, 7, 11);
ROUND3(b, c, d, a, 15, 15);
#undef F
#undef G
#undef H
#undef ROUND1
#undef ROUND2
#undef ROUND3
state[0] += a;
state[1] += b;
state[2] += c;
state[3] += d;
}
static void md4_init(Md4Ctx* ctx) {
ctx->state[0] = 0x67452301u;
ctx->state[1] = 0xefcdab89u;
ctx->state[2] = 0x98badcfeu;
ctx->state[3] = 0x10325476u;
ctx->bit_count = 0;
ctx->buffer_len = 0;
}
static void md4_update(Md4Ctx* ctx, const uint8_t* data, size_t len) {
ctx->bit_count += (uint64_t)len * 8;
while (len > 0) {
size_t space = sizeof(ctx->buffer) - ctx->buffer_len;
size_t take = len < space ? len : space;
memcpy(ctx->buffer + ctx->buffer_len, data, take);
ctx->buffer_len += take;
data += take;
len -= take;
if (ctx->buffer_len == sizeof(ctx->buffer)) {
md4_transform(ctx->state, ctx->buffer);
ctx->buffer_len = 0;
}
}
}
static void md4_final(Md4Ctx* ctx, uint8_t out[16]) {
uint64_t bit_count = ctx->bit_count;
uint8_t pad = 0x80;
md4_update(ctx, &pad, 1);
uint8_t zero = 0;
while (ctx->buffer_len != 56)
md4_update(ctx, &zero, 1);
uint8_t length_le[8];
for (int i = 0; i < 8; i++)
length_le[i] = (uint8_t)((bit_count >> (8 * i)) & 0xff);
md4_update(ctx, length_le, sizeof(length_le));
for (int i = 0; i < 4; i++) {
out[i * 4] = (uint8_t)(ctx->state[i] & 0xff);
out[i * 4 + 1] = (uint8_t)((ctx->state[i] >> 8) & 0xff);
out[i * 4 + 2] = (uint8_t)((ctx->state[i] >> 16) & 0xff);
out[i * 4 + 3] = (uint8_t)((ctx->state[i] >> 24) & 0xff);
}
}
/* One-shot EVP digest (md5/sha1). Returns false when OpenSSL refuses. */
static bool evp_digest(const EVP_MD* md, const void* data, size_t size, uint8_t* out,
size_t out_capacity, size_t* out_len) {
static const uint8_t empty = 0;
const void* input = data ? data : &empty;
unsigned int digest_len = 0;
if (EVP_Digest(input, size, out, &digest_len, md, NULL) != 1)
return false;
if (digest_len > out_capacity)
return false;
*out_len = digest_len;
return true;
}
bool checksum_digest(ChecksumAlgo algo, uint64_t seed, const void* data, size_t size, uint8_t* out,
size_t out_capacity, size_t* out_len) {
if (!out || !out_len || out_capacity < CHECKSUM_MAX_DIGEST_LEN)
@@ -14,44 +173,166 @@ bool checksum_digest(ChecksumAlgo algo, uint64_t seed, const void* data, size_t
if (data == NULL && size != 0)
return false;
if (algo == CHECKSUM_ALGO_XXH64) {
switch (algo) {
case CHECKSUM_ALGO_XXH64: {
uint64_t digest = XXH64(data, size, seed);
memcpy(out, &digest, sizeof(digest));
*out_len = sizeof(digest);
return true;
}
if (algo == CHECKSUM_ALGO_XXH3) {
case CHECKSUM_ALGO_XXH3: {
uint64_t digest = XXH3_64bits_withSeed(data, size, seed);
memcpy(out, &digest, sizeof(digest));
*out_len = sizeof(digest);
return true;
}
if (algo == CHECKSUM_ALGO_XXH128) {
case CHECKSUM_ALGO_XXH128: {
XXH128_hash_t digest = XXH3_128bits_withSeed(data, size, seed);
memcpy(out, &digest, sizeof(digest));
*out_len = sizeof(digest);
return true;
}
if (algo == CHECKSUM_ALGO_MD5) {
case CHECKSUM_ALGO_MD5:
/* md5 takes no seed; the caller's seed is deliberately ignored (documented
* in RSYNC_COMPAT.md). OpenSSL's one-shot EVP_Digest needs a non-NULL
* buffer even for an empty input, so map a NULL data + size==0 to an empty
* buffer. */
static const uint8_t empty = 0;
const void* input = data ? data : &empty;
unsigned int digest_len = 0;
if (EVP_Digest(input, size, out, &digest_len, EVP_md5(), NULL) != 1)
return false;
if (digest_len > out_capacity)
return false;
*out_len = digest_len;
* in RSYNC_COMPAT.md). */
return evp_digest(EVP_md5(), data, size, out, out_capacity, out_len);
case CHECKSUM_ALGO_MD4: {
Md4Ctx ctx;
md4_init(&ctx);
md4_update(&ctx, (const uint8_t*)data, size);
md4_final(&ctx, out);
*out_len = 16;
return true;
}
case CHECKSUM_ALGO_SHA1:
/* sha1 takes no seed; the caller's seed is deliberately ignored. */
return evp_digest(EVP_sha1(), data, size, out, out_capacity, out_len);
case CHECKSUM_ALGO_NONE:
/* No checksum requested: an empty digest is the successful result. */
*out_len = 0;
return true;
}
return false;
}
bool checksum_digest_file(ChecksumAlgo algo, uint64_t seed, const char* path, uint8_t* out,
size_t out_capacity, size_t* out_len) {
if (!path || !out || !out_len || out_capacity < CHECKSUM_MAX_DIGEST_LEN)
return false;
int fd = open(path, O_RDONLY | O_CLOEXEC);
if (fd < 0)
return false;
bool ok = checksum_digest_fd(algo, seed, fd, out, out_capacity, out_len);
close(fd);
return ok;
}
bool checksum_digest_fd(ChecksumAlgo algo, uint64_t seed, int fd, uint8_t* out, size_t out_capacity,
size_t* out_len) {
if (fd < 0 || !out || !out_len || out_capacity < CHECKSUM_MAX_DIGEST_LEN)
return false;
if (algo == CHECKSUM_ALGO_NONE) {
/* No checksum requested: nothing to read; an empty digest succeeds. */
*out_len = 0;
return true;
}
return false;
uint8_t buffer[64 * 1024];
bool ok = false;
lseek(fd, 0, SEEK_SET);
if (algo == CHECKSUM_ALGO_MD5 || algo == CHECKSUM_ALGO_SHA1) {
const EVP_MD* md = algo == CHECKSUM_ALGO_MD5 ? EVP_md5() : EVP_sha1();
EVP_MD_CTX* ctx = EVP_MD_CTX_new();
if (!ctx)
return false;
unsigned int digest_len = 0;
if (EVP_DigestInit_ex(ctx, md, NULL) == 1) {
ok = true;
ssize_t got;
while ((got = read(fd, buffer, sizeof(buffer))) > 0) {
if (EVP_DigestUpdate(ctx, buffer, (size_t)got) != 1) {
ok = false;
break;
}
}
if (got < 0)
ok = false;
if (ok && EVP_DigestFinal_ex(ctx, out, &digest_len) == 1 && digest_len <= out_capacity)
*out_len = digest_len;
else
ok = false;
}
EVP_MD_CTX_free(ctx);
return ok;
}
if (algo == CHECKSUM_ALGO_MD4) {
Md4Ctx ctx;
md4_init(&ctx);
ok = true;
ssize_t got;
while ((got = read(fd, buffer, sizeof(buffer))) > 0)
md4_update(&ctx, buffer, (size_t)got);
if (got < 0)
ok = false;
if (ok) {
md4_final(&ctx, out);
*out_len = 16;
}
return ok;
}
XXH64_state_t xxh64;
XXH3_state_t* xxh3 = NULL;
if (algo == CHECKSUM_ALGO_XXH64) {
XXH64_reset(&xxh64, seed);
} else if (algo == CHECKSUM_ALGO_XXH3 || algo == CHECKSUM_ALGO_XXH128) {
xxh3 = XXH3_createState();
if (!xxh3)
return false;
if (algo == CHECKSUM_ALGO_XXH3)
XXH3_64bits_reset_withSeed(xxh3, seed);
else
XXH3_128bits_reset_withSeed(xxh3, seed);
} else {
return false;
}
ok = true;
ssize_t got;
while ((got = read(fd, buffer, sizeof(buffer))) > 0) {
if (algo == CHECKSUM_ALGO_XXH64)
XXH64_update(&xxh64, buffer, (size_t)got);
else if (XXH3_64bits_update(xxh3, buffer, (size_t)got) == XXH_ERROR) {
ok = false;
break;
}
}
if (got < 0)
ok = false;
if (ok) {
if (algo == CHECKSUM_ALGO_XXH64) {
uint64_t digest = XXH64_digest(&xxh64);
memcpy(out, &digest, sizeof(digest));
*out_len = sizeof(digest);
} else if (algo == CHECKSUM_ALGO_XXH3) {
uint64_t digest = XXH3_64bits_digest(xxh3);
memcpy(out, &digest, sizeof(digest));
*out_len = sizeof(digest);
} else {
XXH128_hash_t digest = XXH3_128bits_digest(xxh3);
memcpy(out, &digest, sizeof(digest));
*out_len = sizeof(digest);
}
}
if (xxh3)
XXH3_freeState(xxh3);
return ok;
}
int checksum_algo_from_name(const char* name) {
@@ -65,6 +346,12 @@ int checksum_algo_from_name(const char* name) {
return (int)CHECKSUM_ALGO_XXH128;
if (strcasecmp(name, "md5") == 0)
return (int)CHECKSUM_ALGO_MD5;
if (strcasecmp(name, "md4") == 0)
return (int)CHECKSUM_ALGO_MD4;
if (strcasecmp(name, "sha1") == 0)
return (int)CHECKSUM_ALGO_SHA1;
if (strcasecmp(name, "none") == 0)
return (int)CHECKSUM_ALGO_NONE;
return -1;
}
@@ -78,13 +365,21 @@ const char* checksum_algo_name(ChecksumAlgo algo) {
return "xxh128";
case CHECKSUM_ALGO_MD5:
return "md5";
case CHECKSUM_ALGO_MD4:
return "md4";
case CHECKSUM_ALGO_SHA1:
return "sha1";
case CHECKSUM_ALGO_NONE:
return "none";
}
return "<unknown>";
}
bool checksum_algo_valid(int algo) {
return algo == (int)CHECKSUM_ALGO_XXH64 || algo == (int)CHECKSUM_ALGO_MD5 ||
algo == (int)CHECKSUM_ALGO_XXH3 || algo == (int)CHECKSUM_ALGO_XXH128;
algo == (int)CHECKSUM_ALGO_XXH3 || algo == (int)CHECKSUM_ALGO_XXH128 ||
algo == (int)CHECKSUM_ALGO_MD4 || algo == (int)CHECKSUM_ALGO_SHA1 ||
algo == (int)CHECKSUM_ALGO_NONE;
}
uint8_t checksum_digest_len(ChecksumAlgo algo) {
@@ -94,7 +389,39 @@ uint8_t checksum_digest_len(ChecksumAlgo algo) {
return 8;
case CHECKSUM_ALGO_XXH128:
case CHECKSUM_ALGO_MD5:
case CHECKSUM_ALGO_MD4:
return 16;
case CHECKSUM_ALGO_SHA1:
return 20;
case CHECKSUM_ALGO_NONE:
return 0;
}
return 0;
}
}
static ChecksumAlgo compiled_checksum_preference_first(void) {
/* rsync 3.4.1 default preference order; every entry is compiled in, so this
* resolves to xxh128. */
static const ChecksumAlgo preference[] = {
CHECKSUM_ALGO_XXH128, CHECKSUM_ALGO_XXH3, CHECKSUM_ALGO_XXH64, CHECKSUM_ALGO_MD5,
CHECKSUM_ALGO_MD4, CHECKSUM_ALGO_SHA1, CHECKSUM_ALGO_NONE,
};
for (size_t i = 0; i < sizeof(preference) / sizeof(preference[0]); i++) {
if (checksum_algo_valid((int)preference[i]))
return preference[i];
}
return CHECKSUM_ALGO_XXH64;
}
int checksum_choice_resolve(void) {
bool specified = false;
int env = env_choice_first("RSYNC_CHECKSUM_LIST", checksum_algo_from_name, &specified);
if (specified)
return env; /* -1 = the list named no supported checksum */
return (int)compiled_checksum_preference_first();
}
ChecksumAlgo checksum_negotiate_default(void) {
int resolved = checksum_choice_resolve();
return resolved >= 0 ? (ChecksumAlgo)resolved : compiled_checksum_preference_first();
}
+50 -15
View File
@@ -8,25 +8,35 @@
/* Whole-file content-digest algorithms selectable with --checksum-choice and
* seeded with --checksum-seed. The ids are the values actually placed on the
* wire (config frame), so they must be kept stable and validated on receive.
* CHECKSUM_ALGO_XXH64 == 0 is the default and is byte-for-byte what FastSync
* computed before these options existed (xxHash64 with seed 0). The set mirrors
* the algorithms rsync 3.4.1 can be built with; the ones FastSync does not
* implement (md4, sha1, none) are rejected by name at parse time. */
* CHECKSUM_ALGO_XXH64 == 0 is the historical FastSync default and its numeric
* value is preserved. The full set mirrors the algorithms rsync 3.4.1 can be
* built with; every one of them is implemented here. */
typedef enum {
CHECKSUM_ALGO_XXH64 = 0,
CHECKSUM_ALGO_MD5 = 1,
CHECKSUM_ALGO_XXH3 = 2,
CHECKSUM_ALGO_XXH128 = 3
CHECKSUM_ALGO_XXH128 = 3,
CHECKSUM_ALGO_MD4 = 4,
CHECKSUM_ALGO_SHA1 = 5,
CHECKSUM_ALGO_NONE = 6
} ChecksumAlgo;
/* xxh128 digest is 16 bytes, the longest supported. */
#define CHECKSUM_MAX_DIGEST_LEN 16
/* FastSync's negotiated default (rsync 3.4.1 auto-negotiates xxh128 first).
* The wire default for Config->checksum_algo is this value. */
#define CHECKSUM_ALGO_DEFAULT CHECKSUM_ALGO_XXH128
/* sha1 digest is 20 bytes, the longest supported. */
#define CHECKSUM_MAX_DIGEST_LEN 20
/* Compute the whole-file digest of the first `size` bytes of `data`.
*
* - CHECKSUM_ALGO_XXH64: xxHash64(data, size, seed) (full 64-bit seed).
* - CHECKSUM_ALGO_MD5: md5(data, size) via OpenSSL EVP.
* md5 has no seed, so `seed` is ignored (documented).
* - CHECKSUM_ALGO_XXH3: XXH3_64bits_withSeed(data, size, seed).
* - CHECKSUM_ALGO_XXH128: XXH3_128bits_withSeed(data, size, seed).
* - CHECKSUM_ALGO_MD5: md5(data, size) via OpenSSL EVP (seed ignored).
* - CHECKSUM_ALGO_MD4: md4(data, size), self-contained RFC 1320 (seed ignored).
* - CHECKSUM_ALGO_SHA1: sha1(data, size) via OpenSSL EVP (seed ignored).
* - CHECKSUM_ALGO_NONE: no digest; *out_len is 0 and nothing is written.
* - `size == 0` hashes the empty input (plus its seed), not a NULL input.
*
* Writes up to `out_capacity` bytes into `out`, storing the digest length in
@@ -35,11 +45,23 @@ typedef enum {
bool checksum_digest(ChecksumAlgo algo, uint64_t seed, const void* data, size_t size, uint8_t* out,
size_t out_capacity, size_t* out_len);
/* Streaming whole-file digest: hash the contents of `path` without holding the
* whole file in memory. Same digest/capacity contract as checksum_digest.
* Returns false on open/read failure or an undersized buffer. */
bool checksum_digest_file(ChecksumAlgo algo, uint64_t seed, const char* path, uint8_t* out,
size_t out_capacity, size_t* out_len);
/* Descriptor form of the streaming digest: rewinds `fd` to the start and hashes
* to EOF without closing it. Used by the --verify-basis path to hash an
* already-open, root-confined basis descriptor. Same contract as
* checksum_digest_file. */
bool checksum_digest_fd(ChecksumAlgo algo, uint64_t seed, int fd, uint8_t* out, size_t out_capacity,
size_t* out_len);
/* Resolve a --checksum-choice string (case-insensitive) to an algorithm id.
* Accepts "xxh64"/"xxhash", "xxh3", "xxh128" and "md5". "auto", rsync's
* default automatic choice, is resolved to the default by the caller (it is not
* a distinct algorithm here). Returns -1 for any name FastSync does not
* implement (md4/sha1/none included). */
* Accepts "xxh64"/"xxhash", "xxh3", "xxh128", "md5", "md4", "sha1", "none".
* "auto" is not an algorithm here; the caller resolves it to the negotiated
* default. Returns -1 for any unrecognized name. */
int checksum_algo_from_name(const char* name);
/* Canonical name of an algorithm (used in CLI error messages). */
@@ -48,7 +70,20 @@ const char* checksum_algo_name(ChecksumAlgo algo);
/* True when `algo` is a supported id (used by config receive validation). */
bool checksum_algo_valid(int algo);
/* Digest length in bytes for an algorithm (xxh64/xxh3 = 8, md5/xxh128 = 16). */
/* Digest length in bytes for an algorithm (xxh64/xxh3 = 8,
* md5/md4/xxh128 = 16, sha1 = 20, none = 0). */
uint8_t checksum_digest_len(ChecksumAlgo algo);
#endif /* CHECKSUM_H */
/* Pick the first algorithm from FastSync's compiled-in preference list that is
* supported on this build (rsync 3.4.1's `--version` order:
* xxh128 xxh3 xxh64 md5 md4 sha1 none). Used to resolve "auto". */
ChecksumAlgo checksum_negotiate_default(void);
/* Resolve "auto" the way rsync does: the first supported name in
* RSYNC_CHECKSUM_LIST (whitespace-separated, client half ends at '&'), then the
* compiled-in preference order when the variable is unset/blank. Returns -1
* when the variable is set but names no supported checksum (rsync's failed
* negotiation), otherwise a valid ChecksumAlgo id. */
int checksum_choice_resolve(void);
#endif /* CHECKSUM_H */
+346 -24
View File
@@ -2,13 +2,17 @@
#include "data.h"
#include "log.h"
#include "protocol.h"
#include "utils.h"
#include <limits.h>
#include <lz4.h>
#include <stdatomic.h>
#include <stdint.h>
#include <stdlib.h>
#include <string.h>
#include <strings.h>
#include <threads.h>
#include <unistd.h>
#include <zlib.h>
#include <zstd.h>
#define INITIAL_DECOMPRESS_BUF_SIZE (1024 * 1024)
@@ -29,6 +33,13 @@
"z " \
"zip zst"
/* Self-describing compressed frames: the first byte is the CompressionAlgo id.
* zlib/lz4 store the uncompressed size as a little-endian uint32 after the
* codec byte so decompression can be exactly pre-sized and bounded. */
#define LZ4_SIZE_PREFIX_LEN 4
static _Atomic int g_compression_algo = COMPRESSION_ALGO_ZSTD;
/* Case-insensitive match of a bare suffix (no leading dot) against a
* space-separated suffix list. */
static bool suffix_in_list(const char* name, const char* list) {
@@ -67,6 +78,122 @@ bool compression_should_skip_with_suffixes(const char* path, char* const* suffix
return false;
}
int compression_algo_from_name(const char* name) {
if (!name)
return -1;
if (strcasecmp(name, "zstd") == 0)
return (int)COMPRESSION_ALGO_ZSTD;
if (strcasecmp(name, "lz4") == 0)
return (int)COMPRESSION_ALGO_LZ4;
if (strcasecmp(name, "zlib") == 0)
return (int)COMPRESSION_ALGO_ZLIB;
if (strcasecmp(name, "zlibx") == 0)
return (int)COMPRESSION_ALGO_ZLIBX;
if (strcasecmp(name, "none") == 0)
return (int)COMPRESSION_ALGO_NONE;
return -1;
}
const char* compression_algo_name(CompressionAlgo algo) {
switch (algo) {
case COMPRESSION_ALGO_NONE:
return "none";
case COMPRESSION_ALGO_ZSTD:
return "zstd";
case COMPRESSION_ALGO_LZ4:
return "lz4";
case COMPRESSION_ALGO_ZLIB:
return "zlib";
case COMPRESSION_ALGO_ZLIBX:
return "zlibx";
}
return "<unknown>";
}
bool compression_algo_valid(int algo) {
return algo == (int)COMPRESSION_ALGO_NONE || algo == (int)COMPRESSION_ALGO_ZSTD ||
algo == (int)COMPRESSION_ALGO_LZ4 || algo == (int)COMPRESSION_ALGO_ZLIB ||
algo == (int)COMPRESSION_ALGO_ZLIBX;
}
bool compression_algo_enabled(CompressionAlgo algo) {
return algo != COMPRESSION_ALGO_NONE;
}
static CompressionAlgo compiled_preference_first(void) {
/* rsync 3.4.1 default preference order; every entry is compiled in, so this
* resolves to zstd. */
static const CompressionAlgo preference[] = {
COMPRESSION_ALGO_ZSTD, COMPRESSION_ALGO_LZ4, COMPRESSION_ALGO_ZLIBX,
COMPRESSION_ALGO_ZLIB, COMPRESSION_ALGO_NONE,
};
for (size_t i = 0; i < sizeof(preference) / sizeof(preference[0]); i++) {
if (compression_algo_valid((int)preference[i]))
return preference[i];
}
return COMPRESSION_ALGO_ZSTD;
}
int compression_choice_resolve(void) {
bool specified = false;
int env = env_choice_first("RSYNC_COMPRESS_LIST", compression_algo_from_name, &specified);
if (specified)
return env; /* -1 = the list named no supported codec */
return (int)compiled_preference_first();
}
CompressionAlgo compression_negotiate_default(void) {
int resolved = compression_choice_resolve();
return resolved >= 0 ? (CompressionAlgo)resolved : compiled_preference_first();
}
int compression_default_level(CompressionAlgo algo) {
switch (algo) {
case COMPRESSION_ALGO_ZSTD:
return ZSTD_CLEVEL_DEFAULT;
case COMPRESSION_ALGO_ZLIB:
case COMPRESSION_ALGO_ZLIBX:
return 6; /* rsync resolves zlib's Z_DEFAULT_COMPRESSION (-1) to 6 */
case COMPRESSION_ALGO_LZ4:
return 1; /* rsync lz4 level is 0/ignored; positive keeps the gate on */
case COMPRESSION_ALGO_NONE:
return 0;
}
return 0;
}
int compression_clamp_level(CompressionAlgo algo, int level) {
switch (algo) {
case COMPRESSION_ALGO_ZSTD:
if (level < 1)
return 1;
if (level > 22)
return 22;
return level;
case COMPRESSION_ALGO_ZLIB:
case COMPRESSION_ALGO_ZLIBX:
if (level < 1)
return 1;
if (level > 9)
return 9;
return level;
case COMPRESSION_ALGO_LZ4:
return 1; /* ignored by lz4_compress; keeps the "compress" gate on */
case COMPRESSION_ALGO_NONE:
return 0;
}
return level;
}
void compression_set_algo(CompressionAlgo algo) {
if (compression_algo_valid((int)algo))
atomic_store(&g_compression_algo, (int)algo);
}
CompressionAlgo compression_get_algo(void) {
return (CompressionAlgo)atomic_load(&g_compression_algo);
}
/* Per-thread cache of zstd contexts plus the grow-only compression scratch
* buffer. zstd contexts are stateful and not safe to share between threads,
* so each thread keeps its own (see compression_get_thread_ctx). The cache is
@@ -157,17 +284,25 @@ static void compression_ctx_put(CompressionThreadCtx* ctx) {
compression_ctx_free(ctx);
}
Data* data_compress(Data* data_to_compress, int compression_level) {
return data_compress_with_threads(data_to_compress, compression_level, 0);
/* Build a frame consisting of a copy of `src` prefixed by `codec`. */
static Data* frame_with_codec(const void* src, size_t size, CompressionAlgo codec) {
if (size > SIZE_MAX - 1)
return NULL;
Data* out = data_create_empty(size + 1);
if (!out)
return NULL;
((uint8_t*)out->data)[0] = (uint8_t)codec;
if (size > 0)
memcpy((uint8_t*)out->data + 1, src, size);
out->size = size + 1;
return out;
}
Data* data_compress_with_threads(Data* data_to_compress, int compression_level,
int compression_threads) {
if (!data_to_compress || (!data_to_compress->data && data_to_compress->size != 0) ||
compression_threads < 0 || compression_threads > COMPRESSION_MAX_THREADS)
static Data* zstd_compress(Data* in, int compression_level, int compression_threads) {
size_t dst_size = ZSTD_compressBound(in->size);
if (dst_size > SIZE_MAX - 1)
return NULL;
log_message(LOG_LEVEL_DEBUG, "Starting to compress data");
size_t dst_size = ZSTD_compressBound(data_to_compress->size);
dst_size += 1; /* codec prefix */
CompressionThreadCtx* ctx = compression_get_thread_ctx();
if (ctx == NULL) {
@@ -218,7 +353,7 @@ Data* data_compress_with_threads(Data* data_to_compress, int compression_level,
if (available_threads > 0) {
/* Streaming compression needs the source size before threaded mode can end a frame. */
size_t zret = ZSTD_CCtx_setPledgedSrcSize(ctx->cctx, data_to_compress->size);
size_t zret = ZSTD_CCtx_setPledgedSrcSize(ctx->cctx, in->size);
if (ZSTD_isError(zret)) {
log_message(LOG_LEVEL_ERROR, "Failed to set compression source size: %s",
ZSTD_getErrorName(zret));
@@ -236,8 +371,8 @@ Data* data_compress_with_threads(Data* data_to_compress, int compression_level,
ctx->out_cap = dst_size;
}
ZSTD_inBuffer input = {data_to_compress->data, data_to_compress->size, 0};
ZSTD_outBuffer output = {ctx->out_buf, dst_size, 0};
ZSTD_inBuffer input = {in->data, in->size, 0};
ZSTD_outBuffer output = {(uint8_t*)ctx->out_buf + 1, dst_size - 1, 0};
size_t ret;
do {
@@ -250,30 +385,192 @@ Data* data_compress_with_threads(Data* data_to_compress, int compression_level,
/* Hand off an exactly-sized copy; the scratch buffer stays cached so the next
* call does not reallocate a ZSTD_compressBound-sized block. */
compressed_data = data_create_empty(output.pos);
compressed_data = data_create_empty(output.pos + 1);
if (compressed_data == NULL) {
log_message(LOG_LEVEL_ERROR, "Failed to allocate compressed data");
goto cleanup;
}
((uint8_t*)compressed_data->data)[0] = (uint8_t)COMPRESSION_ALGO_ZSTD;
if (output.pos > 0)
memcpy(compressed_data->data, ctx->out_buf, output.pos);
compressed_data->size = output.pos;
memcpy((uint8_t*)compressed_data->data + 1, (uint8_t*)ctx->out_buf + 1, output.pos);
compressed_data->size = output.pos + 1;
log_debug_message(LOG_DEBUG_UTIL, "Data succesfully compressed from %zu to %zu",
data_to_compress->size, compressed_data->size);
log_debug_message(LOG_DEBUG_UTIL, "Data succesfully compressed from %zu to %zu", in->size,
compressed_data->size);
cleanup:
compression_ctx_put(ctx);
return compressed_data;
}
Data* data_decompress_limited(Data* compressed_data, size_t maximum_size) {
if (!compressed_data || (!compressed_data->data && compressed_data->size != 0) ||
maximum_size == 0)
static Data* lz4_compress(Data* in) {
int bound = LZ4_compressBound((int)in->size);
if (bound < 0 || in->size > (size_t)INT_MAX)
return NULL;
Data* out = data_create_empty((size_t)bound + 1 + LZ4_SIZE_PREFIX_LEN);
if (!out)
return NULL;
uint32_t raw_size = (uint32_t)in->size;
uint8_t* p = (uint8_t*)out->data;
p[0] = (uint8_t)COMPRESSION_ALGO_LZ4;
for (int i = 0; i < LZ4_SIZE_PREFIX_LEN; i++)
p[1 + i] = (uint8_t)((raw_size >> (8 * i)) & 0xff);
int written = 0;
if (in->size > 0) {
written = LZ4_compress_default((const char*)in->data, (char*)p + 1 + LZ4_SIZE_PREFIX_LEN,
(int)in->size, bound);
if (written <= 0) {
data_destroy(out);
return NULL;
}
}
out->size = (size_t)written + 1 + LZ4_SIZE_PREFIX_LEN;
return out;
}
static Data* zlib_compress(Data* in, CompressionAlgo algo, int compression_level) {
int level = compression_level;
if (level < 1)
level = Z_DEFAULT_COMPRESSION;
if (level > 9)
level = 9;
uLong bound = compressBound((uLong)in->size);
if (in->size > (size_t)ULONG_MAX)
return NULL;
Data* out = data_create_empty((size_t)bound + 1 + LZ4_SIZE_PREFIX_LEN);
if (!out)
return NULL;
uint32_t raw_size = (uint32_t)in->size;
uint8_t* p = (uint8_t*)out->data;
p[0] = (uint8_t)algo;
for (int i = 0; i < LZ4_SIZE_PREFIX_LEN; i++)
p[1 + i] = (uint8_t)((raw_size >> (8 * i)) & 0xff);
uLongf dest_len = bound;
int rc = compress2(p + 1 + LZ4_SIZE_PREFIX_LEN, &dest_len, (const Bytef*)in->data,
(uLong)in->size, level);
if (rc != Z_OK) {
data_destroy(out);
return NULL;
}
out->size = (size_t)dest_len + 1 + LZ4_SIZE_PREFIX_LEN;
return out;
}
Data* data_compress_codec(Data* data_to_compress, CompressionAlgo algo, int compression_level,
int compression_threads) {
if (!data_to_compress || (!data_to_compress->data && data_to_compress->size != 0) ||
compression_threads < 0 || compression_threads > COMPRESSION_MAX_THREADS)
return NULL;
if (!compression_algo_valid((int)algo))
return NULL;
log_message(LOG_LEVEL_DEBUG, "Starting to compress data");
switch (algo) {
case COMPRESSION_ALGO_NONE:
return frame_with_codec(data_to_compress->data, data_to_compress->size, COMPRESSION_ALGO_NONE);
case COMPRESSION_ALGO_ZSTD:
return zstd_compress(data_to_compress, compression_level, compression_threads);
case COMPRESSION_ALGO_LZ4:
return lz4_compress(data_to_compress);
case COMPRESSION_ALGO_ZLIB:
case COMPRESSION_ALGO_ZLIBX:
return zlib_compress(data_to_compress, algo, compression_level);
}
return NULL;
}
Data* data_compress_with_threads(Data* data_to_compress, int compression_level,
int compression_threads) {
return data_compress_codec(data_to_compress, compression_get_algo(), compression_level,
compression_threads);
}
Data* data_compress(Data* data_to_compress, int compression_level) {
return data_compress_codec(data_to_compress, compression_get_algo(), compression_level, 0);
}
static Data* decompress_none(const Data* compressed_data, size_t maximum_size) {
size_t size = compressed_data->size - 1;
if (size > maximum_size)
return NULL;
Data* out = data_create_empty(size);
if (!out)
return NULL;
if (size > 0)
memcpy(out->data, (const uint8_t*)compressed_data->data + 1, size);
out->size = size;
return out;
}
/* Read the 4-byte little-endian raw size stored after the codec byte. */
static bool read_raw_size(const Data* in, uint32_t* raw_size) {
if (in->size < 1 + LZ4_SIZE_PREFIX_LEN)
return false;
const uint8_t* p = (const uint8_t*)in->data;
uint32_t v = 0;
for (int i = 0; i < LZ4_SIZE_PREFIX_LEN; i++)
v |= (uint32_t)p[1 + i] << (8 * i);
*raw_size = v;
return true;
}
static Data* lz4_decompress(Data* compressed_data, size_t maximum_size, size_t hard_limit) {
uint32_t raw_size = 0;
if (!read_raw_size(compressed_data, &raw_size))
return NULL;
if (raw_size > hard_limit || raw_size > maximum_size)
return NULL;
size_t comp_size = compressed_data->size - 1 - LZ4_SIZE_PREFIX_LEN;
Data* out = data_create_empty(raw_size);
if (!out)
return NULL;
if (raw_size == 0) {
out->size = 0;
return out;
}
int rc = LZ4_decompress_safe((const char*)compressed_data->data + 1 + LZ4_SIZE_PREFIX_LEN,
(char*)out->data, (int)comp_size, (int)raw_size);
if (rc < 0 || (uint32_t)rc != raw_size) {
log_message(LOG_LEVEL_ERROR, "LZ4 decompression failed");
data_destroy(out);
return NULL;
}
out->size = raw_size;
return out;
}
static Data* zlib_decompress(Data* compressed_data, size_t maximum_size, size_t hard_limit) {
uint32_t raw_size = 0;
if (!read_raw_size(compressed_data, &raw_size))
return NULL;
if (raw_size > hard_limit || raw_size > maximum_size)
return NULL;
size_t comp_size = compressed_data->size - 1 - LZ4_SIZE_PREFIX_LEN;
Data* out = data_create_empty(raw_size);
if (!out)
return NULL;
if (raw_size == 0) {
out->size = 0;
return out;
}
uLongf dest_len = raw_size;
int rc =
uncompress((Bytef*)out->data, &dest_len,
(const Bytef*)compressed_data->data + 1 + LZ4_SIZE_PREFIX_LEN, (uLong)comp_size);
if (rc != Z_OK || dest_len != raw_size) {
log_message(LOG_LEVEL_ERROR, "zlib decompression failed");
data_destroy(out);
return NULL;
}
out->size = raw_size;
return out;
}
static Data* zstd_decompress(Data* compressed_data, size_t maximum_size) {
/* The zstd frame starts after the codec byte. */
const void* frame = (const uint8_t*)compressed_data->data + 1;
size_t frame_size = compressed_data->size - 1;
log_debug_message(LOG_DEBUG_UTIL, "Start to decompress data");
unsigned long long dst_size =
ZSTD_getFrameContentSize(compressed_data->data, compressed_data->size);
unsigned long long dst_size = ZSTD_getFrameContentSize(frame, frame_size);
/* ZSTD_isError() is also true for ZSTD_CONTENTSIZE_ERROR and
* ZSTD_CONTENTSIZE_UNKNOWN (both are encoded near (size_t)-1), so test the
* sentinels explicitly instead of blanket-rejecting every error-ish value:
@@ -287,9 +584,9 @@ Data* data_decompress_limited(Data* compressed_data, size_t maximum_size) {
// ZSTD_CONTENTSIZE_UNKNOWN (~2^64) can cause massive allocation;
// fall back to a conservative estimate (3x compressed size) when unknown.
if (dst_size == ZSTD_CONTENTSIZE_UNKNOWN) {
if (compressed_data->size > ULLONG_MAX / 3)
if (frame_size > ULLONG_MAX / 3)
return NULL;
dst_size = compressed_data->size * 3;
dst_size = frame_size * 3;
if (dst_size < INITIAL_DECOMPRESS_BUF_SIZE)
dst_size = INITIAL_DECOMPRESS_BUF_SIZE;
}
@@ -326,7 +623,7 @@ Data* data_decompress_limited(Data* compressed_data, size_t maximum_size) {
goto cleanup;
}
ZSTD_inBuffer input = {compressed_data->data, compressed_data->size, 0};
ZSTD_inBuffer input = {frame, frame_size, 0};
ZSTD_outBuffer output = {uncompressed_data->data, buf_size, 0};
size_t ret;
@@ -385,6 +682,31 @@ cleanup:
return uncompressed_data;
}
Data* data_decompress_limited(Data* compressed_data, size_t maximum_size) {
if (!compressed_data || (!compressed_data->data && compressed_data->size != 0) ||
maximum_size == 0)
return NULL;
if (compressed_data->size < 1)
return NULL;
unsigned long long hard_limit =
maximum_size < MAX_DECOMPRESSED_SIZE ? maximum_size : MAX_DECOMPRESSED_SIZE;
uint8_t codec = ((const uint8_t*)compressed_data->data)[0];
if (!compression_algo_valid(codec))
return NULL;
switch ((CompressionAlgo)codec) {
case COMPRESSION_ALGO_NONE:
return decompress_none(compressed_data, (size_t)hard_limit);
case COMPRESSION_ALGO_ZSTD:
return zstd_decompress(compressed_data, (size_t)hard_limit);
case COMPRESSION_ALGO_LZ4:
return lz4_decompress(compressed_data, maximum_size, (size_t)hard_limit);
case COMPRESSION_ALGO_ZLIB:
case COMPRESSION_ALGO_ZLIBX:
return zlib_decompress(compressed_data, maximum_size, (size_t)hard_limit);
}
return NULL;
}
Data* data_decompress(Data* compressed_data) {
return data_decompress_limited(compressed_data, MAX_DECOMPRESSED_SIZE);
}
+69 -1
View File
@@ -6,11 +6,79 @@
#define COMPRESSION_MAX_THREADS 64
/* Compression algorithms selectable with --compress-choice / -z. The ids are
* the values placed on the wire (Config->compression_algo), so they must be
* kept stable. NONE is "no compression"; ZSTD is the historical FastSync
* default and the negotiated "auto" choice. ZLIBX is rsync's zlib-without-
* matched-data variant: FastSync compresses only the delta/token bytes (it does
* not put matched file data in the compression stream), so its zlib codec is
* already the "x" form and zlib/zlibx share the same implementation, recorded
* under distinct ids. */
typedef enum {
COMPRESSION_ALGO_NONE = 0,
COMPRESSION_ALGO_ZSTD = 1,
COMPRESSION_ALGO_LZ4 = 2,
COMPRESSION_ALGO_ZLIB = 3,
COMPRESSION_ALGO_ZLIBX = 4
} CompressionAlgo;
/* Resolve a --compress-choice string (case-insensitive) to an algorithm id.
* Accepts "zstd", "lz4", "zlib", "zlibx", "none". "auto" is not an algorithm
* here; the caller resolves it to the negotiated default. Returns -1 for any
* unrecognized name. */
int compression_algo_from_name(const char* name);
const char* compression_algo_name(CompressionAlgo algo);
bool compression_algo_valid(int algo);
/* Pick the first algorithm from FastSync's compiled-in preference list
* (rsync 3.4.1's `--version` order: zstd lz4 zlibx zlib none). Resolves
* "auto". */
CompressionAlgo compression_negotiate_default(void);
/* Resolve "auto" the way rsync does: the first supported name in
* RSYNC_COMPRESS_LIST (whitespace-separated, client half ends at '&'), then the
* compiled-in preference order when the variable is unset/blank. Returns -1
* when the variable is set but names no supported codec (rsync's failed
* negotiation), otherwise a valid CompressionAlgo id. */
int compression_choice_resolve(void);
/* rsync 3.4.1's per-codec default level, applied when the user did not pass
* --compress-level/--zl. zstd uses ZSTD_CLEVEL_DEFAULT (3) and zlib/zlibx the
* resolved Z_DEFAULT_COMPRESSION (6). lz4 has no tunable level in rsync
* (always the default acceleration); FastSync returns a positive placeholder so
* its "level > 0" compression gate stays engaged, and lz4_compress ignores the
* value, so the output is identical to rsync's. none is 0. */
int compression_default_level(CompressionAlgo algo);
/* Clamp an explicit --compress-level to the codec's accepted range the way
* rsync's init_compression_level() does: zstd 1..22, zlib/zlibx 1..9, lz4
* ignored (fixed positive placeholder), none 0. */
int compression_clamp_level(CompressionAlgo algo, int level);
/* True when the algorithm actually compresses (i.e. is not NONE). */
bool compression_algo_enabled(CompressionAlgo algo);
/* Select the process-wide codec used by the legacy wrappers below. Each
* process serves exactly one transfer config (the server forks per connection,
* the client configures itself before spawning transfer threads), so a
* process-global default is sufficient and constant for the lifetime of a
* transfer. Defaults to ZSTD when never set. Thread-safe. */
void compression_set_algo(CompressionAlgo algo);
CompressionAlgo compression_get_algo(void);
/* Codec-aware primitives. The compressed buffer is self-describing: its first
* byte is the CompressionAlgo id, so decompression never needs the codec passed
* separately (this keeps every existing Decompress call site source-compatible).
* `data_compress_codec` returns NULL on invalid input or an unsupported codec. */
Data* data_compress_codec(Data* data_to_compress, CompressionAlgo algo, int compression_level,
int compression_threads);
Data* data_decompress_limited(Data* compressed_data, size_t maximum_size);
/* Legacy zstd-default wrappers retained for existing callers/tests. */
Data* data_compress(Data* data_to_compress, int compression_level);
Data* data_compress_with_threads(Data* data_to_compress, int compression_level,
int compression_threads);
Data* data_decompress(Data* compressed_data);
Data* data_decompress_limited(Data* compressed_data, size_t maximum_size);
bool compression_should_skip_with_suffixes(const char* path, char* const* suffixes, int count);
/* Release the calling thread's cached zstd contexts (compressor, decompressor
+233 -40
View File
@@ -14,6 +14,7 @@
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <strings.h>
#include <limits.h>
#include <errno.h>
@@ -67,12 +68,17 @@ static void config_set_defaults(Config* config) {
config->human_readable = false;
config->ignore_errors = false;
config->ignore_missing_args = false;
config->checksum_transfer_algo = CHECKSUM_ALGO_DEFAULT;
config->cli_exit_code = 0;
config->compression_level_set = false;
config->checksum_choice_set = false;
config->filters = NULL;
config->files_from = NULL;
config->files_from_set = NULL;
config->from0 = false;
config->cvs_exclude = false;
config->per_dir_filter = false;
config->per_dir_filter_count = 0;
config->one_file_system = false;
config->no_implied_dirs = false;
config->dirs = false;
@@ -196,12 +202,14 @@ static bool validate_received_config(const Config* config) {
valid_wire_bool(config->partial) && valid_wire_bool(config->delete_before) &&
valid_wire_bool(config->checksum) && valid_wire_bool(config->eight_bit_output) &&
valid_wire_bool(config->dry_run) && checksum_algo_valid(config->checksum_algo) &&
identity_wire_valid(config) && valid_wire_bool(config->preserve_atimes) &&
valid_wire_bool(config->preserve_crtimes) && valid_wire_bool(config->omit_dir_times) &&
valid_wire_bool(config->omit_link_times) && valid_wire_bool(config->preserve_perms) &&
valid_wire_bool(config->preserve_times) && valid_wire_bool(config->preserve_owner) &&
valid_wire_bool(config->preserve_group) && valid_wire_bool(config->munge_links) &&
valid_wire_bool(config->keep_dirlinks) && valid_wire_bool(config->fake_super) &&
compression_algo_valid(config->compression_algo) && identity_wire_valid(config) &&
valid_wire_bool(config->preserve_atimes) && valid_wire_bool(config->preserve_crtimes) &&
valid_wire_bool(config->omit_dir_times) && valid_wire_bool(config->omit_link_times) &&
valid_wire_bool(config->preserve_perms) && valid_wire_bool(config->preserve_times) &&
valid_wire_bool(config->preserve_owner) && valid_wire_bool(config->preserve_group) &&
valid_wire_bool(config->munge_links) && valid_wire_bool(config->keep_dirlinks) &&
valid_wire_bool(config->fake_super) && valid_wire_bool(config->report_dest_info) &&
valid_wire_bool(config->report_stats) && valid_wire_bool(config->report_deletes) &&
(!config->copy_as_set || (config->copy_as_uid >= 0 && config->copy_as_gid >= 0)) &&
(!config->use_compression ||
(config->compression_level >= 1 && config->compression_level <= 22)) &&
@@ -228,7 +236,13 @@ Config* config_create(void) {
bool config_delete_timing_early(const Config* config) {
if (!config)
return false;
return config->delete_before || config->delete_during;
return config->delete_before;
}
bool config_delete_timing_per_dir(const Config* config) {
if (!config)
return false;
return config->delete_during || config->delete_delay;
}
/* A delete-timing flag is only meaningful together with --delete. At most one
@@ -329,30 +343,38 @@ bool config_has_basis(const Config* config) {
}
/* A basis-dir path travels from the client to the receiver and is resolved
* below the destination root, so it must be a non-empty relative path with no
* "." or ".." component and no traversal: an absolute or escaping path would
* make the receiver read or link files outside its authorized root.
* below the destination root when relative, or used verbatim when absolute
* (matching rsync). Either form must be non-empty, traversal-free (no "..")
* and free of "." components: an escaping path would make the receiver read or
* link files outside its authorized root. An absolute path is still subject to
* the receiver's root confinement at open time (file_open_secure_parent), so a
* basis outside the authorized root is simply not found rather than an escape.
*
* Returns a malloc'd CANONICAL copy of an accepted path, or NULL when the path
* is rejected. Canonicalization collapses interior empty components ("a//b" ->
* "a/b"), drops "." components and trailing "/"s, so validation, the delete
* walker prefix match and the receiver's basis lookup all agree on one form.
* The normalizer is the single source of truth for both config_basis_path_valid
* and config_basis_append. */
* "a/b"), drops "." components and trailing "/"s, and preserves a leading '/'
* for absolute paths, so validation, the delete walker prefix match and the
* receiver's basis lookup all agree on one form. The normalizer is the single
* source of truth for both config_basis_path_valid and config_basis_append. */
static char* basis_path_normalize(const char* path) {
if (!path || path[0] == '\0' || path[0] == '/' || has_path_traversal(path))
if (!path || path[0] == '\0' || has_path_traversal(path))
return NULL;
if (strcmp(path, ".") == 0)
bool absolute = path[0] == '/';
if (!absolute && strcmp(path, ".") == 0)
return NULL;
if (absolute && strcmp(path, "/") == 0)
return NULL;
char* dup = str_dup(path);
if (!dup)
return NULL;
size_t out_len = 0;
char* out = malloc(strlen(path) + 1);
char* out = malloc(strlen(path) + 2);
if (!out) {
free(dup);
return NULL;
}
if (absolute)
out[out_len++] = '/';
char* saveptr = NULL;
bool ok = true;
for (char* part = strtok_r(dup, "/", &saveptr); part; part = strtok_r(NULL, "/", &saveptr)) {
@@ -362,14 +384,14 @@ static char* basis_path_normalize(const char* path) {
}
if (strcmp(part, ".") == 0)
continue;
if (out_len > 0)
if (out_len > 0 && out[out_len - 1] != '/')
out[out_len++] = '/';
size_t len = strlen(part);
memcpy(out + out_len, part, len);
out_len += len;
}
free(dup);
if (!ok || out_len == 0) {
if (!ok || out_len == 0 || (absolute && out_len == 1)) {
free(out);
return NULL;
}
@@ -769,6 +791,8 @@ void config_delete(Config* config) {
if (config->filters) {
array_list_delete(config->filters);
}
filter_rule_list_free(config->protect_rules);
config->protect_rules = NULL;
/* A --delay-updates staging tree is transient receiver state: remove any
leftovers on every exit path (success already emptied it). */
if (config->delay_context)
@@ -880,6 +904,14 @@ static bool config_receive_checksum_algo(int fd, int* value) {
return true;
}
static bool config_receive_compression_algo(int fd, int* value) {
int algo;
if (!receive_int(fd, &algo) || !compression_algo_valid(algo))
return false;
*value = algo;
return true;
}
static bool config_receive_super_mode(int fd, SuperMode* value) {
int mode;
if (!receive_int(fd, &mode) || mode < SUPER_MODE_AUTO || mode > SUPER_MODE_OFF)
@@ -996,6 +1028,124 @@ static bool receive_basis_entries(int fd, Config* c, ConfigStringBudget* budget)
return true;
}
/* Receiver-side delete-protection rules (protocol 2.28.0). The sender compiles
* its command-line selection rules exactly as the scanner does and streams the
* result as one bounded, self-describing block (count + per-rule records); the
* receiver reconstructs a FilterRuleList for the --delete extras walk. owner
* and pattern are charged through the shared ConfigStringBudget and the block
* additionally enforces MAX_FILTER_RULES / MAX_FILTER_BYTES. */
static bool send_protect_entries(int fd, const Config* c) {
int count = c->filters ? c->filters->size : 0;
const char** texts = NULL;
if (count > 0) {
texts = malloc((size_t)count * sizeof(char*));
if (!texts)
return false;
for (int i = 0; i < count; i++)
texts[i] = (const char*)c->filters->items[i];
}
char err[160];
FilterRuleList* rules =
filter_base_build(texts, count, c->cvs_exclude, c->delete_excluded, err, sizeof(err));
free(texts);
if (!rules) {
log_message(LOG_LEVEL_ERROR, "invalid filter rule: %s", err);
return false;
}
bool ok = send_int(fd, rules->count);
for (int i = 0; ok && i < rules->count; i++) {
const FilterRule* r = rules->items[i];
/* Mirror the receiver's limit so the peer never receives a rule it will
reject as a protocol error. */
if (r->pattern && strlen(r->pattern) > MAX_PROTECT_PATTERN_LEN) {
log_message(LOG_LEVEL_ERROR, "filter pattern exceeds %d bytes", MAX_PROTECT_PATTERN_LEN);
filter_rule_list_free(rules);
return false;
}
ok = send_int(fd, (int)r->action) && send_int(fd, (int)r->sides) &&
send_int(fd, r->anchored ? 1 : 0) && send_int(fd, r->dir_only ? 1 : 0) &&
send_int(fd, r->negate ? 1 : 0) && send_str(fd, r->owner ? r->owner : "") &&
send_str(fd, r->pattern ? r->pattern : "");
}
filter_rule_list_free(rules);
return ok;
}
static bool receive_protect_entries(int fd, Config* c, ConfigStringBudget* budget) {
int count;
if (!receive_int(fd, &count))
return false;
if (count < 0 || count > MAX_FILTER_RULES)
return false;
if (count == 0)
return true;
FilterRuleList* list = filter_rule_list_create();
if (!list)
return false;
size_t pattern_bytes = 0;
for (int i = 0; i < count; i++) {
int action;
int sides;
bool anchored;
bool dir_only;
bool negate;
if (!receive_int(fd, &action) ||
(action != FILTER_ACTION_EXCLUDE && action != FILTER_ACTION_INCLUDE) ||
!receive_int(fd, &sides) || sides < (int)FILTER_SIDE_SENDER ||
sides > (int)(FILTER_SIDE_SENDER | FILTER_SIDE_RECEIVER) ||
!receive_wire_bool(fd, &anchored) || !receive_wire_bool(fd, &dir_only) ||
!receive_wire_bool(fd, &negate))
goto fail;
char* owner = config_receive_str(fd, budget);
if (!owner)
goto fail;
char* pattern = config_receive_str(fd, budget);
if (!pattern || pattern[0] == '\0') {
free(owner);
free(pattern);
goto fail;
}
/* A pattern too long to be evaluated by glob_match against a PATH_MAX path
would silently fail to match and leave a protect rule inert (fail-open:
the entry is then deleted). Reject it up front as a protocol error
rather than accept a rule that can never shield anything. */
if (strlen(pattern) > MAX_PROTECT_PATTERN_LEN) {
free(owner);
free(pattern);
goto fail;
}
size_t bytes = strlen(owner) + strlen(pattern);
if (bytes > MAX_FILTER_BYTES - pattern_bytes) {
free(owner);
free(pattern);
goto fail;
}
pattern_bytes += bytes;
FilterRule* rule = calloc(1, sizeof(FilterRule));
if (!rule) {
free(owner);
free(pattern);
goto fail;
}
rule->action = (FilterAction)action;
rule->sides = (unsigned)sides;
rule->anchored = anchored;
rule->dir_only = dir_only;
rule->negate = negate;
rule->owner = owner;
rule->pattern = pattern;
if (!filter_rule_list_add(list, rule)) {
filter_rule_free(rule);
goto fail;
}
}
c->protect_rules = list;
return true;
fail:
filter_rule_list_free(list);
return false;
}
static bool send_identity_entries(int fd, const IdentityMap* map, int count) {
for (int i = 0; i < count; i++) {
if (!send_int(fd, map[i].from) || !send_int(fd, map[i].from_hi) || !send_int(fd, map[i].to) ||
@@ -1081,6 +1231,9 @@ fail:
#define CONFIG_SEND_INT_CHECKSUM_ALGO(name) send_int(fd, c->name)
#define CONFIG_RECV_INT_CHECKSUM_ALGO(name) config_receive_checksum_algo(fd, &c->name)
#define CONFIG_SEND_INT_COMPRESSION_ALGO(name) send_int(fd, c->name)
#define CONFIG_RECV_INT_COMPRESSION_ALGO(name) config_receive_compression_algo(fd, &c->name)
#define CONFIG_SEND_SUPERMODE(name) send_int(fd, (int)c->name)
#define CONFIG_RECV_SUPERMODE(name) config_receive_super_mode(fd, &c->name)
@@ -1120,6 +1273,9 @@ fail:
#define CONFIG_RECV_BLOCK_IDMAP(name) \
receive_identity_entries(fd, budget, c->name##_count, &c->name)
#define CONFIG_SEND_BLOCK_PROTECT_RULES(name) send_protect_entries(fd, c)
#define CONFIG_RECV_BLOCK_PROTECT_RULES(name) receive_protect_entries(fd, c, budget)
/* One table entry, applied in sequence. XSEND/XRECV are statement macros so
* consecutive entries read as a plain sequence of assignments. */
#define XSEND(name, ctype, def, kind) ok = ok && (CONFIG_SEND_##kind(name));
@@ -1156,6 +1312,8 @@ CONFIG_DEFINE_SEND(send_iconv_spec, CONFIG_WIRE_ICONV_FIELDS)
CONFIG_DEFINE_SEND(send_privilege_options, CONFIG_WIRE_PRIVILEGE_FIELDS)
CONFIG_DEFINE_SEND(send_copy_as_options, CONFIG_WIRE_COPY_AS_FIELDS)
CONFIG_DEFINE_SEND(send_output_options, CONFIG_WIRE_OUTPUT_FIELDS)
CONFIG_DEFINE_SEND(send_codec_options, CONFIG_WIRE_CODEC_FIELDS)
CONFIG_DEFINE_SEND(send_protect_options, CONFIG_WIRE_PROTECT_FIELDS)
CONFIG_DEFINE_RECV(receive_core_fields, CONFIG_WIRE_CORE_FIELDS)
CONFIG_DEFINE_RECV(receive_delta_fields, CONFIG_WIRE_DELTA_FIELDS)
@@ -1175,6 +1333,8 @@ CONFIG_DEFINE_RECV(receive_iconv_spec, CONFIG_WIRE_ICONV_FIELDS)
CONFIG_DEFINE_RECV(receive_privilege_options, CONFIG_WIRE_PRIVILEGE_FIELDS)
CONFIG_DEFINE_RECV(receive_copy_as_options, CONFIG_WIRE_COPY_AS_FIELDS)
CONFIG_DEFINE_RECV(receive_output_options, CONFIG_WIRE_OUTPUT_FIELDS)
CONFIG_DEFINE_RECV(receive_codec_options, CONFIG_WIRE_CODEC_FIELDS)
CONFIG_DEFINE_RECV(receive_protect_options, CONFIG_WIRE_PROTECT_FIELDS)
#undef XSEND
#undef XRECV
@@ -1292,7 +1452,9 @@ bool config_send_wire_block(int file_descriptor, const Config* config) {
send_iconv_spec(file_descriptor, config) &&
send_privilege_options(file_descriptor, config) &&
send_copy_as_options(file_descriptor, config) &&
send_output_options(file_descriptor, config);
send_output_options(file_descriptor, config) &&
send_codec_options(file_descriptor, config) &&
send_protect_options(file_descriptor, config);
}
bool config_send(int file_descriptor, const Config* config) {
@@ -1363,29 +1525,60 @@ Config* config_receive_with_validate(int file_descriptor, ConfigValidateFunc val
!receive_iconv_spec(file_descriptor, config, &budget) ||
!receive_privilege_options(file_descriptor, config, &budget) ||
!receive_copy_as_options(file_descriptor, config, &budget) ||
!receive_output_options(file_descriptor, config, &budget))
!receive_output_options(file_descriptor, config, &budget) ||
!receive_codec_options(file_descriptor, config, &budget) ||
!receive_protect_options(file_descriptor, config, &budget))
goto error;
if (config->compress_choice[0] != '\0' && strcmp(config->compress_choice, "zstd") != 0 &&
strcmp(config->compress_choice, "none") != 0 &&
strcmp(config->compress_choice, "auto") != 0) {
char* escaped_choice = output_escape(config->compress_choice, config->eight_bit_output);
log_message(LOG_LEVEL_ERROR, "Unsupported compression choice: %s",
escaped_choice ? escaped_choice : "<allocation failed>");
char detail[128];
snprintf(detail, sizeof(detail), "unsupported compression choice: %s",
escaped_choice ? escaped_choice : "<allocation failed>");
send_error_detail(file_descriptor, detail);
free(escaped_choice);
/* Validate/normalize the negotiated codec. compress_choice is the human
* spelling (NULL or "" when -z was not given); compression_algo is the
* concrete codec id the sender used. They must agree, and "auto" is
* canonicalized to FastSync's negotiated default so the stored spelling is
* always concrete (a hostile/older client may still send "auto"). */
if (config->compress_choice && config->compress_choice[0] != '\0') {
int choice_algo = compression_algo_from_name(config->compress_choice);
if (choice_algo < 0 && strcasecmp(config->compress_choice, "auto") != 0) {
char* escaped_choice = output_escape(config->compress_choice, config->eight_bit_output);
log_message(LOG_LEVEL_ERROR, "Unsupported compression choice: %s",
escaped_choice ? escaped_choice : "<allocation failed>");
char detail[160];
snprintf(detail, sizeof(detail), "unsupported compression choice: %s",
escaped_choice ? escaped_choice : "<allocation failed>");
send_error_detail(file_descriptor, detail);
free(escaped_choice);
goto error;
}
if (choice_algo < 0)
choice_algo = (int)compression_negotiate_default();
if (strcasecmp(config->compress_choice, "auto") == 0 ||
choice_algo == (int)COMPRESSION_ALGO_NONE) {
const char* canonical = compression_algo_name((CompressionAlgo)choice_algo);
char* dup = str_dup(canonical);
if (!dup)
goto error;
free(config->compress_choice);
config->compress_choice = dup;
}
if (config->compression_algo != choice_algo) {
log_message(LOG_LEVEL_ERROR, "Compression choice '%s' does not match codec id %d",
config->compress_choice, config->compression_algo);
send_error_detail(file_descriptor, "compression choice/codec mismatch");
goto error;
}
}
/* The concrete codec must exist only when compression is on. A client that
* left -z off has no codec in effect, but the field keeps whatever id it
* carried (the receiver never dispatches on it without use_compression), so
* the wire value round-trips untouched. */
if (config->use_compression && config->compression_algo == (int)COMPRESSION_ALGO_NONE) {
log_message(LOG_LEVEL_ERROR, "Compression requested with the 'none' codec");
send_error_detail(file_descriptor, "compression requested with the none codec");
goto error;
}
/* Defensive: an older/hostile client may still send "auto"; canonicalize it
to zstd (its effective choice) so the stored value is always concrete. */
if (strcmp(config->compress_choice, "auto") == 0) {
char* canonical = str_dup("zstd");
if (!canonical)
goto error;
free(config->compress_choice);
config->compress_choice = canonical;
/* rsync: "none" as the pre-transfer checksum is invalid with --checksum. */
if (config->checksum && config->checksum_algo == (int)CHECKSUM_ALGO_NONE) {
log_message(LOG_LEVEL_ERROR, "Invalid checksum-choice for --checksum: none");
send_error_detail(file_descriptor, "checksum-choice 'none' cannot be used with --checksum");
goto error;
}
if (!validate_received_config(config)) {
log_message(LOG_LEVEL_ERROR, "Invalid configuration received from client");
+191 -22
View File
@@ -3,6 +3,8 @@
#include "array_list.h"
#include "checksum.h"
#include "compression.h"
#include "filter.h"
#include <stdbool.h>
#include <stdint.h>
#include <stdio.h>
@@ -81,7 +83,7 @@ typedef struct {
typedef enum SuperMode { SUPER_MODE_AUTO = 0, SUPER_MODE_ON = 1, SUPER_MODE_OFF = 2 } SuperMode;
/* ===========================================================================
* Config wire-field table (single source of truth for protocol 2.23.0).
* Config wire-field table (single source of truth for protocol 2.28.0).
*
* Every field below crosses the wire. The table is the ONLY place a
* serialized field is named: config.h expands CONFIG_WIRE_FIELDS() to declare
@@ -196,14 +198,23 @@ typedef enum SuperMode { SUPER_MODE_AUTO = 0, SUPER_MODE_ON = 1, SUPER_MODE_OFF
X(skip_compress_count, int, 0, INT_SKIPCOUNT) \
X(skip_compress_suffixes, char**, NULL, BLOCK_SKIP_SUFFIXES)
/* FastSync-only --verify-basis (protocol 2.28.0, no version bump by project
* decision): restores the stricter content equality on a basis hit. By
* default a basis hit is accepted on rsync's metadata quick-check alone (equal
* size plus equal mtime, or size alone under --size-only); with this flag the
* receiver ALSO requires the basis bytes' whole-file digest (the negotiated
* --checksum-choice algorithm) to equal the sender's, exactly FastSync's
* historical behavior. It is a receiver policy and crosses the wire so the
* receiver knows whether to read and hash the basis content. */
#define CONFIG_WIRE_BASIS_FIELDS(X) \
X(basis_count, int, 0, INT_BASISCOUNT) \
X(basis_dirs, BasisDest*, NULL, BLOCK_BASIS)
X(basis_dirs, BasisDest*, NULL, BLOCK_BASIS) \
X(verify_basis, bool, false, BOOL)
#define CONFIG_WIRE_FUZZY_FIELDS(X) X(fuzzy, bool, false, BOOL)
#define CONFIG_WIRE_CHECKSUM_FIELDS(X) \
X(checksum_algo, int, CHECKSUM_ALGO_XXH64, INT_CHECKSUM_ALGO) \
X(checksum_algo, int, CHECKSUM_ALGO_DEFAULT, INT_CHECKSUM_ALGO) \
X(checksum_seed, uint64_t, 0, RAW)
#define CONFIG_WIRE_IDENTITY_FIELDS(X) \
@@ -250,8 +261,61 @@ typedef enum SuperMode { SUPER_MODE_AUTO = 0, SUPER_MODE_ON = 1, SUPER_MODE_OFF
* answer every per-file STATUS_CHECK with a STATUS_DEST_INFO snapshot of the
* pre-transfer destination entry (see protocol.h). It is set by the client
* only when -i/--itemize-changes or --out-format asks for per-file change
* output; the transfer decision itself is unchanged.
*
* Wire-stats wave (protocol 2.25.0). report_stats tells the receiver to send a
* STATUS_STATS frame immediately before its terminal success status carrying
* the receiver-only counters (matched data, deleted-file count) and,
* for -n/--dry-run --delete, the destination-relative paths it WOULD have
* deleted. It is set by the client only when --stats, --progress/-P, an
* --out-format token needs a wire counter (%b/%c), or a dry-run carries
* --delete; the transfer decision itself is unchanged.
*
* --info wave (protocol 2.27.0). report_deletes tells the receiver to include
* the destination-relative paths it ACTUALLY removed in its terminal
* STATUS_STATS record (the same path-list field the dry-run would-delete report
* uses), so the sender can print rsync's `deleting PATH`/`*deleting` lines for a
* real (non-dry-run) deletion. It is set when --delete is active and any of
* --info=del, -i/--itemize-changes or --out-format requests per-file change
* output; the transfer decision itself is unchanged. */
#define CONFIG_WIRE_OUTPUT_FIELDS(X) X(report_dest_info, bool, false, BOOL)
#define CONFIG_WIRE_OUTPUT_FIELDS(X) \
X(report_dest_info, bool, false, BOOL) \
X(report_stats, bool, false, BOOL) X(report_deletes, bool, false, BOOL)
/* Codec-negotiation wave (protocol 2.26.0). compression_algo is the concrete
* codec the client selected for this transfer (a CompressionAlgo id) and is the
* value the receiver validates and installs. It is the resolved result of
* --compress-choice / the "auto" negotiation so both peers agree exactly.
*
* Negotiation model: FastSync enforces a strict same-version handshake, so both
* peers carry the identical compiled-in codec set. The client resolves the
* effective algorithm deterministically and serializes it here; "auto" picks
* the first entry of the rsync 3.4.1 preference order
* (compression: zstd lz4 zlibx zlib none; checksum: xxh128 xxh3 xxh64 md5 md4
* sha1 none), and an explicit request wins. The receiver rejects (before
* STATUS_OK) any algorithm outside its own supported set, which is rsync's
* "no common choice is an error" behavior. The same resolver runs on both
* sides (compression_negotiate_default / checksum_negotiate_default), so the
* fallback is consistent.
*
* The field is appended after the output block so every pre-2.26 field keeps
* its wire position. */
#define CONFIG_WIRE_CODEC_FIELDS(X) \
X(compression_algo, int, COMPRESSION_ALGO_ZSTD, INT_COMPRESSION_ALGO)
/* Receiver-side delete-protection filter rules (protocol 2.28.0). The sender
* compiles its root-level selection rules exactly as the scanner does
* (filter_base_build over --filter/-f/--exclude/--include/-C) and streams them
* as one self-describing, bounded block (count followed by per-rule records).
* The receiver reconstructs `protect_rules` and evaluates them against
* DESTINATION-ONLY entries during the --delete extras walk, so a
* `protect`/`P` rule protects an extra that never appeared on the sender
* (rsync re-derives deletion protection from the filter list; FastSync
* historically derived it only from the source scan). `protect_rules` is NULL
* on the sender and is owned/freed by the receiver Config. Bounded by
* MAX_FILTER_RULES and MAX_FILTER_BYTES; an unknown action/sides is a protocol
* error. */
#define CONFIG_WIRE_PROTECT_FIELDS(X) X(protect_rules, FilterRuleList*, NULL, BLOCK_PROTECT_RULES)
/* All serialized fields, in exact wire order. Concatenating the per-segment
* lists here is what keeps the declaration order = the wire order. */
@@ -274,7 +338,9 @@ typedef enum SuperMode { SUPER_MODE_AUTO = 0, SUPER_MODE_ON = 1, SUPER_MODE_OFF
CONFIG_WIRE_ICONV_FIELDS(X) \
CONFIG_WIRE_PRIVILEGE_FIELDS(X) \
CONFIG_WIRE_COPY_AS_FIELDS(X) \
CONFIG_WIRE_OUTPUT_FIELDS(X)
CONFIG_WIRE_OUTPUT_FIELDS(X) \
CONFIG_WIRE_CODEC_FIELDS(X) \
CONFIG_WIRE_PROTECT_FIELDS(X)
typedef struct Config {
/* -j/--threads=N: number of parallel scanner worker threads for the -m
@@ -369,6 +435,22 @@ typedef struct Config {
* enters the keep-set. Implied by --delete-missing-args. */
bool ignore_missing_args;
/* Codec-negotiation CLI state (all client-only, never serialized). The
* effective pre-transfer checksum is Config->checksum_algo (serialized);
* checksum_transfer_algo is the rsync "transfer" half of a two-name
* --checksum-choice form (validated and used only to mirror rsync's
* whole-file forcing, since FastSync's per-block strong hash is fixed).
* cli_exit_code carries a parser-requested process exit status (rsync uses 4
* for an unsupported checksum/compress algorithm) so main() can mirror it. */
int checksum_transfer_algo;
int cli_exit_code;
/* Client-only "the user explicitly chose" bits. They let the per-codec
* default level / checksum list be applied only when the corresponding
* rsync option was omitted (an explicit --compress-level / --checksum-choice
* always wins). Never serialized. */
bool compression_level_set;
bool checksum_choice_set;
// Issue #129: Advanced file selection. These fields are CLIENT-ONLY: they are
// never serialized to the wire (the receiver must not learn them).
ArrayList* filters; /* --filter=RULE rule strings, in order */
@@ -377,9 +459,15 @@ typedef struct Config {
bool from0; /* -0/--from0: NUL-delimited *-from files */
bool cvs_exclude; /* -C/--cvs-exclude: standard CVS ignore set */
bool per_dir_filter; /* -F: apply per-directory .rsync-filter files */
/* -F click count. rsync's single -F means --filter='dir-merge
* /.rsync-filter' (the .rsync-filter files themselves are transferred); a
* repeated -F adds --filter='- .rsync-filter' so they are excluded too. */
int per_dir_filter_count;
bool one_file_system; /* -x/--one-file-system: do not cross filesystem boundaries */
/* --no-implied-dirs: client-only. With -R + --files-from, refuse to place a
* listed file whose ancestor directory is not itself explicitly listed. */
/* --no-implied-dirs: client-only. With -R, do not transfer the source
* metadata of the parent directories implied by a listed path; an unlisted
* implied parent is still created (with default attributes) so the listed
* file can be placed, matching rsync. */
bool no_implied_dirs;
/* -d/--dirs: client-only. Transfer the directory entries named by the
* source argument / --files-from list without recursing into contents. */
@@ -547,13 +635,21 @@ typedef struct Config {
/* rsync deletion-timing family (real from Phase 3). At most one of
delete_before / delete_during / delete_delay / delete_after may be set, and
only together with use_delete (the CLI implies --delete for each of them).
delete_before and delete_during select the EARLY engine mode: the keep-set
delete_before selects the EARLY engine mode: the whole-tree keep-set
manifest is transmitted before any file data and extras are removed then,
acknowledged, before the first data byte. delete_delay and delete_after
select the LATE commit mode: extras are removed only after the whole
transfer has succeeded (plain --delete keeps this mode). The exact
semantics and the divergences from rsync are documented in RSYNC_COMPAT.md
and in config_delete_timing_early() below. */
acknowledged, before the first data byte. delete_during and delete_delay
select the per-directory delete-plan mode (protocol 2.24.0): one plan per
source directory is streamed in directory order, and the receiver removes
each directory's extras when its plan arrives (during) or snapshots them
and removes them only after a successful transfer (delay). delete_after
keeps the whole-tree commit mode: extras are removed from a fresh
end-of-transfer destination scan only after the whole transfer succeeded.
A plain --delete with no explicit timing flag defaults to delete_during on
the client (cli_finalize_config), matching rsync's --del default; the old
late-commit behavior is selected explicitly by --delete-after or the
FastSync-only long spelling --delete-commit (an exact alias for
--delete-after, mapped onto the same wire field). See
config_delete_timing_early()/config_delete_timing_per_dir() below. */
/* partial_dir */
// PR #174: Partial transfer resumption
/* suffix */
@@ -911,12 +1007,78 @@ typedef struct Config {
* snapshot of the old entry) before its ordinary verdict when the config frame
* carries the new report_dest_info bool appended after the --copy-as block.
* This is both a config-frame layout change (one trailing bool) and a frame
* sequence change (the new status). */
#define PROTOCOL_VERSION "2.23.0"
* sequence change (the new status).
*
* (4) Delete timing (protocol 2.24.0): the sender streams one delete plan per
* source directory so --delete-during/--delete-delay reproduce rsync's deletion
* timing (the plan fields and STATUS_DELETE_PLAN are documented at the keep-set
* / delete-plan definitions below).
*
* (5) Wire-stats parity (protocol 2.25.0): --stats, --progress/-P and the
* --out-format %b/%c tokens need receiver-only and wire counters that the push
* sender cannot observe, and -n/--dry-run --delete must report the extras it
* would have removed without deleting anything. The config frame gains one
* trailing report_stats bool and the receiver emits a new STATUS_STATS frame
* (carrying matched data, the deleted-file count and the would-delete path
* list) immediately before its terminal success status.
*
* (6) Codec breadth + negotiation (protocol 2.26.0): the config frame gains one
* trailing int, compression_algo (a CompressionAlgo id), appended after the
* output block. It is the negotiated/effective compression codec and is what
* the receiver's self-describing decompressor validates against its own
* supported set. The checksum_algo wire value now also accepts md4/sha1/none,
* and its default changes to the rsync 3.4.1 auto-negotiated xxh128.
*
* Any config-frame layout change must bump the protocol version: a peer that
* does not parse the new trailing bytes would desynchronize on the frame
* boundary, and the strict same-version handshake (config_receive rejects a
* mismatched version before parsing anything else) keeps mixed deployments from
* ever reaching that state. */
/* (7) --info=del report (protocol 2.27.0): the config frame gains one trailing
* bool, report_deletes, appended after report_stats. When set, the receiver
* lists the paths it actually removed in the terminal STATUS_STATS path list
* (the same count-delimited list the -n/--dry-run would-delete report uses), so
* the sender can print rsync's `deleting PATH` lines for a real deletion. No
* change to the fixed STATUS_STATS record itself; only a new trailing config
* bool, which still requires the version bump for the strict lockstep. */
/* (8) --stats receiver-observed counters (protocol 2.28.0): the fixed
* STATUS_STATS record grows from three counters to eight. The receiver now
* reports the bytes it literally stored (`literal_data`) and the count of
* destination entries it newly CREATED, split by type
* (reg/dir/link/special), so the sender can print rsync's exact
* `Number of created files: N (reg: X, dir: Y, link: Z, special: W)` line and
* an exact `Literal data` total even for delta transfers. The config-frame
* LAYOUT is unchanged (no new config field), but the STATUS_STATS body grows,
* so a 2.27 peer that does not consume the five new fixed-width counters would
* desynchronize on the trailing would-delete path list; the strict
* same-version handshake (config_receive rejects a mismatched version before
* parsing anything else) keeps mixed deployments from ever reaching that
* state. */
/* (9) Receiver-side delete protection (still protocol 2.28.0): the config frame
* gains one trailing self-describing block carrying the sender's compiled base
* filter rules so the receiver can protect DESTINATION-ONLY entries from
* --delete with `protect`/`risk` rules (rsync parity). The block appends after
* compression_algo; see CONFIG_WIRE_PROTECT_FIELDS. */
#define PROTOCOL_VERSION "2.28.0"
#define DEFAULT_CHUNK_SIZE (10 * 1024 * 1024)
/* Upper bound on total basis-dir entries (rsync caps --link-dest at 20). */
#define MAX_BASIS_DIRS 64
/* Bounds on the received receiver-side delete-protection rule block. The rule
* count and the aggregate pattern+owner bytes are each capped so a hostile
* peer cannot pin unbounded pre-auth memory; both are validated strictly on
* receive (alongside the per-string ConfigStringBudget). */
/* A peer may supply protect rules; cap the list so a crafted config cannot make
* the receiver's delete walk evaluate an unbounded number of glob patterns per
* destination entry (glob_match is O(pattern x path)). 1024 is far above any
* legitimate selection. */
#define MAX_FILTER_RULES 1024
#define MAX_FILTER_BYTES (256 * 1024)
/* glob_match's DP is capped at 64 Mi work units; a pattern longer than this
* could exceed the cap against a PATH_MAX path and silently stop matching,
* leaving a protect rule inert. Reject such a rule at receive time. */
#define MAX_PROTECT_PATTERN_LEN 8192
/* Upper bound on the number of --skip-compress suffixes accepted from the wire.
* Each suffix is an independent wire string (up to MAX_STRING_SIZE = 64 KiB), so
* without this a hostile pre-auth client could otherwise retain
@@ -1006,16 +1168,23 @@ int config_parse_daemon_dest(Config* config);
* 0. */
int config_parse_transport_dest(Config* config);
/* True when the negotiated delete timing performs the extra-file deletion
* BEFORE the transfer data (--delete-before / --delete-during). The flag is
* a pure function of the config and is used identically on the sender (to pick
/* True for the whole-tree delete-before timing: a complete keep-set manifest is
* transmitted before any data and committed (with an ack) before the first data
* byte. Pure function of the config, used identically on the sender (to pick
* the manifest-first frame order) and the receiver (to delete when the early
* manifest arrives). When false the deletion is committed only after the whole
* transfer succeeded (--delete / --delete-after / --delete-delay). */
* manifest arrives). */
bool config_delete_timing_early(const Config* config);
/* True for the per-directory timings (--delete-during / --delete-delay). The
* sender streams a delete plan per source directory in directory order; the
* receiver applies each plan on arrival (during) or snapshots its extras and
* commits them only after a fully-successful transfer (delay). */
bool config_delete_timing_per_dir(const Config* config);
/* Delete-timing sanity: with deletion enabled at most one timing flag may be
* set (none = the default delete-after commit timing); without deletion no
* timing flag may be set (each timing flag implies --delete). */
* set; without deletion no timing flag may be set (each timing flag implies
* --delete). A plain --delete is normalized to delete_during by
* cli_finalize_config on the client, so a transmitted use_delete config always
* carries exactly one timing; the zero-timing case remains valid only for a
* config that has not been through the CLI. */
bool config_has_valid_delete_timing(const Config* config);
/* Single source of truth for the cross-field ("combination") invariants a
File diff suppressed because it is too large. Load diff
+95
View File
@@ -0,0 +1,95 @@
#ifndef DELETE_PLAN_H
#define DELETE_PLAN_H
#include "array_list.h"
#include "config.h"
#include "file_receive.h"
#include "protocol.h"
#include "utils.h"
#include <stdbool.h>
/* Per-directory delete plans (protocol 2.24.0).
*
* rsync's --delete-during removes a directory's extras while the generator
* processes that directory, and --delete-delay records the deletion list during
* the scan but applies it only after a fully-successful transfer. FastSync has
* no per-directory generator pass; instead the sender streams one plan per
* source directory, in directory order, and the receiver applies it when it
* arrives (during) or snapshots its extras and commits them at the end (delay).
*
* The sender side builds a plan set from the path-only pre-scan (it needs every
* directory's complete direct-child list before the first data byte of that
* directory). The receiver side is a session that carries the global protected
* prefixes (filter-excluded and size-skipped source mirrors), the
* --delete-missing-args exact deletions, the shared --max-delete budget and,
* for --delete-delay, the snapshotted extras. */
/* ---- Sender: plan builder ---- */
typedef struct DeletePlanSender DeletePlanSender;
DeletePlanSender* delete_plan_sender_create(void);
void delete_plan_sender_destroy(DeletePlanSender* sender);
/* Record one transmitted entry. `path` is the destination-relative wire path;
* is_dir marks an explicit directory entry (--dirs, a -x mount point). */
bool delete_plan_sender_add(DeletePlanSender* sender, const char* path, bool is_dir);
/* Drop plans for directories outside `synced_dirs` (the --files-from
* synchronization scope; pass NULL when a full recursive transfer synchronized
* every directory). The receive root is the "." sentinel.
*
* `walk_root` scopes a general -R transfer: when non-NULL it is the
* reconstructed destination prefix the run actually transferred, and only the
* plan for that prefix (and directories below it) is ever transmitted, so the
* prefix's parent-directory siblings are never walked. Pass NULL for a plain
* recursive transfer and for --files-from. */
void delete_plan_sender_finalize(DeletePlanSender* sender, const ArrayList* synced_dirs,
const char* walk_root);
/* True when no transmitted FILE entry was recorded (an ambiguous empty scan).
Directory keep entries do not count, so an I/O error that hid every file
still refuses to delete. */
bool delete_plan_sender_empty(const DeletePlanSender* sender);
/* Attach the global config sections advertised on the first plan frame. The
* block is always transmitted by delete_plan_send_root(), on a config-only
* carrier frame when the scope allows no directory plan. */
void delete_plan_sender_set_config(DeletePlanSender* sender, const ArrayList* protected_prefixes,
const ArrayList* size_skipped, const ArrayList* missing_args);
/* Send the root plan (even before any data, so root extras are handled like
* rsync's first generator directory), after transmitting the per-run config
* block on its own carrier frame. Returns -1 on I/O error. */
int delete_plan_send_root(int fd, DeletePlanSender* sender);
/* Send the plans for every ancestor of `path` (root-first) and, when is_dir,
* for `path` itself; already-sent plans are skipped. */
int delete_plan_send_for_path(int fd, DeletePlanSender* sender, const char* path, bool is_dir);
/* Send the plan for every directory in `dirs` that has not been transmitted
* yet. Called after the data stream so an empty source directory's plan still
* clears its destination extras even though no file frame triggered it. */
int delete_plan_send_remaining(int fd, DeletePlanSender* sender, const ArrayList* dirs);
/* ---- Receiver: delete session ---- */
typedef struct DeletePlanSession DeletePlanSession;
DeletePlanSession* delete_plan_session_create(const Config* config);
void delete_plan_session_destroy(DeletePlanSession* session);
/* Read one STATUS_DELETE_PLAN frame (the leading status already consumed) and
* act on it. Returns 0 on success (including a dry-run/disabled no-op) and -1
* after signalling STATUS_ERROR on a malformed frame or a deletion failure. */
int delete_plan_session_receive(DeletePlanSession* session, const Config* config, int fd);
/* Apply the deferred snapshot (--delete-delay) and the missing-args deletions.
* Safe to call once; returns the commit outcome. */
DeleteCommitResult delete_plan_session_commit(DeletePlanSession* session, const Config* config);
/* True once the shared --max-delete budget stopped part of a deletion. */
bool delete_plan_session_limit_reached(const DeletePlanSession* session);
/* Number of destination entries the session actually removed, for the
end-of-transfer stats. For --delete-delay this excludes a snapshotted entry
that survived (e.g. a refilled directory that failed ENOTEMPTY), even though
that entry already consumed --max-delete budget at snapshot time. */
size_t delete_plan_session_deleted(const DeletePlanSession* session);
/* Install an observer invoked for every destination-relative path the session
truly removes (including the deferred --delete-delay commit), so the receiver
can report rsync's `deleting PATH` lines through the terminal STATUS_STATS
record. Pass NULL/0 to clear. */
void delete_plan_session_set_delete_observer(DeletePlanSession* session,
DeletePathObserver observer, void* context);
#endif
+408 -38
View File
@@ -15,6 +15,7 @@
#include <unistd.h>
#include "data.h"
#include "checksum.h"
#include "delta.h"
#include "file.h"
#include "file_store.h"
@@ -24,6 +25,13 @@
#include "utils.h"
#include "protocol.h"
#include "xattr.h"
#include <fcntl.h>
#include <unistd.h>
/* Files larger than this are not loaded whole for transfer (the sender streams
* them); a whole-file digest is computed from the path instead. Kept in sync
* with the sender's streaming threshold. */
#define STREAM_THRESHOLD (64ULL * 1024 * 1024)
static bool write_all(int fd, const void* data, unsigned long long size) {
const unsigned char* p = data;
@@ -39,20 +47,53 @@ static bool write_all(int fd, const void* data, unsigned long long size) {
return true;
}
/* Streaming copy of an open source descriptor into the just-created destination
`fd` (already at offset 0). Used by the --copy-dest basis install so a basis
larger than any in-memory whole-file bound still materializes without
buffering the entire file. `expected_size` is the caller-verified basis
size; the copy must produce exactly that many bytes (a short source is a hard
error, never a silently truncated destination). The final ftruncate drops
any residual tail a raced-in longer source might have left. */
static bool copy_fd_all(int dst_fd, int src_fd, unsigned long long expected_size) {
unsigned char buf[1 << 20];
unsigned long long done = 0;
while (done < expected_size) {
unsigned long long remaining = expected_size - done;
size_t want = remaining < sizeof(buf) ? (size_t)remaining : sizeof(buf);
ssize_t n = read(src_fd, buf, want);
if (n < 0 && errno == EINTR)
continue;
if (n <= 0)
return false;
if (!write_all(dst_fd, buf, (unsigned long long)n))
return false;
done += (unsigned long long)n;
}
return ftruncate(dst_fd, (off_t)expected_size) == 0;
}
/* Preallocate `size` bytes on `fd` before any data is written (--preallocate).
* posix_fallocate reserves real disk blocks, so an out-of-space condition
* fallocate(2) reserves real disk blocks, so an out-of-space condition
* (ENOSPC/EDQUOT) surfaces up front instead of partway through a transfer;
* unavoidable fragmentation of a streamed file is also reduced. Some
* filesystems (e.g. tmpfs, ZFS) do not support it and return EOPNOTSUPP/ENOSYS,
* where we fall back to ftruncate, which still extends the logical size so the
* fail-fast/contiguity intent degrades gracefully but never fails. Genuine
* allocation failures are propagated as the error code (caller fails the write).
* posix_fallocate leaves the fd's file offset unchanged, so the subsequent
* write_all at offset 0 is unaffected. Returns 0 on success (including the
* fallback) or a nonzero error code. */
* unavoidable fragmentation of a streamed file is also reduced. rsync favors
* the syscall over glibc posix_fallocate (whose emulation can be subtly
* different), so try fallocate(2) first and only fall back to posix_fallocate,
* then to ftruncate on filesystems (e.g. tmpfs, ZFS) that support neither. The
* logical size is always extended, so the fail-fast/contiguity intent degrades
* gracefully but never fails on an unsupported filesystem; genuine allocation
* failures are propagated as the error code (caller fails the write). Neither
* leaves the fd's file offset guaranteed, so the caller seeks back to 0 before
* writing. Returns 0 on success (including the fallback) or a nonzero error
* code. */
static int preallocate_fd(int fd, unsigned long long size) {
if (size == 0)
return 0;
#ifdef __linux__
if (fallocate(fd, 0, 0, (off_t)size) == 0)
return 0;
if (errno != EOPNOTSUPP && errno != ENOSYS && errno != EINVAL)
return errno;
#endif
int rc = posix_fallocate(fd, 0, (off_t)size);
if (rc == EOPNOTSUPP || rc == ENOSYS) {
if (ftruncate(fd, (off_t)size) == 0)
@@ -132,6 +173,12 @@ bool file_checksum(File* file, ChecksumAlgo algo, uint64_t seed, uint8_t* out, s
if (file->data->size == 0) {
return checksum_digest(algo, seed, "", 0, out, out_capacity, out_len);
}
/* A streamed source (data not loaded) may exceed any in-memory whole-file
bound; hash it from the file path in bounded buffers instead of forcing a
full load. This is the same digest the receiver recomputes on the basis. */
if (!file->data->data && file->path && file->data->size > STREAM_THRESHOLD &&
checksum_digest_file(algo, seed, file->path, out, out_capacity, out_len))
return true;
if (!file->data->data && !file_load_data(file))
return false;
return checksum_digest(algo, seed, file->data->data, file->data->size, out, out_capacity,
@@ -168,6 +215,7 @@ File* file_create(const char* path) {
file->is_dir = false;
file->dir_time_only = false;
file->basis_link = NULL;
file->basis_copy = NULL;
file->link_group = 0;
file->link_first = false;
file->hardlink_target = NULL;
@@ -178,6 +226,8 @@ File* file_create(const char* path) {
file->rdev_minor = 0;
file->xattrs = NULL;
file->dest_state = (OutputDestState){0};
file->matched_bytes = 0;
file->literal_bytes = 0;
return file;
}
@@ -195,6 +245,8 @@ void file_destroy(void* item) {
file->send_path = NULL;
free(file->basis_link);
file->basis_link = NULL;
free(file->basis_copy);
file->basis_copy = NULL;
free(file->hardlink_target);
file->hardlink_target = NULL;
free(file->symlink_target);
@@ -605,6 +657,103 @@ static int open_dir_beneath_root(const char* resolved, const char* root) {
}
int file_open_secure_parent(const char* path, char** leaf_out, bool create_dirs) {
return file_open_secure_parent_counted(path, leaf_out, create_dirs, NULL, NULL);
}
/* The logical transfer root expressed in the same coordinate as the secure
* parent walk's `rel_buf` (relative to the authorized root, with a leading
* '/'), used as the floor at or below which a created directory is a real
* file-list entry. The on-disk transfer root is the receive root joined to the
* wire path; the mirror scaffolding above it (the absolute source path below
* the destination root) is not an rsync entry. Returns an allocated string or
* NULL (count every created component). */
static char* transfer_root_floor(const Config* config) {
if (!config || !config->send_directory || config->send_directory[0] == '\0')
return NULL;
const char* spec = config->send_directory;
const char* after = spec;
if (spec[0] == '.' && spec[1] == '/') {
after = spec + 2;
} else {
const char* cut = strstr(spec, "/./");
if (cut)
after = cut + 3;
}
while (*after == '/')
after++;
char* wire_root = str_dup(after);
if (!wire_root)
return NULL;
size_t wlen = strlen(wire_root);
while (wlen > 0 && wire_root[wlen - 1] == '/')
wire_root[--wlen] = '\0';
if (wlen == 0) {
free(wire_root);
return NULL;
}
char* disk_root = config->receive_root_directory
? path_cat(config->receive_root_directory, wire_root)
: str_dup(wire_root);
free(wire_root);
if (!disk_root)
return NULL;
const char* root_path = utils_get_authorized_root_path();
const char* floor = disk_root;
if (root_path && root_path[0] == '/') {
size_t rl = strlen(root_path);
while (rl > 0 && root_path[rl - 1] == '/')
rl--;
if (strncmp(disk_root, root_path, rl) == 0 && (disk_root[rl] == '/' || disk_root[rl] == '\0'))
floor = disk_root + rl;
}
while (*floor == '/')
floor++;
char* out = str_dup(floor);
free(disk_root);
if (!out)
return NULL;
if (out[0] == '\0') {
free(out);
return NULL;
}
return out;
}
/* A created parent component counts toward `Number of created files` only when
* its receive-root-relative path is at or below the logical transfer root
* (`count_floor`). The transfer root itself corresponds to rsync's `.` entry
* (created on a fresh destination, pre-existing otherwise); the mirror
* scaffolding above it is FastSync's absolute-path layout, not an rsync entry. */
static bool created_dir_counts(const char* count_floor, const char* rel_buf,
const char* component) {
if (!count_floor)
return true;
char candidate[PATH_MAX];
int n = snprintf(candidate, sizeof(candidate), "%s/%s", rel_buf, component);
if (n < 0 || (size_t)n >= sizeof(candidate))
return false;
const char* cand = candidate;
while (*cand == '/')
cand++;
size_t fl = strlen(count_floor);
if (strncmp(cand, count_floor, fl) != 0)
return false;
return cand[fl] == '\0' || cand[fl] == '/';
}
/* Public wrapper for the receiver's created-directory accounting: the logical
* transfer root expressed receive-root-relative, or NULL when the wire paths
* carry no mirror scaffolding above it (--relative and --files-from, whose
* paths are already relative to the transfer root). The caller frees a
* non-NULL result. */
char* file_transfer_root_floor(const Config* config) {
if (!config || config->relative || config->files_from_set != NULL)
return NULL;
return transfer_root_floor(config);
}
int file_open_secure_parent_counted(const char* path, char** leaf_out, bool create_dirs,
unsigned* dirs_created, const char* count_floor) {
char* copy = str_dup(path);
if (!copy)
return -1;
@@ -663,8 +812,16 @@ int file_open_secure_parent(const char* path, char** leaf_out, bool create_dirs)
if (strcmp(component, ".") != 0) {
int next = openat(fd, component, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
if (next < 0 && create_dirs && errno == ENOENT) {
bool created = mkdirat(fd, component, 0755) == 0;
bool created = mkdirat(fd, component, (mode_t)(0777 & ~(mode_t)file_process_umask())) == 0;
if (created || errno == EEXIST) {
/* Protocol 2.28.0: only directories the logical file list would
create count toward `Number of created files`; the mirror
scaffolding above the transfer root (e.g. the absolute source path
under the destination root) is not an rsync entry. `count_floor`
is a receive-root-relative prefix that must be reached before a
created component is counted. */
if (created && dirs_created && created_dir_counts(count_floor, rel_buf, component))
(*dirs_created)++;
/* P7 Wave E: --copy-as owns EVERY entry, including the intermediate
directories this walk creates implicitly. Its target ids are a
global policy, so they are available here without per-entry source
@@ -792,12 +949,31 @@ bool file_ensure_directory_secure(const char* path) {
int dir_fd = openat(parent_fd, leaf, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
bool created = false;
if (dir_fd < 0 && errno == ENOENT) {
if (mkdirat(parent_fd, leaf, 0755) == 0) {
if (mkdirat(parent_fd, leaf, (mode_t)(0777 & ~(mode_t)file_process_umask())) == 0) {
created = true;
dir_fd = openat(parent_fd, leaf, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
} else if (errno == EEXIST) {
dir_fd = openat(parent_fd, leaf, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
}
} else if (dir_fd < 0 && errno == ENOTDIR) {
/* rsync replaces a destination non-directory (regular file) with an
incoming directory. Confined to the already-opened secure parent fd:
the leaf is unlinked by name (never followed) and only a non-directory
is ever removed, so this cannot escape the authorized root or remove a
pre-existing directory tree. A symlink is left alone (openat with
O_NOFOLLOW reports ELOOP, which takes no branch here), since replacing
it is not required for FastSync's transferred directories and keeps
--keep-dirlinks semantics untouched. */
struct stat leaf_st;
if (fstatat(parent_fd, leaf, &leaf_st, AT_SYMLINK_NOFOLLOW) == 0 && !S_ISDIR(leaf_st.st_mode) &&
!S_ISLNK(leaf_st.st_mode)) {
if (unlinkat(parent_fd, leaf, 0) == 0) {
if (mkdirat(parent_fd, leaf, (mode_t)(0777 & ~(mode_t)file_process_umask())) == 0)
created = true;
/* On failure dir_fd stays < 0 below, so the caller still sees it. */
dir_fd = openat(parent_fd, leaf, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
}
}
}
bool ok = dir_fd >= 0;
/* --copy-as owns a directory this call just created (the final component;
@@ -998,15 +1174,14 @@ static void restore_extra_fd(int fd, const FileMetadata* metadata, const FileXat
}
}
static bool file_to_disk_secure_impl(const char* path, const void* data,
unsigned long long data_size, bool inplace, bool sparse,
bool preallocate, const FileMetadata* metadata,
FileAttrPolicy policy, bool update, bool no_replace,
bool use_fsync, const char* temp_dir,
const FileXattrList* xattrs, bool fake_super,
bool keep_partial) {
static bool
file_to_disk_secure_impl(const char* path, const void* data, unsigned long long data_size,
bool inplace, bool sparse, bool preallocate, const FileMetadata* metadata,
FileAttrPolicy policy, bool update, bool no_replace, bool use_fsync,
const char* temp_dir, const FileXattrList* xattrs, bool fake_super,
bool keep_partial, unsigned* dirs_created, const char* count_floor) {
char* leaf = NULL;
int dirfd = file_open_secure_parent(path, &leaf, true);
int dirfd = file_open_secure_parent_counted(path, &leaf, true, dirs_created, count_floor);
if (dirfd < 0)
return false;
int fd = -1;
@@ -1066,11 +1241,10 @@ static bool file_to_disk_secure_impl(const char* path, const void* data,
} else {
/* Preallocate the expected payload size before writing so an
out-of-space condition fails cleanly up front (--preallocate).
--sparse takes precedence: posix_fallocate would allocate every
block, defeating the holes the sparse writer would create, so the
two never combine here (the ftruncate presize below stays). */
rsync lets --preallocate win over --sparse (the reserved blocks
survive the sparse writer's seeks), so both flags can be active. */
int prealloc_rc = 0;
if (preallocate && !sparse && data_size > 0) {
if (preallocate && data_size > 0) {
prealloc_rc = preallocate_fd(fd, data_size);
if (prealloc_rc != 0) {
char* escaped_path = output_escape(path, log_get_8_bit_output());
@@ -1196,7 +1370,7 @@ static bool file_to_disk_secure_impl(const char* path, const void* data,
if (fd < 0)
continue; /* EEXIST (or a transient open error): try a fresh name. */
int prealloc_rc = 0;
if (preallocate && !sparse && data_size > 0) {
if (preallocate && data_size > 0) {
prealloc_rc = preallocate_fd(fd, data_size);
if (prealloc_rc != 0) {
char* escaped_path = output_escape(path, log_get_8_bit_output());
@@ -1298,7 +1472,7 @@ static bool file_to_disk_secure_impl(const char* path, const void* data,
"non-atomic copy into the destination directory");
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
policy, update, no_replace, use_fsync, NULL, xattrs, fake_super,
keep_partial);
keep_partial, dirs_created, count_floor);
}
return ok;
}
@@ -1307,7 +1481,8 @@ bool file_to_disk_secure(const char* path, const void* data, unsigned long long
bool inplace, bool sparse, bool preallocate, const FileMetadata* metadata,
FileAttrPolicy policy, const char* temp_dir) {
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
policy, false, false, false, temp_dir, NULL, false, false);
policy, false, false, false, temp_dir, NULL, false, false, NULL,
NULL);
}
bool file_to_disk_secure_update(const char* path, const void* data, unsigned long long data_size,
@@ -1315,7 +1490,8 @@ bool file_to_disk_secure_update(const char* path, const void* data, unsigned lon
const FileMetadata* metadata, FileAttrPolicy policy,
const char* temp_dir) {
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
policy, true, false, false, temp_dir, NULL, false, false);
policy, true, false, false, temp_dir, NULL, false, false, NULL,
NULL);
}
bool file_to_disk_secure_with_fsync(const char* path, const void* data,
@@ -1323,7 +1499,8 @@ bool file_to_disk_secure_with_fsync(const char* path, const void* data,
bool preallocate, const FileMetadata* metadata,
FileAttrPolicy policy, bool use_fsync, const char* temp_dir) {
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
policy, false, false, use_fsync, temp_dir, NULL, false, false);
policy, false, false, use_fsync, temp_dir, NULL, false, false,
NULL, NULL);
}
bool file_to_disk_secure_no_replace(const char* path, const void* data,
@@ -1331,7 +1508,8 @@ bool file_to_disk_secure_no_replace(const char* path, const void* data,
const FileMetadata* metadata, FileAttrPolicy policy,
const char* temp_dir) {
return file_to_disk_secure_impl(path, data, data_size, false, sparse, preallocate, metadata,
policy, false, true, false, temp_dir, NULL, false, false);
policy, false, true, false, temp_dir, NULL, false, false, NULL,
NULL);
}
/* Receiver write-path variant that also applies the per-file xattrs (-X/-A)
@@ -1344,9 +1522,21 @@ bool file_to_disk_secure_attrs(const char* path, const void* data, unsigned long
const FileMetadata* metadata, FileAttrPolicy policy, bool update,
bool no_replace, bool use_fsync, const FileXattrList* xattrs,
bool fake_super, bool keep_partial, const char* temp_dir) {
return file_to_disk_secure_attrs_counted(path, data, data_size, inplace, sparse, preallocate,
metadata, policy, update, no_replace, use_fsync, xattrs,
fake_super, keep_partial, temp_dir, NULL, NULL);
}
bool file_to_disk_secure_attrs_counted(const char* path, const void* data,
unsigned long long data_size, bool inplace, bool sparse,
bool preallocate, const FileMetadata* metadata,
FileAttrPolicy policy, bool update, bool no_replace,
bool use_fsync, const FileXattrList* xattrs, bool fake_super,
bool keep_partial, const char* temp_dir,
unsigned* dirs_created, const char* count_floor) {
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
policy, update, no_replace, use_fsync, temp_dir, xattrs,
fake_super, keep_partial);
fake_super, keep_partial, dirs_created, count_floor);
}
/* Atomic --link-dest install. The destination is replaced (via a temporary
@@ -1364,16 +1554,176 @@ bool file_to_disk_secure_attrs(const char* path, const void* data, unsigned long
* basis). Likewise `xattrs`/`fake_super` are applied only on the copy
* fallback, so a fallback copy preserves the per-file attributes instead of
* silently dropping them. */
/* Streaming --copy-dest basis install: atomically materialize `path` from the
* bytes of `basis_path` without holding the file in memory, so a basis larger
* than any whole-file bound still works. Mirrors the ordinary secure store
* path (confined parent walk, temp + rename, --update/--ignore-existing/
* --preallocate/--temp-dir) but sources the data from the basis descriptor
* rather than a caller buffer, and applies the SOURCE metadata (rsync copies
* then fixes attributes). A hard-link install that falls back to a byte copy
* also routes through here when the caller supplies the basis path. */
static bool file_copy_basis_stream_impl(const char* path, const char* basis_path,
unsigned long long expected_size, bool preallocate,
const FileMetadata* metadata, FileAttrPolicy policy,
bool update, bool no_replace, bool use_fsync,
const FileXattrList* xattrs, bool fake_super,
const char* temp_dir, unsigned* dirs_created,
const char* count_floor) {
if (!path || !basis_path)
return false;
char* leaf = NULL;
int dirfd = file_open_secure_parent_counted(path, &leaf, true, dirs_created, count_floor);
if (dirfd < 0)
return false;
char* basis_leaf = NULL;
int basis_dirfd = file_open_secure_parent(basis_path, &basis_leaf, false);
int src_fd = -1;
if (basis_dirfd >= 0 && basis_leaf != NULL) {
/* O_NONBLOCK rejects a raced-in FIFO without blocking; the S_ISREG gate
below is the real type check. */
src_fd = openat(basis_dirfd, basis_leaf, O_RDONLY | O_CLOEXEC | O_NOFOLLOW | O_NONBLOCK);
struct stat src_st;
if (src_fd >= 0 && (fstat(src_fd, &src_st) != 0 || !S_ISREG(src_st.st_mode))) {
close(src_fd);
src_fd = -1;
}
}
if (basis_dirfd >= 0)
close(basis_dirfd);
free(basis_leaf);
if (src_fd < 0) {
close(dirfd);
free(leaf);
return false;
}
struct stat destination_stat;
bool destination_is_regular = fstatat(dirfd, leaf, &destination_stat, AT_SYMLINK_NOFOLLOW) == 0 &&
S_ISREG(destination_stat.st_mode);
if (update && metadata && destination_is_regular && stat_is_newer(&destination_stat, metadata)) {
close(src_fd);
close(dirfd);
free(leaf);
return true;
}
if (no_replace && file_path_exists_secure(path)) {
close(src_fd);
close(dirfd);
free(leaf);
return true;
}
int scratch_dirfd = -1;
if (temp_dir) {
scratch_dirfd = file_open_temp_dir(temp_dir);
if (scratch_dirfd < 0) {
int saved_errno = errno;
log_message(LOG_LEVEL_ERROR,
"--temp-dir '%s' could not be opened (rsync requires it to already exist): %s",
temp_dir, strerror(saved_errno));
close(src_fd);
close(dirfd);
free(leaf);
return false;
}
}
int target_dirfd = scratch_dirfd >= 0 ? scratch_dirfd : dirfd;
int tmp_size = snprintf(NULL, 0, ".%s.tmp.%ld.%llu", leaf, (long)getpid(), ~0ULL);
char* tmp = NULL;
bool ok = false;
if (tmp_size >= 0)
tmp = malloc((size_t)tmp_size + 1);
if (tmp) {
for (unsigned int i = 0; i < 100 && !ok; ++i) {
if (scratch_dirfd >= 0)
snprintf(tmp, (size_t)tmp_size + 1, ".%s.tmp.%ld.%llu", leaf, (long)getpid(),
next_temp_sequence());
else
snprintf(tmp, (size_t)tmp_size + 1, ".%s.tmp.%ld.%u", leaf, (long)getpid(), i);
int fd =
openat(target_dirfd, tmp, O_WRONLY | O_CREAT | O_EXCL | O_CLOEXEC | O_NOFOLLOW, 0600);
if (fd < 0) {
if (errno != EEXIST)
break;
continue;
}
bool wrote = true;
if (preallocate && expected_size > 0 && preallocate_fd(fd, expected_size) != 0)
wrote = false;
if (wrote)
wrote = copy_fd_all(fd, src_fd, expected_size);
if (wrote && metadata) {
if (!policy.perms &&
fchmod(fd, file_mode_base(metadata, destination_is_regular,
destination_is_regular ? destination_stat.st_mode & 0777
: 0)) != 0)
wrote = false;
if (wrote)
wrote = file_restore_metadata_fd(fd, metadata, policy);
} else if (wrote && fchmod(fd, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH) != 0) {
wrote = false;
}
if (wrote)
restore_extra_fd(fd, metadata, xattrs, fake_super, policy);
if (wrote && use_fsync)
wrote = fsync(fd) == 0;
if (close(fd) != 0)
wrote = false;
if (wrote && renameat(target_dirfd, tmp, dirfd, leaf) != 0)
wrote = false;
if (!wrote)
unlinkat(target_dirfd, tmp, 0);
ok = wrote;
}
free(tmp);
}
if (!ok && scratch_dirfd >= 0) {
/* Retry once with no scratch dir (rsync's EXDEV fallback). */
close(scratch_dirfd);
close(src_fd);
close(dirfd);
free(leaf);
return file_copy_basis_stream_impl(path, basis_path, expected_size, preallocate, metadata,
policy, update, no_replace, use_fsync, xattrs, fake_super,
NULL, dirs_created, count_floor);
}
if (scratch_dirfd >= 0)
close(scratch_dirfd);
close(src_fd);
close(dirfd);
free(leaf);
return ok;
}
/* --copy-dest basis install (streaming). Applies the source metadata and the
per-file xattrs / --fake-super record. */
bool file_copy_basis_stream_attrs(const char* path, const char* basis_path,
unsigned long long expected_size, bool preallocate,
const FileMetadata* metadata, FileAttrPolicy policy, bool update,
bool use_fsync, const FileXattrList* xattrs, bool fake_super,
const char* temp_dir) {
return file_copy_basis_stream_impl(path, basis_path, expected_size, preallocate, metadata, policy,
update, false, use_fsync, xattrs, fake_super, temp_dir, NULL,
NULL);
}
static bool file_to_disk_secure_link_impl(const char* path, const char* basis_path,
const void* data, unsigned long long data_size,
bool preallocate, const FileMetadata* metadata,
FileAttrPolicy policy, bool use_fsync,
const FileXattrList* xattrs, bool fake_super,
const char* temp_dir) {
const char* temp_dir, unsigned* dirs_created,
const char* count_floor) {
if (!path || !basis_path)
return false;
/* The caller-supplied buffer is no longer used: the copy fallback streams
from the basis path (which may hold an over-limit file). Kept in the
signature for the existing API. */
(void)data;
char* leaf = NULL;
int dirfd = file_open_secure_parent(path, &leaf, true);
int dirfd = file_open_secure_parent_counted(path, &leaf, true, dirs_created, count_floor);
if (dirfd < 0)
return false;
@@ -1455,10 +1805,17 @@ static bool file_to_disk_secure_link_impl(const char* path, const char* basis_pa
close(dirfd);
free(leaf);
/* The basis file could not be linked in (missing, cross-device, refused
by the filesystem). Write a byte-identical local copy instead. */
return file_to_disk_secure_attrs(path, data, data_size, false, false, preallocate, metadata,
policy, false, false, use_fsync, xattrs, fake_super, false,
temp_dir);
by the filesystem). Stream a byte-identical local copy from the basis
itself (never the possibly-absent caller buffer) so an over-limit basis
still materializes. When the basis path is not a readable regular file
(e.g. a directory raced in), fall back to the caller-supplied bytes. */
if (file_copy_basis_stream_impl(path, basis_path, data_size, preallocate, metadata, policy,
false, false, use_fsync, xattrs, fake_super, temp_dir,
dirs_created, count_floor))
return true;
return file_to_disk_secure_attrs_counted(
path, data, data_size, false, false, preallocate, metadata, policy, false, false, use_fsync,
xattrs, fake_super, false, temp_dir, dirs_created, count_floor);
}
if (scratch_dirfd >= 0)
@@ -1473,7 +1830,7 @@ bool file_to_disk_secure_link(const char* path, const char* basis_path, const vo
const FileMetadata* metadata, FileAttrPolicy policy, bool use_fsync,
const char* temp_dir) {
return file_to_disk_secure_link_impl(path, basis_path, data, data_size, preallocate, metadata,
policy, use_fsync, NULL, false, temp_dir);
policy, use_fsync, NULL, false, temp_dir, NULL, NULL);
}
bool file_to_disk_secure_link_attrs(const char* path, const char* basis_path, const void* data,
@@ -1481,8 +1838,21 @@ bool file_to_disk_secure_link_attrs(const char* path, const char* basis_path, co
const FileMetadata* metadata, FileAttrPolicy policy,
bool use_fsync, const FileXattrList* xattrs, bool fake_super,
const char* temp_dir) {
return file_to_disk_secure_link_attrs_counted(path, basis_path, data, data_size, preallocate,
metadata, policy, use_fsync, xattrs, fake_super,
temp_dir, NULL, NULL);
}
bool file_to_disk_secure_link_attrs_counted(const char* path, const char* basis_path,
const void* data, unsigned long long data_size,
bool preallocate, const FileMetadata* metadata,
FileAttrPolicy policy, bool use_fsync,
const FileXattrList* xattrs, bool fake_super,
const char* temp_dir, unsigned* dirs_created,
const char* count_floor) {
return file_to_disk_secure_link_impl(path, basis_path, data, data_size, preallocate, metadata,
policy, use_fsync, xattrs, fake_super, temp_dir);
policy, use_fsync, xattrs, fake_super, temp_dir,
dirs_created, count_floor);
}
bool file_write_to_disk(const char* path, const void* data, unsigned long long data_size,
+37
View File
@@ -87,6 +87,11 @@ bool file_path_exists_secure(const char* path);
bool file_stat_secure(const char* path, struct stat* st);
bool file_destination_is_newer_secure(const char* path, const FileMetadata* metadata);
int file_open_secure_parent(const char* path, char** leaf_out, bool create_dirs);
/* Protocol 2.28.0 variant: also increments *dirs_created for every missing
* parent directory this walk creates that lies strictly below `count_floor`
* (a receive-root-relative path, or NULL to count all of them). */
int file_open_secure_parent_counted(const char* path, char** leaf_out, bool create_dirs,
unsigned* dirs_created, const char* count_floor);
bool file_ensure_directory_secure(const char* path);
bool file_directory_exists_secure(const char* path);
bool file_rename_secure(const char* old_path, const char* new_path);
@@ -158,5 +163,37 @@ bool file_to_disk_secure_link_attrs(const char* path, const char* basis_path, co
const FileMetadata* metadata, FileAttrPolicy policy,
bool use_fsync, const FileXattrList* xattrs, bool fake_super,
const char* temp_dir);
/* Streaming --copy-dest install: atomically materialize `path` by copying the
* bytes of `basis_path` through a bounded buffer (no whole-file buffering, so
* an arbitrarily large basis works), applying the SOURCE metadata and the
* per-file xattrs / --fake-super record. `update` honors a newer destination;
* a --temp-dir scratch location falls back to a direct write on EXDEV. */
bool file_copy_basis_stream_attrs(const char* path, const char* basis_path,
unsigned long long expected_size, bool preallocate,
const FileMetadata* metadata, FileAttrPolicy policy, bool update,
bool use_fsync, const FileXattrList* xattrs, bool fake_super,
const char* temp_dir);
/* Protocol 2.28.0 receiver-stat variants: like the two above but additionally
* report through `dirs_created` (when non-NULL) how many parent directories the
* confined secure walk had to create that lie strictly below `count_floor` (a
* receive-root-relative prefix, or NULL for all). Used to reproduce rsync's
* `Number of created files` directory count on a fresh destination. */
bool file_to_disk_secure_attrs_counted(const char* path, const void* data,
unsigned long long data_size, bool inplace, bool sparse,
bool preallocate, const FileMetadata* metadata,
FileAttrPolicy policy, bool update, bool no_replace,
bool use_fsync, const FileXattrList* xattrs, bool fake_super,
bool keep_partial, const char* temp_dir,
unsigned* dirs_created, const char* count_floor);
bool file_to_disk_secure_link_attrs_counted(const char* path, const char* basis_path,
const void* data, unsigned long long data_size,
bool preallocate, const FileMetadata* metadata,
FileAttrPolicy policy, bool use_fsync,
const FileXattrList* xattrs, bool fake_super,
const char* temp_dir, unsigned* dirs_created,
const char* count_floor);
/* The logical transfer root expressed receive-root-relative, or NULL when the
* wire paths carry no mirror scaffolding above it. Caller frees non-NULL. */
char* file_transfer_root_floor(const Config* config);
#endif
+698 -241
View File
File diff suppressed because it is too large. Load diff
+63
View File
@@ -3,6 +3,7 @@
#include "config.h"
#include "file_types.h"
#include "utils.h"
#include <stdbool.h>
/* Server-side file receive/save path. */
@@ -23,6 +24,16 @@ File* file_receive_hardlink(int file_descriptor);
File* file_receive_symlink(int file_descriptor, const Config* config);
File* file_receive_special(int file_descriptor);
bool file_special_rdev_valid(int32_t major, int32_t minor, mode_t mode);
/* Testable basis quick-check / verification policy. file_basis_quick_match is
* rsync's metadata quick-check for a basis candidate (equal size is required
* separately by the caller; this adds the --size-only / mtime / --modify-window
* leg). file_basis_content_required reports whether a hit must ALSO be
* confirmed by a whole-file content digest (--verify-basis; false is the
* default rsync-parity behavior). */
bool file_basis_quick_match(const Config* config, const struct stat* st, time_t check_mtime,
long check_mtime_nsec);
bool file_basis_content_required(const Config* config);
File* receive_incremental_check(int fd, const Config* config, bool* skipped);
/* Extended variant used by the receiver. `would_transfer` (may be NULL) is set
* true only on the server-contacting --dry-run path when the file is not up to
@@ -118,6 +129,21 @@ bool manifest_delete_extras(const Config* config, DeleteManifest* manifest);
confinement or I/O error (the run then fails); tolerated per-path cases are
reported and skipped. */
bool manifest_delete_missing_args(const Config* config, DeleteManifest* manifest);
/* Budgeted form of manifest_delete_missing_args for the per-directory delete
session: each removed mirror draws from `max_delete` (SIZE_MAX = unlimited)
and the tallies are accumulated into `*deleted`/`*skipped`. `*limit_hit` is set
when the budget stopped the pass with entries left over. Returns false only
on a genuine deletion error. */
bool manifest_delete_missing_args_limited(const Config* config, DeleteManifest* manifest,
size_t max_delete, size_t* deleted, size_t* skipped,
bool* limit_hit);
/* Observer-aware form of manifest_delete_missing_args_limited: `observer` (may
be NULL) is invoked for every destination-relative path truly removed. */
bool manifest_delete_missing_args_limited_observed(const Config* config, DeleteManifest* manifest,
size_t max_delete, size_t* deleted,
size_t* skipped, bool* limit_hit,
DeletePathObserver observer,
void* observer_context);
/* Outcome of committing a delete manifest. LIMIT_REACHED reports rsync's
partial --max-delete result: the budget allowed some deletions and the rest
were skipped (the run still stores all file data but the client exits 25). */
@@ -134,6 +160,27 @@ typedef enum {
do or everything committed, DELETE_COMMIT_LIMIT_REACHED when the budget
stopped part of the work, or DELETE_COMMIT_ERROR on a genuine failure. */
DeleteCommitResult manifest_delete_all(const Config* config, DeleteManifest* manifest);
/* Like manifest_delete_all, but reports how many destination entries the commit
removed (for the end-of-transfer wire stats). `deleted` may be NULL. */
DeleteCommitResult manifest_delete_all_counted(const Config* config, DeleteManifest* manifest,
size_t* deleted);
/* Observer-aware form of manifest_delete_all_counted: `observer` (may be NULL)
is invoked for every destination-relative path truly removed. */
DeleteCommitResult manifest_delete_all_observed(const Config* config, DeleteManifest* manifest,
size_t* deleted, DeletePathObserver observer,
void* observer_context);
/* -n/--dry-run --delete would-delete reporting: walk the destination exactly as
the delete pass would and append (strdup'd) destination-relative paths that
WOULD be removed to `out`, without touching disk. Uses the same staging-dir,
basis-dir and protected-prefix skips as the real commit. Returns true on a
clean walk; `*count_out` receives the number of paths appended. */
bool manifest_would_delete_list(const Config* config, DeleteManifest* manifest, ArrayList* out,
size_t* count_out);
/* Convert one basis-directory path to the receive-root-relative protection
prefix the delete walker uses (NULL when it lies outside the root). Exposed
for unit tests of the root-of-"/" and normalization edge cases. */
char* file_receive_basis_delete_relative(const Config* config, const char* path);
/* Outcome of a single file_save_to_disk operation. The receiver needs to
distinguish "written" from "skipped" so --remove-source-files can be told
@@ -142,6 +189,22 @@ typedef enum { FILE_SAVE_ERROR = 0, FILE_SAVE_WRITTEN = 1, FILE_SAVE_SKIPPED = 2
FileSaveResult file_save_to_disk_full(const char* root_directory, const File* file,
const Config* config);
/* Protocol 2.28.0 variant: also reports through `created` (when non-NULL)
* whether the destination entry did not exist before this save, and through
* `created_dirs` how many parent directories the confined walk created, so the
* receiver can build rsync's `Number of created files` breakdown. The plain
* file_save_to_disk_full() is this with both out-params NULL. */
FileSaveResult file_save_to_disk_full_ex(const char* root_directory, const File* file,
const Config* config, bool* created,
unsigned* created_dirs);
bool file_save_to_disk(const char* root_directory, const File* file, const Config* config);
/* Protocol 2.28.0 receiver counter accumulator: fold one successfully saved
* entry into `stats`, adding its receiver-observed literal bytes and, when
* `created`, the matching created-by-type counter (regular file / symlink /
* special) plus `created_dirs` implicitly-created parent directories.
* Non-first hardlink siblings contribute no literal bytes. */
void receiver_stats_note_saved(ReceiverStats* stats, const File* file, bool created,
unsigned created_dirs);
#endif
+1
View File
@@ -182,6 +182,7 @@ bool file_send_sendfile_with_skip(File* file, int file_descriptor, bool use_meta
close(fd);
return false;
}
protocol_note_bytes_written((unsigned long long)sent);
}
close(fd);
+14
View File
@@ -57,6 +57,11 @@ typedef struct {
* equals the incoming file, and `data` is kept as the cross-filesystem
* fallback (a local copy) if the hard link cannot be created. */
char* basis_link;
/* Receiver-only, --copy-dest: when set (and basis_link is NULL), stream the
* basis file's bytes into the destination instead of `data`/`data->size`.
* This lets a basis larger than any whole-file bound materialize without
* buffering it; the source metadata on `metadata` is applied afterwards. */
char* basis_copy;
/* --hard-links (-H), sender + receiver wire state. link_group is a run-local
* id shared by every member of one source inode (0 = not part of a group).
* The FIRST member (link_first == true) carries its data on the wire and is
@@ -93,6 +98,15 @@ typedef struct {
* no report was requested/received, in which case -i/--out-format treats the
* entry conservatively as newly created. */
OutputDestState dest_state;
/* Receiver-only wire-stats tally: the number of bytes reconstructed from the
* basis file (matched delta blocks) for this entry. 0 when the file was sent
* whole. Accumulated into ReceiverStats.matched_data by the receiver sink. */
unsigned long long matched_bytes;
/* Receiver-only (protocol 2.28.0) wire-stats tally: the literal delta fragment
* bytes this entry carried (DELTA_INSTR_LITERAL). 0 when the file was sent
* whole; the sink then falls back to the whole payload size. Accumulated
* into ReceiverStats.literal_bytes. */
unsigned long long literal_bytes;
} File;
/* The path that should be sent on the wire and used for the receiver-side
+600 -225
View File
@@ -1,163 +1,27 @@
#include "filter.h"
#include "log.h"
#include "utils.h"
#include <ctype.h>
#include <errno.h>
#include <limits.h>
#include <stdarg.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
/* ---- Single rule parsing ---- */
static bool rule_text_is_unsupported_word(const char* p, size_t len) {
static const char* const words[] = {"merge", "dir-merge", "hide", "show",
"protect", "risk", "clear"};
for (size_t i = 0; i < sizeof(words) / sizeof(words[0]); i++) {
size_t wl = strlen(words[i]);
if (len == wl && strncmp(p, words[i], wl) == 0)
return true;
}
return false;
/* Write a diagnostic message into the caller's optional buffer. A NULL `err`
* (or a zero size) is a no-op, so a caller that only needs the boolean status
* may pass NULL without the snprintf-on-NULL undefined behaviour. */
static void filter_set_error(char* err, size_t err_size, const char* fmt, ...) {
if (!err || err_size == 0)
return;
va_list ap;
va_start(ap, fmt);
vsnprintf(err, err_size, fmt, ap);
va_end(ap);
}
/* rsync include/exclude rule modifiers we do NOT implement. A rule whose +/- is
* immediately followed by one of these is rejected instead of being silently
* parsed as a literal pattern. */
static bool is_unsupported_rule_modifier(char c) {
return c == '!' || c == 'C' || c == 's' || c == 'r' || c == 'p' || c == 'x';
}
FilterRule* filter_rule_parse(const char* line, char* err, size_t err_size) {
if (err && err_size > 0)
err[0] = '\0';
if (!line)
return NULL;
char* text = str_dup(line);
if (!text) {
if (err)
snprintf(err, err_size, "memory allocation failed");
return NULL;
}
size_t len = strlen(text);
while (len > 0 && (text[len - 1] == '\n' || text[len - 1] == '\r'))
text[--len] = '\0';
const char* p = text;
while (*p == ' ' || *p == '\t')
p++;
if (*p == '\0') {
snprintf(err, err_size, "empty filter rule");
free(text);
return NULL;
}
FilterAction action = FILTER_ACTION_EXCLUDE;
if (*p == '+' || *p == '-') {
action = *p == '+' ? FILTER_ACTION_INCLUDE : FILTER_ACTION_EXCLUDE;
p++;
/* rsync attaches rule modifiers directly to the +/- (e.g. "-s foo"). Only
* the '/' anchor modifier is supported; anything else is a clear error
* rather than a silently-ignored literal. */
if (*p != ' ' && *p != '\t' && *p != '\0' && is_unsupported_rule_modifier(*p)) {
snprintf(err, err_size,
"filter rule modifier '%c' is not supported (only the '/' anchor after +/- "
"is implemented; put a space between +/- and the pattern)",
*p);
free(text);
return NULL;
}
while (*p == ' ' || *p == '\t')
p++;
} else {
/* ':' (dir-merge) and '.' (merge) are rsync filter-rule shorthands. At the
* start of a rule they mean "merge this file", so reject them instead of
* silently turning them into inert exclude patterns. */
if (*p == ':' || *p == '.' || *p == '!') {
snprintf(err, err_size,
"filter rule starting with '%c' is not supported (merge/dir-merge/list-clear "
"shorthands are not implemented; use +/- include/exclude rules)",
*p);
free(text);
return NULL;
}
const char* sp = p;
while (*sp != '\0' && *sp != ' ' && *sp != '\t')
sp++;
size_t word_len = (size_t)(sp - p);
if (rule_text_is_unsupported_word(p, word_len)) {
snprintf(err, err_size,
"'%.*s' filter directives are not supported (only +/- include/exclude rules "
"with an optional '/' anchor and trailing '/' dir marker)",
(int)word_len, p);
free(text);
return NULL;
}
if (word_len == strlen("include") && strncmp(p, "include", word_len) == 0) {
action = FILTER_ACTION_INCLUDE;
p = sp;
} else if (word_len == strlen("exclude") && strncmp(p, "exclude", word_len) == 0) {
action = FILTER_ACTION_EXCLUDE;
p = sp;
}
while (*p == ' ' || *p == '\t')
p++;
}
if (*p == '\0') {
snprintf(err, err_size, "filter rule has no pattern");
free(text);
return NULL;
}
/* A pattern beginning with '/' is anchored (either as "-/foo" or "- /foo"). */
bool anchored = false;
if (*p == '/') {
anchored = true;
p++;
while (*p == ' ' || *p == '\t')
p++;
}
if (*p == '\0') {
snprintf(err, err_size, "filter rule has no pattern after '/' anchor");
free(text);
return NULL;
}
/* Pattern runs to the end of the rule; a single trailing '/' marks dir-only. */
size_t pat_len = strlen(p);
bool dir_only = false;
if (pat_len > 1 && p[pat_len - 1] == '/') {
dir_only = true;
pat_len--;
} else if (pat_len == 1 && p[0] == '/') {
/* "//" anchored with nothing after: meaningless. */
snprintf(err, err_size, "filter rule has no pattern");
free(text);
return NULL;
}
FilterRule* rule = calloc(1, sizeof(FilterRule));
if (!rule) {
snprintf(err, err_size, "memory allocation failed");
free(text);
return NULL;
}
rule->pattern = malloc(pat_len + 1);
if (!rule->pattern) {
free(rule);
snprintf(err, err_size, "memory allocation failed");
free(text);
return NULL;
}
memcpy(rule->pattern, p, pat_len);
rule->pattern[pat_len] = '\0';
rule->action = action;
rule->anchored = anchored;
rule->dir_only = dir_only;
rule->owner = NULL;
free(text);
return rule;
}
/* ---- Ordered rule lists ---- */
void filter_rule_free(FilterRule* rule) {
if (!rule)
@@ -167,8 +31,6 @@ void filter_rule_free(FilterRule* rule) {
free(rule);
}
/* ---- Ordered rule lists ---- */
FilterRuleList* filter_rule_list_create(void) {
return calloc(1, sizeof(FilterRuleList));
}
@@ -190,28 +52,42 @@ bool filter_rule_list_add(FilterRuleList* list, FilterRule* rule) {
return true;
}
bool filter_rule_list_parse_append(FilterRuleList* list, const char* line, char* err,
size_t err_size) {
FilterRule* rule = filter_rule_parse(line, err, err_size);
if (!rule)
return false;
if (!filter_rule_list_add(list, rule)) {
filter_rule_free(rule);
snprintf(err, err_size, "memory allocation failed");
return false;
}
return true;
}
void filter_rule_list_free(FilterRuleList* list) {
if (!list)
return;
for (int i = 0; i < list->count; i++)
filter_rule_free(list->items[i]);
for (int i = 0; i < list->dir_merge_count; i++)
free(list->dir_merge_names[i]);
free(list->dir_merge_names);
free(list->items);
free(list);
}
/* Register a per-directory merge-file basename (for "dir-merge NAME"/": NAME"
* and -F's .rsync-filter). Duplicate names are ignored. */
bool filter_rule_list_add_dir_merge(FilterRuleList* list, const char* name) {
if (!list || !name || name[0] == '\0')
return false;
for (int i = 0; i < list->dir_merge_count; i++) {
if (strcmp(list->dir_merge_names[i], name) == 0)
return true;
}
if (list->dir_merge_count == list->dir_merge_capacity) {
int new_cap = list->dir_merge_capacity > 0 ? list->dir_merge_capacity * 2 : 4;
char** grown = realloc(list->dir_merge_names, (size_t)new_cap * sizeof(char*));
if (!grown)
return false;
list->dir_merge_names = grown;
list->dir_merge_capacity = new_cap;
}
char* dup = str_dup(name);
if (!dup)
return false;
list->dir_merge_names[list->dir_merge_count++] = dup;
return true;
}
static bool set_rule_owner(FilterRule* rule, const char* owner) {
char* dup = str_dup(owner ? owner : "");
if (!dup)
@@ -221,7 +97,315 @@ static bool set_rule_owner(FilterRule* rule, const char* owner) {
return true;
}
/* ---- CVS default excludes (-C) ---- */
/* ---- Rule parsing ---- */
/* A short rule prefix is a single character; a long rule name is alphabetic
* (with '-'). `is_short` distinguishes the modifier-attachment rules. */
typedef enum {
RULE_KIND_EXCLUDE,
RULE_KIND_INCLUDE,
RULE_KIND_HIDE,
RULE_KIND_SHOW,
RULE_KIND_PROTECT,
RULE_KIND_RISK,
RULE_KIND_MERGE,
RULE_KIND_DIR_MERGE,
RULE_KIND_CLEAR,
RULE_KIND_UNKNOWN,
} RuleKind;
static bool short_rule_char(char c, RuleKind* kind) {
switch (c) {
case '-':
*kind = RULE_KIND_EXCLUDE;
return true;
case '+':
*kind = RULE_KIND_INCLUDE;
return true;
case 'H':
*kind = RULE_KIND_HIDE;
return true;
case 'S':
*kind = RULE_KIND_SHOW;
return true;
case 'P':
*kind = RULE_KIND_PROTECT;
return true;
case 'R':
*kind = RULE_KIND_RISK;
return true;
case '.':
*kind = RULE_KIND_MERGE;
return true;
case ':':
*kind = RULE_KIND_DIR_MERGE;
return true;
case '!':
*kind = RULE_KIND_CLEAR;
return true;
default:
return false;
}
}
static bool long_rule_name(const char* name, size_t len, RuleKind* kind) {
struct {
const char* word;
RuleKind kind;
} table[] = {
{"exclude", RULE_KIND_EXCLUDE}, {"include", RULE_KIND_INCLUDE},
{"hide", RULE_KIND_HIDE}, {"show", RULE_KIND_SHOW},
{"protect", RULE_KIND_PROTECT}, {"risk", RULE_KIND_RISK},
{"merge", RULE_KIND_MERGE}, {"dir-merge", RULE_KIND_DIR_MERGE},
{"clear", RULE_KIND_CLEAR},
};
for (size_t i = 0; i < sizeof(table) / sizeof(table[0]); i++) {
if (strlen(table[i].word) == len && strncmp(name, table[i].word, len) == 0) {
*kind = table[i].kind;
return true;
}
}
return false;
}
static bool is_modifier_char(char c) {
return c == 's' || c == 'r' || c == 'p' || c == 'x' || c == '/' || c == '!' || c == 'C';
}
/* Parse "RULE[,MODIFIERS] [PATTERN]". On success `kind`, `sides`,
* `sides_explicit`, `negate`, `anchored_mod`, `perishable`, `xattr`,
* `cvs_inject` and the pattern span (`pat_start`/`pat_len`, possibly 0 for
* merge/clear) are filled. Returns true on success. */
static bool parse_rule_syntax(const char* text, RuleKind* kind, unsigned* sides,
bool* sides_explicit, bool* negate, bool* anchored_mod,
bool* perishable, bool* xattr, bool* cvs_inject,
const char** pat_start, size_t* pat_len) {
const char* p = text;
*sides = FILTER_SIDE_SENDER | FILTER_SIDE_RECEIVER;
*sides_explicit = false;
*negate = false;
*anchored_mod = false;
*perishable = false;
*xattr = false;
*cvs_inject = false;
*pat_start = NULL;
*pat_len = 0;
bool is_short = false;
if (short_rule_char(*p, kind)) {
is_short = true;
p++;
} else {
const char* name_start = p;
while (isalpha((unsigned char)*p) || *p == '-')
p++;
size_t name_len = (size_t)(p - name_start);
if (name_len == 0 || !long_rule_name(name_start, name_len, kind))
return false;
/* A long name must be followed by a separator, a comma or the end. */
if (*p != '\0' && *p != ',' && *p != ' ' && *p != '_')
return false;
}
/* Modifiers: long names require a comma; short names may attach directly.
Only commit a modifier run that terminates at a separator or the end, so a
pattern such as "*.tmp" written as "-*.tmp" is not mistaken for modifiers. */
const char* mod_start = p;
const char* mod_end = p;
if (*p == ',') {
p++;
mod_start = p;
while (is_modifier_char(*p))
p++;
mod_end = p;
} else if (is_short) {
const char* scan = p;
while (is_modifier_char(*scan))
scan++;
if (*scan == '\0' || *scan == ' ' || *scan == '_') {
mod_start = p;
mod_end = scan;
p = scan;
}
}
for (const char* m = mod_start; m < mod_end; m++) {
switch (*m) {
case 's':
*sides = FILTER_SIDE_SENDER;
*sides_explicit = true;
break;
case 'r':
*sides = FILTER_SIDE_RECEIVER;
*sides_explicit = true;
break;
case '!':
*negate = true;
break;
case '/':
*anchored_mod = true;
break;
case 'p':
*perishable = true;
break;
case 'x':
*xattr = true;
break;
case 'C':
*cvs_inject = true;
break;
default:
break;
}
}
/* A single space or underscore separates the rule/modifiers from the
pattern; further spaces/underscores belong to the pattern. */
const char* pat = p;
if (*pat == ' ' || *pat == '_')
pat++;
/* Trim a trailing newline/CR (the caller may pass a raw file line). */
*pat_start = pat;
*pat_len = strlen(pat);
while (*pat_len > 0 && (pat[*pat_len - 1] == '\n' || pat[*pat_len - 1] == '\r'))
(*pat_len)--;
return true;
}
FilterRule* filter_rule_parse(const char* line, const FilterParseOptions* opts, char* err,
size_t err_size) {
if (err && err_size > 0)
err[0] = '\0';
if (!line)
return NULL;
const char* p = line;
while (*p == ' ' || *p == '\t')
p++;
if (*p == '\0' || *p == '\n' || *p == '\r') {
filter_set_error(err, err_size, "empty filter rule");
return NULL;
}
RuleKind kind = RULE_KIND_UNKNOWN;
unsigned sides;
bool sides_explicit, negate, anchored_mod, perishable, xattr, cvs_inject;
const char* pat;
size_t pat_len;
if (!parse_rule_syntax(p, &kind, &sides, &sides_explicit, &negate, &anchored_mod, &perishable,
&xattr, &cvs_inject, &pat, &pat_len)) {
filter_set_error(err, err_size, "unrecognized filter rule syntax");
return NULL;
}
if (cvs_inject) {
/* The C modifier expands to the CVS defaults in place; the rule itself
carries no pattern and is handled by the caller. */
filter_set_error(err, err_size, "the C modifier is handled by the rule-list parser");
return NULL;
}
if (xattr) {
filter_set_error(err, err_size, "xattr-name filter rules (the x modifier) are not supported");
return NULL;
}
if (kind == RULE_KIND_MERGE || kind == RULE_KIND_DIR_MERGE) {
filter_set_error(err, err_size, "merge/dir-merge rules are handled by the rule-list parser");
return NULL;
}
if (kind == RULE_KIND_CLEAR) {
if (pat_len != 0) {
filter_set_error(err, err_size, "clear takes no pattern");
return NULL;
}
FilterRule* rule = calloc(1, sizeof(FilterRule));
if (!rule) {
filter_set_error(err, err_size, "memory allocation failed");
return NULL;
}
rule->action = FILTER_ACTION_NONE; /* clear marker: no pattern */
rule->sides = 0;
return rule;
}
FilterAction action;
switch (kind) {
case RULE_KIND_INCLUDE:
case RULE_KIND_SHOW:
case RULE_KIND_RISK:
action = FILTER_ACTION_INCLUDE;
break;
default:
action = FILTER_ACTION_EXCLUDE;
break;
}
if (kind == RULE_KIND_HIDE)
sides = FILTER_SIDE_SENDER;
else if (kind == RULE_KIND_SHOW)
sides = FILTER_SIDE_SENDER;
else if (kind == RULE_KIND_PROTECT)
sides = FILTER_SIDE_RECEIVER;
else if (kind == RULE_KIND_RISK)
sides = FILTER_SIDE_RECEIVER;
if (kind == RULE_KIND_HIDE || kind == RULE_KIND_SHOW || kind == RULE_KIND_PROTECT ||
kind == RULE_KIND_RISK)
sides_explicit = true;
/* --delete-excluded turns an unqualified (no explicit s/r) rule into a
sender-side-only rule, so it no longer protects the receiver. */
if (opts && opts->delete_excluded && !sides_explicit)
sides = FILTER_SIDE_SENDER;
if (pat_len == 0) {
filter_set_error(err, err_size, "filter rule has no pattern");
return NULL;
}
bool anchored = anchored_mod;
const char* pat_begin = pat;
if (*pat_begin == '/') {
anchored = true;
pat_begin++;
/* Drop the spaces that could follow the anchor in the "-/ foo" form. */
while (*pat_begin == ' ' || *pat_begin == '\t')
pat_begin++;
pat_len = strlen(pat_begin);
while (pat_len > 0 && (pat_begin[pat_len - 1] == '\n' || pat_begin[pat_len - 1] == '\r'))
pat_len--;
}
if (pat_len == 0) {
filter_set_error(err, err_size, "filter rule has no pattern after '/' anchor");
return NULL;
}
bool dir_only = false;
if (pat_len > 1 && pat_begin[pat_len - 1] == '/') {
dir_only = true;
pat_len--;
}
if (pat_len == 0) {
filter_set_error(err, err_size, "filter rule has no pattern");
return NULL;
}
FilterRule* rule = calloc(1, sizeof(FilterRule));
if (!rule) {
filter_set_error(err, err_size, "memory allocation failed");
return NULL;
}
rule->pattern = malloc(pat_len + 1);
if (!rule->pattern) {
free(rule);
filter_set_error(err, err_size, "memory allocation failed");
return NULL;
}
memcpy(rule->pattern, pat_begin, pat_len);
rule->pattern[pat_len] = '\0';
rule->action = action;
rule->sides = sides;
rule->anchored = anchored;
rule->dir_only = dir_only;
rule->negate = negate;
rule->perishable = perishable;
(void)xattr; /* xattr-name rules never match file/dir names; accepted/ignored */
return rule;
}
/* ---- CVS default excludes (-C and the C modifier) ---- */
typedef struct {
const char* pattern;
@@ -240,12 +424,13 @@ static const CvsDefaultRule CVS_DEFAULTS[] = {
{".svn/", true}, {".git/", true}, {".hg/", true}, {".bzr/", true},
};
static bool cvs_rule_list_append(FilterRuleList* list) {
static bool filter_list_append_cvs(FilterRuleList* list, unsigned sides) {
for (size_t i = 0; i < sizeof(CVS_DEFAULTS) / sizeof(CVS_DEFAULTS[0]); i++) {
FilterRule* rule = calloc(1, sizeof(FilterRule));
if (!rule)
return false;
rule->action = FILTER_ACTION_EXCLUDE;
rule->sides = sides;
rule->dir_only = CVS_DEFAULTS[i].dir_only;
size_t plen = strlen(CVS_DEFAULTS[i].pattern);
if (rule->dir_only && plen > 0 && CVS_DEFAULTS[i].pattern[plen - 1] == '/')
@@ -269,76 +454,236 @@ static bool cvs_rule_list_append(FilterRuleList* list) {
return true;
}
#define FILTER_MAX_MERGE_DEPTH 16
static bool filter_list_parse_append_depth(FilterRuleList* list, const char* line,
const FilterParseOptions* opts, const char* base_dir,
int depth, char* err, size_t err_size);
/* Read a merge file and splice its rules into `list`. A relative path is
* resolved below `base_dir` when given, else used as-is (rsync resolves a
* command-line merge file relative to the current directory). */
static bool filter_list_merge_file(FilterRuleList* list, const char* name,
const FilterParseOptions* opts, const char* base_dir, int depth,
char* err, size_t err_size) {
if (name[0] == '\0') {
filter_set_error(err, err_size, "merge requires a filename");
return false;
}
char* path =
(base_dir && base_dir[0] && name[0] != '/') ? path_cat(base_dir, name) : str_dup(name);
if (!path) {
filter_set_error(err, err_size, "memory allocation failed");
return false;
}
FILE* fp = fopen(path, "r");
if (!fp) {
filter_set_error(err, err_size, "could not read merge file '%s': %s", path, strerror(errno));
free(path);
return false;
}
char* line = NULL;
size_t cap = 0;
bool ok = true;
while (true) {
ssize_t n = utils_getdelim_bounded(fp, &line, &cap, '\n', UTILS_MAX_LINE_LEN);
if (n < 0) {
filter_set_error(err, err_size, "error reading merge file '%s'", path);
ok = false;
break;
}
if (n == 0)
break;
const char* lp = line;
while (*lp == ' ' || *lp == '\t')
lp++;
if (*lp == '\0' || *lp == '\n' || *lp == '\r' || *lp == '#')
continue;
if (!filter_list_parse_append_depth(list, lp, opts, base_dir, depth + 1, err, err_size)) {
ok = false;
break;
}
}
free(line);
fclose(fp);
free(path);
return ok;
}
/* Parse one line and append/merge it into `list`. Handles clear, merge and
* dir-merge at the list level. */
static bool filter_list_parse_append_depth(FilterRuleList* list, const char* line,
const FilterParseOptions* opts, const char* base_dir,
int depth, char* err, size_t err_size) {
if (depth > FILTER_MAX_MERGE_DEPTH) {
filter_set_error(err, err_size, "merge files nested too deeply");
return false;
}
const char* p = line;
while (*p == ' ' || *p == '\t')
p++;
if (*p == '\0' || *p == '\n' || *p == '\r')
return true;
RuleKind kind = RULE_KIND_UNKNOWN;
unsigned sides;
bool sides_explicit, negate, anchored_mod, perishable, xattr, cvs_inject;
const char* pat;
size_t pat_len;
if (!parse_rule_syntax(p, &kind, &sides, &sides_explicit, &negate, &anchored_mod, &perishable,
&xattr, &cvs_inject, &pat, &pat_len)) {
filter_set_error(err, err_size, "unrecognized filter rule syntax: %s", p);
return false;
}
(void)sides_explicit;
(void)negate;
(void)anchored_mod;
(void)perishable;
(void)xattr;
if (cvs_inject) {
/* "C" injects the CVS defaults in place; no pattern is expected. */
return filter_list_append_cvs(list, sides);
}
if (kind == RULE_KIND_CLEAR) {
if (pat_len != 0) {
filter_set_error(err, err_size, "clear takes no pattern");
return false;
}
for (int i = 0; i < list->count; i++)
filter_rule_free(list->items[i]);
list->count = 0;
return true;
}
if (kind == RULE_KIND_MERGE) {
if (pat_len == 0) {
filter_set_error(err, err_size, "merge requires a filename");
return false;
}
char* name = malloc(pat_len + 1);
if (!name) {
filter_set_error(err, err_size, "memory allocation failed");
return false;
}
memcpy(name, pat, pat_len);
name[pat_len] = '\0';
bool ok = filter_list_merge_file(list, name, opts, base_dir, depth, err, err_size);
free(name);
return ok;
}
if (kind == RULE_KIND_DIR_MERGE) {
if (pat_len == 0) {
filter_set_error(err, err_size, "dir-merge requires a filename");
return false;
}
char* name = malloc(pat_len + 1);
if (!name) {
filter_set_error(err, err_size, "memory allocation failed");
return false;
}
memcpy(name, pat, pat_len);
name[pat_len] = '\0';
bool ok = filter_rule_list_add_dir_merge(list, name);
free(name);
if (!ok) {
filter_set_error(err, err_size, "memory allocation failed");
return false;
}
return true;
}
FilterRule* rule = filter_rule_parse(p, opts, err, err_size);
if (!rule)
return false;
if (!filter_rule_list_add(list, rule)) {
filter_rule_free(rule);
filter_set_error(err, err_size, "memory allocation failed");
return false;
}
return true;
}
bool filter_rule_list_parse_append(FilterRuleList* list, const char* line,
const FilterParseOptions* opts, const char* merge_base_dir,
char* err, size_t err_size) {
if (err && err_size > 0)
err[0] = '\0';
if (!list)
return false;
return filter_list_parse_append_depth(list, line, opts, merge_base_dir, 0, err, err_size);
}
FilterRuleList* filter_base_build(const char* const* rule_texts, int rule_count, bool cvs_exclude,
char* err, size_t err_size) {
bool delete_excluded, char* err, size_t err_size) {
if (err && err_size > 0)
err[0] = '\0';
FilterRuleList* list = filter_rule_list_create();
if (!list) {
snprintf(err, err_size, "memory allocation failed");
filter_set_error(err, err_size, "memory allocation failed");
return NULL;
}
FilterParseOptions opts = {.delete_excluded = delete_excluded, .cvs_exclude = cvs_exclude};
for (int i = 0; i < rule_count; i++) {
if (!rule_texts || !rule_texts[i])
continue;
FilterRule* rule = filter_rule_parse(rule_texts[i], err, err_size);
if (!rule) {
if (!filter_rule_list_parse_append(list, rule_texts[i], &opts, NULL, err, err_size)) {
filter_rule_list_free(list);
return NULL;
}
if (!set_rule_owner(rule, "")) {
filter_rule_free(rule);
filter_rule_list_free(list);
snprintf(err, err_size, "memory allocation failed");
return NULL;
}
if (!filter_rule_list_add(list, rule)) {
filter_rule_free(rule);
filter_rule_list_free(list);
snprintf(err, err_size, "memory allocation failed");
return NULL;
}
}
if (cvs_exclude && !cvs_rule_list_append(list)) {
if (cvs_exclude && !filter_list_append_cvs(list, FILTER_SIDE_SENDER | FILTER_SIDE_RECEIVER)) {
filter_rule_list_free(list);
snprintf(err, err_size, "memory allocation failed");
filter_set_error(err, err_size, "memory allocation failed");
return NULL;
}
return list;
}
/* ---- Per-directory .rsync-filter files ---- */
/* ---- Per-directory merge files ---- */
FilterRuleList* filter_file_read(const char* dir_path, const char* owner_rel, bool* exists,
char* err, size_t err_size) {
/* Undo the rules and dir-merge registrations that one merge file appended,
* leaving the caller's earlier content intact. A "clear" rule inside the file
* frees every rule, including the caller's; clamp to the surviving count so
* those already-freed rules are never resurrected and freed a second time. */
static void filter_file_rollback(FilterRuleList* list, int rules_before, int dir_merges_before) {
int first = rules_before < list->count ? rules_before : list->count;
for (int i = first; i < list->count; i++)
filter_rule_free(list->items[i]);
list->count = first;
for (int i = dir_merges_before; i < list->dir_merge_count; i++)
free(list->dir_merge_names[i]);
list->dir_merge_count = dir_merges_before;
}
bool filter_file_append(FilterRuleList* list, const char* dir_path, const char* name,
const char* owner_rel, const FilterParseOptions* opts, bool* exists,
char* err, size_t err_size) {
if (err && err_size > 0)
err[0] = '\0';
if (exists)
*exists = false;
char* filter_path = path_cat(dir_path, ".rsync-filter");
if (!list)
return false;
char* filter_path = path_cat(dir_path, name);
if (!filter_path) {
snprintf(err, err_size, "memory allocation failed");
return NULL;
filter_set_error(err, err_size, "memory allocation failed");
return false;
}
FILE* fp = fopen(filter_path, "r");
free(filter_path);
if (!fp) {
if (errno == ENOENT || errno == ENOTDIR)
return filter_rule_list_create();
return true;
char* escaped_dir = output_escape(dir_path, log_get_8_bit_output());
log_message(LOG_LEVEL_WARNING, "Could not read .rsync-filter in %s: %s",
log_message(LOG_LEVEL_WARNING, "Could not read %s in %s: %s", name,
escaped_dir ? escaped_dir : "<allocation failed>", strerror(errno));
free(escaped_dir);
return filter_rule_list_create();
return true;
}
if (exists)
*exists = true;
FilterRuleList* list = filter_rule_list_create();
if (!list) {
fclose(fp);
snprintf(err, err_size, "memory allocation failed");
return NULL;
}
int rules_before = list->count;
int dir_merges_before = list->dir_merge_count;
char* line = NULL;
size_t line_cap = 0;
bool ok = true;
@@ -346,9 +691,10 @@ FilterRuleList* filter_file_read(const char* dir_path, const char* owner_rel, bo
ssize_t n = utils_getdelim_bounded(fp, &line, &line_cap, '\n', UTILS_MAX_LINE_LEN);
if (n < 0) {
if (errno == EFBIG) {
snprintf(err, err_size, "line in .rsync-filter exceeds %d bytes", (int)UTILS_MAX_LINE_LEN);
filter_set_error(err, err_size, "line in %s exceeds %d bytes", name,
(int)UTILS_MAX_LINE_LEN);
} else {
snprintf(err, err_size, "error reading .rsync-filter: %s", strerror(errno));
filter_set_error(err, err_size, "error reading %s: %s", name, strerror(errno));
}
ok = false;
break;
@@ -360,20 +706,9 @@ FilterRuleList* filter_file_read(const char* dir_path, const char* owner_rel, bo
p++;
if (*p == '\0' || *p == '\n' || *p == '\r' || *p == '#')
continue;
FilterRule* rule = filter_rule_parse(p, err, err_size);
if (!rule) {
ok = false;
break;
}
if (!set_rule_owner(rule, owner_rel)) {
filter_rule_free(rule);
snprintf(err, err_size, "memory allocation failed");
ok = false;
break;
}
if (!filter_rule_list_add(list, rule)) {
filter_rule_free(rule);
snprintf(err, err_size, "memory allocation failed");
/* Merge files inside a per-directory file resolve relative to that
directory. */
if (!filter_list_parse_append_depth(list, p, opts, dir_path, 0, err, err_size)) {
ok = false;
break;
}
@@ -381,12 +716,40 @@ FilterRuleList* filter_file_read(const char* dir_path, const char* owner_rel, bo
free(line);
fclose(fp);
if (!ok) {
filter_file_rollback(list, rules_before, dir_merges_before);
return false;
}
for (int i = rules_before; i < list->count; i++) {
if (!set_rule_owner(list->items[i], owner_rel)) {
filter_set_error(err, err_size, "memory allocation failed");
filter_file_rollback(list, rules_before, dir_merges_before);
return false;
}
}
return true;
}
FilterRuleList* filter_file_read_named(const char* dir_path, const char* name,
const char* owner_rel, const FilterParseOptions* opts,
bool* exists, char* err, size_t err_size) {
FilterRuleList* list = filter_rule_list_create();
if (!list) {
if (err && err_size > 0)
filter_set_error(err, err_size, "memory allocation failed");
return NULL;
}
if (!filter_file_append(list, dir_path, name, owner_rel, opts, exists, err, err_size)) {
filter_rule_list_free(list);
return NULL;
}
return list;
}
FilterRuleList* filter_file_read(const char* dir_path, const char* owner_rel, bool* exists,
char* err, size_t err_size) {
return filter_file_read_named(dir_path, ".rsync-filter", owner_rel, NULL, exists, err, err_size);
}
/* ---- Rule matching ---- */
/* Match a pattern that contains '/' (non-anchored) against the end of the
@@ -402,10 +765,10 @@ static bool glob_suffix_match(const char* pattern, const char* str) {
}
static FilterAction rule_matches(const FilterRule* rule, const char* rel_path, const char* leaf,
bool is_dir) {
bool is_dir, unsigned side) {
if (!rule || !rule->pattern)
return FILTER_ACTION_NONE;
if (rule->dir_only && !is_dir)
if (!(rule->sides & side))
return FILTER_ACTION_NONE;
/* A rule applies only to entries below its owner directory. */
const char* rel2 = rel_path;
@@ -420,24 +783,36 @@ static FilterAction rule_matches(const FilterRule* rule, const char* rel_path, c
if (rel2[0] == '\0')
return FILTER_ACTION_NONE;
bool matched;
if (rule->anchored) {
if (rule->dir_only && !is_dir)
matched = false;
else if (rule->anchored)
matched = glob_match(rule->pattern, rel2);
} else if (strchr(rule->pattern, '/') != NULL) {
else if (strchr(rule->pattern, '/') != NULL)
matched = glob_suffix_match(rule->pattern, rel2);
} else {
else
matched = glob_match(rule->pattern, leaf);
}
return matched ? rule->action : FILTER_ACTION_NONE;
if (rule->negate)
matched = !matched;
if (!matched)
return FILTER_ACTION_NONE;
if (side == FILTER_SIDE_RECEIVER)
return rule->action == FILTER_ACTION_EXCLUDE ? FILTER_ACTION_PROTECT : FILTER_ACTION_RISK;
return rule->action;
}
FilterAction filter_rules_apply(const FilterRuleList* list, const char* rel_path, const char* leaf,
bool is_dir) {
FilterAction filter_rules_apply_side(const FilterRuleList* list, const char* rel_path,
const char* leaf, bool is_dir, unsigned side) {
if (!list)
return FILTER_ACTION_NONE;
for (int i = 0; i < list->count; i++) {
FilterAction action = rule_matches(list->items[i], rel_path, leaf, is_dir);
FilterAction action = rule_matches(list->items[i], rel_path, leaf, is_dir, side);
if (action != FILTER_ACTION_NONE)
return action;
}
return FILTER_ACTION_NONE;
}
FilterAction filter_rules_apply(const FilterRuleList* list, const char* rel_path, const char* leaf,
bool is_dir) {
return filter_rules_apply_side(list, rel_path, leaf, is_dir, FILTER_SIDE_SENDER);
}
+96 -42
View File
@@ -4,79 +4,133 @@
#include <stdbool.h>
#include <stddef.h>
/* rsync-style filter rule engine (client-side file selection).
/* rsync-style filter rule engine (client-side file selection and the
* receiver-side protection set it feeds).
*
* Supported rule syntax (documented subset):
* [+|-] [anchored '/' prefix] pattern [trailing '/' for dir-only]
*
* "+ PATTERN" include rule (first match wins)
* "- PATTERN" exclude rule
* "PATTERN" implicit exclude rule (rsync default)
* "include PATTERN" / "exclude PATTERN" word forms
* leading '/' after the +/- anchors the pattern to its owner directory
* (the transfer root for command-line/-C rules, the directory that
* contains a .rsync-filter file for per-directory rules)
* a trailing '/' makes the rule match directories only
*
* Rejected explicitly (no silent no-ops): the rsync merge/dir-merge/list-clear
* shorthands written as a rule that starts with ':' or '.' or '!', the
* merge/dir-merge/hide/show/protect/risk/clear words, and every include/exclude
* rule modifier other than '/' (! C s r p x). The pattern must be separated
* from +/- by a space (or a single '/' anchor), exactly like rsync's
* "-s foo"/"-p ..." modifier syntax is refused.
* Rule syntax (see the rsync man page FILTER RULES section):
* RULE [PATTERN_OR_FILENAME]
* RULE,MODIFIERS [PATTERN_OR_FILENAME]
* Short RULE names may attach MODIFIERS directly ("-sr foo"); the long name
* form requires the comma. The pattern/filename is separated from the rule by
* one space or underscore. Rule names:
* exclude/- exclude (by default both sender-hide and receiver-protect)
* include/+ include (by default both sender-show and receiver-risk)
* hide/H sender-only exclude
* show/S sender-only include
* protect/P receiver-only exclude (protect from deletion)
* risk/R receiver-only include (allow deletion)
* merge/. read a client-side merge file for more rules
* dir-merge/: per-directory merge file (registered for the scanner)
* clear/! clear the current rule list (takes no argument)
* Modifiers: '/' absolute anchor, '!' negate match, 'C' inject CVS defaults,
* 's' sender side, 'r' receiver side, 'p' perishable, 'x' xattr name rule.
* A trailing '/' makes a pattern match directories only. A leading '/' anchors
* the pattern to its owner directory.
*/
typedef enum {
FILTER_ACTION_NONE = 0, /* no rule matched */
FILTER_ACTION_EXCLUDE = -1,
FILTER_ACTION_INCLUDE = 1
FILTER_ACTION_INCLUDE = 1,
/* Receiver-side-only verdicts: the entry is transferred but its destination
* mirror is protected from --delete (PROTECT) or explicitly left at risk
* (RISK). */
FILTER_ACTION_PROTECT = 2,
FILTER_ACTION_RISK = 3,
} FilterAction;
typedef struct {
FilterAction action;
bool anchored; /* pattern anchored to the rule's owner directory */
bool dir_only; /* pattern had a trailing '/': matches directories only */
char* owner; /* owning directory rel path ("" == transfer root) */
char* pattern; /* cleaned glob pattern (no leading '/', no trailing '/') */
} FilterRule;
#define FILTER_SIDE_SENDER 1u
#define FILTER_SIDE_RECEIVER 2u
typedef struct {
FilterAction action; /* EXCLUDE or INCLUDE (the base pattern action) */
unsigned sides; /* FILTER_SIDE_SENDER | FILTER_SIDE_RECEIVER */
bool anchored; /* pattern anchored to the rule's owner directory */
bool dir_only; /* pattern had a trailing '/': matches directories only */
bool negate; /* '!' modifier: match succeeds when the pattern does not */
bool perishable; /* 'p' modifier (ignored in deleted directories) */
char* owner; /* owning directory rel path ("" == transfer root) */
char* pattern; /* cleaned glob pattern (no leading '/', no trailing '/') */
} FilterRule;
typedef struct FilterRuleList {
FilterRule** items; /* owned array of rule pointers */
int count;
int capacity;
/* Per-directory merge-file basenames registered by "dir-merge NAME"/": NAME"
* or by -F (.rsync-filter). Owned strings; the scanner reads each name in
* every directory it traverses. */
char** dir_merge_names;
int dir_merge_count;
int dir_merge_capacity;
} FilterRuleList;
/* Context needed while parsing a rule list (merge files, --delete-excluded). */
typedef struct {
bool delete_excluded; /* --delete-excluded: default sides become sender-only */
bool cvs_exclude; /* -C: expand the CVS default excludes */
} FilterParseOptions;
/* Parse a single filter-rule line (no trailing newline required). Returns an
* owned rule, or NULL on unsupported/invalid syntax with a message in `err`. */
FilterRule* filter_rule_parse(const char* line, char* err, size_t err_size);
* owned rule, or NULL on unsupported/invalid syntax with a message in `err`.
* `opts` may be NULL (no merge expansion / no delete-excluded). */
FilterRule* filter_rule_parse(const char* line, const FilterParseOptions* opts, char* err,
size_t err_size);
void filter_rule_free(FilterRule* rule);
FilterRuleList* filter_rule_list_create(void);
/* Append a fully-parsed rule (takes ownership). Returns false on OOM. */
bool filter_rule_list_add(FilterRuleList* list, FilterRule* rule);
/* Parse `line` and append it. Returns false and fills `err` on bad syntax. */
bool filter_rule_list_parse_append(FilterRuleList* list, const char* line, char* err,
size_t err_size);
/* Register a per-directory merge-file basename (idempotent). Returns false on
* OOM. Used by the scanner to read custom "dir-merge" files. */
bool filter_rule_list_add_dir_merge(FilterRuleList* list, const char* name);
/* Parse `line` and append it. Handles "clear"/"!" (resets the list), "merge
* FILE"/". FILE" (splices the file's rules) and "dir-merge NAME"/": NAME"
* (registers a per-directory filename). Returns false and fills `err` on bad
* syntax or an unreadable merge file. `merge_base_dir` resolves a relative
* merge-file path (NULL means the process working directory). */
bool filter_rule_list_parse_append(FilterRuleList* list, const char* line,
const FilterParseOptions* opts, const char* merge_base_dir,
char* err, size_t err_size);
void filter_rule_list_free(FilterRuleList* list);
/* Build the command-line filter set: `rule_texts` (--filter=RULE in the order
* given, 0..rule_count) followed by the -C CVS default excludes when
* cvs_exclude is true. All rules are owned by "" (the transfer root).
* cvs_exclude is true. All rules are owned by "" (the transfer root).
* Returns NULL on unsupported rule text (message in `err`). */
FilterRuleList* filter_base_build(const char* const* rule_texts, int rule_count, bool cvs_exclude,
char* err, size_t err_size);
bool delete_excluded, char* err, size_t err_size);
/* Read "<dir_path>/.rsync-filter" and return its rules, each owned by
* `owner_rel`. A missing file yields an empty list with *exists=false; an
* unreadable file is treated as missing. Returns NULL only on parse or
* allocation failure (message in `err`). */
/* Read "<dir_path>/<name>" and return its rules, each owned by `owner_rel`. A
* missing file yields an empty list with *exists=false; an unreadable file is
* treated as missing. Returns NULL only on parse or allocation failure
* (message in `err`). `opts` may be NULL. */
FilterRuleList* filter_file_read_named(const char* dir_path, const char* name,
const char* owner_rel, const FilterParseOptions* opts,
bool* exists, char* err, size_t err_size);
/* Append the rules of "<dir_path>/<name>" into an existing list (each owned by
* `owner_rel`). A missing file yields *exists=false and no error. Returns
* false only on parse/allocation failure (message in `err`). */
bool filter_file_append(FilterRuleList* list, const char* dir_path, const char* name,
const char* owner_rel, const FilterParseOptions* opts, bool* exists,
char* err, size_t err_size);
/* filter_file_read_named with the default ".rsync-filter" name. */
FilterRuleList* filter_file_read(const char* dir_path, const char* owner_rel, bool* exists,
char* err, size_t err_size);
/* Evaluate an entry against one ordered rule list. Returns FILTER_ACTION_NONE
* when no rule matched, otherwise the first matching rule's action.
* `rel_path` is the entry's path relative to the transfer root ("" == root),
* `leaf` its final name, `is_dir` whether it is a directory. */
/* Evaluate an entry against one ordered rule list for one side. Returns
* FILTER_ACTION_NONE when no rule matched, otherwise the first matching rule's
* action (for the receiver side an EXCLUDE is reported as
* FILTER_ACTION_PROTECT and an INCLUDE as FILTER_ACTION_RISK). `rel_path` is
* the entry's path relative to the transfer root ("" == root), `leaf` its final
* name, `is_dir` whether it is a directory. */
FilterAction filter_rules_apply_side(const FilterRuleList* list, const char* rel_path,
const char* leaf, bool is_dir, unsigned side);
/* Sender-side convenience wrapper (kept for callers/tests that only need the
* transfer decision). */
FilterAction filter_rules_apply(const FilterRuleList* list, const char* rel_path, const char* leaf,
bool is_dir);
+28
View File
@@ -101,3 +101,31 @@ bool format_dest_state_receive(int fd, OutputDestState* state) {
state->gid = gid;
return true;
}
bool format_stats_send(int fd, const ReceiverStats* stats) {
if (!stats)
return false;
unsigned long long fields[8] = {
stats->matched_data, stats->deleted_files, stats->would_delete_count, stats->literal_bytes,
stats->created_reg, stats->created_dir, stats->created_link, stats->created_special,
};
return send_n_data(fd, fields, sizeof(fields));
}
bool format_stats_receive(int fd, ReceiverStats* stats) {
if (!stats)
return false;
unsigned long long fields[8] = {0};
if (!receive_n_data(fd, fields, sizeof(fields)))
return false;
memset(stats, 0, sizeof(*stats));
stats->matched_data = fields[0];
stats->deleted_files = fields[1];
stats->would_delete_count = fields[2];
stats->literal_bytes = fields[3];
stats->created_reg = fields[4];
stats->created_dir = fields[5];
stats->created_link = fields[6];
stats->created_special = fields[7];
return true;
}
+52
View File
@@ -56,4 +56,56 @@ bool format_rsync_datetime(time_t when, bool dash, char* buffer, size_t buffer_s
bool format_dest_state_send(int fd, const OutputDestState* state);
bool format_dest_state_receive(int fd, OutputDestState* state);
/* End-of-transfer receiver counters reported through STATUS_STATS (protocol
* 2.25.0, extended in 2.28.0) when the wire config carries report_stats.
* `would_delete_count` is the number of destination-relative paths the receiver
* would have deleted in a -n/--dry-run --delete run; that many wire strings
* immediately follow the fixed record (sent/read by the caller).
*
* Protocol 2.28.0 adds the receiver-observed counters the sender cannot see:
* `literal_bytes` is the file data the receiver actually stored literally
* (whole files plus the literal fragments of a delta) and the four `created_*`
* counters split the destination entries the receiver newly created by type,
* reproducing rsync's `Number of created files` breakdown and an exact
* `Literal data` for a delta run. */
typedef struct {
unsigned long long matched_data;
unsigned long long deleted_files;
unsigned long long would_delete_count;
unsigned long long literal_bytes;
unsigned long long created_reg;
unsigned long long created_dir;
unsigned long long created_link;
unsigned long long created_special;
} ReceiverStats;
/* Fixed-width STATUS_STATS counter record. The status frame and the optional
* would-delete path list are sent/received by the caller. Returns false on I/O
* failure. */
bool format_stats_send(int fd, const ReceiverStats* stats);
bool format_stats_receive(int fd, ReceiverStats* stats);
/* Sender-side file-list accounting for rsync's `--stats` block. Filled while
* the scan/send loops walk each entry: the flist counters describe every
* scanned source entry (transferred or skipped), while the transferred/literal
* counters describe only the regular files the receiver actually stored. The
* type split lets the client print rsync's `Number of files` breakdown; the
* receiver-only counters (matched data, deleted, created) come from
* STATUS_STATS. */
typedef struct {
unsigned long long flist_reg;
unsigned long long flist_dir;
unsigned long long flist_link;
unsigned long long flist_special;
unsigned long long total_file_size; /* sum of entry sizes (link target len) */
unsigned long long transferred_regular; /* regular files actually stored */
unsigned long long transferred_file_size; /* source size of those files */
/* Whole-file accuracy: the `--stats` "Literal data" row. The sender counts
* the source size of every stored file, so a whole-file transfer matches
* rsync. A delta run actually ships only the literal fragments of the diff
* (the rest is matched/copied), so here the value is an upper bound, not
* rsync's literal-byte total; see RSYNC_COMPAT.md's `--stats` row. */
unsigned long long literal_data;
} TransferStats;
#endif
+68 -21
View File
@@ -589,6 +589,67 @@ static int identity_split_chown(const char* value, char** puser, char** pgroup)
return 0;
}
/* --chown is rsync's shorthand for "--usermap=*:USER --groupmap=*:GROUP", so a
* name TO value must be resolved on the RECEIVER, not on the sender. Append the
* equivalent map rule (FROM matches every id). The numeric/'*' forms are stored
* numerically exactly as rsync's id_parse/user_to_uid would. Returns 0 on
* success, -1 on a malformed numeric token or allocation failure. */
static int identity_append_chown_rule(Config* config, bool is_group, const char* token) {
IdentityMap rule;
memset(&rule, 0, sizeof(rule));
rule.from = IDENTITY_MATCH_ANY;
rule.from_hi = IDENTITY_MATCH_ANY;
if (strcmp(token, "*") == 0) {
rule.to = IDENTITY_CURRENT;
} else if (identity_all_digits(token[0] == '@' ? token + 1 : token)) {
if (identity_resolve_token(token, is_group, &rule.to) != 0) {
log_message(LOG_LEVEL_ERROR, "--chown numeric id is out of range: %s", token);
return -1;
}
} else {
rule.to = 0;
rule.to_name = str_dup(token);
if (!rule.to_name)
return -1;
}
if (identity_append_rule(is_group ? &config->groupmap : &config->usermap,
is_group ? &config->groupmap_count : &config->usermap_count,
&rule) != 0) {
free(rule.to_name);
log_message(LOG_LEVEL_ERROR, "--chown has too many rules (max %d)", MAX_IDENTITY_MAP);
return -1;
}
return 0;
}
/* Resolve/record one --chown side. The source-side numeric value is kept in
* chown_uid/chown_gid purely as a fallback (the appended map rule resolves the
* name on the receiver and wins); a name that does not exist on the sender is
* accepted and left to receiver-side resolution, matching rsync. */
static int identity_parse_chown_side(Config* config, bool is_group, const char* token) {
if (identity_append_chown_rule(config, is_group, token) != 0)
return -1;
bool numeric = identity_all_digits(token[0] == '@' ? token + 1 : token);
int32_t resolved;
if (identity_resolve_token(token, is_group, &resolved) == 0) {
if (is_group) {
config->chown_gid = resolved;
config->chown_gid_set = true;
} else {
config->chown_uid = resolved;
config->chown_uid_set = true;
}
return 0;
}
if (numeric) {
log_message(LOG_LEVEL_ERROR, "--chown could not resolve numeric id '%s'", token);
return -1;
}
/* Unknown sender-side name: rsync accepts it and resolves it (or warns) on
* the receiver; do the same instead of failing the whole run. */
return 0;
}
int identity_parse_chown(Config* config, const char* value) {
if (!config || !value || *value == '\0') {
log_message(LOG_LEVEL_ERROR, "--chown requires a value (USER:GROUP, USER, or :GROUP)");
@@ -627,32 +688,18 @@ int identity_parse_chown(Config* config, const char* value) {
if (*user == '\0') {
log_message(LOG_LEVEL_ERROR, "--chown requires a user or group (got '%s')", value);
ret = -1;
} else if (identity_resolve_token(user, false, &config->chown_uid) != 0) {
log_message(LOG_LEVEL_ERROR,
"--chown could not resolve user '%s' (use a name that exists "
"on the source, '*', or @N)",
value);
} else if (identity_parse_chown_side(config, false, user) != 0) {
ret = -1;
} else {
config->chown_uid_set = true;
}
} else {
/* --chown=USER:GROUP, --chown=:GROUP, --chown=USER: */
if (*user != '\0') {
if (identity_resolve_token(user, false, &config->chown_uid) != 0) {
log_message(LOG_LEVEL_ERROR, "--chown could not resolve user '%s'", value);
ret = -1;
goto done;
}
config->chown_uid_set = true;
if (*user != '\0' && identity_parse_chown_side(config, false, user) != 0) {
ret = -1;
goto done;
}
if (*group != '\0') {
if (identity_resolve_token(group, true, &config->chown_gid) != 0) {
log_message(LOG_LEVEL_ERROR, "--chown could not resolve group '%s'", value);
ret = -1;
goto done;
}
config->chown_gid_set = true;
if (*group != '\0' && identity_parse_chown_side(config, true, group) != 0) {
ret = -1;
goto done;
}
if (!*user && !*group) {
log_message(LOG_LEVEL_ERROR, "--chown must set a user, a group, or both (got '%s')", value);
+20 -1
View File
@@ -21,7 +21,26 @@ typedef enum {
LOG_INFO_MISC = 1u << 1,
LOG_INFO_SKIP = 1u << 2,
LOG_INFO_STATS = 1u << 3,
LOG_INFO_ALL = LOG_INFO_COPY | LOG_INFO_MISC | LOG_INFO_SKIP | LOG_INFO_STATS,
/* rsync categories that map to a FastSync event (emitted in rsync's line
* format): del (deletions), remove (sender-side source removal), name
* (transferred entry names), flist (file-list header), nonreg (skipped
* non-regular files), progress (per-file progress). rsync's `backup`
* category is accepted for CLI parity but stays silent: the receiver does the
* backing-up and FastSync has no backup event to report from the sender. */
LOG_INFO_DEL = 1u << 4,
LOG_INFO_REMOVE = 1u << 5,
LOG_INFO_NAME = 1u << 6,
LOG_INFO_FLIST = 1u << 7,
LOG_INFO_NONREG = 1u << 8,
LOG_INFO_PROGRESS = 1u << 9,
/* Marker for `--info=name2` and higher: also print rsync's
"NAME is uptodate" line for entries the receiver already has. It rides in
the info_level bitset (there is no separate Config field) and is never set
by --info=all (which selects level 1). */
LOG_INFO_NAME_UPTODATE = 1u << 10,
LOG_INFO_ALL = LOG_INFO_COPY | LOG_INFO_MISC | LOG_INFO_SKIP | LOG_INFO_STATS | LOG_INFO_DEL |
LOG_INFO_REMOVE | LOG_INFO_NAME | LOG_INFO_FLIST | LOG_INFO_NONREG |
LOG_INFO_PROGRESS,
} LogInfoFlag;
void log_message(LogLevel log_level, const char* message, ...);
+8
View File
@@ -32,14 +32,18 @@ PipelineContextSender* pipeline_context_sender_create(Config* config, Queue* que
context->excluded_paths = NULL;
context->size_skipped_paths = NULL;
context->synced_dirs = NULL;
context->plan_dirs = NULL;
context->missing_args = NULL;
context->scan_had_io_error = false;
context->remove_source_files = NULL;
context->early_delete = false;
context->delete_plans = NULL;
context->delete_suppressed = false;
context->scan_stopped_early = false;
context->total_files = 0;
context->progress_bytes = 0;
context->total_bytes = 0;
memset(&context->stats, 0, sizeof(context->stats));
context->sender_done = false;
atomic_init(&context->cancelled, false);
protocol_session_init(&context->allocation_session, -1, -1);
@@ -187,12 +191,16 @@ void pipeline_context_sender_destroy(PipelineContextSender* context) {
if (context->manifest) {
array_list_delete(context->manifest);
}
if (context->delete_plans)
delete_plan_sender_destroy(context->delete_plans);
if (context->excluded_paths)
array_list_delete(context->excluded_paths);
if (context->size_skipped_paths)
array_list_delete(context->size_skipped_paths);
if (context->synced_dirs)
array_list_delete(context->synced_dirs);
if (context->plan_dirs)
array_list_delete(context->plan_dirs);
if (context->missing_args)
array_list_delete(context->missing_args);
if (context->remove_source_files)
+28 -4
View File
@@ -7,7 +7,9 @@
#include "array_list.h"
#include "chunk.h"
#include "config.h"
#include "delete_plan.h"
#include "file.h"
#include "format.h"
#include "protocol.h"
#include "queue.h"
#include "stop_condition.h"
@@ -54,6 +56,11 @@ typedef struct {
"delete only in synchronized directories" (notably for --files-from).
Populated by the scanner thread or the early pre-scan. */
ArrayList* synced_dirs;
/* Destination-relative paths of every traversed source directory, for the
per-directory delete plan keep set (so an empty source directory survives
--delete rather than being removed as an extra). Prebuilt by the path-only
pre-scan on the calling thread. */
ArrayList* plan_dirs;
/* --delete-missing-args: the destination-relative mirrors of the --files-from
entries that are missing under the source. Computed by the preflight on
the calling thread before the pipeline starts; the sender thread transmits
@@ -66,15 +73,32 @@ typedef struct {
--ignore-errors kept the run going. */
bool scan_had_io_error;
ArrayList* remove_source_files;
/* True when --delete-before/--delete-during require the keep-set manifest to
be transmitted before any file data: context->manifest is then prebuilt by
a path-only pre-scan on the calling thread and the pipeline scanner must
not append to it. Set once before the worker threads start. */
/* True when --delete-before requires the whole-tree keep-set manifest to be
transmitted before any file data: context->manifest is then prebuilt by a
path-only pre-scan on the calling thread and the pipeline scanner must not
append to it. Set once before the worker threads start. */
bool early_delete;
/* Non-NULL for --delete-during/--delete-delay: the per-directory plan set
prebuilt by the path-only pre-scan on the calling thread. The sender
thread transmits the root plan before any data and the remaining plans
alongside the chunks. Set once before the worker threads start. */
DeletePlanSender* delete_plans;
/* A scan I/O error without --ignore-errors suppressed deletion: the prebuilt
keep-set/plans were dropped, and the streaming scanner must not build a
fresh manifest or re-send the per-directory plans. Set once before the
worker threads start. */
bool delete_suppressed;
mtx_t mutex_progress;
int total_files;
unsigned long long progress_bytes;
unsigned long long total_bytes;
/* Per-type flist / transferred accounting for the rsync --stats breakdown and
the progress `to-chk` denominator. Owned by the sender thread: it is the
only writer (the entry/transfer notes in send_chunks_multithreaded) and it
reads the totals in its completion tail, so no lock is needed. This is NOT
guarded by mutex_progress (which covers total_files/progress_bytes/
total_bytes/sender_done). */
TransferStats stats;
bool sender_done;
atomic_bool cancelled;
ProtocolSession allocation_session;
+45 -3
View File
@@ -29,6 +29,13 @@ static unsigned long long io_bwlimit = 0;
static mtx_t bw_mutex;
static once_flag bw_mutex_once = ONCE_FLAG_INIT;
/* Process-wide wire byte counters, used by the client to render rsync's
* --stats/--progress totals and the --out-format %b/%c tokens. The zero-copy
* sendfile path bypasses protocol_send_n_data, so it reports its bytes through
* protocol_note_bytes_written. */
static atomic_ullong io_bytes_written = 0;
static atomic_ullong io_bytes_read = 0;
static unsigned long long global_bwlimit(void);
static bool protocol_reserve_memory(ProtocolSession* session, size_t charge) {
@@ -176,12 +183,28 @@ void io_set_bwlimit(unsigned long long bytes_per_sec) {
mtx_unlock(&bw_mutex);
}
unsigned long long io_get_bwlimit(void) {
return global_bwlimit();
}
/* rsync's throttle (io.c sleep_for_bwlimit) sleeps once its unslept debt
* reaches ~100 ms of bandwidth, so its effective initial burst is about 0.1 s
* worth of bytes, not a full second. FastSync models the same with a token
* bucket whose capacity is bwlimit/10, so a throttled run paces like rsync
* instead of sending a full second's worth up front. */
static long long bw_burst_capacity(unsigned long long bwlimit) {
if (bwlimit == 0)
return 0;
long long burst = (long long)(bwlimit / 10);
return burst > 0 ? burst : 1;
}
void protocol_session_set_bwlimit(ProtocolSession* session, unsigned long long bytes_per_sec) {
if (!session)
return;
session->bwlimit =
bytes_per_sec > (unsigned long long)LLONG_MAX ? (unsigned long long)LLONG_MAX : bytes_per_sec;
session->bw_tokens = (long long)session->bwlimit;
session->bw_tokens = bw_burst_capacity(session->bwlimit);
struct timespec now;
clock_gettime(CLOCK_MONOTONIC, &now);
session->bw_last_refill_sec = now.tv_sec;
@@ -215,8 +238,9 @@ static void bw_throttle_session(ProtocolSession* session, size_t bytes_written)
long long tokens_to_add = (long long)((double)session->bwlimit * elapsed_ns / 1000000000.0);
session->bw_tokens += tokens_to_add;
if (session->bw_tokens > (long long)session->bwlimit)
session->bw_tokens = (long long)session->bwlimit;
long long burst = bw_burst_capacity(session->bwlimit);
if (session->bw_tokens > burst)
session->bw_tokens = burst;
session->bw_tokens -= bytes_written;
@@ -227,7 +251,11 @@ static void bw_throttle_session(ProtocolSession* session, size_t bytes_written)
poll(NULL, 0, (int)(deficit_us / 1000));
else
usleep((useconds_t)deficit_us);
/* Reset the bucket AFTER the sleep: crediting the sleep duration as elapsed
refill time would cancel half the throttle (the next call would see the
whole sleep as refill and immediately grant a fresh burst). */
session->bw_tokens = 0;
clock_gettime(CLOCK_MONOTONIC, &now);
session->bw_last_refill_sec = now.tv_sec;
session->bw_last_refill_nsec = now.tv_nsec;
}
@@ -242,6 +270,18 @@ SSL* io_get_ssl(void) {
return io_ssl;
}
unsigned long long protocol_bytes_written(void) {
return atomic_load(&io_bytes_written);
}
unsigned long long protocol_bytes_read(void) {
return atomic_load(&io_bytes_read);
}
void protocol_note_bytes_written(unsigned long long bytes) {
atomic_fetch_add(&io_bytes_written, bytes);
}
static ProtocolSession* legacy_session(int read_fd, int write_fd) {
if (bound_session)
return bound_session;
@@ -343,6 +383,7 @@ bool protocol_send_n_data(ProtocolSession* session, const void* data, size_t dat
wait_events = POLLOUT;
}
log_debug_message(LOG_DEBUG_IO, " Send n Data: %zu", total_bytes_send);
atomic_fetch_add(&io_bytes_written, (unsigned long long)total_bytes_send);
return true;
}
@@ -430,6 +471,7 @@ static bool protocol_receive_n_data_until(ProtocolSession* session, void* data,
wait_events = POLLIN;
}
log_debug_message(LOG_DEBUG_IO, " Received n Data: %zu", total_bytes_received);
atomic_fetch_add(&io_bytes_read, (unsigned long long)total_bytes_received);
return true;
}
+33 -1
View File
@@ -181,14 +181,46 @@ enum NET_STATUS {
* (new vs modified, and which of size/time/perms/owner/group differ) without
* changing the transfer decision itself. Appended after
* STATUS_DELETE_LIMIT so no existing status is renumbered. */
STATUS_DEST_INFO
STATUS_DEST_INFO,
/* Per-directory delete plan (protocol 2.24.0). The sender of a
* --delete-during/--delete-delay transfer streams one frame per source
* directory in directory order instead of a single whole-tree keep-set
* manifest. The receiver applies the plan when it arrives
* (--delete-during removes that directory's extras immediately) or records
* the extras and applies them only after the whole transfer succeeded
* (--delete-delay). Payload: an int32 has_config flag (1 on the first plan
* of the run, 0 afterwards); when set, the three global config sections
* (protected-prefix count+paths, size-skipped count+paths, missing-args
* count+paths); then an int32 apply flag (1 for a real plan, 0 for a
* config-only carrier frame that must not walk a directory); then the
* destination-relative directory path wire string
* ("." for the receive root); then the child-directory count + names and the
* child-file count + names that must be kept. Appended after
* STATUS_DEST_INFO so no existing status is renumbered. */
STATUS_DELETE_PLAN,
/* End-of-transfer receiver counter report (protocol 2.25.0). When the wire
* config carries report_stats=true, the receiver sends this status once,
* immediately before its terminal success status, followed by a fixed stats
* record (see format_stats_send/receive in format.h) and, when the run is a
* --dry-run with --delete, the would-delete path list. Appended after
* STATUS_DELETE_PLAN so no existing status is renumbered. */
STATUS_STATS
};
void io_set_fds(int read_fd, int write_fd);
void io_set_bwlimit(unsigned long long bytes_per_sec);
unsigned long long io_get_bwlimit(void);
void io_set_ssl(SSL* ssl);
SSL* io_get_ssl(void);
/* Process-wide wire byte counters. protocol_send_n_data/protocol_receive_n_data
* update them; the zero-copy sendfile path reports through
* protocol_note_bytes_written. Used by the client to render rsync's
* --stats/--progress totals and the --out-format %b/%c tokens. */
unsigned long long protocol_bytes_written(void);
unsigned long long protocol_bytes_read(void);
void protocol_note_bytes_written(unsigned long long bytes);
void protocol_session_init(ProtocolSession* session, int read_fd, int write_fd);
/* Transitional bridge for helpers whose signatures still carry only an fd. */
void protocol_session_bind(ProtocolSession* session);
+264 -14
View File
@@ -2,6 +2,7 @@
#include "array_list.h"
#include "log.h"
#include <arpa/inet.h>
#include <ctype.h>
#include <dirent.h>
#include <errno.h>
#include <fcntl.h>
@@ -52,6 +53,31 @@ bool path_is_within_root(const char* root, const char* path) {
return strncmp(root, path, root_len) == 0 && (path[root_len] == '\0' || path[root_len] == '/');
}
/* Borrowed transfer-relative view of `path`: strip any leading '/' and then a
* `root` prefix (its own leading/trailing slashes tolerated), returning a
* pointer into `path`. Non-allocating, so it is safe on the hot scan/print
* paths. A NULL/empty root, or a path not under `root`, leaves only the
* leading-slash strip. `path` must be NUL-terminated and live in the caller. */
const char* utils_strip_transfer_root(const char* path, const char* root) {
if (path == NULL)
return NULL;
const char* rel = path;
while (*rel == '/')
rel++;
if (root == NULL)
return rel;
while (*root == '/')
root++;
size_t root_len = strlen(root);
while (root_len > 0 && root[root_len - 1] == '/')
root_len--;
if (root_len == 0)
return rel;
if (strncmp(rel, root, root_len) == 0 && (rel[root_len] == '/' || rel[root_len] == '\0'))
return rel + root_len + (rel[root_len] == '/' ? 1 : 0);
return rel;
}
/* Open the destination root directory itself, confined to the authorized root.
* NOTE (do not merge with file_open_secure_parent): this walk opens dest_root
* (a directory that must already exist) and returns its fd, whereas
@@ -60,7 +86,7 @@ bool path_is_within_root(const char* root, const char* path) {
* two differ in create-vs-no-create, in what path component they stop at, and
* in the extra receiver policies they apply, so they are intentionally kept
* separate. Both rely on the shared lexical path_is_within_root check. */
static int open_authorized_destination(const char* dest_root) {
int utils_open_authorized_destination(const char* dest_root) {
int root_fd = utils_get_authorized_root_fd();
const char* root_path = utils_get_authorized_root_path();
if (root_fd < 0 || !root_path || !dest_root || !path_is_within_root(root_path, dest_root))
@@ -117,6 +143,47 @@ char* str_dup(const char* string) {
return new_string;
}
int env_choice_first(const char* env_name, int (*resolve)(const char*), bool* specified) {
if (specified)
*specified = false;
if (!env_name || !resolve)
return -1;
const char* env = getenv(env_name);
if (!env)
return -1;
bool saw_nonblank = false;
const char* p = env;
while (*p) {
if (*p == '&')
break;
if (isspace((unsigned char)*p)) {
p++;
continue;
}
saw_nonblank = true;
char token[64];
size_t len = 0;
while (*p && *p != '&' && !isspace((unsigned char)*p)) {
if (len < sizeof(token) - 1)
token[len++] = *p;
p++;
}
token[len] = '\0';
if (len > 0) {
int id = resolve(token);
if (id >= 0) {
if (specified)
*specified = true;
return id;
}
}
}
if (specified)
*specified = saw_nonblank;
return -1;
}
#define STR_HASH_SET_MIN_CAPACITY 16
static size_t str_hash_set_hash(const char* key, size_t len) {
@@ -615,8 +682,10 @@ static bool is_synced_dir(const PathIndex* dirs, const char* rel) {
like any other non-directory extra (never followed). */
static bool delete_extras_fd(int dirfd, const char* rel_path, const PathIndex* keep,
const PathIndex* dirs, DeleteBudget* budget,
const DeleteSkipEntry* skips, int skip_count, bool parent_deletable,
bool* all_removed) {
const DeleteSkipEntry* skips, int skip_count,
const FilterRuleList* protect_rules, bool parent_deletable,
bool* all_removed, DeletePathObserver observer,
void* observer_context) {
/* openat(dirfd, ".") opens an independent file description: a dup() would
share dirfd's file offset and a prior pass could leave the stream drained. */
int scanfd = openat(dirfd, ".", O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
@@ -661,12 +730,23 @@ static bool delete_extras_fd(int dirfd, const char* rel_path, const PathIndex* k
free(child_rel);
continue;
}
if (S_ISDIR(st.st_mode)) {
bool is_dir = S_ISDIR(st.st_mode);
if (protect_rules && filter_rules_apply_side(protect_rules, child_rel, entry->d_name, is_dir,
FILTER_SIDE_RECEIVER) == FILTER_ACTION_PROTECT) {
/* A first-match protect rule shields the extra; for a directory the whole
subtree is shielded (rsync prunes an excluded directory), so do not
descend. */
local_survives = true;
free(child_rel);
continue;
}
if (is_dir) {
int childfd = openat(dirfd, entry->d_name, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
bool child_all_removed = false;
if (childfd >= 0) {
if (!delete_extras_fd(childfd, child_rel, keep, dirs, budget, skips, skip_count, deletable,
&child_all_removed))
if (!delete_extras_fd(childfd, child_rel, keep, dirs, budget, skips, skip_count,
protect_rules, deletable, &child_all_removed, observer,
observer_context))
operation_ok = false;
close(childfd);
} else if (errno != ENOENT) {
@@ -693,6 +773,8 @@ static bool delete_extras_fd(int dirfd, const char* rel_path, const PathIndex* k
local_survives = true;
} else {
budget->deleted++;
if (observer)
observer(observer_context, child_rel);
}
} else {
local_survives = true;
@@ -713,6 +795,8 @@ static bool delete_extras_fd(int dirfd, const char* rel_path, const PathIndex* k
local_survives = true;
} else {
budget->deleted++;
if (observer)
observer(observer_context, child_rel);
char* escaped_path = output_escape(child_rel, log_get_8_bit_output());
fprintf(stderr, " Deleted: %s\n", escaped_path ? escaped_path : "<allocation failed>");
free(escaped_path);
@@ -725,10 +809,165 @@ static bool delete_extras_fd(int dirfd, const char* rel_path, const PathIndex* k
return operation_ok;
}
DeleteWalkResult delete_extras_limited(const char* dest_root, const ArrayList* manifest,
const ArrayList* synced_dirs, size_t max_delete,
const DeleteSkipEntry* skips, int skip_count,
size_t* deleted_out, size_t* skipped_out) {
/* Read-only mirror of delete_extras_fd: records the paths that WOULD be removed
without unlinking anything. A child directory is reported after its own
reportable children (depth-first), matching the delete pass's ordering. */
static bool list_extras_fd(int dirfd, const char* rel_path, const PathIndex* keep,
const PathIndex* dirs, ArrayList* out, size_t* recorded,
const DeleteSkipEntry* skips, int skip_count,
const FilterRuleList* protect_rules, bool parent_deletable,
bool* all_removed) {
int scanfd = openat(dirfd, ".", O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
if (scanfd < 0)
return false;
DIR* dir = fdopendir(scanfd);
if (!dir) {
close(scanfd);
return false;
}
bool operation_ok = true;
bool local_survives = false;
bool deletable = parent_deletable || is_synced_dir(dirs, rel_path);
const struct dirent* entry;
while ((entry = readdir(dir)) != NULL) {
if (strcmp(entry->d_name, ".") == 0 || strcmp(entry->d_name, "..") == 0)
continue;
char* child_rel = path_cat((char*)rel_path, entry->d_name);
if (!child_rel) {
operation_ok = false;
continue;
}
if (path_under_skip_prefix(child_rel, rel_path[0] == '\0', skips, skip_count)) {
local_survives = true;
free(child_rel);
continue;
}
struct stat st;
if (fstatat(dirfd, entry->d_name, &st, AT_SYMLINK_NOFOLLOW) != 0) {
if (errno != ENOENT)
operation_ok = false;
free(child_rel);
continue;
}
bool is_dir = S_ISDIR(st.st_mode);
if (protect_rules && filter_rules_apply_side(protect_rules, child_rel, entry->d_name, is_dir,
FILTER_SIDE_RECEIVER) == FILTER_ACTION_PROTECT) {
/* Mirror the delete walk: a protected entry is never reported as a
would-delete and a protected directory's subtree is not enumerated. */
local_survives = true;
free(child_rel);
continue;
}
if (is_dir) {
int childfd = openat(dirfd, entry->d_name, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
bool child_all_removed = false;
if (childfd >= 0) {
if (!list_extras_fd(childfd, child_rel, keep, dirs, out, recorded, skips, skip_count,
protect_rules, deletable, &child_all_removed))
operation_ok = false;
close(childfd);
} else if (errno != ENOENT) {
operation_ok = false;
}
bool child_synced = dirs && path_index_contains(dirs, child_rel);
if (child_synced || keep_is_dir(keep, child_rel)) {
local_survives = true;
} else if (child_all_removed && deletable) {
size_t len = strlen(child_rel);
char* copy = malloc(len + 2);
if (!copy) {
operation_ok = false;
} else {
memcpy(copy, child_rel, len);
copy[len] = '/';
copy[len + 1] = '\0';
if (!array_list_add(out, copy)) {
free(copy);
operation_ok = false;
} else {
(*recorded)++;
}
}
} else {
local_survives = true;
}
} else {
bool found = keep_is_file(keep, child_rel);
if (found || !deletable) {
local_survives = true;
} else {
char* copy = str_dup(child_rel);
if (!copy || !array_list_add(out, copy)) {
free(copy);
operation_ok = false;
} else {
(*recorded)++;
}
}
}
free(child_rel);
}
closedir(dir);
*all_removed = !local_survives;
return operation_ok;
}
bool delete_extras_list(const char* dest_root, const ArrayList* manifest,
const ArrayList* synced_dirs, const DeleteSkipEntry* skips, int skip_count,
const FilterRuleList* protect_rules, ArrayList* out, size_t* count_out) {
if (count_out)
*count_out = 0;
if (!manifest || !out)
return false;
PathIndex keep;
if (!build_keep_index(manifest, &keep))
return false;
PathIndex dirs;
bool have_dirs = synced_dirs != NULL;
if (have_dirs &&
!path_index_build(&dirs, (const char* const*)synced_dirs->items, (size_t)synced_dirs->size)) {
path_index_free(&keep);
return false;
}
int rootfd;
int root_fd = utils_get_authorized_root_fd();
if (root_fd >= 0) {
if (utils_get_authorized_root_path())
rootfd = utils_open_authorized_destination(dest_root);
else if (dest_root == NULL)
rootfd = dup(root_fd);
else
rootfd = -1;
} else {
rootfd = open(dest_root, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
}
if (rootfd < 0) {
path_index_free(&keep);
if (have_dirs)
path_index_free(&dirs);
return false;
}
bool all_removed = false;
size_t recorded = 0;
bool ok = list_extras_fd(rootfd, "", &keep, have_dirs ? &dirs : NULL, out, &recorded, skips,
skip_count, protect_rules, false, &all_removed);
if (close(rootfd) != 0)
ok = false;
path_index_free(&keep);
if (have_dirs)
path_index_free(&dirs);
if (count_out)
*count_out = recorded;
return ok;
}
DeleteWalkResult delete_extras_limited_observed(const char* dest_root, const ArrayList* manifest,
const ArrayList* synced_dirs, size_t max_delete,
const DeleteSkipEntry* skips, int skip_count,
const FilterRuleList* protect_rules,
size_t* deleted_out, size_t* skipped_out,
DeletePathObserver observer,
void* observer_context) {
if (deleted_out)
*deleted_out = 0;
if (skipped_out)
@@ -752,7 +991,7 @@ DeleteWalkResult delete_extras_limited(const char* dest_root, const ArrayList* m
int root_fd = utils_get_authorized_root_fd();
if (root_fd >= 0) {
if (utils_get_authorized_root_path())
rootfd = open_authorized_destination(dest_root);
rootfd = utils_open_authorized_destination(dest_root);
else if (dest_root == NULL)
rootfd = dup(root_fd);
else
@@ -768,8 +1007,9 @@ DeleteWalkResult delete_extras_limited(const char* dest_root, const ArrayList* m
}
DeleteBudget budget = {.max_delete = max_delete, .deleted = 0, .skipped = 0, .limit_hit = false};
bool all_removed = false;
bool ok = delete_extras_fd(rootfd, "", &keep, have_dirs ? &dirs : NULL, &budget, skips,
skip_count, false, &all_removed);
bool ok =
delete_extras_fd(rootfd, "", &keep, have_dirs ? &dirs : NULL, &budget, skips, skip_count,
protect_rules, false, &all_removed, observer, observer_context);
if (close(rootfd) != 0)
ok = false;
path_index_free(&keep);
@@ -784,8 +1024,18 @@ DeleteWalkResult delete_extras_limited(const char* dest_root, const ArrayList* m
return budget.limit_hit ? DELETE_WALK_LIMIT_REACHED : DELETE_WALK_OK;
}
DeleteWalkResult delete_extras_limited(const char* dest_root, const ArrayList* manifest,
const ArrayList* synced_dirs, size_t max_delete,
const DeleteSkipEntry* skips, int skip_count,
const FilterRuleList* protect_rules, size_t* deleted_out,
size_t* skipped_out) {
return delete_extras_limited_observed(dest_root, manifest, synced_dirs, max_delete, skips,
skip_count, protect_rules, deleted_out, skipped_out, NULL,
NULL);
}
bool delete_extras(const char* dest_root, const ArrayList* manifest) {
return delete_extras_limited(dest_root, manifest, NULL, SIZE_MAX, NULL, 0, NULL, NULL) ==
return delete_extras_limited(dest_root, manifest, NULL, SIZE_MAX, NULL, 0, NULL, NULL, NULL) ==
DELETE_WALK_OK;
}
+51 -1
View File
@@ -2,6 +2,7 @@
#define UTILS_H
#include "array_list.h"
#include "filter.h"
#include <stddef.h>
#include <stdbool.h>
#include <stdio.h>
@@ -81,6 +82,17 @@ bool path_index_has_descendant(const PathIndex* index, const char* path);
char* str_dup(const char* string);
char* output_escape(const char* string, bool eight_bit_output);
/* Resolve the first supported name from a rsync algorithm-preference
* environment variable (RSYNC_COMPRESS_LIST / RSYNC_CHECKSUM_LIST). `resolve`
* maps a case-insensitive name to an algorithm id (>= 0) or -1 for an unknown
* name. rsync's syntax is a whitespace-separated list (comma/colon are NOT
* separators); the client-side half ends at '&'. Unknown entries are skipped
* and the first resolvable one wins. *specified is set true when the variable
* holds at least one non-blank character. Returns the first resolvable id, or
* -1 when the variable is unset/blank or names no supported algorithm. */
int env_choice_first(const char* env_name, int (*resolve)(const char*), bool* specified);
/* Upper bound on one line/token read from a local list file (--files-from,
* --exclude-from/--include-from, .rsync-filter). Mirrors MAX_STRING_SIZE and
* stops a hostile multi-gigabyte line from forcing unbounded allocation. */
@@ -137,8 +149,41 @@ bool path_under_skip_prefix(const char* child_rel, bool at_root, const DeleteSki
DeleteWalkResult delete_extras_limited(const char* dest_root, const ArrayList* manifest,
const ArrayList* synced_dirs, size_t max_delete,
const DeleteSkipEntry* skips, int skip_count,
size_t* deleted_out, size_t* skipped_out);
const FilterRuleList* protect_rules, size_t* deleted_out,
size_t* skipped_out);
/* Optional per-deletion observer: called for each destination-relative path
actually removed (a file, symlink, or directory), in removal order, so the
receiver can stream rsync's `--info=del`/`--info=remove` lines. */
typedef void (*DeletePathObserver)(void* context, const char* rel_path);
/* `delete_extras_limited_observed` is delete_extras_limited with an optional
* observer; the observer is invoked only for entries truly removed. When
* `protect_rules` is non-NULL its receiver-side verdict is evaluated for every
* candidate extra: a first-match PROTECT leaves the entry (and, for a
* directory, its whole subtree) in place, while RISK/NONE fall through to the
* ordinary skip-prefix/keep-set logic. */
DeleteWalkResult delete_extras_limited_observed(const char* dest_root, const ArrayList* manifest,
const ArrayList* synced_dirs, size_t max_delete,
const DeleteSkipEntry* skips, int skip_count,
const FilterRuleList* protect_rules,
size_t* deleted_out, size_t* skipped_out,
DeletePathObserver observer,
void* observer_context);
/* Read-only companion to delete_extras_limited: walk the destination exactly as
the delete pass would and APPEND (strdup'd) destination-relative paths that
WOULD be removed, without touching disk. Used for -n/--dry-run --delete
would-delete reporting. Returns true on a clean walk; the caller owns the
strings appended to `out` and receives their count in *count_out. */
bool delete_extras_list(const char* dest_root, const ArrayList* manifest,
const ArrayList* synced_dirs, const DeleteSkipEntry* skips, int skip_count,
const FilterRuleList* protect_rules, ArrayList* out, size_t* count_out);
bool delete_extras(const char* dest_root, const ArrayList* manifest);
/* Open the existing destination directory at `dest_root`, confined to the
authorized root with an O_NOFOLLOW component walk (the same confinement the
deletion walker uses for its root). Returns a new fd the caller owns, or -1
on error (including a destination that does not exist). */
int utils_open_authorized_destination(const char* dest_root);
bool utils_set_authorized_root(int fd, const char* canonical_path);
/* The fd-only compatibility form is fail-closed for path-based operations;
* callers should use utils_set_authorized_root with the canonical identity. */
@@ -164,6 +209,11 @@ const char* utils_get_authorized_root_path(void);
* callers guarantee this); this is containment by string, not by resolved
* symlinks. Shared by the utils and file secure-walk root confinement. */
bool path_is_within_root(const char* root, const char* path);
/* Non-allocating transfer-relative view of `path`: strip any leading '/' and
* then a `root` prefix (leading/trailing slashes tolerated), returning a
* borrowed pointer into `path`. A NULL/empty root, or a path not under
* `root`, yields just the leading-slash strip. `path`/`root` must stay alive. */
const char* utils_strip_transfer_root(const char* path, const char* root);
/* True when `path` contains a ".." component. This is a purely lexical
* dot-dot check: an absolute path is NOT rejected here, because default
* (non-relative) transfers legitimately put the sender's absolute source path
+4 -4
View File
@@ -409,10 +409,10 @@ bool fake_super_restore_fd(int fd, FileAttrPolicy policy) {
(void)ul_gid;
/* Mode is applied only when the per-attribute policy asks for it, through the
SAME shared helper the normal metadata path uses (metadata_mode_for_policy):
group/other write bits are never granted, so a recorded source mode of 0666
restores as 0644 — identical to a non-fake-super --preserve run, never a
privilege-granting regression — and the -E rule derives exec bits from the
destination's read bits exactly like file_restore_metadata_fd. */
under --perms the recorded source mode is copied exactly, including
group/other write and setuid/setgid/sticky bits (rsync parity), and the -E
rule derives exec bits from the destination's read bits exactly like
file_restore_metadata_fd. */
if (policy.perms || policy.executability) {
struct stat cur;
mode_t want = 0;
+3 -2
View File
@@ -105,8 +105,9 @@ void fake_super_store_fd(int fd, uint32_t uid, uint32_t gid, uint32_t mode, int6
* absence of the xattr or a malformed record is a silent no-op that never fails
* the transfer. The MODE leg is applied only when policy.perms||policy.
* executability and the MTIME leg only when policy.times, so the fake-super
* replay cannot bypass the per-attribute split; the mode is sanitized exactly
* like the normal metadata path (group/other write bits never granted).
* replay cannot bypass the per-attribute split; the mode follows the normal
* metadata path exactly (under --perms the source mode is copied verbatim,
* special and group/other write bits included).
* Returns true when the xattr was present and parsed. */
bool fake_super_restore_fd(int fd, FileAttrPolicy policy);
+1
View File
@@ -0,0 +1 @@
OLDDEST
@@ -0,0 +1 @@
NEWCONTENT
+1
View File
@@ -0,0 +1 @@
NEWCONTENT
+7
View File
@@ -119,6 +119,13 @@ static void build_canonical_frame(void) {
cfg->usermap[0].to = MAP_TO;
cfg->usermap[0].to_name = NULL;
}
/* Force a non-empty receiver delete-protection block so the fuzzer mutates
* its rule count, action/sides codes and pattern strings. */
cfg->filters = array_list_create(free);
if (cfg->filters) {
array_list_add(cfg->filters, str_dup("P *.log"));
array_list_add(cfg->filters, str_dup("+r keep/**"));
}
if (!cfg->send_directory || !cfg->receive_root_directory || !cfg->usermap) {
config_delete(cfg);
return;
+80
View File
@@ -0,0 +1,80 @@
# Integration tests
The integration suite drives the built `build/server` and `build/client`
against local corpora. Unit tests live in `tests/` (the custom C framework);
the Python suite here covers the full transfer pipeline, transports, features,
and rsync parity.
## Running
```bash
# Full suite (excludes privilege-dependent tests on CI runners)
python3 -m pytest tests/integration/ -n 4 --dist=load -m "not setpriv"
# Fast PR subset only
python3 -m pytest tests/integration/ -n 4 --dist=load -m ci
```
The tests expect `build/server` and `build/client` (configure/build with CMake
first); `common.py` derives `BUILD_DIR` from the repository root.
## Differential rsync-parity gate
`test_differential_parity.py` runs the **same** transfer with real
`rsync 3.4.1` and with FastSync over separate destinations, then compares:
- the destination trees — relative paths, file content hashes, symlink
targets, modes (where the case is about perms), and hard-link grouping;
- the normalized stdout for output-oriented flags (`-i`,
`--out-format=...`, `--stats`), after stripping volatile fields
(timings, rates, wire byte counts) and directory-only itemize lines that
FastSync's recursive scanner documents as absent.
FastSync mirrors the absolute source path under its receive root (see
`get_dest_received_dir`); the harness normalizes that layout (and the
`-R`/`--files-from` layouts) before comparing.
```bash
# Fast subset that guards the ✅ surface on pull requests
python3 -m pytest tests/integration/test_differential_parity.py -n 4 --dist=load -m parity_ci
# Full differential case table (`_CASES`): every row is marked `parity`, and a
# case with an allowlisted residual in `parity_caveats.py` is included too.
python3 -m pytest tests/integration/test_differential_parity.py -n 4 --dist=load -m parity
```
`-m parity` selects only the `_CASES` table in this module. Differential
coverage for options outside that table (`--temp-dir`, `--delay-updates`,
`--dry-run`, `--fuzzy`, the basis-dir options, `-M` over daemon/TCP, and
receiver filter-protect) lives in dedicated modules (`test_option_parity.py`,
`test_parity_blockers.py`, `test_parity_quickwins.py`, ...) and is not part of
this gate. The suite skips cleanly when `rsync` is not installed.
## Allowlist (`parity_caveats.py`)
`parity_caveats.py` is the single data-driven allowlist of known differences.
Each entry maps a case id to the aspects that may differ (`tree`, `stdout`,
`extra`, `rc`) and cites the governing row in `RSYNC_COMPAT.md`:
```python
CAVEATS = {
# no known residuals at present -- the burn-down reached zero
# "some_case_id": {"tree": "documented residual ... ref: RSYNC_COMPAT.md ..."},
}
```
A differential mismatch in an aspect that is **not** listed fails the gate with
a readable tree/stdout diff.
If a case is allowlisted but now matches rsync, the gate emits a loud warning
naming the stale entry — that is the parity burn-down signal. Run with
`FASTSYNC_PARITY_STRICT=1` to make stale entries fail instead (the full CI
parity job sets this). To add a residual:
1. Reproduce it with `-m parity` and read the failure's tree/stdout diff.
2. Confirm it is a documented `⚠️`/`❌` residual (or get the `✅` row
reclassified) and cite the row.
3. Add the case id and aspect(s) to `CAVEATS`, keeping the reason concise.
Do not allowlist an undocumented divergence from a `✅` row — fix it or get the
row reclassified first.
+11 -5
View File
@@ -101,9 +101,15 @@ class CountingProxy:
return
counter[0] += len(data)
def run(self, cmd):
def run(self, cmd, join_timeout=20):
"""Forward one client run (the full command list) to the real server and
return the CompletedProcess after the counts have settled."""
return the CompletedProcess after the counts have settled.
``join_timeout`` bounds how long to wait for the forwarding threads. The
client->server count is published as soon as the client side reaches EOF
(i.e. once the client process has exited), so callers that only need that
count can pass a small value instead of waiting for the server to close
its idle socket."""
def serve():
try:
@@ -119,15 +125,15 @@ class CountingProxy:
a.start()
b.start()
a.join()
b.join()
self.client_to_server = c2s[0]
b.join()
self.server_to_client = s2c[0]
self._listener.close()
thread = threading.Thread(target=serve)
thread = threading.Thread(target=serve, daemon=True)
thread.start()
result = subprocess.run(cmd, capture_output=True, text=True, timeout=180)
thread.join(20)
thread.join(join_timeout)
return result
+34
View File
@@ -0,0 +1,34 @@
"""Data-driven allowlist for the differential rsync-parity gate.
Every entry maps a case id (see ``test_differential_parity.py``) to the aspects
that are *known* to differ from ``rsync 3.4.1`` and the documented reason. A
differential mismatch in an aspect that is **not** listed here fails the gate.
Aspect keys
-----------
``tree`` destination tree differs (paths, file hashes, symlink targets,
modes, hardlink grouping)
``stdout`` normalized output for ``-i`` / ``--stats`` / ``--out-format``
``extra`` a case-specific assertion differs (basis/inode checks, ...)
``rc`` exit status differs
Burn-down
---------
If a case is listed here but now matches rsync, the gate emits a loud
``pytest`` warning naming the stale entry: delete the entry (and, when the
underlying row in ``RSYNC_COMPAT.md`` is now parity, update that row). Set
``FASTSYNC_PARITY_STRICT=1`` to turn stale entries into failures in CI.
Keep the values concise but cite the governing row so the entry can be
re-triaged when the row moves.
"""
# case id -> {aspect: "reason (ref: RSYNC_COMPAT.md ...)"}
CAVEATS = {}
# Accepted aspect names (guards against typos in this file).
ASPECTS = ("tree", "stdout", "extra", "rc")
def caveat_for(case_id: str) -> dict:
return CAVEATS.get(case_id, {})
+584
View File
@@ -0,0 +1,584 @@
"""Differential rsync-parity harness.
Runs the SAME transfer with real ``rsync`` and with FastSync over separate
destinations and compares the resulting trees and (optionally) normalized
stdout. ``test_differential_parity.py`` drives this module with a table of
cases; ``parity_caveats.py`` is the data-driven allowlist of documented
residuals.
Design notes
------------
FastSync mirrors the *absolute* source path below its receive root, while
rsync copies the source contents directly into the destination. ``Case.layout``
tells the harness which pair of directory roots to compare:
* ``MIRROR`` -- rsync ``DEST/`` vs FastSync ``DEST/<abs-src>/`` (the common
case; matches ``common.get_dest_received_dir``).
* ``MIRROR_ABS`` -- ``rsync -R`` without a cut lays the full absolute path
under the destination, so rsync ``DEST/<abs-src>/`` is compared against the
same FastSync mirror path.
* ``RELATIVE`` -- ``rsync -R --files-from`` lays bare relative paths under the
destination and FastSync does the same, so both destination roots compare
directly.
Only ``tests/integration/common.py`` is used to reach the build products and the
server manager; the harness never duplicates that plumbing.
"""
import difflib
import hashlib
import os
import re
import shutil
import subprocess
import sys
from dataclasses import dataclass
from typing import Callable, Dict, List, Optional, Tuple
sys.path.insert(0, os.path.dirname(__file__))
from common import ( # noqa: E402 (path bootstrap above)
TEST_DATA_DIR,
clean_dir,
get_dest_received_dir,
run_client,
)
RSYNC = shutil.which("rsync")
# Comparison layouts (see module docstring).
MIRROR = "mirror"
MIRROR_ABS = "mirror_abs"
RELATIVE = "relative"
# stdout comparators.
STDOUT_NONE = None
STDOUT_ITEMIZE = "itemize"
STDOUT_OUTFMT = "outfmt"
STDOUT_STATS = "stats"
STDOUT_PROGRESS = "progress"
# rsync --stats lines that are protocol-independent and must match exactly.
# `Number of files` and `Number of created files` carry rsync's per-type
# breakdown; protocol 2.28.0 reports the receiver-created split over
# STATUS_STATS. Deliberately excluded: Total bytes sent/received (protocol
# framing differs, see the `--stats` row in RSYNC_COMPAT.md).
STATS_KEYS = (
"Number of files",
"Number of created files",
"Number of deleted files",
"Number of regular files transferred",
"Total file size",
"Total transferred file size",
"Literal data",
"Matched data",
"File list size",
)
_ITEMIZE_RE = re.compile(r"^(<|>|c|h|\.|\*)[fdLDS][.+\-][.+\-][.+\-][.+\-]")
_PROGRESS_TOTAL_RE = re.compile(r"to-chk=\d+/(\d+)")
_PROGRESS_XFR_RE = re.compile(r"xfr#(\d+)")
@dataclass
class Case:
"""One differential scenario: a corpus, a flag set, and how to compare."""
id: str
corpus: str
flags: List[str]
fastsync_flags: Optional[List[str]] = None
layout: str = MIRROR
server_args: Tuple[str, ...] = ("--allow-super",)
seed: Optional[Callable] = None
stdout: Optional[str] = STDOUT_NONE
compare_modes: bool = False
compare_hardlinks: bool = False
ignore_paths: Tuple[str, ...] = ()
extra_check: Optional[Callable] = None
files_from: Optional[Tuple[str, ...]] = None
# rsync receives ``src + "/"``; FastSync mirrors the path it is given, so a
# trailing-slash-sensitive case must hand FastSync the same form.
fs_src_suffix: str = ""
# Some cases have an unspecified result (e.g. which extras survive a
# partial --max-delete abort): assert the case-specific invariants via
# extra_check and skip the exact-tree comparison.
compare_tree: bool = True
ci: bool = False
ref: str = ""
def fs_flags(self) -> List[str]:
return list(self.flags if self.fastsync_flags is None else self.fastsync_flags)
# ---------------------------------------------------------------------------
# Corpora
# ---------------------------------------------------------------------------
# Deterministic mtimes so quick-check decisions are reproducible.
_SRC_MTIME = 1_600_000_000
def _write(path: str, data: bytes, mode: Optional[int] = None) -> None:
os.makedirs(os.path.dirname(path), exist_ok=True)
with open(path, "wb") as fh:
fh.write(data)
os.utime(path, (_SRC_MTIME, _SRC_MTIME))
if mode is not None:
os.chmod(path, mode)
def _set_mode(path: str, mode: int) -> None:
os.chmod(path, mode)
def corpus_basic(root: str) -> None:
"""Regular files + nested dirs (dirs are implied by their files)."""
clean_dir(root)
_write(os.path.join(root, "a.txt"), b"hello world\n")
_write(os.path.join(root, "sub", "b.bin"),
bytes((i * 7) & 0xFF for i in range(5000)))
_write(os.path.join(root, "sub", "deep", "c.txt"), "w\u00f6rld\n".encode())
def corpus_unicode(root: str) -> None:
clean_dir(root)
_write(os.path.join(root, "uni \u00f1\u6587.txt"), b"unicode\n")
_write(os.path.join(root, "sub", "sp ace \u00e9.dat"), b"spaced\n")
_set_mode(os.path.join(root, "sub"), 0o750)
def corpus_links(root: str) -> None:
corpus_basic(root)
os.symlink("a.txt", os.path.join(root, "rel_link"))
os.symlink("/etc/hostname", os.path.join(root, "abs_link"))
os.symlink("nowhere/target", os.path.join(root, "broken_link"))
def corpus_hardlinks(root: str) -> None:
clean_dir(root)
_write(os.path.join(root, "h1.txt"), b"hardlinked payload\n")
os.link(os.path.join(root, "h1.txt"), os.path.join(root, "h2.txt"))
_write(os.path.join(root, "other.txt"), b"other\n")
def corpus_sparse(root: str) -> None:
clean_dir(root)
_write(os.path.join(root, "small.txt"), b"small\n")
sparse = os.path.join(root, "sparse.bin")
with open(sparse, "wb") as fh:
fh.seek(1024 * 1024 - 1)
fh.write(b"\0")
os.utime(sparse, (_SRC_MTIME, _SRC_MTIME))
def corpus_filters(root: str) -> None:
clean_dir(root)
_write(os.path.join(root, "keep.txt"), b"keep\n")
_write(os.path.join(root, "drop.log"), b"log\n")
_write(os.path.join(root, "sub", "keep2.txt"), b"keep2\n")
_write(os.path.join(root, "sub", "drop2.log"), b"log2\n")
_write(os.path.join(root, "sub", "data.bin"), b"bin\n")
def corpus_empty_dir(root: str) -> None:
clean_dir(root)
_write(os.path.join(root, "keep.txt"), b"keep\n")
os.makedirs(os.path.join(root, "emptydir"), exist_ok=True)
os.utime(os.path.join(root, "emptydir"), (_SRC_MTIME, _SRC_MTIME))
_write(os.path.join(root, "nonempty", "f.txt"), b"f\n")
def corpus_multidir(root: str) -> None:
"""Multi-directory tree for the --progress file-list naming/denominator.
Nested files, a directory-only branch, an empty directory and a symlink
exercise every file-list entry type rsync counts in `to-chk` but FastSync's
streaming scanner never emits as a transfer entry.
"""
clean_dir(root)
_write(os.path.join(root, "a.txt"), b"alpha\n")
_write(os.path.join(root, "b.txt"), b"bravo\n")
_write(os.path.join(root, "sub1", "c.txt"), b"charlie\n")
_write(os.path.join(root, "sub1", "deep", "d.txt"), b"delta\n")
_write(os.path.join(root, "sub2", "e.txt"), b"echo\n")
os.symlink("a.txt", os.path.join(root, "link1"))
os.makedirs(os.path.join(root, "emptydir"), exist_ok=True)
os.utime(os.path.join(root, "emptydir"), (_SRC_MTIME, _SRC_MTIME))
def corpus_relative(root: str) -> None:
"""Tree for the -R/--files-from cases."""
clean_dir(root)
_write(os.path.join(root, "a.txt"), b"a\n")
_write(os.path.join(root, "b.txt"), b"b\n")
_write(os.path.join(root, "sub", "x.txt"), b"x\n")
_write(os.path.join(root, "sub", "y.txt"), b"y\n")
os.makedirs(os.path.join(root, "dir1"), exist_ok=True)
os.utime(os.path.join(root, "dir1"), (_SRC_MTIME, _SRC_MTIME))
_write(os.path.join(root, "dir1", "keep.txt"), b"keep\n")
def corpus_iconv(root: str) -> None:
"""Latin-1 (ISO-8859-1) encoded filenames, matching the --iconv direction."""
clean_dir(root)
for rel, data in ((b"caf\xe9.txt", b"caf\xe9\n"),
(os.path.join(b"sub", b"\xfcber.txt"), b"\xfcber\n")):
full = os.path.join(os.fsencode(root), rel)
os.makedirs(os.path.dirname(full), exist_ok=True)
with open(full, "wb") as fh:
fh.write(data)
os.utime(full, (_SRC_MTIME, _SRC_MTIME))
# Payload for the --fuzzy basis corpus: large enough for the delta engine's
# 16 KiB minimum and with repeated content so a coinciding basis yields a
# non-zero (and identical) Matched data count in both tools.
FUZZY_PAYLOAD = (b"the quick brown fox jumps over the lazy dog\n" * 2000)[:65536]
def corpus_fuzzy(root: str) -> None:
"""A named regular file; the `fuzzy` seed adds the similar-suffix sibling."""
clean_dir(root)
_write(os.path.join(root, "report_v2.txt"), FUZZY_PAYLOAD)
CORPORA: Dict[str, Callable[[str], None]] = {
"basic": corpus_basic,
"unicode": corpus_unicode,
"links": corpus_links,
"hardlinks": corpus_hardlinks,
"sparse": corpus_sparse,
"filters": corpus_filters,
"empty_dir": corpus_empty_dir,
"multidir": corpus_multidir,
"relative": corpus_relative,
"iconv": corpus_iconv,
"fuzzy": corpus_fuzzy,
}
# ---------------------------------------------------------------------------
# Tree snapshotting / comparison
# ---------------------------------------------------------------------------
def snapshot(root: str, compare_modes: bool = False) -> Dict[str, tuple]:
"""Map relative path -> descriptor for every entry below ``root``.
Files hash their contents with SHA-256 (structural comparison, so differing
quick-check metadata cannot mask a payload difference). Symlinks record
their target. Empty directories are included (as ``("dir", ...)``) so the
recursive-empty-directory residual is observable.
"""
out: Dict[str, tuple] = {}
if not os.path.isdir(root):
return out
def describe(path: str) -> Optional[tuple]:
st = os.lstat(path)
if os.path.islink(path):
return ("link", os.readlink(path))
if os.path.isdir(path):
mode = oct(st.st_mode & 0o7777) if compare_modes else None
return ("dir", mode)
h = hashlib.sha256()
with open(path, "rb") as fh:
for chunk in iter(lambda: fh.read(65536), b""):
h.update(chunk)
mode = oct(st.st_mode & 0o7777) if compare_modes else None
return ("file", h.hexdigest()[:16], mode)
# The comparison root itself is not part of the tree diff: a no-transfer
# result legitimately leaves FastSync's mirror directory absent while rsync
# leaves an existing (empty) destination root.
for dirpath, dirnames, filenames in os.walk(root, followlinks=False):
dirnames.sort()
for name in sorted(dirnames):
p = os.path.join(dirpath, name)
rel = os.path.relpath(p, root)
if os.path.islink(p):
out[rel] = ("link", os.readlink(p))
dirnames.remove(name)
else:
out[rel] = describe(p)
for name in sorted(filenames):
p = os.path.join(dirpath, name)
out[os.path.relpath(p, root)] = describe(p)
return out
def _hardlink_groups(root: str) -> Dict[str, str]:
"""Assign a stable group letter to each inode shared by >1 regular file."""
inodes: Dict[tuple, List[str]] = {}
for dirpath, _dirs, filenames in os.walk(root, followlinks=False):
for name in filenames:
p = os.path.join(dirpath, name)
if os.path.islink(p):
continue
st = os.lstat(p)
if st.st_nlink > 1:
inodes.setdefault((st.st_dev, st.st_ino), []).append(
os.path.relpath(p, root))
groups: Dict[str, str] = {}
for i, (_key, members) in enumerate(sorted(inodes.items())):
for rel in members:
groups[rel] = chr(ord("A") + i)
return groups
def _drop_ignored(tree: Dict[str, tuple], ignore_paths) -> Dict[str, tuple]:
if not ignore_paths:
return tree
out = {}
for rel, desc in tree.items():
if any(rel == ig or rel.startswith(ig.rstrip("/") + "/") for ig in ignore_paths):
continue
out[rel] = desc
return out
def tree_diff(rsync_root: str, fs_root: str, case: Case) -> List[str]:
"""Return a list of human-readable differences (empty when identical)."""
rtree = _drop_ignored(snapshot(rsync_root, case.compare_modes), case.ignore_paths)
ftree = _drop_ignored(snapshot(fs_root, case.compare_modes), case.ignore_paths)
if case.compare_hardlinks:
rgroups = _hardlink_groups(rsync_root)
fgroups = _hardlink_groups(fs_root)
else:
rgroups = fgroups = {}
diffs: List[str] = []
for rel in sorted(set(rtree) | set(ftree)):
r = rtree.get(rel)
f = ftree.get(rel)
if r == f:
continue
if r is None:
diffs.append(f"+ fastsync-only: {rel!r} {f}")
elif f is None:
diffs.append(f"- rsync-only: {rel!r} {r}")
else:
diffs.append(f"~ differs: {rel!r} rsync={r} fastsync={f}")
if case.compare_hardlinks:
for rel in sorted(set(rgroups) | set(fgroups)):
if rgroups.get(rel) != fgroups.get(rel):
diffs.append(
f"~ hardlink group: {rel!r} rsync={rgroups.get(rel)} "
f"fastsync={fgroups.get(rel)}")
return diffs
# ---------------------------------------------------------------------------
# stdout normalization
# ---------------------------------------------------------------------------
def _parse_bytes(text: str) -> str:
m = re.match(r"([\d,]+)", text.strip())
return m.group(1).replace(",", "") if m else text.strip()
def normalize_stdout(text: str, mode: Optional[str]) -> object:
if mode == STDOUT_ITEMIZE:
lines = []
for line in (text or "").splitlines():
line = line.rstrip()
if not line:
continue
if line.startswith("*deleting"):
lines.append(line)
continue
if not _ITEMIZE_RE.match(line):
continue
# Directories are not transfer entries in FastSync's recursive
# scanner, so rsync's `cd+++++++++ name/` lines have no counterpart
# (documented recursive-empty-dir residual). Compare file/link
# itemization only.
if line.rsplit(" ", 1)[-1].endswith("/"):
continue
lines.append(line)
return sorted(lines)
if mode == STDOUT_OUTFMT:
lines = []
for line in (text or "").splitlines():
line = line.rstrip()
if not line:
continue
# Directory entries are emitted by rsync but not by FastSync's
# recursive scanner (documented residual). Tokens are either
# `%n %l` (path first) or `%i %n` (path last); drop a line when
# either end-token is a directory path.
first = line.split(" ", 1)[0]
last = line.rsplit(" ", 1)[-1]
if first.endswith("/") or last.endswith("/"):
continue
lines.append(line)
return sorted(lines)
if mode == STDOUT_STATS:
found = {}
for line in (text or "").splitlines():
for key in STATS_KEYS:
if line.startswith(key + ":"):
found[key] = _parse_bytes(line.split(":", 1)[1])
return found
if mode == STDOUT_PROGRESS:
# rsync prints the file-list entries in sorted depth-first order while
# FastSync's streaming scan emits them in readdir/BFS order; only the
# entry set and deterministic fields are compared. The transfer-root
# `./` line's trigger condition is a separate documented residual, and
# the per-frame rate/elapsed/xfr#/to-chk numerator are wall-clock- or
# order-dependent, so only the `to-chk` denominator and the name set are
# asserted.
names = []
totals = set()
max_xfr = 0
for line in (text or "").splitlines():
line = line.rstrip()
if not line:
continue
if "%" in line:
m = _PROGRESS_TOTAL_RE.search(line)
if m:
totals.add(int(m.group(1)))
mx = _PROGRESS_XFR_RE.search(line)
if mx:
max_xfr = max(max_xfr, int(mx.group(1)))
continue
if line == "sending incremental file list":
continue
if line.startswith("created directory "):
continue
if line == "./":
continue
names.append(line)
return {"names": sorted(names), "total": sorted(totals), "xfr": max_xfr}
# raw
return sorted(l.rstrip() for l in (text or "").splitlines() if l.strip())
def stdout_diff(rsync_out: str, fs_out: str, mode: Optional[str]) -> List[str]:
r = normalize_stdout(rsync_out, mode)
f = normalize_stdout(fs_out, mode)
if r == f:
return []
if mode == STDOUT_STATS:
return [f"stats rsync={r}", f"stats fastsync={f}"]
if mode == STDOUT_PROGRESS:
return [f"progress rsync={r}", f"progress fastsync={f}"]
return list(difflib.unified_diff(
[str(x) for x in r], [str(x) for x in f],
fromfile="rsync", tofile="fastsync", lineterm=""))
# ---------------------------------------------------------------------------
# Running one case
# ---------------------------------------------------------------------------
def run_rsync(src: str, rdst: str, flags: List[str]) -> subprocess.CompletedProcess:
args = [RSYNC] + list(flags) + [src + "/", rdst + "/"]
return subprocess.run(
args, capture_output=True, text=True,
env=dict(os.environ, LC_ALL="C"), timeout=180)
def run_fastsync(src: str, fdst: str, flags: List[str], port: int):
return run_client(src, fdst, flags=list(flags), port=port)
def run_differential( # noqa: PLR0913 (explicit scenario parameters)
src: str,
rdst: str,
fdst: str,
rs_flags: List[str],
fs_flags: List[str],
server,
layout: str = MIRROR,
seed: Optional[Callable] = None,
stdout: Optional[str] = STDOUT_NONE,
compare_modes: bool = False,
compare_hardlinks: bool = False,
ignore_paths: Tuple[str, ...] = (),
extra_check: Optional[Callable] = None,
files_from: Optional[Tuple[str, ...]] = None,
fs_src_suffix: str = "",
compare_tree: bool = True,
) -> Dict[str, object]:
"""Run one rsync/FastSync pair and return the diff aspects.
Returned dict keys: ``rsync_rc``, ``fastsync_rc``, ``rsync_stderr``,
``fastsync_stderr``, ``tree``, ``stdout``, ``extra``.
"""
clean_dir(rdst)
clean_dir(fdst)
abs_src = os.path.abspath(src)
rel = abs_src.lstrip(os.sep)
if layout == RELATIVE:
rroot, froot = rdst, fdst
elif layout == MIRROR_ABS:
rroot, froot = os.path.join(rdst, rel), get_dest_received_dir(fdst, src)
else:
rroot, froot = rdst, get_dest_received_dir(fdst, src)
# rsync's destination root always exists (clean_dir created it). FastSync's
# logical transfer root is the mirror path below the destination argument,
# so pre-create it too: `Number of created files` counts the root only when
# it is genuinely absent, and the two tools must start from the same state.
os.makedirs(froot, exist_ok=True)
if seed:
seed(src, rroot, froot)
rs_flags = list(rs_flags)
fs_flags = list(fs_flags)
if files_from is not None:
list_path = os.path.join(TEST_DATA_DIR, "parity_" +
os.path.basename(src) + ".list")
write_list(list_path, files_from)
rs_flags.append(f"--files-from={list_path}")
fs_flags.append(f"--files-from={list_path}")
rs = run_rsync(src, rdst, rs_flags)
fs_result, _ = run_fastsync(src + fs_src_suffix, fdst, fs_flags, server.port)
class _View:
"""Adapter so tree_diff/extra_check keep the Case-shaped interface."""
def __init__(self) -> None:
self.compare_modes = compare_modes
self.compare_hardlinks = compare_hardlinks
self.ignore_paths = ignore_paths
result = {
"rsync_rc": rs.returncode,
"fastsync_rc": fs_result.returncode,
"rsync_stderr": rs.stderr,
"fastsync_stderr": fs_result.stderr or fs_result.stdout,
"tree": tree_diff(rroot, froot, _View()) if compare_tree else [],
"stdout": [],
"extra": [],
}
if stdout is not None:
result["stdout"] = stdout_diff(rs.stdout, fs_result.stdout, stdout)
if extra_check:
result["extra"] = list(extra_check(src, rroot, froot, rs, fs_result) or [])
return result
def execute_case(case: Case, server) -> Dict[str, object]:
"""Run a table-driven case and return the diff aspects."""
tag = case.id
src = os.path.join(TEST_DATA_DIR, f"parity_{tag}_src")
rdst = os.path.join(TEST_DATA_DIR, f"parity_{tag}_rdst")
fdst = os.path.join(TEST_DATA_DIR, f"parity_{tag}_fdst")
CORPORA[case.corpus](src)
return run_differential(
src, rdst, fdst,
case.flags, case.fs_flags(), server,
layout=case.layout, seed=case.seed, stdout=case.stdout,
compare_modes=case.compare_modes, compare_hardlinks=case.compare_hardlinks,
ignore_paths=case.ignore_paths, extra_check=case.extra_check,
files_from=case.files_from, fs_src_suffix=case.fs_src_suffix,
compare_tree=case.compare_tree,
)
def write_list(path: str, entries) -> str:
os.makedirs(os.path.dirname(path), exist_ok=True)
with open(path, "w", encoding="utf-8") as fh:
for e in entries:
fh.write(e + "\n")
return path
+213
View File
@@ -0,0 +1,213 @@
"""Differential tests for the client CLI's codec defaults and env lists.
Track 3a of the rsync-parity plan pins two rsync 3.4.1 behaviors that are
resolved entirely on the client:
* the per-codec default ``--compress-level`` (zstd 3, zlib/zlibx 6, lz4
ignored) applied when the user omits ``--compress-level``/``--zl``, with an
explicit level clamped to the codec's range; and
* the ``RSYNC_COMPRESS_LIST`` / ``RSYNC_CHECKSUM_LIST`` preference lists that
rsync's ``auto`` consults before its compiled-in order (whitespace-separated,
unknown names skipped, first supported wins, all-unknown is exit 4).
The rsync side is observed through ``--debug=NSTR1``; FastSync publishes its
resolved codec/level through ``--debug=util``. The checksum side is confirmed
byte-for-byte through ``--out-format %C``. The rsync-based tests skip cleanly
when rsync is not installed.
"""
import os
import re
import shutil
import subprocess
import sys
import pytest
sys.path.insert(0, os.path.dirname(__file__))
from common import (
TEST_DATA_DIR,
run_client,
clean_dir,
get_dest_received_dir,
)
RSYNC = shutil.which("rsync")
requires_rsync = pytest.mark.skipif(RSYNC is None, reason="rsync 3.4.1 not installed")
CODEC_ROOT = os.path.join(TEST_DATA_DIR, "cli_differential")
_COMPRESS_RE = re.compile(r"compress(?:ion)?: (\w+) \(level (-?\d+)\)")
def _rsync(args):
env = dict(os.environ, LC_ALL="C")
return subprocess.run([RSYNC] + args, capture_output=True, text=True, env=env, timeout=120)
def _scratch(tag):
path = os.path.join(CODEC_ROOT, tag)
clean_dir(path)
os.makedirs(path, exist_ok=True)
return path
def _make_corpus(root):
clean_dir(root)
os.makedirs(root, exist_ok=True)
with open(os.path.join(root, "big.bin"), "wb") as fh:
fh.write(b"FastSync codec payload " * 4096)
with open(os.path.join(root, "small.txt"), "wb") as fh:
fh.write(b"hello codec world\n" * 32)
return root
def _rsync_compress_level(choice, level):
src = _make_corpus(_scratch(f"lvl_src_{choice}_{level}"))
dst = _scratch(f"lvl_rsync_{choice}_{level}")
args = ["-a", "-z", f"--zc={choice}"]
if level is not None:
args.append(f"--zl={level}")
args += ["--debug=NSTR1", src + "/", dst + "/"]
result = _rsync(args)
assert result.returncode == 0, result.stderr
match = _COMPRESS_RE.search(result.stdout + result.stderr)
assert match, (result.stdout, result.stderr)
return match.group(1), int(match.group(2))
def _fastsync_compress_level(choice, level, shared_server):
src = _make_corpus(_scratch(f"lvl_src_fs_{choice}_{level}"))
dst = _scratch(f"lvl_fs_{choice}_{level}")
args = ["-a", "-z", f"--zc={choice}"]
if level is not None:
args.append(f"--zl={level}")
args += ["-v", "--debug=util"]
result, _ = run_client(src, dst, flags=args, port=shared_server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
match = _COMPRESS_RE.search(result.stdout)
assert match, result.stdout[:500]
return match.group(1), int(match.group(2))
class TestPerCodecCompressionLevelDefaults:
"""``--compress-level`` defaults and clamping match rsync per codec."""
# FastSync uses a positive lz4 placeholder because its "level > 0" gate
# enables compression; lz4_compress ignores the value, so rsync's level 0
# and FastSync's level 1 produce the same bytes.
CASES = [
("zstd", None, 3),
("zlib", None, 6),
("zlibx", None, 6),
("lz4", None, 1),
("zstd", 10, 10),
("zlib", 15, 9),
("zlib", 3, 3),
("lz4", 15, 1),
]
@requires_rsync
@pytest.mark.ci
@pytest.mark.parametrize("choice,level,fs_level", CASES)
def test_level_matches_rsync(self, choice, level, fs_level, shared_server):
rsync_algo, rsync_level = _rsync_compress_level(choice, level)
fs_algo, fs_level_actual = _fastsync_compress_level(choice, level, shared_server)
assert rsync_algo == choice
assert fs_algo == choice
if choice == "lz4":
assert rsync_level == 0 and fs_level_actual > 0
else:
assert rsync_level == fs_level
assert fs_level_actual == fs_level
class TestEnvPreferenceLists:
"""``RSYNC_COMPRESS_LIST`` / ``RSYNC_CHECKSUM_LIST`` drive auto like rsync."""
# (env value, expected codec, rsync level, FastSync level)
COMPRESS_CASES = [
("zlib lz4", "zlib", 6, 6),
("lz4 zstd", "lz4", 0, 1),
("bogus zstd zlib", "zstd", 3, 3),
(" ", "zstd", 3, 3),
]
@requires_rsync
@pytest.mark.ci
@pytest.mark.parametrize("env,algo,rsync_level,fs_level", COMPRESS_CASES)
def test_compress_list_matches_rsync(self, env, algo, rsync_level, fs_level, shared_server,
monkeypatch):
monkeypatch.setenv("RSYNC_COMPRESS_LIST", env)
src = _make_corpus(_scratch(f"envc_src_{algo}"))
rdst = _scratch(f"envc_rsync_{algo}")
rs = _rsync(["-a", "-z", "--debug=NSTR1", src + "/", rdst + "/"])
assert rs.returncode == 0, rs.stderr
rm = _COMPRESS_RE.search(rs.stdout + rs.stderr)
assert rm, (rs.stdout, rs.stderr)
assert rm.group(1) == algo
assert int(rm.group(2)) == rsync_level
fdst = _scratch(f"envc_fs_{algo}")
result, _ = run_client(src, fdst, flags=["-a", "-z", "-v", "--debug=util"],
port=shared_server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
fm = _COMPRESS_RE.search(result.stdout)
assert fm, result.stdout[:500]
assert fm.group(1) == algo
assert int(fm.group(2)) == fs_level
received = get_dest_received_dir(fdst, src)
assert _tree_bytes(received) == _tree_bytes(src)
@requires_rsync
@pytest.mark.ci
@pytest.mark.parametrize("env,algo", [("md5", "md5"), ("sha1", "sha1"), ("xxh3 md5", "xxh3")])
def test_checksum_list_matches_rsync(self, env, algo, shared_server, monkeypatch):
monkeypatch.setenv("RSYNC_CHECKSUM_LIST", env)
src = _make_corpus(_scratch(f"envcc_src_{algo}"))
rdst = _scratch(f"envcc_rsync_{algo}")
rs = _rsync(["-a", "--checksum", "--out-format=%C %n", src + "/", rdst + "/"])
assert rs.returncode == 0, rs.stderr
rs_digests = _digests(rs.stdout)
fdst = _scratch(f"envcc_fs_{algo}")
result, _ = run_client(src, fdst, flags=["-a", "--checksum", "--out-format=%C %n"],
port=shared_server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
assert _digests(result.stdout) == rs_digests
@requires_rsync
@pytest.mark.ci
def test_all_unknown_lists_fail_like_rsync(self, shared_server, monkeypatch):
src = _make_corpus(_scratch("envbad_src"))
monkeypatch.setenv("RSYNC_COMPRESS_LIST", "bogus")
rs = _rsync(["-a", "-z", src + "/", _scratch("envbad_rsync_c") + "/"])
assert rs.returncode == 4, rs.stderr
result, _ = run_client(src, _scratch("envbad_fs_c"), flags=["-a", "-z"],
port=shared_server.port)
assert result.returncode == 4, (result.stderr or result.stdout)[:200]
monkeypatch.setenv("RSYNC_CHECKSUM_LIST", "bogus")
rs = _rsync(["-a", src + "/", _scratch("envbad_rsync_s") + "/"])
assert rs.returncode == 4, rs.stderr
result, _ = run_client(src, _scratch("envbad_fs_s"), flags=["-a"],
port=shared_server.port)
assert result.returncode == 4, (result.stderr or result.stdout)[:200]
def _tree_bytes(root):
out = {}
for dirpath, _dirs, files in os.walk(root):
for name in files:
path = os.path.join(dirpath, name)
with open(path, "rb") as fh:
out[os.path.relpath(path, root)] = fh.read()
return out
def _digests(output):
out = {}
for line in output.splitlines():
parts = line.split()
if len(parts) == 2 and parts[0]:
out[parts[1]] = parts[0]
return out
+367
View File
@@ -0,0 +1,367 @@
"""Differential tests for --checksum-choice / --compress-choice against rsync 3.4.1.
These pin the accepted/rejected algorithm matrix and exit codes to real rsync,
and verify that every codec FastSync now offers still transfers byte-exactly.
The rsync-based tests skip cleanly when rsync is not installed.
The FastSync server confines transfers to its authorized root (the project
directory when the shared test server is launched), so every scratch tree lives
under ``TEST_DATA_DIR`` rather than pytest's ``tmp_path``.
"""
import os
import random
import shutil
import subprocess
import sys
import pytest
sys.path.insert(0, os.path.dirname(__file__))
from common import (
TEST_DATA_DIR,
run_client,
clean_dir,
get_dest_received_dir,
)
RSYNC = shutil.which("rsync")
requires_rsync = pytest.mark.skipif(RSYNC is None, reason="rsync 3.4.1 not installed")
CHECKSUM_NAMES = ["xxh128", "xxh3", "xxh64", "md5", "md4", "sha1"]
COMPRESS_NAMES = ["zstd", "lz4", "zlib", "zlibx"]
CODEC_ROOT = os.path.join(TEST_DATA_DIR, "codec_differential")
def _rsync(args):
env = dict(os.environ, LC_ALL="C")
return subprocess.run([RSYNC] + args, capture_output=True, text=True, env=env, timeout=120)
def _scratch(tag):
"""A confined, uniquely named scratch directory under the project tree."""
path = os.path.join(CODEC_ROOT, tag)
clean_dir(path)
os.makedirs(path, exist_ok=True)
return path
def _make_corpus(root):
clean_dir(root)
os.makedirs(os.path.join(root, "sub"), exist_ok=True)
# Highly compressible payload so each codec is actually exercised.
with open(os.path.join(root, "big.bin"), "wb") as fh:
fh.write(b"FastSync codec payload " * 4096)
with open(os.path.join(root, "sub", "text.txt"), "wb") as fh:
fh.write(b"hello codec world\n" * 128)
with open(os.path.join(root, "empty"), "wb"):
pass
return root
def _tree_bytes(root):
out = {}
for dirpath, _dirs, files in os.walk(root):
for name in files:
path = os.path.join(dirpath, name)
with open(path, "rb") as fh:
out[os.path.relpath(path, root)] = fh.read()
return out
_CC_DELTA_T0 = 1_600_000_000
_CC_DELTA_T1 = 1_600_000_100
_DELTA_STATS_KEYS = (
"Number of created files",
"Number of regular files transferred",
"Total transferred file size",
"Literal data",
"Matched data",
)
def _pin_tree(root, mtime):
for dirpath, dirnames, filenames in os.walk(root):
for name in dirnames + filenames:
path = os.path.join(dirpath, name)
if not os.path.islink(path):
os.utime(path, (mtime, mtime))
os.utime(root, (mtime, mtime))
def _make_delta_basis(src, size=512 * 1024):
"""Build a source and a matching pre-modification basis tree.
The source's ``big.bin`` is then modified in a few disjoint places and given
a newer mtime so both tools take the delta path. Returns the basis dir.
"""
clean_dir(src)
original = random.Random(20240101).randbytes(size)
with open(os.path.join(src, "big.bin"), "wb") as fh:
fh.write(original)
with open(os.path.join(src, "small.txt"), "wb") as fh:
fh.write(b"hello world\n")
_pin_tree(src, _CC_DELTA_T0)
basis = src.rstrip("/") + "_basis"
clean_dir(basis)
shutil.copy2(os.path.join(src, "big.bin"), os.path.join(basis, "big.bin"))
shutil.copy2(os.path.join(src, "small.txt"), os.path.join(basis, "small.txt"))
_pin_tree(basis, _CC_DELTA_T0)
modified = bytearray(original)
for off in (0, size // 3, 2 * size // 3, size - 64):
for i in range(32):
modified[off + i] ^= 0x5A
with open(os.path.join(src, "big.bin"), "wb") as fh:
fh.write(bytes(modified))
os.utime(os.path.join(src, "big.bin"), (_CC_DELTA_T1, _CC_DELTA_T1))
return basis
def _seed_from_basis(basis, target):
clean_dir(target)
for name in os.listdir(basis):
shutil.copy2(os.path.join(basis, name), os.path.join(target, name))
def _delta_stats(text):
found = {}
for line in text.splitlines():
for key in _DELTA_STATS_KEYS:
if line.startswith(key + ":"):
found[key] = line.split(":", 1)[1].strip()
return found
def _big_bin_outfmt(text):
"""The ``(c, C)`` pair from the ``big.bin`` out-format line (`%c|%C %n`)."""
for line in text.splitlines():
stripped = line.strip()
if "|" not in stripped or not stripped.endswith("big.bin"):
continue
c_field, rest = stripped.split("|", 1)
fields = rest.split()
return c_field.strip(), (fields[0] if fields else "")
return None, None
class TestCodecChoiceMatrix:
"""The CLI accept/reject set and exit codes must match rsync 3.4.1."""
@requires_rsync
@pytest.mark.ci
@pytest.mark.parametrize("name", CHECKSUM_NAMES)
def test_checksum_names_accepted_by_both(self, name, shared_server):
src = _make_corpus(_scratch(f"cc_src_{name}"))
rdst = _scratch(f"cc_rsync_{name}")
rsync_result = _rsync(["-a", f"--cc={name}", src + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
fdst = _scratch(f"cc_fs_{name}")
result, _ = run_client(src, fdst, flags=[f"--cc={name}"], port=shared_server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
@requires_rsync
@pytest.mark.ci
@pytest.mark.parametrize("name", COMPRESS_NAMES)
def test_compress_names_accepted_by_both(self, name, shared_server):
src = _make_corpus(_scratch(f"zc_src_{name}"))
rdst = _scratch(f"zc_rsync_{name}")
rsync_result = _rsync(["-az", f"--zc={name}", src + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
fdst = _scratch(f"zc_fs_{name}")
result, _ = run_client(src, fdst, flags=["-z", f"--zc={name}"], port=shared_server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
@requires_rsync
@pytest.mark.ci
@pytest.mark.parametrize("choice", ["md4,sha1", "sha1,md4", "auto,md5", "none,md5"])
def test_checksum_two_name_accepted_by_both(self, choice, shared_server):
tag = choice.replace(",", "_")
src = _make_corpus(_scratch(f"two_src_{tag}"))
rdst = _scratch(f"two_rsync_{tag}")
rsync_result = _rsync(["-a", "--checksum", f"--cc={choice}", src + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
fdst = _scratch(f"two_fs_{tag}")
result, _ = run_client(src, fdst, flags=["--checksum", f"--cc={choice}"],
port=shared_server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:200]
@requires_rsync
@pytest.mark.ci
@pytest.mark.parametrize("name", ["sha256", "crc32", "md5,", "md4,md5,sha1"])
def test_unknown_checksum_rejected_exit_4_both(self, name, shared_server):
src = _make_corpus(_scratch(f"badcc_src_{name.replace(',', '_').replace(':', '_')}"))
rdst = _scratch(f"badcc_rsync_{name.replace(',', '_').replace(':', '_')}")
rsync_result = _rsync(["-a", f"--cc={name}", src + "/", rdst + "/"])
assert rsync_result.returncode == 4, rsync_result.stderr
fdst = _scratch(f"badcc_fs_{name.replace(',', '_').replace(':', '_')}")
result, _ = run_client(src, fdst, flags=[f"--cc={name}"], port=shared_server.port)
assert result.returncode == 4, (result.stderr or result.stdout)[:200]
@requires_rsync
@pytest.mark.ci
@pytest.mark.parametrize("choice", ["none", "md5,none"])
def test_checksum_none_with_checksum_rejected_exit_4_both(self, choice, shared_server):
tag = choice.replace(",", "_")
src = _make_corpus(_scratch(f"nonecc_src_{tag}"))
rdst = _scratch(f"nonecc_rsync_{tag}")
rsync_result = _rsync(["-a", "--checksum", f"--cc={choice}", src + "/", rdst + "/"])
assert rsync_result.returncode == 4, rsync_result.stderr
fdst = _scratch(f"nonecc_fs_{tag}")
result, _ = run_client(src, fdst, flags=["--checksum", f"--cc={choice}"],
port=shared_server.port)
assert result.returncode == 4, (result.stderr or result.stdout)[:200]
@requires_rsync
@pytest.mark.ci
@pytest.mark.parametrize("name", ["bogus", "zstd,lz4"])
def test_unknown_compress_rejected_exit_4_both(self, name, shared_server):
tag = name.replace(",", "_")
src = _make_corpus(_scratch(f"badzc_src_{tag}"))
rdst = _scratch(f"badzc_rsync_{tag}")
rsync_result = _rsync(["-az", f"--zc={name}", src + "/", rdst + "/"])
assert rsync_result.returncode == 4, rsync_result.stderr
fdst = _scratch(f"badzc_fs_{tag}")
result, _ = run_client(src, fdst, flags=["-z", f"--zc={name}"], port=shared_server.port)
assert result.returncode == 4, (result.stderr or result.stdout)[:200]
class TestCodecTransferDifferential:
"""Each codec lands the same bytes rsync lands."""
@requires_rsync
@pytest.mark.ci
@pytest.mark.parametrize("name", COMPRESS_NAMES + ["none"])
def test_compress_codec_matches_rsync_bytes(self, name, shared_server):
src = _make_corpus(_scratch(f"byteszc_src_{name}"))
rsync_dst = _scratch(f"byteszc_rsync_{name}")
rsync_result = _rsync(["-a", "-z", f"--zc={name}", src + "/", rsync_dst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
fs_dst = _scratch(f"byteszc_fs_{name}")
result, _ = run_client(src, fs_dst, flags=["-a", "-z", f"--zc={name}"],
port=shared_server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
received = get_dest_received_dir(fs_dst, src)
assert _tree_bytes(received) == _tree_bytes(rsync_dst)
@requires_rsync
@pytest.mark.ci
@pytest.mark.parametrize("name", CHECKSUM_NAMES)
def test_checksum_codec_matches_rsync_bytes(self, name, shared_server):
src = _make_corpus(_scratch(f"bytescc_src_{name}"))
rsync_dst = _scratch(f"bytescc_rsync_{name}")
rsync_result = _rsync(["-a", "--checksum", f"--cc={name}", src + "/", rsync_dst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
fs_dst = _scratch(f"bytescc_fs_{name}")
result, _ = run_client(src, fs_dst, flags=["-a", "--checksum", f"--cc={name}"],
port=shared_server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
received = get_dest_received_dir(fs_dst, src)
assert _tree_bytes(received) == _tree_bytes(rsync_dst)
class TestChecksumChoiceDeltaSurface:
"""--checksum-choice does not move the delta-transfer parity surface.
FastSync's delta BLOCK strong checksum is a fixed xxHash32, so the
negotiated algorithm only selects the whole-file comparison digest (and the
``%C`` transfer digest). A pre-seeded delta transfer must therefore land
byte-identical bytes and report the same counters for every choice, while
``%C`` -- the one token that tracks the choice -- stays byte-identical to
rsync. This pins the Track-3b reclassification in RSYNC_COMPAT.md.
"""
CHOICES = ["xxh64", "xxh128", "xxh3", "md5", "md4", "sha1",
"xxh64,sha1", "sha1,xxh64"]
@requires_rsync
@pytest.mark.ci
def test_delta_surface_invariant_to_checksum_choice(self, shared_server):
src = _scratch("ccdelta_src")
basis = _make_delta_basis(src)
source_bytes = _tree_bytes(src)
rsync_c, fastsync_c, digests = {}, {}, {}
rsync_stats, fastsync_stats = {}, {}
for choice in self.CHOICES:
tag = choice.replace(",", "_")
rsync_dst = _scratch(f"ccdelta_rs_{tag}")
fs_dst = _scratch(f"ccdelta_fs_{tag}")
fs_root = get_dest_received_dir(fs_dst, src)
_seed_from_basis(basis, rsync_dst)
_seed_from_basis(basis, fs_root)
# Pin the block size on both ends so the literal/matched split is
# comparable (rsync's adaptive default would otherwise differ from
# FastSync's 8192-byte default).
rsync_result = _rsync(["-a", "--no-whole-file", "-B8192", "--stats",
"--out-format=%c|%C %n", f"--cc={choice}",
src + "/", rsync_dst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
result, _ = run_client(
src, fs_dst,
flags=["-a", "--incremental", "--delta", "-B8192", "--stats",
"--out-format=%c|%C %n", f"--cc={choice}"],
port=shared_server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
assert _tree_bytes(rsync_dst) == source_bytes, choice
assert _tree_bytes(fs_root) == source_bytes, choice
assert _delta_stats(rsync_result.stdout) == _delta_stats(result.stdout), choice
rs_c, rs_C = _big_bin_outfmt(rsync_result.stdout)
fs_c, fs_C = _big_bin_outfmt(result.stdout)
assert rs_C == fs_C, f"{choice}: %C rsync={rs_C!r} fastsync={fs_C!r}"
rsync_c[choice] = rs_c
fastsync_c[choice] = fs_c
digests[choice] = fs_C
rsync_stats[choice] = _delta_stats(rsync_result.stdout)
fastsync_stats[choice] = _delta_stats(result.stdout)
# The choice is only observable in %C, and it is effective (the digests
# are not all the same algorithm's output).
assert len(set(digests.values())) > 1, digests
# The compared --stats counters are invariant across choices in each tool
# (and were asserted equal cross-tool inside the loop).
assert len({tuple(sorted(s.items())) for s in rsync_stats.values()}) == 1, rsync_stats
assert len({tuple(sorted(s.items())) for s in fastsync_stats.values()}) == 1, fastsync_stats
# The block-checksum token (%c) is invariant across choices in each tool.
assert len(set(rsync_c.values())) == 1, rsync_c
assert len(set(fastsync_c.values())) == 1, fastsync_c
class TestCodecNegotiationFallback:
"""FastSync's auto negotiation and deterministic fallback order."""
@pytest.mark.ci
def test_default_checksum_and_compression_agree(self, shared_server):
"""A default transfer (auto on both peers) succeeds; the negotiated
default is xxh128 + zstd."""
src = _make_corpus(_scratch("auto_src"))
fdst = _scratch("auto_fs")
result, _ = run_client(src, fdst, flags=["-a", "-z"], port=shared_server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
received = get_dest_received_dir(fdst, src)
assert _tree_bytes(received) == _tree_bytes(src)
@pytest.mark.ci
def test_explicit_choice_overrides_auto(self, shared_server):
"""An explicit --zc/--cc wins over the negotiated default on both ends,
so the receiver decodes with the sender's codec."""
src = _make_corpus(_scratch("explicit_src"))
fdst = _scratch("explicit_fs")
result, _ = run_client(src, fdst, flags=["-a", "-z", "--zc=lz4", "--cc=sha1"],
port=shared_server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
received = get_dest_received_dir(fdst, src)
assert _tree_bytes(received) == _tree_bytes(src)
@@ -0,0 +1,185 @@
"""Differential coverage for ``--delete-delay`` + ``--max-delete`` with a
refilled deferred directory.
FastSync snapshots a directory's extras at plan time (``defer_add``) but charges
``--max-delete`` only when a path is actually removed, and its deferred commit
re-scans a queued directory and removes content created after the plan -- the
same rules as rsync. These tests run both tools on the same fixture and assert
both sides remove the late content (recursively) and bound the deletion with
``--max-delete`` identically.
They are not part of the fast PR gate because the rsync side needs a wide
real-time injection window (a throttled transfer), while the FastSync side uses
the existing byte-deterministic slicing proxy.
The refilled directory sits at the transfer ROOT, whose delete plan is always
processed before any subdirectory's, so the budget is deterministically charged
to the refilled entry; the second extra lives under ``b`` and is skipped.
"""
import os
import shutil
import subprocess
import sys
import threading
import time
import pytest
sys.path.insert(0, os.path.dirname(__file__))
from common import ( # noqa: E402
TEST_DATA_DIR,
ServerManager,
clean_dir,
get_dest_received_dir,
run_client,
)
from test_delete_timing_parity import _SlicingProxy # noqa: E402
RSYNC = shutil.which("rsync")
requires_rsync = pytest.mark.skipif(RSYNC is None, reason="rsync 3.4.1 not installed")
BIG_BYTES = 8 * 1024 * 1024
MID_TRANSFER_BYTES = 256 * 1024
PROXY_THROTTLE = 0.001
# rsync is driven locally, so the refill is injected on a wall-clock delay while
# a throttled ~8 s transfer is in flight. 1.5 s is safely after rsync's plan
# scan (t=0) and well before the deferred commit at the end.
RSYNC_BWLIMIT = 1024 # 1 MiB/s
RSYNC_INJECT_DELAY = 1.5
def _write(path, content):
os.makedirs(os.path.dirname(path), exist_ok=True)
with open(path, "wb") as fh:
fh.write(content)
def _seed_source(tag):
source = os.path.join(TEST_DATA_DIR, f"ddb_{tag}_src")
clean_dir(source)
_write(os.path.join(source, "a", "keep.bin"), b"B" * BIG_BYTES)
_write(os.path.join(source, "b", "keep.txt"), b"keep\n")
return source
def _seed_fastsync(tag):
"""FastSync mirrors the absolute source path under its receive root, so the
extras live below ``received``."""
source = _seed_source(tag)
dest = os.path.join(TEST_DATA_DIR, f"ddb_{tag}_dst")
clean_dir(dest)
received = get_dest_received_dir(dest, source)
os.makedirs(os.path.join(received, "xdir"), exist_ok=True)
os.makedirs(os.path.join(received, "b", "ydir"), exist_ok=True)
return source, dest, received
def _seed_rsync(tag):
"""rsync mirrors the source contents directly into the destination, so the
extras are flat under ``rsync_dst``."""
source = _seed_source(tag)
rsync_dst = os.path.join(TEST_DATA_DIR, f"ddb_{tag}_dst")
clean_dir(rsync_dst)
os.makedirs(os.path.join(rsync_dst, "xdir"), exist_ok=True)
os.makedirs(os.path.join(rsync_dst, "b", "ydir"), exist_ok=True)
return source, rsync_dst
def _deleted_count(text):
for line in text.splitlines():
if line.startswith("Number of deleted files:"):
return int(line.split(":", 1)[1].split()[0])
return None
def _rsync(args, timeout=120):
env = dict(os.environ, LC_ALL="C")
return subprocess.run([RSYNC] + args, capture_output=True, text=True, env=env, timeout=timeout)
class TestDeleteDelayRefilledDirVsRsync:
"""Both tools charge --max-delete on actual removals and recurse."""
def _fastsync_refilled(self, tag, max_delete=None):
"""Run FastSync with the refill injected deterministically by the proxy
hook (fired once the receiver has processed the plan frames)."""
source, dest, received = _seed_fastsync(tag)
late = os.path.join(received, "xdir", "new.txt")
def hook():
_write(late, b"created mid-transfer\n")
flags = ["--delete-delay", "--incremental", "--ignore-times", "--stats"]
if max_delete is not None:
flags.append(f"--max-delete={max_delete}")
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
proxy = _SlicingProxy(server.port, hook=hook, hook_after=MID_TRANSFER_BYTES,
throttle=PROXY_THROTTLE, wait_for_reply=True)
result, _ = run_client(source, dest, flags=flags, port=proxy.port)
proxy.finish()
assert proxy.hook_called.is_set(), "refill hook never fired"
return result, received, late
@requires_rsync
def test_max_delete_budget_bound_matches(self):
# --- FastSync: the one actual removal is the late file; dirs survive ---
result, received, late = self._fastsync_refilled("budget_fs", max_delete=1)
assert result.returncode == 25, (result.stderr or result.stdout)[:300]
assert _deleted_count(result.stdout) == 1, result.stdout
assert not os.path.exists(late), "FastSync kept the late content of a queued dir"
assert os.path.isdir(os.path.join(received, "xdir"))
assert os.path.isdir(os.path.join(received, "b", "ydir")), (
"FastSync did not bound the deletion with --max-delete=1"
)
# --- rsync: same budget rule and recursive removal ---
source, rsync_dst = _seed_rsync("budget_rsync")
def inject():
time.sleep(RSYNC_INJECT_DELAY)
_write(os.path.join(rsync_dst, "xdir", "new.txt"), b"created mid-transfer\n")
t = threading.Thread(target=inject)
t.start()
rsync_result = _rsync(
["-a", "--delete-delay", "--max-delete=1", "--stats",
f"--bwlimit={RSYNC_BWLIMIT}", source + "/", rsync_dst + "/"]
)
t.join()
assert rsync_result.returncode == 25, rsync_result.stderr
assert _deleted_count(rsync_result.stdout) == _deleted_count(result.stdout)
assert not os.path.exists(os.path.join(rsync_dst, "xdir", "new.txt")), (
"rsync kept late content inside a queued directory"
)
assert os.path.isdir(os.path.join(rsync_dst, "b", "ydir")), (
"rsync did not bound the deletion with --max-delete=1"
)
assert os.path.isdir(os.path.join(received, "b", "ydir"))
@requires_rsync
def test_refilled_extra_dir_recursive_removal_matches(self):
"""Without --max-delete both tools remove the refilled extra directory
(and its late content)."""
result, received, late = self._fastsync_refilled("recur_fs")
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
assert not os.path.exists(late), "FastSync kept the refilled directory's late content"
assert not os.path.isdir(os.path.join(received, "xdir"))
source, rsync_dst = _seed_rsync("recur_rsync")
def inject():
time.sleep(RSYNC_INJECT_DELAY)
_write(os.path.join(rsync_dst, "xdir", "new.txt"), b"created mid-transfer\n")
t = threading.Thread(target=inject)
t.start()
rsync_result = _rsync(
["-a", "--delete-delay", "--stats", f"--bwlimit={RSYNC_BWLIMIT}",
source + "/", rsync_dst + "/"]
)
t.join()
assert rsync_result.returncode == 0, rsync_result.stderr
assert not os.path.exists(os.path.join(rsync_dst, "xdir")), (
"rsync did not recursively remove the refilled extra directory"
)
@@ -0,0 +1,539 @@
"""Differential + regression coverage for rsync's delete timing.
``--delete-during``/``--delete-delay`` stream a per-directory delete plan instead
of one whole-tree manifest, so the timing is observable:
* ``--delete-during`` removes a directory's extras as it processes that
directory (so an interrupted transfer has already removed the extras of the
directories it reached);
* ``--delete-delay`` snapshots those extras while scanning and commits the
removals only after a fully-successful transfer (so an extra created in the
destination after its directory's plan survives, and a failed transfer
removes nothing);
* ``--delete-after`` re-scans the destination at the end (so that same
late-created extra is removed).
The final-state tests compare against real ``rsync 3.4.1`` where a deterministic
comparison exists; the timing tests use a byte-slicing proxy to force a
mid-transfer failure or to create a destination entry while the transfer is in
flight.
"""
import os
import select
import shutil
import socket
import struct
import subprocess
import sys
import threading
import time
import pytest
sys.path.insert(0, os.path.dirname(__file__))
from common import ( # noqa: E402
BUILD_DIR,
TEST_DATA_DIR,
ServerManager,
clean_dir,
get_dest_received_dir,
run_client,
)
# Every test here is deterministic (the proxy throttles until the delete-plan
# frames are processed), so the PR gate runs the whole module.
pytestmark = pytest.mark.ci
RSYNC = shutil.which("rsync")
requires_rsync = pytest.mark.skipif(RSYNC is None, reason="rsync 3.4.1 not installed")
BIG_BYTES = 8 * 1024 * 1024
# Forward/cut this far into the stream: past the (small) delete-plan frames and
# well into the big payload, so the receiver has already processed the plan.
MID_TRANSFER_BYTES = 256 * 1024
# Throttle the proxy so the receiver keeps up with the (fast) client and the
# plan frames are provably processed before the hook/cut offset is reached.
PROXY_THROTTLE = 0.001
def _write(path, content):
os.makedirs(os.path.dirname(path), exist_ok=True)
with open(path, "wb") as fh:
fh.write(content)
def _seed_pair(tag, big=False):
"""Create a source tree and a destination mirror seeded with extras.
The tree is a single directory ``d`` containing the transferred files plus,
in the destination, an extra ``d/old_extra``.
"""
source = os.path.join(TEST_DATA_DIR, f"dtp_{tag}_src")
dest = os.path.join(TEST_DATA_DIR, f"dtp_{tag}_dst")
clean_dir(source)
clean_dir(dest)
_write(os.path.join(source, "d", "keep.txt"), b"kept payload\n")
if big:
_write(os.path.join(source, "d", "big.bin"), b"B" * BIG_BYTES)
received = get_dest_received_dir(dest, source)
os.makedirs(os.path.join(received, "d"), exist_ok=True)
_write(os.path.join(received, "d", "old_extra"), b"stale extra\n")
return source, dest, received
def _tree(root):
"""Sorted relative paths of every entry below root (files and dirs)."""
out = []
for dirpath, dirs, files in os.walk(root):
for name in dirs:
out.append(os.path.relpath(os.path.join(dirpath, name), root))
for name in files:
out.append(os.path.relpath(os.path.join(dirpath, name), root))
return sorted(out)
def _rsync(args):
env = dict(os.environ, LC_ALL="C")
return subprocess.run([RSYNC] + args, capture_output=True, text=True, env=env, timeout=120)
class _SlicingProxy:
"""Forward the client stream to a server, optionally cutting it or invoking a
hook after a byte threshold. ``forward_limit`` mode resets both ends after
that many client bytes (a mid-transfer failure). ``hook`` mode calls the
hook once and keeps forwarding to completion.
With ``wait_for_reply`` the hook is a real barrier, not a timing guess: it
fires only after the server has sent *any* reply, which the receiver does
only after it has consumed the frames that precede the payload (the
per-directory delete plan for ``--delete-delay``). The caller pairs it with
``--incremental`` so a per-file handshake reply is guaranteed mid-transfer.
"""
def __init__(self, target_port, forward_limit=None, hook=None, hook_after=0,
throttle=0.0, wait_for_reply=False):
self.target = ("127.0.0.1", target_port)
self.forward_limit = forward_limit
self.hook = hook
self.hook_after = hook_after
self.throttle = throttle
self.wait_for_reply = wait_for_reply
self.server_replied = threading.Event()
self.hook_called = threading.Event()
self.listener = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.listener.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self.listener.bind(("127.0.0.1", 0))
self.listener.listen(1)
self.listener.settimeout(20)
self.port = self.listener.getsockname()[1]
self._thread = threading.Thread(target=self._serve, daemon=True)
self._thread.start()
def _serve(self):
try:
client, _ = self.listener.accept()
except OSError:
return
try:
backend = socket.create_connection(self.target, timeout=10)
except OSError:
client.close()
return
client.settimeout(20)
backend.settimeout(20)
forwarded = 0
socks = [client, backend]
try:
while socks:
ready, _, _ = select.select(socks, [], [], 20)
if not ready:
break
for sock in ready:
data = sock.recv(65536)
if not data:
socks.remove(sock)
peer = backend if sock is client else client
try:
peer.shutdown(socket.SHUT_WR)
except OSError:
pass
continue
if sock is client:
if self.forward_limit is not None:
room = self.forward_limit - forwarded
if room <= 0:
socks = []
break
data = data[:room]
backend.sendall(data)
forwarded += len(data)
self._maybe_hook(forwarded)
if self.forward_limit is not None and forwarded >= self.forward_limit:
socks = []
break
if self.throttle > 0:
time.sleep(self.throttle)
else:
client.sendall(data)
# Any server reply proves the receiver consumed the
# frames that precede it, so the hook barrier is met.
self.server_replied.set()
self._maybe_hook(forwarded)
except OSError:
pass
for sock in (client, backend):
try:
sock.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack("ii", 1, 0))
except OSError:
pass
try:
sock.close()
except OSError:
pass
try:
self.listener.close()
except OSError:
pass
def _maybe_hook(self, forwarded):
"""Fire the one-shot hook once its barrier is satisfied: enough client
bytes have been forwarded and, when ``wait_for_reply`` is set, the
server has sent a reply proving it processed the preceding frames."""
if self.hook is None or self.hook_called.is_set():
return
if forwarded < self.hook_after:
return
if self.wait_for_reply and not self.server_replied.is_set():
return
self.hook()
self.hook_called.set()
def finish(self):
self._thread.join(30)
try:
self.listener.close()
except OSError:
pass
class TestDeleteTimingFinalStateParity:
"""On a successful transfer the per-directory timings match rsync's result.
Plain ``--delete`` has no rsync-incompatible spelling: it defaults to
delete-during on both tools, so it is compared against rsync's own default.
``--delete-commit`` is FastSync-only and selects the late whole-tree commit,
which is rsync's ``--delete-after`` timing.
"""
# (fastsync flag, rsync flag)
PAIRS = [
("--delete", "--delete"),
("--delete-during", "--delete-during"),
("--delete-delay", "--delete-delay"),
("--delete-commit", "--delete-after"),
]
def _run_fastsync(self, tag, timing):
source, dest, received = _seed_pair(tag)
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
result, _ = run_client(source, dest, flags=[timing], port=server.port)
return result, received
@pytest.mark.parametrize("fs_timing,rs_timing", PAIRS)
@requires_rsync
def test_success_final_state_matches_rsync(self, fs_timing, rs_timing):
# Worker-safe names: xdist may run the parametrizations concurrently, so
# the flags are part of every fixture path.
label = f"{fs_timing.lstrip('-')}_vs_{rs_timing.lstrip('-')}"
# Build the rsync fixture from the same seed so both sides start equal.
source, dest, received = _seed_pair(f"parity_rsync_{label}")
source2 = source
rsync_dst = os.path.join(TEST_DATA_DIR, f"dtp_rsync_{label}_dst")
clean_dir(rsync_dst)
# rsync mirrors src/ into dst/; seed the same extra.
_write(os.path.join(rsync_dst, "d", "old_extra"), b"stale extra\n")
rsync_result = _rsync(["-a", rs_timing, source2 + "/", rsync_dst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
rsync_tree = _tree(rsync_dst)
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
result, _ = run_client(source, dest, flags=[fs_timing], port=server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
fastsync_tree = _tree(received)
assert fastsync_tree == rsync_tree, (
f"{fs_timing} vs rsync {rs_timing}: fastsync tree {fastsync_tree} != "
f"rsync tree {rsync_tree}"
)
class TestDeleteTimingTypeConflictParity:
"""A destination entry whose type differs from the source is replaced, in
both per-directory timings and in both directions, exactly like rsync."""
@pytest.mark.parametrize("timing", ["--delete-during", "--delete-delay"])
@requires_rsync
def test_type_conflicts_match_rsync(self, timing):
label = timing.lstrip("-")
source = os.path.join(TEST_DATA_DIR, f"dtc_{label}_src")
clean_dir(source)
_write(os.path.join(source, "foo"), b"now a file\n")
_write(os.path.join(source, "bar", "inner.txt"), b"now a dir\n")
def seed_dest(root):
clean_dir(root)
_write(os.path.join(root, "foo", "inner.txt"), b"was a dir\n")
_write(os.path.join(root, "bar"), b"was a file\n")
rsync_dst = os.path.join(TEST_DATA_DIR, f"dtc_{label}_rsync_dst")
seed_dest(rsync_dst)
rsync_result = _rsync(["-a", timing, source + "/", rsync_dst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
rsync_tree = _tree(rsync_dst)
dest = os.path.join(TEST_DATA_DIR, f"dtc_{label}_dst")
clean_dir(dest)
received = get_dest_received_dir(dest, source)
seed_dest(received)
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
result, _ = run_client(source, dest, flags=[timing], port=server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
assert _tree(received) == rsync_tree, (
f"{timing}: fastsync tree {_tree(received)} != rsync tree {rsync_tree}"
)
class TestDeleteTimingFailure:
"""A mid-transfer failure distinguishes the during timings from the late
commit timings.
Plain ``--delete`` must behave like ``--delete-during`` (the rsync default),
removing the extras of the directories already reached; ``--delete-commit``
must behave like ``--delete-after`` and remove nothing until the transfer
has fully succeeded.
"""
@pytest.mark.parametrize("mt", [False, True])
def test_during_removes_delay_preserves_on_failure(self, mt):
source, dest, received = _seed_pair(f"failure_mt{int(mt)}", big=True)
extra = os.path.join(received, "d", "old_extra")
assert os.path.exists(extra)
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
for timing, expect_removed in (
("--delete-during", True),
("--delete", True),
("--delete-delay", False),
("--delete-commit", False),
("--delete-after", False)):
# Re-seed the extra before each run.
_write(extra, b"stale extra\n")
proxy = _SlicingProxy(server.port, forward_limit=MID_TRANSFER_BYTES, throttle=PROXY_THROTTLE)
flags = [timing] + (["--threads"] if mt else [])
result, _ = run_client(source, dest, flags=flags, port=proxy.port)
proxy.finish()
assert result.returncode != 0, f"{timing}: truncated transfer succeeded"
present = os.path.exists(extra)
assert present != expect_removed, (
f"{timing} (mt={mt}): extra present={present}, expected "
f"removed={expect_removed}"
)
class TestDeleteDelayDeletedCount:
"""The reported deleted count must reflect entries actually removed."""
def test_refilled_deferred_dir_is_recursively_removed_and_counted(self):
"""A directory snapshotted into a --delete-delay plan that is refilled
before the commit is re-scanned and removed recursively (rsync parity):
the late file and the directory are both counted as deleted."""
source = os.path.join(TEST_DATA_DIR, "ddc_src")
dest = os.path.join(TEST_DATA_DIR, "ddc_dst")
clean_dir(source)
clean_dir(dest)
_write(os.path.join(source, "d", "keep.txt"), b"kept payload\n")
_write(os.path.join(source, "d", "big.bin"), b"B" * BIG_BYTES)
received = get_dest_received_dir(dest, source)
extra_dir = os.path.join(received, "d", "extradir")
os.makedirs(extra_dir, exist_ok=True)
def hook():
# Runs while big.bin is in flight, after d's delete plan was processed.
_write(os.path.join(extra_dir, "new.txt"), b"created mid-transfer\n")
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
proxy = _SlicingProxy(server.port, hook=hook, hook_after=MID_TRANSFER_BYTES,
throttle=PROXY_THROTTLE, wait_for_reply=True)
flags = ["--delete-delay", "--incremental", "--ignore-times", "--stats"]
result, _ = run_client(source, dest, flags=flags, port=proxy.port)
proxy.finish()
assert result.returncode == 0, (result.stderr or result.stdout)[:400]
assert proxy.hook_called.is_set(), "hook never fired"
assert not os.path.exists(os.path.join(extra_dir, "new.txt")), "late file survived"
assert not os.path.isdir(extra_dir), "refilled extra dir survived"
deleted = None
for line in result.stdout.splitlines():
if line.startswith("Number of deleted files:"):
deleted = int(line.split(":", 1)[1].split()[0])
assert deleted == 2, (deleted, result.stdout)
class TestDeleteDelayMaxDeleteParity:
"""--max-delete with --delete-delay: a partial deletion still reports the
number of entries actually removed, matching rsync (the exact surviving set
can differ; only the count is compared)."""
@requires_rsync
def test_max_delete_count_matches_rsync(self):
source = os.path.join(TEST_DATA_DIR, "ddm_src")
rsync_dst = os.path.join(TEST_DATA_DIR, "ddm_rsync_dst")
clean_dir(source)
clean_dir(rsync_dst)
_write(os.path.join(source, "d", "keep.txt"), b"keep\n")
for i in range(1, 6):
_write(os.path.join(rsync_dst, "d", f"e{i}.txt"), f"extra{i}\n".encode())
rsync_result = _rsync(["-a", "--delete-delay", "--max-delete=2", "--stats",
source + "/", rsync_dst + "/"])
# rsync exits 25 ("the --max-delete limit stopped deletions").
assert rsync_result.returncode == 25, rsync_result.stderr
rsync_count = _deleted_count(rsync_result.stdout)
assert rsync_count == 2, rsync_result.stdout
dest = os.path.join(TEST_DATA_DIR, "ddm_dst")
clean_dir(dest)
received = get_dest_received_dir(dest, source)
for i in range(1, 6):
_write(os.path.join(received, "d", f"e{i}.txt"), f"extra{i}\n".encode())
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
result, _ = run_client(
source, dest,
flags=["--delete-delay", "--max-delete=2", "--stats"],
port=server.port,
)
# A capped --max-delete commit is a successful transfer that both tools
# report with exit 25.
assert result.returncode == 25, (result.stderr or result.stdout)[:300]
assert _deleted_count(result.stdout) == rsync_count, result.stdout
def _deleted_count(text):
for line in text.splitlines():
if line.startswith("Number of deleted files:"):
return int(line.split(":", 1)[1].split()[0])
return None
class TestDeleteDelayVsAfterSnapshot:
"""A destination entry created after its directory's scan survives under
--delete-delay but is removed by --delete-after's fresh end scan."""
@pytest.mark.parametrize("mt", [False, True])
def test_late_created_extra_survives_delay_not_after(self, mt):
source, dest, received = _seed_pair(f"latecreate_mt{int(mt)}", big=True)
old_extra = os.path.join(received, "d", "old_extra")
new_extra = os.path.join(received, "d", "new_extra")
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
for timing, new_survives in (("--delete-delay", True),
("--delete-after", False)):
_write(old_extra, b"stale extra\n")
if os.path.exists(new_extra):
os.unlink(new_extra)
def hook():
# Runs on the proxy thread while the big file is in flight,
# after the directory's plan (delay) has been processed.
_write(new_extra, b"created mid-transfer\n")
# --incremental gives the receiver a mid-transfer handshake
# reply; the proxy waits for it (wait_for_reply) so the hook is
# causally after the plan frame, never a timing guess.
# --ignore-times forces the big file to transfer on the second
# timing too (the first run already installed it), keeping the
# mid-transfer reply present in both iterations.
proxy = _SlicingProxy(server.port, hook=hook,
hook_after=MID_TRANSFER_BYTES,
throttle=PROXY_THROTTLE, wait_for_reply=True)
flags = [timing, "--incremental", "--ignore-times"] + (["--threads"] if mt else [])
result, _ = run_client(source, dest, flags=flags, port=proxy.port)
proxy.finish()
assert result.returncode == 0, (
f"{timing}: {(result.stderr or result.stdout)[:300]}"
)
assert proxy.hook_called.is_set(), f"{timing}: hook never fired"
assert not os.path.exists(old_extra), f"{timing}: old extra survived"
assert os.path.exists(new_extra) == new_survives, (
f"{timing} (mt={mt}): new_extra present="
f"{os.path.exists(new_extra)}, expected survives={new_survives}"
)
class TestDeleteAfterThreadsKeepSet:
"""Regression: -j/--threads must still transmit the delete keep-set in every
timing. PipelineContextSender.delete_suppressed was left uninitialized, so a
garbage true silently skipped the late keep-set manifest under --threads.
Plain --delete now uses the per-directory plans, while --delete-commit /
--delete-after keep exercising the late whole-tree manifest."""
@pytest.mark.parametrize("delete_flag", ["--delete", "--delete-commit", "--delete-after"])
def test_threads_delete_after_sends_keep_set(self, delete_flag):
source, dest, received = _seed_pair("mtkeep")
extra = os.path.join(received, "d", "old_extra")
assert os.path.exists(extra)
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
result, _ = run_client(source, dest, flags=["--threads", delete_flag],
port=server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
assert not os.path.exists(extra), (
f"{delete_flag} --threads did not remove an extra: delete keep-set was suppressed"
)
class TestDeleteDelayMaxDeleteRefilledDir:
"""--delete-delay charges the --max-delete budget on ACTUAL removals: the
refilled directory's late content is removed first (consuming the one slot),
so the directory itself and a later extra are skipped, matching rsync.
The refilled directory is at the destination ROOT (its plan is always sent
first) and the skipped extra is under a separate source directory, so the
ordering that decides the budget charge is deterministic -- not readdir
order. The refill is injected through the byte-barrier proxy so it is
causally after the plan frame."""
def test_budget_charged_on_actual_removal(self):
source = os.path.join(TEST_DATA_DIR, "ddmb_src")
dest = os.path.join(TEST_DATA_DIR, "ddmb_dst")
clean_dir(source)
clean_dir(dest)
_write(os.path.join(source, "a", "keep.bin"), b"B" * BIG_BYTES)
_write(os.path.join(source, "b", "keep.txt"), b"keep\n")
received = get_dest_received_dir(dest, source)
refilled_dir = os.path.join(received, "xdir")
os.makedirs(refilled_dir, exist_ok=True)
later_dir = os.path.join(received, "b", "ydir")
os.makedirs(later_dir, exist_ok=True)
def hook():
_write(os.path.join(refilled_dir, "new.txt"), b"created mid-transfer\n")
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
proxy = _SlicingProxy(server.port, hook=hook, hook_after=MID_TRANSFER_BYTES,
throttle=PROXY_THROTTLE, wait_for_reply=True)
flags = ["--delete-delay", "--max-delete=1", "--incremental", "--ignore-times", "--stats"]
result, _ = run_client(source, dest, flags=flags, port=proxy.port)
proxy.finish()
assert result.returncode == 25, (result.stderr or result.stdout)[:400]
assert proxy.hook_called.is_set(), "hook never fired"
# The late content consumes the single budget slot; the refilled
# directory itself and the later extra are skipped.
assert not os.path.exists(os.path.join(refilled_dir, "new.txt")), "late file survived"
assert os.path.isdir(later_dir), "later extra was not skipped by the budget"
# The one actual removal is reported.
assert _deleted_count(result.stdout) == 1, result.stdout
@@ -0,0 +1,719 @@
"""Differential rsync-parity gate.
Runs real ``rsync 3.4.1`` and FastSync over the same corpora and flags, then
compares the destination trees and the normalized output of the
output-oriented flags. This is the executable counterpart of
``RSYNC_COMPAT.md``: the fast subset (``-m parity_ci``) guards the ✅ surface on
every pull request, and the full set (``-m parity``) burns the documented
⚠️/❌ residuals down.
Known, documented differences live in ``parity_caveats.py``; anything else
fails with a readable tree/stdout diff. A stale allowlist entry is reported
loudly (and fails when ``FASTSYNC_PARITY_STRICT=1``).
Run locally::
python3 -m pytest tests/integration/test_differential_parity.py -n 4 --dist=load -m parity_ci
python3 -m pytest tests/integration/test_differential_parity.py -n 4 --dist=load -m parity
"""
import os
import shutil
import sys
import warnings
import pytest
sys.path.insert(0, os.path.dirname(__file__))
from common import ( # noqa: E402
ServerManager,
TEST_DATA_DIR,
clean_dir,
get_dest_received_dir,
)
from parity_caveats import ASPECTS, caveat_for # noqa: E402
import parity_harness as H # noqa: E402
RSYNC = shutil.which("rsync")
requires_rsync = pytest.mark.skipif(RSYNC is None, reason="rsync 3.4.1 not installed")
# `--allow-super` matches the rest of the integration suite; `--allow-delete`
# is needed only by the delete cases.
SUPER = ("--allow-super",)
DELETE = ("--allow-super", "--allow-delete")
_OLD_MTIME = 1_500_000_000
parity = pytest.mark.parity
parity_ci = pytest.mark.parity_ci
@pytest.fixture(scope="session")
def parity_server_factory():
"""Lazily start one server per distinct extra-argument set, per xdist worker."""
servers = {}
def get(extra):
key = tuple(extra)
if key not in servers:
s = ServerManager()
s.start(extra_args=list(extra))
servers[key] = s
return servers[key]
yield get
for s in servers.values():
s.stop()
def _pin(path, mtime):
os.utime(path, (mtime, mtime))
def _mk(path, data, mtime=None):
os.makedirs(os.path.dirname(path), exist_ok=True)
with open(path, "wb") as fh:
fh.write(data)
if mtime is not None:
_pin(path, mtime)
# --- destination seeds ------------------------------------------------------
def seed_extras(_src, rroot, froot):
for root in (rroot, froot):
_mk(os.path.join(root, "extra.txt"), b"extra\n")
_mk(os.path.join(root, "extradir", "z.txt"), b"z\n")
def seed_update(_src, rroot, froot):
for root in (rroot, froot):
p = os.path.join(root, "a.txt")
_mk(p, b"destination is newer and longer\n", 2_000_000_000)
def seed_ignore_existing(_src, rroot, froot):
for root in (rroot, froot):
_mk(os.path.join(root, "a.txt"), b"destination-kept\n", _OLD_MTIME)
def seed_append(_src, rroot, froot):
for root in (rroot, froot):
_mk(os.path.join(root, "a.txt"), b"hello ", _OLD_MTIME)
def seed_backup(_src, rroot, froot):
for root in (rroot, froot):
_mk(os.path.join(root, "a.txt"), b"OLD-CONTENT\n", _OLD_MTIME)
def seed_size_only(_src, rroot, froot):
for root in (rroot, froot):
_mk(os.path.join(root, "a.txt"), b"XXXXXXXXXXX\n", _OLD_MTIME)
def seed_delete_excluded(_src, rroot, froot):
for root in (rroot, froot):
_mk(os.path.join(root, "drop.log"), b"stale log\n", _OLD_MTIME)
_mk(os.path.join(root, "extra.txt"), b"extra\n", _OLD_MTIME)
_mk(os.path.join(root, "keep.txt"), b"keep\n", _OLD_MTIME)
def seed_filter_protect(_src, rroot, froot):
"""Destination-only entries, including nested ones, for the receiver-side
`protect` rule: the `.log` extras must survive --delete, the rest go."""
for root in (rroot, froot):
_mk(os.path.join(root, "extra.log"), b"dest-only log\n", _OLD_MTIME)
_mk(os.path.join(root, "other.txt"), b"dest-only other\n", _OLD_MTIME)
_mk(os.path.join(root, "sub", "extra2.log"), b"nested dest-only log\n", _OLD_MTIME)
_mk(os.path.join(root, "sub", "other2.txt"), b"nested dest-only other\n", _OLD_MTIME)
def seed_max_delete(_src, rroot, froot):
for root in (rroot, froot):
_mk(os.path.join(root, "extra1.txt"), b"e1\n", _OLD_MTIME)
_mk(os.path.join(root, "extra2.txt"), b"e2\n", _OLD_MTIME)
def fuzzy_basis_seed(_src, rroot, froot):
"""Seed a same-suffix sibling whose name is one edit from the source and
whose content matches it, with a DIFFERENT mtime so rsync's exact
size+mtime pass cannot fire: both tools must select it via the
name-distance pass. Where the two tools' basis choices coincide the
block-level results are identical when the block size is pinned."""
for root in (rroot, froot):
_mk(os.path.join(root, "report_v1.txt"), H.FUZZY_PAYLOAD, _OLD_MTIME)
def max_delete_count_check(_src, rroot, froot, _rs, _fs):
"""The exact survivor set is order-dependent; the count must still match."""
r = H.snapshot(rroot)
f = H.snapshot(froot)
if len(r) != len(f):
return [f"survivor count differs: rsync={len(r)} fastsync={len(f)}"]
return []
# --- case table -------------------------------------------------------------
_CASES = [
# --- core archive / recursion -----------------------------------------
H.Case("archive", "basic", ["-a"], ci=True, ref="-a/--archive"),
H.Case("recursive", "basic", ["-r"], ci=True, ref="-r/--recursive"),
H.Case("unicode_names", "unicode", ["-a"], ci=True, ref="-a unicode names"),
H.Case("links_archive", "links", ["-a"], ci=True, ref="-l/--links"),
H.Case("copy_links", "links", ["-aL"], ref="-L/--copy-links"),
H.Case("hardlinks", "hardlinks", ["-a", "-H"], compare_hardlinks=True,
ci=True, ref="-H/--hard-links"),
H.Case("hardlinks_without_H", "hardlinks", ["-a"], compare_hardlinks=True,
ref="hardlinks without -H"),
H.Case("sparse", "sparse", ["-a", "-S"], ref="-S/--sparse"),
# --- compression / checksums ------------------------------------------
H.Case("compress_zstd", "basic", ["-a", "-z"], ci=True, ref="-z/--compress"),
H.Case("checksum", "basic", ["-a", "-c"], ref="-c/--checksum"),
H.Case("checksum_choice_xxh64", "basic",
["-a", "-c", "--checksum-choice=xxh64"], ref="--checksum-choice"),
# --- selection --------------------------------------------------------
H.Case("exclude", "filters", ["-a", "--exclude=*.log"], ci=True,
ref="--exclude"),
H.Case("include_exclude", "filters",
["-a", "--include=*.txt", "--exclude=*"], ci=True,
ref="--include/--exclude ordering"),
H.Case("filter_rules", "filters",
["-a", "-f", "- *.log", "-f", "+ *.txt", "-f", "- *"],
ref="--filter/-f grammar"),
H.Case("max_size", "basic", ["-a", "--max-size=1000"], ref="--max-size"),
H.Case("min_size", "basic", ["-a", "--min-size=1000"], ref="--min-size"),
# --- output-oriented --------------------------------------------------
H.Case("stats", "basic", ["-a", "--stats"], stdout=H.STDOUT_STATS,
ci=True, ref="--stats"),
H.Case("itemize", "links", ["-a", "-i"], stdout=H.STDOUT_ITEMIZE,
ci=True, ref="-i/--itemize-changes"),
H.Case("out_format_n_l", "basic", ["-a", "--out-format=%n %l"],
stdout=H.STDOUT_OUTFMT, ref="--out-format %n %l"),
H.Case("out_format_i_n", "basic", ["-a", "--out-format=%i %n"],
stdout=H.STDOUT_OUTFMT, ref="--out-format %i %n"),
H.Case("progress", "multidir", ["-a", "--progress"], stdout=H.STDOUT_PROGRESS,
ci=True, ref="--progress multi-directory file list"),
H.Case("progress_threads", "multidir", ["-a", "--progress"],
fastsync_flags=["-a", "--progress", "--threads"],
stdout=H.STDOUT_PROGRESS, ci=True,
ref="--progress multi-directory file list (--threads)"),
# --- transfer modifications -------------------------------------------
H.Case("update", "basic", ["-a", "--update"], seed=seed_update,
ref="-u/--update"),
H.Case("ignore_existing", "basic", ["-a", "--ignore-existing"],
seed=seed_ignore_existing, ci=True, ref="--ignore-existing"),
H.Case("size_only", "basic",
["-a", "--size-only"], fastsync_flags=["-a", "--incremental", "--size-only"],
seed=seed_size_only, ref="--size-only"),
H.Case("append", "basic", ["-a", "--append"], seed=seed_append,
ref="--append"),
H.Case("append_verify", "basic", ["-a", "--append-verify"], seed=seed_append,
ref="--append-verify"),
H.Case("backup", "basic", ["-a", "--backup"], seed=seed_backup,
ref="--backup"),
H.Case("chmod", "basic", ["-a", "--chmod=Fu+rwx"], compare_modes=True,
ci=True, ref="--chmod"),
# --- delta / similar-file basis (--fuzzy) -----------------------------
# Basis choices coincide here (same-suffix sibling, name distance one edit,
# content identical); with the block size pinned both tools report the same
# Matched/Literal/transferred counters. The residual (FastSync's narrower
# delta size window) is covered by TestFuzzy in test_parity_quickwins.py.
H.Case("fuzzy_basis", "fuzzy",
["-a", "--no-whole-file", "--fuzzy", "--stats", "-B8192"],
fastsync_flags=["-a", "--incremental", "--delta", "--fuzzy",
"--stats", "--delta-block=8192"],
seed=fuzzy_basis_seed, stdout=H.STDOUT_STATS, ci=True,
ref="-y/--fuzzy similar-file basis"),
# --- deletion ---------------------------------------------------------
H.Case("delete", "basic", ["-a", "--delete"], seed=seed_extras,
server_args=DELETE, ci=True, ref="--delete"),
H.Case("delete_before", "basic", ["-a", "--delete-before"], seed=seed_extras,
server_args=DELETE, ref="--delete-before"),
H.Case("delete_during", "basic", ["-a", "--delete-during"], seed=seed_extras,
server_args=DELETE, ref="--delete-during"),
H.Case("delete_delay", "basic", ["-a", "--delete-delay"], seed=seed_extras,
server_args=DELETE, ref="--delete-delay"),
H.Case("delete_after", "basic", ["-a", "--delete-after"], seed=seed_extras,
server_args=DELETE, ref="--delete-after"),
H.Case("delete_commit", "basic", ["-a", "--delete-after"], seed=seed_extras,
fastsync_flags=["-a", "--delete-commit"], server_args=DELETE,
ref="FastSync-only --delete-commit == rsync --delete-after"),
H.Case("delete_excluded", "filters",
["-a", "--delete", "--delete-excluded", "--exclude=*.log"],
seed=seed_delete_excluded, server_args=DELETE, ref="--delete-excluded"),
H.Case("exclude_protect_dest_only", "filters",
["-a", "--delete", "--exclude=*.log"],
seed=seed_delete_excluded, server_args=DELETE, ci=True,
ref="--delete protects a destination-only excluded entry like rsync"),
H.Case("max_delete", "basic", ["-a", "--delete", "--max-delete=1"],
seed=seed_max_delete, server_args=DELETE,
extra_check=max_delete_count_check, compare_tree=False,
ref="--max-delete"),
H.Case("filter_protect", "filters",
["-a", "--delete", "--filter=P *.log"],
seed=seed_filter_protect, server_args=DELETE, ci=True,
ref="--filter P/--protect receiver-side delete protection (default during)"),
H.Case("filter_protect_during", "filters",
["-a", "--delete-during", "--filter=P *.log"],
seed=seed_filter_protect, server_args=DELETE, ci=True,
ref="--filter P/--protect under --delete-during"),
H.Case("filter_protect_delay", "filters",
["-a", "--delete-delay", "--filter=P *.log"],
seed=seed_filter_protect, server_args=DELETE, ci=True,
ref="--filter P/--protect under --delete-delay"),
H.Case("filter_protect_after", "filters",
["-a", "--delete-after", "--filter=P *.log"],
seed=seed_filter_protect, server_args=DELETE, ci=True,
ref="--filter P/--protect under the whole-tree --delete-after commit"),
# --- relative / dirs --------------------------------------------------
H.Case("relative_general", "basic", ["-a", "-R"], layout=H.MIRROR_ABS,
compare_modes=True, ref="-R/--relative"),
H.Case("relative_no_implied_dirs", "basic",
["-a", "-R", "--no-implied-dirs"], layout=H.MIRROR_ABS,
compare_modes=True, ref="--no-implied-dirs"),
H.Case("files_from", "relative", ["--dirs", "-R"],
files_from=("dir1", "sub/x.txt"), layout=H.RELATIVE, ci=True,
ref="-d/--dirs + --files-from"),
H.Case("dirs_plain", "basic", ["-d"], fs_src_suffix="/",
ref="-d/--dirs (plain)"),
H.Case("empty_dirs_recursive", "empty_dir", ["-a"],
ref="recursive empty-directory residual"),
H.Case("empty_dirs_files_from", "empty_dir", ["--dirs", "-R"],
files_from=("emptydir",), layout=H.RELATIVE, ci=True,
ref="-d/--dirs explicit empty directory"),
# --- codecs -----------------------------------------------------------
H.Case("iconv_identity", "basic", ["-a", "--iconv=UTF-8,UTF-8"],
ref="--iconv identity"),
H.Case("iconv_convert", "iconv",
["-a", "--iconv=ISO-8859-1,UTF-8"],
server_args=("--allow-super", "--iconv=UTF-8"),
ref="--iconv conversion (receiver declares its own charset)"),
# rsync's spec is LOCAL,REMOTE and the destination end's charset is REMOTE
# on a push, so a default server writes the wire (UTF-8) names verbatim.
H.Case("iconv_default_server", "iconv",
["-a", "--iconv=ISO-8859-1,UTF-8"],
ref="--iconv push direction (default receiver charset = REMOTE)"),
# --- partial ----------------------------------------------------------
H.Case("partial_complete", "basic", ["-a", "--partial"], ref="--partial"),
]
# Cases that must always be tolerated (documented ⚠️/❌ residuals) get an
# allowlist entry; the table below stays the exact ✅ surface.
ALL_CASES = _CASES
def _params():
out = []
for case in ALL_CASES:
marks = [parity]
if case.ci:
marks.append(parity_ci)
out.append(pytest.param(case, id=case.id, marks=marks))
return out
def _aspects_to_check(result):
return {
"tree": result["tree"],
"stdout": result["stdout"],
"extra": result["extra"],
}
def _assert_no_unexpected(case_id, mismatches, caveat, ref=""):
unexpected = {a: v for a, v in mismatches.items() if v and a not in caveat}
if unexpected:
lines = [f"differential parity mismatch for case {case_id!r}:"]
lines.append(f" ref: {ref or 'see RSYNC_COMPAT.md'}")
for aspect, detail in unexpected.items():
lines.append(f" --- {aspect} ---")
lines.extend(" " + str(d) for d in detail)
lines.append("If this is a documented residual, add it to "
"tests/integration/parity_caveats.py with a RSYNC_COMPAT.md "
"reference. Do not allowlist an undocumented divergence.")
pytest.fail("\n".join(lines))
stale = [a for a in ASPECTS
if a in caveat and a != "rc" and not mismatches.get(a)]
if stale:
msg = (f"stale parity allowlist entry for case {case_id!r}, aspect(s) "
f"{stale}: FastSync now matches rsync. Remove it from "
f"parity_caveats.py (and update RSYNC_COMPAT.md if the row moved).")
if os.environ.get("FASTSYNC_PARITY_STRICT") == "1":
pytest.fail(msg)
warnings.warn(msg, stacklevel=2)
@requires_rsync
@pytest.mark.parametrize("case", _params())
def test_differential_case(case, parity_server_factory):
server = parity_server_factory(case.server_args)
result = H.execute_case(case, server)
caveat = caveat_for(case.id)
mismatches = _aspects_to_check(result)
if result["rsync_rc"] != result["fastsync_rc"]:
mismatches["rc"] = [
f"rsync rc={result['rsync_rc']} fastsync rc={result['fastsync_rc']} "
f"(rsync stderr: {result['rsync_stderr'][:200]!r}, "
f"fastsync stderr: {result['fastsync_stderr'][:200]!r})"]
_assert_no_unexpected(case.id, mismatches, caveat, ref=case.ref)
# ---------------------------------------------------------------------------
# Multi-run and setup-heavy scenarios (kept as explicit tests)
# ---------------------------------------------------------------------------
def _result_aspects(result):
return _aspects_to_check(result)
_STANDALONE_REFS = {
"incremental_modified": "-i/--itemize-changes + incremental second run",
"compare_dest": "--compare-dest",
"copy_dest": "--copy-dest",
"link_dest": "--link-dest",
"link_dest_stats": "--link-dest + --stats",
"verify_basis": "--verify-basis (FastSync-only)",
"verify_basis_default": "--verify-basis (default quick-check vs rsync)",
"added_and_deleted": "--delete across two runs",
"added_and_deleted_seed": "--delete across two runs",
"one_file_system": "-x/--one-file-system",
}
def _run_and_check(case_id, result, ref=""):
mismatches = _result_aspects(result)
if result["rsync_rc"] != result["fastsync_rc"]:
mismatches["rc"] = [
f"rsync rc={result['rsync_rc']} fastsync rc={result['fastsync_rc']} "
f"(rsync stderr: {result['rsync_stderr'][:200]!r}, "
f"fastsync stderr: {result['fastsync_stderr'][:200]!r})"]
_assert_no_unexpected(case_id, mismatches, caveat_for(case_id),
ref=ref or _STANDALONE_REFS.get(case_id, ""))
@requires_rsync
@parity
def test_incremental_modified_file(parity_server_factory):
"""A second run sends only the modified file; destinations stay identical."""
case_id = "incremental_modified"
src = os.path.join(TEST_DATA_DIR, "parity_inc_src")
rdst = os.path.join(TEST_DATA_DIR, "parity_inc_rdst")
fdst = os.path.join(TEST_DATA_DIR, "parity_inc_fdst")
H.CORPORA["basic"](src)
server = parity_server_factory(SUPER)
# Seed both destinations with the initial content.
H.run_differential(src, rdst, fdst, ["-a"], ["-a"], server,
extra_check=lambda *a: [])
with open(os.path.join(src, "a.txt"), "wb") as fh:
fh.write(b"hello world, now modified and longer\n")
_pin(os.path.join(src, "a.txt"), 1_650_000_000)
result = H.run_differential(
src, rdst, fdst, ["-a", "-i"], ["-a", "-i", "--incremental"], server,
stdout=H.STDOUT_ITEMIZE)
_run_and_check(case_id, result)
def _seed_basis(rel_entries):
def seed(src, rroot, froot):
for root in (rroot, froot):
os.makedirs(root, exist_ok=True)
for rel, data in rel_entries.items():
_mk(os.path.join(root, rel), data)
return seed
@requires_rsync
@parity
def test_compare_dest_skips_basis(parity_server_factory):
"""--compare-dest: a file present in the basis is not copied."""
case_id = "compare_dest"
src = os.path.join(TEST_DATA_DIR, "parity_cmpd_src")
rdst = os.path.join(TEST_DATA_DIR, "parity_cmpd_rdst")
fdst = os.path.join(TEST_DATA_DIR, "parity_cmpd_fdst")
clean_dir(src)
_mk(os.path.join(src, "f.txt"), b"basis-content\n")
_pin(os.path.join(src, "f.txt"), _OLD_MTIME)
server = parity_server_factory(SUPER)
rel = os.path.abspath(src).lstrip(os.sep)
# rsync resolves --compare-dest relative to the destination dir; FastSync
# resolves it under the receive root and appends the mirrored source path.
# Both rely on rsync's size+mtime quick-check, so the basis mtime is pinned
# to the source's to keep the match deterministic across a second boundary.
def seed(_src, rroot, froot):
_mk(os.path.join(rroot, "basis", "f.txt"), b"basis-content\n", _OLD_MTIME)
_mk(os.path.join(fdst, "basis", rel, "f.txt"), b"basis-content\n", _OLD_MTIME)
def extra(_src, rroot, froot, _rs, _fs):
out = []
for label, root in (("rsync", rroot), ("fastsync", froot)):
if os.path.exists(os.path.join(root, "f.txt")):
out.append(f"{label} copied a file that is present in the "
f"compare basis")
return out
result = H.run_differential(
src, rdst, fdst,
["-a", "--compare-dest=basis"],
["-a", f"--compare-dest={os.path.join(fdst, 'basis')}", "--incremental"],
server, seed=seed, ignore_paths=("basis",), extra_check=extra)
_run_and_check(case_id, result)
@requires_rsync
@parity
def test_link_dest_hardlinks_basis(parity_server_factory):
"""--link-dest: an unchanged file is hard-linked to the basis, not copied."""
case_id = "link_dest"
src = os.path.join(TEST_DATA_DIR, "parity_linkd_src")
rdst = os.path.join(TEST_DATA_DIR, "parity_linkd_rdst")
fdst = os.path.join(TEST_DATA_DIR, "parity_linkd_fdst")
clean_dir(src)
_mk(os.path.join(src, "f.txt"), b"link-basis-content\n")
_pin(os.path.join(src, "f.txt"), _OLD_MTIME)
server = parity_server_factory(SUPER)
rel = os.path.abspath(src).lstrip(os.sep)
def seed(_src, rroot, froot):
_mk(os.path.join(rroot, "basis", "f.txt"), b"link-basis-content\n", _OLD_MTIME)
_mk(os.path.join(fdst, "basis", rel, "f.txt"), b"link-basis-content\n", _OLD_MTIME)
def extra(_src, rroot, froot, _rs, _fs):
r_basis = os.stat(os.path.join(rroot, "basis", "f.txt")).st_ino
f_basis = os.stat(os.path.join(fdst, "basis", rel, "f.txt")).st_ino
out = []
for label, root, basis in (("rsync", rroot, r_basis),
("fastsync", froot, f_basis)):
target = os.path.join(root, "f.txt")
if not os.path.exists(target):
out.append(f"{label}: f.txt missing")
elif os.stat(target).st_ino != basis:
out.append(f"{label}: f.txt is not hard-linked to the basis")
return out
result = H.run_differential(
src, rdst, fdst,
["-a", "--link-dest=basis"],
["-a", f"--link-dest={os.path.join(fdst, 'basis')}", "--incremental"],
server, seed=seed, ignore_paths=("basis",), extra_check=extra)
_run_and_check(case_id, result)
@requires_rsync
@parity
def test_link_dest_stats_matches_rsync(parity_server_factory):
"""A basis hit must not be counted as created or literal data: rsync reports
zero for both, so FastSync's receiver tallies must too (regression for the
basis materialization over-report)."""
case_id = "link_dest_stats"
src = os.path.join(TEST_DATA_DIR, "parity_linkds_src")
rdst = os.path.join(TEST_DATA_DIR, "parity_linkds_rdst")
fdst = os.path.join(TEST_DATA_DIR, "parity_linkds_fdst")
clean_dir(src)
_mk(os.path.join(src, "f.txt"), b"link-basis-content\n")
_pin(os.path.join(src, "f.txt"), _OLD_MTIME)
server = parity_server_factory(SUPER)
rel = os.path.abspath(src).lstrip(os.sep)
def seed(_src, rroot, froot):
_mk(os.path.join(rroot, "basis", "f.txt"), b"link-basis-content\n", _OLD_MTIME)
_mk(os.path.join(fdst, "basis", rel, "f.txt"), b"link-basis-content\n", _OLD_MTIME)
result = H.run_differential(
src, rdst, fdst,
["-a", "--link-dest=basis", "--stats"],
["-a", f"--link-dest={os.path.join(fdst, 'basis')}", "--incremental", "--stats"],
server, seed=seed, ignore_paths=("basis",), stdout=H.STDOUT_STATS)
_run_and_check(case_id, result, ref="--link-dest + --stats")
@requires_rsync
@parity
def test_copy_dest_copies_basis(parity_server_factory):
"""--copy-dest: a basis match is materialized as an independent copy with the
source's attributes, matching rsync (copy then fix attributes)."""
case_id = "copy_dest"
src = os.path.join(TEST_DATA_DIR, "parity_copyd_src")
rdst = os.path.join(TEST_DATA_DIR, "parity_copyd_rdst")
fdst = os.path.join(TEST_DATA_DIR, "parity_copyd_fdst")
clean_dir(src)
_mk(os.path.join(src, "f.txt"), b"copy-basis-content\n")
_pin(os.path.join(src, "f.txt"), 1_600_000_000)
os.chmod(os.path.join(src, "f.txt"), 0o755)
server = parity_server_factory(SUPER)
rel = os.path.abspath(src).lstrip(os.sep)
def seed(_src, rroot, froot):
# Basis content matches the source; give the basis a different mode so a
# wrong "keep basis attributes" implementation is visible.
_mk(os.path.join(rroot, "basis", "f.txt"), b"copy-basis-content\n",
1_600_000_000)
os.chmod(os.path.join(rroot, "basis", "f.txt"), 0o644)
_mk(os.path.join(fdst, "basis", rel, "f.txt"), b"copy-basis-content\n",
1_600_000_000)
os.chmod(os.path.join(fdst, "basis", rel, "f.txt"), 0o644)
def extra(_src, rroot, froot, _rs, _fs):
out = []
bases = {"rsync": os.path.join(rroot, "basis", "f.txt"),
"fastsync": os.path.join(fdst, "basis", rel, "f.txt")}
for label, root in (("rsync", rroot), ("fastsync", froot)):
target = os.path.join(root, "f.txt")
if not os.path.exists(target):
out.append(f"{label}: f.txt missing")
continue
if os.stat(target).st_ino == os.stat(bases[label]).st_ino:
out.append(f"{label}: f.txt is hard-linked, not copied")
if (os.stat(target).st_mode & 0o777) != 0o755:
out.append(f"{label}: f.txt mode "
f"{oct(os.stat(target).st_mode & 0o777)} != 0o755")
return out
result = H.run_differential(
src, rdst, fdst,
["-a", "--copy-dest=basis"],
["-a", f"--copy-dest={os.path.join(fdst, 'basis')}", "--incremental"],
server, seed=seed, ignore_paths=("basis",), extra_check=extra,
compare_modes=True)
_run_and_check(case_id, result)
@requires_rsync
@parity
def test_verify_basis_restores_strict_content(parity_server_factory):
"""Default matches rsync's metadata quick-check; FastSync-only
`--verify-basis` restores strict content equality and transfers the source
when a same-size/different-content basis would otherwise be trusted."""
case_id = "verify_basis"
src = os.path.join(TEST_DATA_DIR, "parity_vbasis_src")
rdst = os.path.join(TEST_DATA_DIR, "parity_vbasis_rdst")
fdst = os.path.join(TEST_DATA_DIR, "parity_vbasis_fdst")
clean_dir(src)
_mk(os.path.join(src, "f.txt"), b"AAAA\n")
_pin(os.path.join(src, "f.txt"), _OLD_MTIME)
server = parity_server_factory(SUPER)
rel = os.path.abspath(src).lstrip(os.sep)
def seed(_src, rroot, froot):
# Same size and mtime as the source, different bytes: a metadata
# quick-check trusts it; --verify-basis must not.
for root, basis_rel in ((rroot, os.path.join("basis", "f.txt")),
(fdst, os.path.join("basis", rel, "f.txt"))):
_mk(os.path.join(root, basis_rel), b"BBBB\n", _OLD_MTIME)
# Default: both tools trust the basis (rsync's quick check), so the
# destination carries the basis bytes and the trees match.
result = H.run_differential(
src, rdst, fdst,
["-a", "--link-dest=basis"],
["-a", f"--link-dest={os.path.join(fdst, 'basis')}", "--incremental"],
server, seed=seed, ignore_paths=("basis",))
_run_and_check(case_id + "_default", result)
# --verify-basis (FastSync only): the digest mismatch rejects the basis and
# the source is transferred, so the destination is the source bytes. rsync
# has no such flag; assert the FastSync outcome directly against the source.
fdst2 = os.path.join(TEST_DATA_DIR, "parity_vbasis_fdst2")
clean_dir(fdst2)
for root, basis_rel in ((fdst2, os.path.join("basis", rel, "f.txt")),):
_mk(os.path.join(root, basis_rel), b"BBBB\n", _OLD_MTIME)
result, _ = H.run_fastsync(src, fdst2,
["-a", f"--link-dest={os.path.join(fdst2, 'basis')}",
"--incremental", "--verify-basis"], server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
target = os.path.join(get_dest_received_dir(fdst2, src), "f.txt")
with open(target, "rb") as fh:
assert fh.read() == b"AAAA\n", \
"--verify-basis must reject the same-size/different-content basis"
@requires_rsync
@parity
def test_added_and_deleted_between_runs(parity_server_factory):
"""A source deletion and addition sync correctly under --delete."""
case_id = "added_and_deleted"
src = os.path.join(TEST_DATA_DIR, "parity_addel_src")
rdst = os.path.join(TEST_DATA_DIR, "parity_addel_rdst")
fdst = os.path.join(TEST_DATA_DIR, "parity_addel_fdst")
server = parity_server_factory(DELETE)
H.CORPORA["basic"](src)
seed = seed_extras
result = H.run_differential(
src, rdst, fdst, ["-a", "--delete"], ["-a", "--delete"], server,
seed=seed)
_run_and_check(case_id + "_seed", result)
os.remove(os.path.join(src, "a.txt"))
_mk(os.path.join(src, "added.txt"), b"added between runs\n")
result = H.run_differential(
src, rdst, fdst, ["-a", "--delete", "-i"],
["-a", "--delete", "-i", "--incremental"], server,
stdout=H.STDOUT_ITEMIZE)
_run_and_check(case_id, result)
@requires_rsync
@parity
def test_one_file_system(parity_server_factory):
"""-x emits the mount-point directory but not its contents."""
case_id = "one_file_system"
local = os.stat(".")
shm = "/dev/shm"
if not os.path.isdir(shm):
pytest.skip("/dev/shm not available")
if os.stat(shm).st_dev == local.st_dev:
pytest.skip("no cross-device filesystem available")
src = os.path.join(TEST_DATA_DIR, "parity_ofs_src")
rdst = os.path.join(TEST_DATA_DIR, "parity_ofs_rdst")
fdst = os.path.join(TEST_DATA_DIR, "parity_ofs_fdst")
clean_dir(src)
_mk(os.path.join(src, "keep.txt"), b"keep\n")
probe = os.path.join(shm, f"fastsync_ofs_{os.getpid()}")
shutil.rmtree(probe, ignore_errors=True)
os.makedirs(probe)
_mk(os.path.join(probe, "inside.txt"), b"cross\n")
try:
os.symlink(probe, os.path.join(src, "nested_link"))
server = parity_server_factory(SUPER)
result = H.run_differential(
src, rdst, fdst,
["-a", "--copy-links", "-x"],
["-a", "--copy-links", "-x"], server)
_run_and_check(case_id, result)
finally:
shutil.rmtree(probe, ignore_errors=True)
@requires_rsync
@parity
def test_parity_caveats_reference_known_cases():
"""Every allowlist entry must name a real case id and aspect."""
from parity_caveats import CAVEATS
known = {c.id for c in ALL_CASES} | {
"incremental_modified", "compare_dest", "link_dest",
"added_and_deleted", "added_and_deleted_seed", "one_file_system",
}
problems = []
for case_id, entry in CAVEATS.items():
if case_id not in known:
problems.append(f"unknown case id in parity_caveats.py: {case_id!r}")
for aspect in entry:
if aspect not in ASPECTS:
problems.append(
f"{case_id!r}: unknown aspect {aspect!r} (expected {ASPECTS})")
assert not problems, "\n".join(problems)
+1 -1
View File
@@ -36,7 +36,7 @@ from common import ( # noqa: E402
verify_transfer,
)
PROTOCOL_VERSION = b"2.23.0"
PROTOCOL_VERSION = b"2.28.0"
STATUS_MANIFEST = 5
STATUS_OK = 0
+489 -106
View File
@@ -583,6 +583,57 @@ class TestRemoteDryRun:
assert os.path.exists(extra), f"{flags} deleted an extra in dry-run"
assert _snapshot_tree(received) == before, f"{flags} mutated the destination"
@pytest.mark.skipif(shutil.which("rsync") is None, reason="rsync not installed")
def test_dry_run_delete_lines_match_rsync(self):
"""`-n --delete` lists exactly the destination extras rsync would remove.
Covers the three cases that a real run protects: the file being updated
(in the keep set), a filter-excluded source entry (protected prefix), and
a --max-size-pruned source entry (always-protected prefix). Track 4a
adds a fourth: a destination-only entry matching the exclude rule is
re-derived on the receiver and also protected, so only the genuine
destination-only `extra.txt` appears.
"""
source = os.path.join(TEST_DATA_DIR, "dryrep_src")
rdst = os.path.join(TEST_DATA_DIR, "dryrep_rdst")
fdst = os.path.join(TEST_DATA_DIR, "dryrep_fdst")
clean_dir(source)
clean_dir(rdst)
clean_dir(fdst)
for name, data in (("a.txt", b"new content\n"), ("keep.log", b"log\n"),
("big.bin", b"B" * 2000)):
with open(os.path.join(source, name), "wb") as fh:
fh.write(data)
os.utime(os.path.join(source, "a.txt"), (1_700_000_000, 1_700_000_000))
received = get_dest_received_dir(fdst, source)
os.makedirs(received, exist_ok=True)
for root in (rdst, received):
for name, data in (("a.txt", b"old\n"), ("keep.log", b"log\n"),
("big.bin", b"B" * 2000), ("extra.txt", b"extra\n"),
("stray.log", b"dest only\n")):
with open(os.path.join(root, name), "wb") as fh:
fh.write(data)
os.utime(os.path.join(root, name), (1_500_000_000, 1_500_000_000))
flags = ["-a", "-n", "-i", "--delete", "--exclude=*.log", "--max-size=1000"]
r = subprocess.run(["rsync", "-an", "-i", "--delete", "--exclude=*.log",
"--max-size=1000", source + "/", rdst + "/"],
capture_output=True, text=True,
env=dict(os.environ, LC_ALL="C"))
assert r.returncode == 0, r.stderr
rsync_del = sorted(l for l in r.stdout.splitlines() if l.startswith("*deleting"))
assert rsync_del == ["*deleting extra.txt"], f"unexpected rsync set: {rsync_del}"
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
result, _ = run_client(source, fdst, flags=flags, port=server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
fs_del = sorted(l for l in (result.stdout or "").splitlines()
if l.startswith("*deleting"))
assert fs_del == rsync_del, f"rsync={rsync_del}\nfastsync={fs_del}"
assert os.path.exists(os.path.join(received, "stray.log")), \
"destination-only exclude match must be protected in the dry-run report"
@pytest.mark.ci
def test_remote_dry_run_quiet_is_silent(self, shared_server):
source = os.path.join(TEST_DATA_DIR, "remote_dry_quiet_src")
@@ -1491,20 +1542,101 @@ class TestChecksumChoice:
assert fh.read() == b"same content\n"
@pytest.mark.ci
def test_checksum_choice_md4_single_name_rejected(self, shared_server):
for bad in ("md4", "sha1", "none", "xxh64,md5"):
@pytest.mark.parametrize("algo", ["xxh128", "xxh3", "xxh64", "md5", "md4", "sha1"])
def test_checksum_choice_all_algorithms_transfer(self, shared_server, algo):
"""Every rsync 3.4.1 checksum algorithm is accepted and transfers
byte-exactly. 'none' is covered separately (it needs no digest)."""
clean_dir(DEST_DIR)
flags = ["--preserve", "--incremental", "--checksum", f"--checksum-choice={algo}"]
result, _ = run_client(SOURCE_DIR, DEST_DIR, flags=flags, port=shared_server.port)
assert result.returncode == 0, \
f"checksum-choice={algo} failed: {(result.stderr or result.stdout)[:300]}"
received = get_dest_received_dir(DEST_DIR, SOURCE_DIR)
mismatches, missing = verify_transfer(SOURCE_DIR, received)
assert not missing, f"Missing: {missing}"
assert not mismatches, f"Mismatch: {mismatches}"
@pytest.mark.ci
def test_checksum_choice_two_name_form(self, shared_server):
"""The rsync 'TRANSFER,PRE-TRANSFER' form is accepted; FastSync uses the
second (pre-transfer) algorithm for its whole-file digest."""
clean_dir(DEST_DIR)
flags = ["--preserve", "--incremental", "--checksum", "--cc=md4,sha1"]
result, _ = run_client(SOURCE_DIR, DEST_DIR, flags=flags, port=shared_server.port)
assert result.returncode == 0, \
f"two-name --cc=md4,sha1 failed: {(result.stderr or result.stdout)[:300]}"
received = get_dest_received_dir(DEST_DIR, SOURCE_DIR)
mismatches, missing = verify_transfer(SOURCE_DIR, received)
assert not missing and not mismatches, f"missing={missing} mismatches={mismatches}"
@pytest.mark.ci
def test_checksum_choice_none_accepted_without_checksum(self, shared_server):
clean_dir(DEST_DIR)
result, _ = run_client(SOURCE_DIR, DEST_DIR,
flags=["--preserve", "--incremental", "--cc=none"],
port=shared_server.port)
assert result.returncode == 0, \
f"--cc=none failed: {(result.stderr or result.stdout)[:300]}"
received = get_dest_received_dir(DEST_DIR, SOURCE_DIR)
mismatches, missing = verify_transfer(SOURCE_DIR, received)
assert not missing and not mismatches, f"missing={missing} mismatches={mismatches}"
@pytest.mark.ci
def test_checksum_choice_none_rejected_with_checksum(self, shared_server):
"""rsync rejects 'none' as the pre-transfer checksum with --checksum and
exits 4; mirror both the rejection and the exit code."""
for choice in ("none", "md5,none"):
result, _ = run_client(SOURCE_DIR, DEST_DIR,
flags=["--checksum", f"--cc={choice}"],
port=shared_server.port)
assert result.returncode == 4, \
f"--cc={choice} --checksum must exit 4, got {result.returncode}: " \
f"{(result.stderr or result.stdout)[:200]}"
@pytest.mark.ci
def test_checksum_choice_unknown_rejected_exit_4(self, shared_server):
for bad in ("sha256", "bogus", "md5,", "md4,md5,sha1"):
result, _ = run_client(SOURCE_DIR, DEST_DIR,
flags=[f"--checksum-choice={bad}"],
port=shared_server.port)
assert result.returncode != 0, f"{bad} must be rejected"
assert result.returncode == 4, \
f"--checksum-choice={bad} must exit 4, got {result.returncode}"
@pytest.mark.ci
def test_compress_choice_unsupported_rejected(self, shared_server):
for bad in ("lz4", "zlib", "zlibx"):
@pytest.mark.parametrize("algo", ["zstd", "lz4", "zlib", "zlibx"])
def test_compress_choice_all_algorithms_transfer(self, shared_server, algo):
"""Every rsync 3.4.1 compression codec is accepted and transfers
byte-exactly through its own codec."""
clean_dir(DEST_DIR)
flags = ["-z", f"--compress-choice={algo}"]
result, _ = run_client(SOURCE_DIR, DEST_DIR, flags=flags, port=shared_server.port)
assert result.returncode == 0, \
f"--compress-choice={algo} failed: {(result.stderr or result.stdout)[:300]}"
received = get_dest_received_dir(DEST_DIR, SOURCE_DIR)
mismatches, missing = verify_transfer(SOURCE_DIR, received)
assert not missing, f"Missing: {missing}"
assert not mismatches, f"Mismatch: {mismatches}"
@pytest.mark.ci
def test_compress_choice_unknown_rejected_exit_4(self, shared_server):
for bad in ("bogus", "zstd,lz4", ""):
result, _ = run_client(SOURCE_DIR, DEST_DIR,
flags=[f"--compress-choice={bad}"],
port=shared_server.port)
assert result.returncode != 0, f"{bad} must be rejected"
assert result.returncode == 4, \
f"--compress-choice={bad} must exit 4, got {result.returncode}"
@pytest.mark.ci
def test_compress_choice_none_disables_compression(self, shared_server):
clean_dir(DEST_DIR)
result, _ = run_client(SOURCE_DIR, DEST_DIR,
flags=["-z", "--compress-choice=none"],
port=shared_server.port)
assert result.returncode == 0, \
f"--compress-choice=none failed: {(result.stderr or result.stdout)[:300]}"
received = get_dest_received_dir(DEST_DIR, SOURCE_DIR)
mismatches, missing = verify_transfer(SOURCE_DIR, received)
assert not missing and not mismatches, f"missing={missing} mismatches={mismatches}"
@pytest.mark.ci
def test_compress_choice_auto_transfers(self, shared_server):
@@ -1855,8 +1987,8 @@ class TestDelete:
)
assert result.returncode == 0, f"Exit {result.returncode}: {result.stderr[:100]}"
output = result.stdout + result.stderr
assert "Sent " in output and "MB" in output, "--progress produced no stable byte marker"
assert "Done." in output, "--progress did not report completion"
assert "sending incremental file list" in output, "--progress produced no rsync header"
assert "(xfr#" in output, "--progress produced no per-file xfr block"
def test_human_readable_stats(self, shared_server):
clean_dir(DEST_DIR)
@@ -1891,8 +2023,8 @@ class TestDelete:
)
assert result.returncode == 0, f"Exit {result.returncode}: {result.stderr[:100]}"
output = result.stdout + result.stderr
assert "Sent " in output
assert "Done." in output
assert "sending incremental file list" in output
assert "(xfr#" in output
class TestInfo:
@@ -2295,6 +2427,61 @@ class TestTempDir:
assert not mismatches, f"Mismatch: {mismatches}"
self._assert_clean_scratch(os.path.join(dest, "scratch"))
@pytest.mark.skipif(shutil.which("rsync") is None, reason="rsync not installed")
def test_relative_temp_dir_matches_rsync_absolute_rejected(self):
"""Differential: a relative --temp-dir is resolved under the destination
by both (rsync 3.4.1 and FastSync), producing identical trees. An
absolute --temp-dir is used verbatim by rsync standalone, but the
receiver deliberately confines it to the receive root (security
invariant), so FastSync rejects it without writing outside the root.
"""
source = self._make_source("tempdir_diff_src")
rdst = os.path.join(TEST_DATA_DIR, "tempdir_diff_rdst")
fdst = os.path.join(TEST_DATA_DIR, "tempdir_diff_fdst")
clean_dir(rdst)
clean_dir(fdst)
os.makedirs(os.path.join(rdst, "scratch"), exist_ok=True)
os.makedirs(os.path.join(fdst, "scratch"), exist_ok=True)
r = subprocess.run(["rsync", "-a", "--temp-dir=scratch", source + "/", rdst + "/"],
capture_output=True, text=True,
env=dict(os.environ, LC_ALL="C"))
assert r.returncode == 0, r.stderr
with ServerManager() as server:
server.start()
result, _ = run_client(source, fdst, flags=["--temp-dir=scratch"],
port=server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
# rsync lays the source contents directly in rdst; FastSync mirrors the
# absolute source path below fdst. Compare the mirrored content trees
# (the scratch dir lives at each destination root).
rtree = sorted(os.path.relpath(os.path.join(dp, n), rdst)
for dp, dn, fn in os.walk(rdst)
for n in dn + fn if os.path.join(dp, n) != os.path.join(rdst, "scratch"))
mirror = get_dest_received_dir(fdst, source)
ftree = sorted(os.path.relpath(os.path.join(dp, n), mirror)
for dp, dn, fn in os.walk(mirror) for n in dn + fn)
assert rtree == ftree, f"relative temp-dir tree mismatch: {rtree} != {ftree}"
assert _walk_tmp_files(os.path.join(rdst, "scratch")) == []
assert _walk_tmp_files(os.path.join(fdst, "scratch")) == []
# Absolute temp dir: rsync accepts it; FastSync rejects it safely.
abs_scratch = os.path.join(TEST_DATA_DIR, "tempdir_diff_abs")
clean_dir(abs_scratch)
rdst2 = os.path.join(TEST_DATA_DIR, "tempdir_diff_rdst2")
clean_dir(rdst2)
r2 = subprocess.run(["rsync", "-a", "--temp-dir=" + abs_scratch, source + "/", rdst2 + "/"],
capture_output=True, text=True,
env=dict(os.environ, LC_ALL="C"))
assert r2.returncode == 0, r2.stderr
fdst2 = os.path.join(TEST_DATA_DIR, "tempdir_diff_fdst2")
clean_dir(fdst2)
with ServerManager() as server:
server.start()
result2, _ = run_client(source, fdst2, flags=["--temp-dir", abs_scratch],
port=server.port)
assert result2.returncode != 0, "an absolute --temp-dir must be rejected (confined)"
assert os.listdir(abs_scratch) == [], "receiver wrote into an unconfined temp dir"
def test_default_behavior_has_no_scratch_dir(self, shared_server):
source = self._make_source("tempdir_default_src")
dest = os.path.join(TEST_DATA_DIR, "tempdir_default_dst")
@@ -2756,6 +2943,41 @@ class TestDelayUpdates:
assert not os.path.isdir(os.path.join(delay_dest, self.STAGING)), \
"staging directory left behind after a successful delayed transfer"
@pytest.mark.skipif(shutil.which("rsync") is None, reason="rsync not installed")
def test_delay_updates_staging_name_collision_residual(self):
"""Documented residual (RSYNC_COMPAT.md `--delay-updates` row): FastSync
uses a fixed `.fastsync-stage` staging name and wipes a pre-existing tree
of that name at the start of a delayed run (crash-leftover cleanup),
even without `--delete`; rsync leaves a genuine destination entry of that
name untouched. Pins the divergence that keeps the row Divergent."""
source = self._make_source("delay_collide_src")
rdst = os.path.join(TEST_DATA_DIR, "delay_collide_rdst")
fdst = os.path.join(TEST_DATA_DIR, "delay_collide_fdst")
clean_dir(rdst)
clean_dir(fdst)
for root in (rdst, fdst):
with open(os.path.join(root, "top.txt"), "wb") as fh:
fh.write(b"old\n")
stage = os.path.join(root, self.STAGING)
os.makedirs(stage, exist_ok=True)
with open(os.path.join(stage, "keepme.txt"), "wb") as fh:
fh.write(b"genuine user data\n")
r = subprocess.run(["rsync", "-a", "--delay-updates", source + "/", rdst + "/"],
capture_output=True, text=True,
env=dict(os.environ, LC_ALL="C"))
assert r.returncode == 0, r.stderr
assert os.path.exists(os.path.join(rdst, self.STAGING, "keepme.txt")), \
"rsync removed an unrelated destination entry named like the staging dir"
with ServerManager() as server:
server.start()
result, _ = run_client(source, fdst, flags=["--delay-updates"],
port=server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
assert not os.path.exists(os.path.join(fdst, self.STAGING)), \
"FastSync did not wipe the reserved staging name (residual changed)"
@pytest.mark.parametrize("mt", [False, True])
def test_delay_updates_incremental_rerun_no_leftovers(self, shared_server, mt):
source = self._make_source("delay_rerun_src")
@@ -3157,15 +3379,21 @@ class TestMissingArgs:
assert not os.path.exists(os.path.join(received, "gone1.txt"))
@pytest.mark.parametrize("mt", [False, True])
def test_empty_list_stays_a_hard_error(self, shared_server, mt):
def test_empty_list_succeeds_transferring_nothing(self, shared_server, mt):
"""rsync 3.4.1 treats an empty --files-from list as "nothing to
transfer" and exits 0 (verified with the real binary), so fastsync must
too rather than reporting a hard error."""
source = self._make_source("mg_empty_src")
dest = os.path.join(TEST_DATA_DIR, "mg_empty_dst")
clean_dir(dest)
lst = _write_rel_list(b"")
flags = ["--files-from", lst, "--ignore-missing-args"] + (["--threads"] if mt else [])
result, _ = run_client(source, dest, flags=flags, port=shared_server.port)
assert result.returncode != 0, "an empty --files-from list must stay a hard error"
assert "contains no entries" in (result.stderr or result.stdout)
assert result.returncode == 0, \
f"an empty --files-from list must succeed like rsync: {result.stderr[:300]}"
received = get_dest_received_dir(dest, source)
assert not os.path.exists(os.path.join(received, "a.txt")), \
"an empty --files-from list must transfer nothing"
@pytest.mark.parametrize("mt", [False, True])
def test_delete_missing_removes_mirror_not_unrelated(self, mt):
@@ -3436,23 +3664,25 @@ class TestMissingArgs:
class TestNoImpliedDirs:
"""--no-implied-dirs (only meaningful with -R + --files-from) refuses to
place a listed file whose parent directory is not itself listed."""
"""--no-implied-dirs (meaningful with -R) omits the source metadata of a
listed path's implied parent directories but still creates those parents
with default attributes, matching rsync 3.4.1."""
def _make(self):
return _make_relative_source("noimplied_src")
@pytest.mark.parametrize("mt", [False, True])
def test_implied_dir_only_fails_entry(self, shared_server, mt):
def test_implied_dir_created_with_default_attrs(self, shared_server, mt):
source = self._make()
dest = os.path.join(TEST_DATA_DIR, "noimplied_dst")
clean_dir(dest)
lst = _write_rel_list(b"a/b.txt\n") # "a" itself is not listed
flags = ["--files-from", lst, "-R", "--no-implied-dirs"] + (["--threads"] if mt else [])
result, _ = run_client(source, dest, flags=flags, port=shared_server.port)
assert result.returncode != 0, "implied parent directory was not rejected"
assert "--no-implied-dirs" in (result.stderr or result.stdout)
assert not os.path.exists(os.path.join(dest, "a", "b.txt"))
assert result.returncode == 0, \
f"implied parent directory was not created: {result.stderr[:200]}"
assert os.path.isdir(os.path.join(dest, "a")), "implied parent 'a' was not created"
assert _read_file(os.path.join(dest, "a", "b.txt")) == b"nested\n"
@pytest.mark.parametrize("mt", [False, True])
def test_listed_dir_allows_file(self, shared_server, mt):
@@ -3579,9 +3809,10 @@ class TestDirs:
files.extend(os.path.relpath(os.path.join(root, n), mirror) for n in names)
assert files == [], f"--dirs descended into contents: {files}"
def test_dirs_listed_dir_colliding_with_file_fails(self, shared_server):
"""A listed directory that already exists as a regular file at the
destination fails the transfer cleanly instead of clobbering the file."""
def test_dirs_listed_dir_replaces_blocking_file(self, shared_server):
"""rsync parity: a listed directory replaces a regular file already at
its destination path (rsync removes the non-directory and creates the
directory)."""
source = self._make()
dest = os.path.join(TEST_DATA_DIR, "dirs_coll_dst")
clean_dir(dest)
@@ -3591,8 +3822,10 @@ class TestDirs:
lst = _write_rel_list(b"dir1\n")
result, _ = run_client(source, dest, flags=["--files-from", lst, "--dirs", "-R"],
port=shared_server.port)
assert result.returncode != 0, "dir entry over an existing file did not fail"
assert os.path.isfile(blocker), "blocking regular file was clobbered"
assert result.returncode == 0, \
f"dir entry over an existing file failed: {(result.stderr or result.stdout)[:300]}"
assert os.path.isdir(blocker) and not os.path.islink(blocker), \
"blocking regular file was not replaced by the incoming directory"
class TestMkpath:
@@ -3749,15 +3982,16 @@ class TestDeleteTiming:
assert _read_file(os.path.join(received, "sub", "deep.txt")) == b"deeply nested file\n", \
f"{flag}: nested file was not written after the early deletion"
@pytest.mark.parametrize("flag", ["--delete", "--delete-after", "--delete-delay"])
@pytest.mark.parametrize("flag", ["--delete-commit", "--delete-after"])
@pytest.mark.parametrize("mt", [False, True])
def test_late_flags_commit_only_after_success(self, flag, mt):
"""Plain --delete/--delete-after/--delete-delay defer deletion until the
whole transfer succeeds: a mid-transfer write failure must leave every
extra in place (commit-style safety). The -m receiver must also keep
the extras: the deferred keep-set is committed by the server only after
the disk-writer thread has finished, and a failing writer means the
manifest is freed, never applied."""
"""--delete-commit/--delete-after defer deletion until the whole transfer
succeeds: a mid-transfer write failure must leave every extra in place
(commit-style safety). Plain --delete no longer defers (it defaults to
delete-during), so only the explicitly late timings are exercised here.
The --threads receiver must also keep the extras: the deferred keep-set is
committed by the server only after the disk-writer thread has finished,
and a failing writer means the manifest is freed, never applied."""
source = self._seed("late")
dest = os.path.join(TEST_DATA_DIR, "deltiming_late_dst")
clean_dir(dest)
@@ -3783,10 +4017,42 @@ class TestDeleteTiming:
assert os.path.isfile(blocker), \
f"{flag} (mt={mt}) deleted the blocker although the transfer failed"
def test_early_flag_respected_when_server_refuses_delete(self, shared_server):
"""With an --allow-delete-less server the client's early timing still
completes (no deadlock on the pre-delete ack) and simply never deletes,
exactly like the plain server policy."""
@pytest.mark.parametrize("mt", [False, True])
def test_delete_delay_clears_type_conflict_like_rsync(self, mt):
"""rsync clears a destination file that blocks a source directory even
when the deletion itself is deferred (--delete-delay); the type conflict
is resolved immediately so the nested write succeeds. The transfer must
therefore succeed and the unrelated extra must still be removed."""
source = self._seed("delayconflict")
dest = os.path.join(TEST_DATA_DIR, "deltiming_delayconflict_dst")
clean_dir(dest)
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
result, _ = run_client(source, dest, port=server.port)
assert result.returncode == 0, f"seed sync failed: {result.stderr[:200]}"
received = get_dest_received_dir(dest, source)
extra = os.path.join(received, "extra.txt")
with open(extra, "wb") as fh:
fh.write(b"extra file")
blocker = os.path.join(received, "sub")
shutil.rmtree(blocker)
with open(blocker, "wb") as fh:
fh.write(b"blocks the nested destination directory")
flags = ["--delete-delay"] + (["--threads"] if mt else [])
result, _ = run_client(source, dest, flags=flags, port=server.port)
assert result.returncode == 0, \
f"--delete-delay (mt={mt}) did not clear the type conflict: " \
f"{(result.stderr or result.stdout)[:300]}"
assert os.path.isdir(blocker), "blocker file was not replaced by the source directory"
assert _read_file(os.path.join(received, "sub", "deep.txt")) == b"deeply nested file\n"
assert not os.path.exists(extra), "--delete-delay did not remove the extra"
@pytest.mark.parametrize("flag", ["--delete-before", "--delete-during", "--delete-delay"])
def test_early_flag_respected_when_server_refuses_delete(self, flag, shared_server):
"""With an --allow-delete-less server the client's timing still completes
(no deadlock on the pre-delete ack, no per-directory deletion) and simply
never deletes, exactly like the plain server policy."""
source = self._seed("refused")
dest = os.path.join(TEST_DATA_DIR, "deltiming_refused_dst")
clean_dir(dest)
@@ -3796,9 +4062,9 @@ class TestDeleteTiming:
extra = os.path.join(received, "extra.txt")
with open(extra, "wb") as fh:
fh.write(b"extra file")
result, _ = run_client(source, dest, flags=["--delete-before"], port=shared_server.port)
result, _ = run_client(source, dest, flags=[flag], port=shared_server.port)
assert result.returncode == 0, \
f"--delete-before against a refuse-delete server failed: {result.stderr[:300]}"
f"{flag} against a refuse-delete server failed: {result.stderr[:300]}"
assert os.path.exists(extra), "unauthorized delete removed an extra file"
@@ -3893,6 +4159,31 @@ class TestDeleteScope:
finally:
server.stop()
@pytest.mark.parametrize("mt", [False, True])
@pytest.mark.parametrize("timing", ["--delete-during", "--delete-delay"])
@pytest.mark.ci
def test_files_from_per_dir_timing_confined_to_listed_dirs(self, mt, timing):
"""The per-directory timings honor the same --files-from scope: an extra
inside a listed directory is removed, while unlisted siblings and the
receive-root extra survive."""
source, dest, received, server = self._seed(f"pd_{timing.strip('-')}_{mt}")
try:
listed = _write_rel_list(b"listed.txt\nsub/\n")
flags = ["--files-from", listed, timing] + (["--threads"] if mt else [])
result, _ = run_client(source, dest, flags=flags, port=server.port)
assert result.returncode == 0, f"{timing} delete failed: {result.stderr[:300]}"
assert not os.path.exists(os.path.join(received, "sub", "extra.txt")), \
f"{timing} did not delete the in-scope extra"
assert os.path.isfile(os.path.join(received, "sub", "x.txt"))
assert os.path.exists(os.path.join(received, "unlisted.txt")), \
f"{timing} deleted an unlisted path (data loss)"
assert os.path.exists(os.path.join(received, "other", "c.txt")), \
f"{timing} deleted an unlisted sibling directory (data loss)"
assert os.path.exists(os.path.join(received, "rootextra.txt")), \
f"{timing} deleted the receive-root extra (data loss)"
finally:
server.stop()
class TestDeleteExtraneousSymlinks:
"""#290 (3): --delete unlinks extraneous destination symlinks (never follows
@@ -3980,7 +4271,8 @@ class TestDeletePolicy:
@pytest.mark.parametrize("mt", [False, True])
@pytest.mark.parametrize("timing",
["--delete", "--delete-before", "--delete-after", "--delete-delay"])
["--delete", "--delete-before", "--delete-after", "--delete-delay",
"--delete-during"])
def test_delete_protects_excluded_by_default_and_delete_excluded_removes(self, mt, timing):
"""rsync parity: with a --delete timing the destination mirror path whose
source was excluded survives (protected by default); --delete-excluded
@@ -4064,7 +4356,7 @@ class TestDeletePolicy:
"--delete-excluded did not remove the excluded dir subtree"
@pytest.mark.parametrize("mt", [False, True])
@pytest.mark.parametrize("timing", ["--delete", "--delete-before"])
@pytest.mark.parametrize("timing", ["--delete", "--delete-before", "--delete-during"])
def test_max_delete_exceeded_deletes_up_to_cap_and_exits_25(self, mt, timing):
"""rsync parity: --max-delete=N deletes up to N extras, skips the rest and
still succeeds as a transfer, exiting 25 with a diagnostic."""
@@ -4318,11 +4610,11 @@ class TestDeletePolicy:
@pytest.mark.parametrize("mt", [False, True])
@pytest.mark.setpriv
def test_ignore_errors_keeps_deletion_active_on_scan_error(self, mt):
"""A source I/O error (unreadable subdirectory) aborts the run so no
deletion happens by default; --ignore-errors continues, still transfers
the readable tree and still deletes, single-threaded and under -m. Run
as an unprivileged user so the mode-000 directory is genuinely
unreadable."""
"""rsync's --ignore-errors semantics: a source I/O error (unreadable
subdirectory) makes the run continue and transfer the readable tree, but
the default suppresses deletion ("IO error encountered -- skipping file
deletion"); --ignore-errors lets deletion proceed. Both exit 23. Run as
an unprivileged user so the mode-000 directory is genuinely unreadable."""
if os.geteuid() != 0 or shutil.which("setpriv") is None:
pytest.skip("requires root + setpriv to drop privileges for the client")
tag = f"ioerr_{os.getpid()}_{mt}"
@@ -4342,11 +4634,15 @@ class TestDeletePolicy:
try:
os.chmod(os.path.join(source, "locked"), 0)
# Default: scan error aborts the run; nothing is deleted.
# Default: the scan continues past the unreadable dir and the
# readable tree transfers, but deletion is skipped (exit 23).
self._write(os.path.join(received, "extra.txt"), b"extra\n")
flags = ["--delete"] + (["--threads"] if mt else [])
result = self._run_client_as_nobody(source, dest, server.port, flags)
assert result.returncode != 0, "unreadable source dir did not fail the run"
assert result.returncode == 23, \
f"unreadable source dir should exit 23 (got {result.returncode})"
assert os.path.exists(os.path.join(received, "top.txt")), \
"readable tree did not transfer past the I/O error"
assert os.path.exists(os.path.join(received, "extra.txt")), \
"default run deleted although the scan hit an I/O error"
@@ -4354,6 +4650,8 @@ class TestDeletePolicy:
self._write(os.path.join(received, "extra.txt"), b"extra\n")
flags = ["--delete", "--ignore-errors"] + (["--threads"] if mt else [])
result = self._run_client_as_nobody(source, dest, server.port, flags)
assert result.returncode == 23, \
f"--ignore-errors run should still exit 23 (got {result.returncode})"
assert not os.path.exists(os.path.join(received, "extra.txt")), \
f"--ignore-errors did not keep deletion active: {result.stderr[:300]}"
assert not os.path.exists(os.path.join(received, "locked")), \
@@ -4398,11 +4696,11 @@ class TestDeletePolicy:
finally:
os.chmod(source, 0o755)
def test_delete_excluded_protection_is_sender_derived(self):
def test_delete_protection_reapplied_on_receiver(self):
"""Plain --delete protects destination mirrors of files the SOURCE scan
excluded, but a destination-only file that merely matches an exclude
rule is still an extra and is removed (protection never re-applies rules
to the destination)."""
excluded, and (track 4a) also protects a destination-only file matching
an exclude rule because the compiled rule set is re-applied on the
receiver, matching rsync."""
source = os.path.join(TEST_DATA_DIR, "senderderived_src")
clean_dir(source)
self._write(os.path.join(source, "keep.txt"), b"kept\n")
@@ -4422,8 +4720,8 @@ class TestDeletePolicy:
f"delete sync failed: {(result.stderr or result.stdout)[:300]}"
assert os.path.exists(os.path.join(received, "secret.log")), \
"source-excluded mirror was deleted under plain --delete"
assert not os.path.exists(os.path.join(received, "stray.log")), \
"destination-only file matching the exclude rule was left (should be deleted)"
assert os.path.exists(os.path.join(received, "stray.log")), \
"destination-only file matching the exclude rule must be protected like rsync"
def _pin_mtime(path, ts):
@@ -4443,11 +4741,12 @@ class TestBasisDestDirs:
STAGING = ".fastsync-stage"
TS = 1577836800 # 2020-01-01 00:00:00 UTC, used to pin matching mtimes
# fixture files: source and basis share the mtime pin, so a basis "match"
# is decided purely by content (xxHash). unchanged.txt is byte-identical;
# changed.txt is byte-DIFFERENT but has the SAME SIZE as the source (and
# the same pinned mtime), which is what forces the content-hash gate;
# added.txt does not exist in the basis at all.
# fixture files: source and basis share the mtime pin, so the DEFAULT
# (rsync-parity) quick-check is a size+mtime match and trusts the basis even
# when the body differs. unchanged.txt is byte-identical; changed.txt is
# byte-DIFFERENT but has the SAME SIZE as the source (and the same pinned
# mtime), which is what the FastSync-only --verify-basis content gate
# rejects; added.txt does not exist in the basis at all.
UNCHANGED = "unchanged.txt"
CHANGED = "changed.txt"
ADDED = "added.txt"
@@ -4487,18 +4786,19 @@ class TestBasisDestDirs:
}
def _basis_tree(self, prefix):
# unchanged.txt is identical to the source; changed.txt has the SAME
# byte size and pinned mtime but a different body (equal size forces
# the xxHash gate); added.txt is missing from the basis.
# unchanged.txt is identical to the source; changed.txt has a DIFFERENT
# size (and body) so the size leg of the quick-check fails and it is
# transferred normally; added.txt is missing from the basis.
return {
self.UNCHANGED: b"stable content v1\n",
self.CHANGED: b"CHANGED CONTENT NOW\n",
self.CHANGED: b"CHANGED CONTENT NOW AND LONGER\n",
}
def test_same_size_different_content_is_not_a_basis_match(self, shared_server):
# Core safety property: equal size + pinned mtime but different content
# must NEVER be hard-linked or copied from the basis -- the xxHash gate
# rejects it and the sender's data is transferred instead.
def test_same_size_different_content_default_trusts_quick_check(self, shared_server):
# Default rsync-parity behavior: equal size + pinned mtime is a basis
# match, so the basis body is materialized/linked without reading it.
# This mirrors rsync 3.4.1's quick check (differential-tested in
# test_differential_parity.py::test_verify_basis_restores_strict_content).
for flag, basis_dir in (("--link-dest", "szlb"), ("--copy-dest", "szcp"),
("--compare-dest", "szcmp")):
source = self._make_source("basis_same_size_src",
@@ -4510,7 +4810,36 @@ class TestBasisDestDirs:
result, _ = run_client(source, dest, flags=[f"{flag}={basis_dir}"],
port=shared_server.port)
assert result.returncode == 0, \
f"{flag} same-size mismatch failed: {result.stderr[:300]}"
f"{flag} same-size quick-check failed: {result.stderr[:300]}"
received = get_dest_received_dir(dest, source)
dest_file = os.path.join(received, self.UNCHANGED)
if flag == "--compare-dest":
assert not os.path.exists(dest_file), \
f"{flag}: compare-dest must leave a matching file sparse"
else:
assert _read_file(dest_file) == b"SAME LENGTH BODY!", \
f"{flag}: default quick-check did not trust the basis body"
if flag == "--link-dest":
assert os.stat(dest_file).st_ino == os.stat(basis_file).st_ino, \
f"{flag}: basis was not hard-linked"
def test_verify_basis_rejects_same_size_different_content(self, shared_server):
# FastSync-only --verify-basis: the whole-file digest gate rejects the
# same-size/different-content basis, so the source data is transferred
# instead of the wrong basis bytes.
for flag, basis_dir in (("--link-dest", "vszlb"), ("--copy-dest", "vszcp"),
("--compare-dest", "vszcmp")):
source = self._make_source("basis_verify_src",
{self.UNCHANGED: b"same length body\n"})
dest = os.path.join(TEST_DATA_DIR, f"basis_verify_dst_{basis_dir}")
clean_dir(dest)
basis_file = self._seed_basis_file(dest, source, basis_dir, self.UNCHANGED,
b"SAME LENGTH BODY!")
result, _ = run_client(source, dest,
flags=[f"{flag}={basis_dir}", "--verify-basis"],
port=shared_server.port)
assert result.returncode == 0, \
f"{flag} --verify-basis failed: {result.stderr[:300]}"
received = get_dest_received_dir(dest, source)
dest_file = os.path.join(received, self.UNCHANGED)
assert _read_file(dest_file) == b"same length body\n", \
@@ -4542,11 +4871,13 @@ class TestBasisDestDirs:
self._source_tree("c")[self.ADDED], "added file not transferred"
@pytest.mark.ci
def test_dry_run_compare_dest_does_not_read_basis(self, shared_server):
# A dry-run --compare-dest must never read/hash the basis file: doing so
# is a 1-bit content oracle against the client-supplied digest. Even a
# byte-identical basis with a matching size+mtime is therefore reported
# as would-transfer, and nothing is created.
def test_dry_run_compare_dest_quick_check_does_not_read_basis(self, shared_server):
# A dry-run --compare-dest must never read/hash the basis file. Under
# the default metadata quick-check a matching basis is reported as a
# skip (matching rsync) without reading it; nothing is created. Under
# --verify-basis, which would require hashing, the dry-run cannot
# confirm the hit (that would be a 1-bit content oracle) and reports
# would-transfer instead.
source = self._make_source("basis_dry_src", {self.UNCHANGED: b"stable content v1\n"})
dest = os.path.join(TEST_DATA_DIR, "basis_dry_dst")
clean_dir(dest)
@@ -4557,11 +4888,30 @@ class TestBasisDestDirs:
port=shared_server.port)
assert result.returncode == 0, \
f"dry-run compare-dest failed: {result.stderr[:300]}"
assert self.UNCHANGED in result.stdout, (
"dry-run compare-dest silently skipped: receiver read the basis content"
assert self.UNCHANGED not in result.stdout, (
"dry-run compare-dest did not honor the metadata quick-check "
"(reported would-transfer for a matching basis)"
)
assert _snapshot_tree(dest) == before, "dry-run compare-dest mutated the destination"
# --verify-basis: the hit needs the basis content, which a dry-run must
# not read, so the file is reported as would-transfer.
dest2 = os.path.join(TEST_DATA_DIR, "basis_dry_verify_dst")
clean_dir(dest2)
self._seed_basis(dest2, source, "drybasis", {self.UNCHANGED: b"stable content v1\n"})
before2 = _snapshot_tree(dest2)
result, _ = run_client(source, dest2,
flags=["--compare-dest=drybasis", "--dry-run",
"--verify-basis"],
port=shared_server.port)
assert result.returncode == 0, \
f"dry-run --verify-basis compare-dest failed: {result.stderr[:300]}"
assert self.UNCHANGED in result.stdout, (
"dry-run --verify-basis must not read the basis to confirm a hit"
)
assert _snapshot_tree(dest2) == before2, \
"dry-run --verify-basis compare-dest mutated the destination"
def test_compare_dest_content_mismatch_forces_transfer(self, shared_server):
# The basis holds a file with a DIFFERENT body: even though it shares
# the mtime pin, the xxHash check fails and the data must be sent.
@@ -4800,27 +5150,36 @@ class TestBasisDestDirs:
assert os.stat(dest_file).st_ino != os.stat(basis_file).st_ino, \
"--ignore-times must not hard-link to a basis file"
def test_basis_refuses_file_above_whole_file_limit(self, shared_server):
# Every whole-file payload path in FastSync (basis dirs included) is
# bounded by MAX_RECEIVE_WHOLE_FILE_SIZE. rsync supports basis dirs for
# arbitrary sizes; FastSync refuses such a run up front with a clear
# diagnostic instead of letting the receiver abort the whole transfer
# mid-stream with no client-side explanation.
def test_basis_handles_file_above_whole_file_limit(self, shared_server):
# Track 5a: a basis hit streams the copy (and the --verify-basis digest
# streams the basis), so a source larger than the whole-file payload
# bound is supported for basis dirs exactly like rsync. A basis MISS
# still falls back to the normal transfer, which keeps its own bound.
source = self._make_source("basis_oversize_src", {"small.txt": b"ok\n"})
big = os.path.join(source, "huge.bin")
with open(big, "wb") as fh:
os.ftruncate(fh.fileno(), 256 * 1024 * 1024 + 4096)
dest = os.path.join(TEST_DATA_DIR, "basis_oversize_dst")
clean_dir(dest)
result, _ = run_client(source, dest, flags=["--link-dest=nope"],
port=shared_server.port)
assert result.returncode != 0, \
"basis run with an over-limit file unexpectedly succeeded"
assert "larger than" in result.stderr, \
f"no clear over-limit diagnostic: {result.stderr[:300]}"
received = get_dest_received_dir(dest, source)
assert not os.path.exists(received), \
"over-limit basis run transferred files before failing"
rel = os.path.relpath(received, dest)
basis_big = os.path.join(dest, "ob", rel, "huge.bin")
os.makedirs(os.path.dirname(basis_big), exist_ok=True)
shutil.copyfile(big, basis_big)
os.utime(basis_big, (self.TS, self.TS))
os.utime(big, (self.TS, self.TS))
result, _ = run_client(source, dest,
flags=["--link-dest=ob", "--incremental"],
port=shared_server.port)
assert result.returncode == 0, \
f"over-limit basis run failed: {result.stderr[:300]}"
dest_big = os.path.join(received, "huge.bin")
assert os.path.exists(dest_big), "over-limit basis hit was not materialized"
assert os.path.getsize(dest_big) == 256 * 1024 * 1024 + 4096
assert os.stat(dest_big).st_ino == os.stat(basis_big).st_ino, \
"over-limit --link-dest did not hard-link to the basis"
assert _read_file(os.path.join(received, "small.txt")) == b"ok\n"
def _random_payloads(size=2 * 1024 * 1024, changed=64 * 1024, seed=1234):
@@ -4957,15 +5316,20 @@ class TestFuzzy:
"no-candidate fuzzy run should have sent the whole file"
def test_dissimilar_sibling_is_not_used(self, shared_server):
# The destination holds a large sibling whose basename is too different
# from the incoming name; the name gate must reject it and fall back to
# a whole-file transfer.
# A sibling whose basename is too different from the incoming name is
# rejected by rsync's fuzzy distance window (the length gap exceeds
# 25), so the run falls back to a whole-file transfer. A distinct
# mtime keeps rsync's exact size+mtime first pass from accepting it.
source, dest = self._prepare("dissim")
old_bytes, new_bytes = _random_payloads()
self._seed_dest(source, dest, {"totally-unrelated-notes.bin": old_bytes},
long_name = "totally-unrelated-notes-with-a-very-long-name.bin"
self._seed_dest(source, dest, {long_name: old_bytes},
shared_server.port)
with open(os.path.join(source, self.NEW_NAME), "wb") as fh:
fh.write(new_bytes)
received_dir = get_dest_received_dir(dest, source)
os.utime(os.path.join(received_dir, long_name), (self.TS, self.TS))
os.utime(os.path.join(source, self.NEW_NAME), (self.TS + 100000, self.TS + 100000))
result, proxy = self._run_measured(source, dest, ["--fuzzy"], shared_server.port)
assert result.returncode == 0, \
f"--fuzzy dissimilar-sibling run failed: {(result.stderr or result.stdout)[:300]}"
@@ -4974,6 +5338,28 @@ class TestFuzzy:
assert proxy.client_to_server > len(new_bytes) // 2, \
"a dissimilar-named sibling must not be used as a fuzzy basis"
def test_exact_size_mtime_sibling_is_used(self, shared_server):
# rsync's fuzzy first pass accepts a sibling with an exact size+mtime
# match regardless of how unrelated its name is (its content is almost
# certainly the same).
source, dest = self._prepare("exact")
old_bytes, new_bytes = _random_payloads()
self._seed_dest(source, dest, {"unrelated-blob.bin": old_bytes},
shared_server.port)
with open(os.path.join(source, self.NEW_NAME), "wb") as fh:
fh.write(new_bytes)
received_dir = get_dest_received_dir(dest, source)
ts = 1600000000
os.utime(os.path.join(received_dir, "unrelated-blob.bin"), (ts, ts))
os.utime(os.path.join(source, self.NEW_NAME), (ts, ts))
result, proxy = self._run_measured(source, dest, ["--fuzzy"], shared_server.port)
assert result.returncode == 0, \
f"--fuzzy exact size+mtime run failed: {(result.stderr or result.stdout)[:300]}"
received = get_dest_received_dir(dest, source)
assert _read_file(os.path.join(received, self.NEW_NAME)) == new_bytes
assert proxy.client_to_server < len(new_bytes) // 4, \
"an exact size+mtime sibling should be used as a fuzzy basis"
def test_fuzzy_helps_when_dest_holds_an_unsuitable_file(self, shared_server):
# The destination DOES hold the exact new name, but it is a tiny stale
# file (below the delta engine's minimum, ratio far outside its window),
@@ -6518,11 +6904,10 @@ class TestDirectoryAndSymlinkTimes:
@pytest.mark.ci
@pytest.mark.parametrize("mt", [False, True])
def test_preserve_does_not_create_empty_source_dir(self, shared_server, mt):
"""P7 Wave D #1: a captured-but-EMPTY source directory is never created
at the destination. The scanner records its time (it is transmitted via
STATUS_DIR_TIMES), but the receiver treats that entry as record-only, so
`-a` keeps the documented "empty dirs are never transferred" behavior."""
def test_preserve_creates_empty_source_dir(self, shared_server, mt):
"""rsync parity: a recursive `-a` transfer recreates an empty source
directory at the destination (the scanner emits it as an explicit
directory entry)."""
source = os.path.join(TEST_DATA_DIR, f"empty_dir_{'m' if mt else 's'}_src")
dest = os.path.join(TEST_DATA_DIR, f"empty_dir_{'m' if mt else 's'}_dst")
clean_dir(source)
@@ -6533,8 +6918,8 @@ class TestDirectoryAndSymlinkTimes:
flags = ["-a"] + (["--threads"] if mt else [])
received = self._run(source, dest, flags, shared_server)
assert os.path.isfile(os.path.join(received, "keep.txt")), "regular file missing"
assert not os.path.lexists(os.path.join(received, "empty_sub")), \
f"-a created an empty source directory at {received}/empty_sub"
assert os.path.isdir(os.path.join(received, "empty_sub")), \
f"-a did not recreate the empty source directory at {received}/empty_sub"
@pytest.mark.ci
@pytest.mark.parametrize("mt", [False, True])
@@ -6557,10 +6942,10 @@ class TestDirectoryAndSymlinkTimes:
@pytest.mark.ci
@pytest.mark.parametrize("mt", [False, True])
def test_collision_at_dir_time_path_does_not_abort(self, shared_server, mt):
"""P7 Wave D #1: a pre-existing regular file at a source-empty-dir's
mirror path must not abort the transfer (the old mkdir failed and failed
the run) and must not be clobbered."""
def test_collision_at_empty_dir_path_replaces_blocker(self, shared_server, mt):
"""rsync parity: a pre-existing regular file at a source empty-dir's
mirror path is replaced by the incoming directory (rsync removes the
non-directory and creates the directory); the run succeeds."""
source = os.path.join(TEST_DATA_DIR, f"dirtime_collide_{'m' if mt else 's'}_src")
dest = os.path.join(TEST_DATA_DIR, f"dirtime_collide_{'m' if mt else 's'}_dst")
clean_dir(source)
@@ -6579,10 +6964,8 @@ class TestDirectoryAndSymlinkTimes:
assert result.returncode == 0, \
f"-a aborted on a pre-existing file at an empty-dir path: " \
f"{(result.stderr or result.stdout)[:400]}"
assert os.path.isfile(blocker) and not os.path.islink(blocker), \
"the pre-existing blocker was replaced by a directory"
with open(blocker, "rb") as fh:
assert fh.read() == b"pre-existing blocker\n", "the blocker file was clobbered"
assert os.path.isdir(blocker) and not os.path.islink(blocker), \
"the pre-existing blocker was not replaced by the incoming directory"
assert os.path.isfile(os.path.join(received, "keep.txt")), "regular file missing"
+34 -23
View File
@@ -1,10 +1,12 @@
"""--iconv=CONVERT_SPEC file-NAME charset conversion integration tests.
The client converts every source file name from LOCAL to REMOTE before it goes
on the wire, and the receiver converts it back from REMOTE to LOCAL, so a
source tree using one charset can be written into a destination tree using
another (rsync compatibility; content bytes are never touched).
rsync's spec is ``--iconv=LOCAL,REMOTE`` (the order is the same push or pull).
The sender converts each source name from LOCAL to REMOTE for the wire, and on
a PUSH the receiver's charset is the spec's REMOTE half, so it writes the wire
bytes verbatim (only a server with its own ``--iconv`` declares a different
destination charset and re-converts). Content bytes are never touched.
"""
import codecs
import os
import shutil
@@ -16,6 +18,11 @@ LATIN1_NAME = b"caf\xe9.txt"
UTF8_NAME = "caf\u00e9.txt".encode("utf-8")
def _to_utf8(name_bytes):
"""The UTF-8 encoding of a name that is stored as ISO-8859-1 bytes."""
return codecs.encode(codecs.decode(name_bytes, "iso-8859-1"), "utf-8")
def _make(tag):
source = os.path.join(TEST_DATA_DIR, f"iconv_{tag}_src")
dest = os.path.join(TEST_DATA_DIR, f"iconv_{tag}_dst")
@@ -41,10 +48,10 @@ def _dest_file(source, dest, name):
@pytest.mark.ci
def test_iconv_latin1_roundtrip(shared_server):
"""A source file whose name is ISO-8859-1 bytes is transferred with
--iconv=iso-8859-1,utf-8 and lands on the destination with the ORIGINAL
latin1 name (the wire carried it as UTF-8)."""
def test_iconv_latin1_to_utf8_dest(shared_server):
"""rsync push parity: --iconv=iso-8859-1,utf-8 converts a latin1 source name
to the spec's REMOTE (UTF-8) on the wire and the default receiver writes it
verbatim, so the destination name is UTF-8 (not the source's latin1)."""
source, dest = _make("latin1")
_place_bytes(source, LATIN1_NAME)
@@ -53,8 +60,10 @@ def test_iconv_latin1_roundtrip(shared_server):
)
assert result.returncode == 0, (result.stderr or result.stdout)[:400]
dst = _dest_file(source, dest, LATIN1_NAME)
assert os.path.exists(dst), f"dest latin1-named file not found under {dest}"
dst = _dest_file(source, dest, UTF8_NAME)
assert os.path.exists(dst), f"dest UTF-8-named file not found under {dest}"
assert not os.path.exists(_dest_file(source, dest, LATIN1_NAME)), \
"destination kept the latin1 name instead of the wire (UTF-8) charset"
@pytest.mark.ci
@@ -153,7 +162,7 @@ def test_iconv_expanding_name_growth(shared_server):
)
assert result.returncode == 0, (result.stderr or result.stdout)[:400]
assert os.path.exists(_dest_file(source, dest, name_bytes))
assert os.path.exists(_dest_file(source, dest, _to_utf8(name_bytes)))
def test_iconv_symlink_path_and_target(shared_server):
@@ -170,11 +179,13 @@ def test_iconv_symlink_path_and_target(shared_server):
)
assert result.returncode == 0, (result.stderr or result.stdout)[:400]
dst_target = _dest_file(source, dest, target)
dst_link = _dest_file(source, dest, b"link\xe9")
assert os.path.exists(dst_target), "dest latin1 target file missing"
assert os.path.islink(dst_link), "dest latin1 symlink missing"
assert os.readlink(dst_link) == target, "symlink target not preserved/decoded"
utf8_target = _to_utf8(target)
utf8_link = _to_utf8(b"link\xe9")
dst_target = _dest_file(source, dest, utf8_target)
dst_link = _dest_file(source, dest, utf8_link)
assert os.path.exists(dst_target), "dest UTF-8 target file missing"
assert os.path.islink(dst_link), "dest UTF-8 symlink missing"
assert os.readlink(dst_link) == utf8_target, "symlink target not wire-converted"
with open(dst_link, "rb") as fh:
assert fh.read() == b"t\n"
@@ -198,8 +209,8 @@ def test_iconv_hardlink_path_and_target(shared_server):
)
assert result.returncode == 0, (result.stderr or result.stdout)[:400]
dst_a = _dest_file(source, dest, a)
dst_b = _dest_file(source, dest, b)
dst_a = _dest_file(source, dest, _to_utf8(a))
dst_b = _dest_file(source, dest, _to_utf8(b))
assert os.path.exists(dst_a) and os.path.exists(dst_b)
assert os.stat(dst_a).st_ino == os.stat(dst_b).st_ino, \
"hard-link relationship not preserved across the transfer"
@@ -221,16 +232,16 @@ def test_iconv_delete_manifest_consistent(shared_server):
flags = ["--iconv=iso-8859-1,utf-8"]
result, _ = run_client(source, dest, flags=flags, port=server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:400]
assert os.path.exists(_dest_file(source, dest, keep))
assert os.path.exists(_dest_file(source, dest, gone))
assert os.path.exists(_dest_file(source, dest, _to_utf8(keep)))
assert os.path.exists(_dest_file(source, dest, _to_utf8(gone)))
os.remove(os.path.join(os.fsencode(source), gone))
result, _ = run_client(
source, dest, flags=flags + ["--delete"], port=server.port
)
assert result.returncode == 0, (result.stderr or result.stdout)[:400]
assert os.path.exists(_dest_file(source, dest, keep)), "kept file deleted"
assert not os.path.exists(_dest_file(source, dest, gone)), \
assert os.path.exists(_dest_file(source, dest, _to_utf8(keep))), "kept file deleted"
assert not os.path.exists(_dest_file(source, dest, _to_utf8(gone))), \
"missing file was not deleted"
@@ -247,4 +258,4 @@ def test_iconv_chunk_serialization_blob(shared_server):
)
assert result.returncode == 0, (result.stderr or result.stdout)[:400]
assert os.path.exists(_dest_file(source, dest, name))
assert os.path.exists(_dest_file(source, dest, _to_utf8(name)))
+538
View File
@@ -0,0 +1,538 @@
"""Differential parity tests for the option wave (bwlimit, --info=*, -M,
--ignore-errors, --filter protect).
Every differential here runs the SAME scenario with real ``rsync 3.4.1`` and
with fastsync and compares the observable result, so the modules are skipped
when rsync is unavailable. The privilege-dependent --ignore-errors differential
drops the client to an unprivileged uid so a mode-000 source directory is
genuinely unreadable; it is marked ``setpriv`` (run as root locally, excluded
from the root PR gate exactly like the other privilege tests).
"""
import os
import shutil
import subprocess
import sys
import time
import pytest
sys.path.insert(0, os.path.dirname(__file__))
from common import ( # noqa: E402
CLIENT_CMD,
TEST_DATA_DIR,
ServerManager,
clean_dir,
get_dest_received_dir,
run_client,
)
RSYNC = shutil.which("rsync")
requires_rsync = pytest.mark.skipif(RSYNC is None, reason="rsync 3.4.1 not installed")
def _rsync(args, timeout=120, as_nobody=False):
env = dict(os.environ, LC_ALL="C")
cmd = [RSYNC] + args
if as_nobody:
cmd = ["setpriv", "--reuid=65534", "--regid=65534", "--clear-groups"] + cmd
return subprocess.run(cmd, capture_output=True, text=True, env=env, timeout=timeout)
def _write(path, content):
os.makedirs(os.path.dirname(path), exist_ok=True)
with open(path, "wb") as fh:
fh.write(content)
class TestBwlimitParity:
"""--bwlimit must accept rsync 3.4.1's spellings and pace like it."""
ACCEPTED = ["100", "0", "1.5", "100K", "100KB", "100KiB", "1M", "1MB", "1m", "1G", "512"]
REJECTED = ["-1", "abc", "1x", "1 000"]
@requires_rsync
@pytest.mark.ci
def test_parse_acceptance_matches_rsync(self, shared_server):
source = os.path.join(TEST_DATA_DIR, "bwp_src")
clean_dir(source)
_write(os.path.join(source, "f.txt"), b"payload\n")
for value in self.ACCEPTED + self.REJECTED:
rdst = os.path.join(TEST_DATA_DIR, "bwp_rdst")
clean_dir(rdst)
rsync_result = _rsync(["-a", "--bwlimit=" + value, source + "/", rdst + "/"])
dest = os.path.join(TEST_DATA_DIR, "bwp_dst")
clean_dir(dest)
result, _ = run_client(source, dest, flags=["-a", "--bwlimit=" + value],
port=shared_server.port)
assert (result.returncode == 0) == (rsync_result.returncode == 0), (
f"--bwlimit={value}: fastsync rc={result.returncode} "
f"({(result.stderr or result.stdout)[:120]!r}) "
f"rsync rc={rsync_result.returncode} ({rsync_result.stderr[:120]!r})"
)
@requires_rsync
@pytest.mark.ci
def test_throttle_rate_matches_rsync(self, shared_server):
"""A 4 MiB transfer at --bwlimit=2048 (2 MiB/s) must take about the same
wall-clock time for both tools (~2 s with rsync's leaky bucket)."""
source = os.path.join(TEST_DATA_DIR, "bwt_src")
clean_dir(source)
_write(os.path.join(source, "big.bin"), os.urandom(4 * 1024 * 1024))
dest = os.path.join(TEST_DATA_DIR, "bwt_dst")
rdst = os.path.join(TEST_DATA_DIR, "bwt_rdst")
clean_dir(rdst)
start = time.monotonic()
rsync_result = _rsync(["-a", "--bwlimit=2048", source + "/", rdst + "/"])
rsync_secs = time.monotonic() - start
assert rsync_result.returncode == 0, rsync_result.stderr
clean_dir(dest)
result, fast_secs = run_client(source, dest, flags=["-a", "--bwlimit=2048"],
port=shared_server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:200]
# 4 MiB at 2 MiB/s rendezvous near 2 s. Use a coarse band on each side
# (an unthrottled transfer finishes well under 1.5 s) plus a generous
# cross-tolerance so a loaded CI runner cannot flake the parity assert.
lo, hi = 1.5, 4.5
assert lo <= fast_secs <= hi, f"fastsync throttle out of band: {fast_secs:.2f}s"
assert lo <= rsync_secs <= hi, f"rsync throttle out of band: {rsync_secs:.2f}s"
assert abs(fast_secs - rsync_secs) < 2.0, (
f"fastsync {fast_secs:.2f}s vs rsync {rsync_secs:.2f}s"
)
def _output_tree(root):
clean_dir(root)
os.makedirs(os.path.join(root, "sub"))
_write(os.path.join(root, "a.txt"), b"top\n")
_write(os.path.join(root, "sub", "b.txt"), b"nested\n")
os.symlink("a.txt", os.path.join(root, "link"))
class TestInfoParity:
"""The --info categories that map to a FastSync event must print rsync's
line format."""
@requires_rsync
@pytest.mark.ci
def test_info_flist_matches_rsync(self, shared_server):
source = os.path.join(TEST_DATA_DIR, "inf_fl_src")
dest = os.path.join(TEST_DATA_DIR, "inf_fl_dst")
rdst = os.path.join(TEST_DATA_DIR, "inf_fl_rdst")
_output_tree(source)
clean_dir(dest)
clean_dir(rdst)
rsync_result = _rsync(["-a", "--info=flist", source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
result, _ = run_client(source, dest, flags=["-a", "--info=flist"],
port=shared_server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:200]
assert "sending incremental file list" in result.stdout
assert "sending incremental file list" in rsync_result.stdout
@requires_rsync
@pytest.mark.ci
def test_info_name_matches_rsync(self, shared_server):
source = os.path.join(TEST_DATA_DIR, "inf_nm_src")
dest = os.path.join(TEST_DATA_DIR, "inf_nm_dst")
rdst = os.path.join(TEST_DATA_DIR, "inf_nm_rdst")
_output_tree(source)
clean_dir(dest)
clean_dir(rdst)
rsync_result = _rsync(["-a", "--info=name", source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
result, _ = run_client(source, dest, flags=["-a", "--info=name"],
port=shared_server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:200]
def entries(text):
# Compare the transferred entries only: rsync also prints the
# transfer-root `./` and every directory (FastSync records dirs),
# which are a separate documented divergence.
out = []
for line in text.splitlines():
if not line or line.startswith("sending ") or line.startswith("created "):
continue
if line == "./" or line.endswith("/"):
continue
out.append(line)
return sorted(out)
assert entries(result.stdout) == entries(rsync_result.stdout), (
f"rsync={entries(rsync_result.stdout)} fastsync={entries(result.stdout)}"
)
@requires_rsync
@pytest.mark.ci
def test_info_name_root_line_matches_rsync(self, shared_server):
"""A fresh destination: rsync prints `created directory`, then the
transfer-root `./` name line before the entries; FastSync must emit the
same `./` line."""
source = os.path.join(TEST_DATA_DIR, "inf_root_src")
dest = os.path.join(TEST_DATA_DIR, "inf_root_dst")
rdst = os.path.join(TEST_DATA_DIR, "inf_root_rdst")
clean_dir(source)
_write(os.path.join(source, "f.bin"), b"payload\n")
clean_dir(dest)
shutil.rmtree(rdst, ignore_errors=True)
rsync_result = _rsync(["-a", "--info=name", source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
result, _ = run_client(source, dest, flags=["-a", "--info=name"],
port=shared_server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:200]
def names(text):
return [l for l in text.splitlines()
if l and not l.startswith("created directory")
and not (l.endswith("/") and l != "./")]
assert names(rsync_result.stdout) == ["./", "f.bin"], names(rsync_result.stdout)
assert names(result.stdout) == ["./", "f.bin"], names(result.stdout)
@requires_rsync
@pytest.mark.ci
def test_info_name2_uptodate_matches_rsync(self, shared_server):
"""--info=name2 prints `NAME is uptodate` for entries the receiver
already has, matching rsync byte-for-byte."""
source = os.path.join(TEST_DATA_DIR, "inf_up_src")
dest = os.path.join(TEST_DATA_DIR, "inf_up_dst")
rdst = os.path.join(TEST_DATA_DIR, "inf_up_rdst")
clean_dir(source)
os.makedirs(os.path.join(source, "sub"))
_write(os.path.join(source, "a.txt"), b"a\n")
_write(os.path.join(source, "sub", "b.txt"), b"b\n")
clean_dir(rdst)
assert _rsync(["-a", source + "/", rdst + "/"]).returncode == 0
rsync_result = _rsync(["-a", "--info=name2", source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
clean_dir(dest)
seed, _ = run_client(source, dest, flags=["-a", "--incremental"],
port=shared_server.port)
assert seed.returncode == 0, (seed.stderr or seed.stdout)[:200]
result, _ = run_client(source, dest, flags=["-a", "--incremental", "--info=name2"],
port=shared_server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:200]
rsync_lines = sorted(l for l in rsync_result.stdout.splitlines()
if l.endswith("is uptodate"))
fast_lines = sorted(l for l in result.stdout.splitlines()
if l.endswith("is uptodate"))
assert fast_lines == rsync_lines, (rsync_lines, fast_lines)
assert fast_lines == ["a.txt is uptodate", "sub/b.txt is uptodate"], fast_lines
@requires_rsync
@pytest.mark.ci
def test_info_nonreg_matches_rsync(self, shared_server):
source = os.path.join(TEST_DATA_DIR, "inf_nr_src")
dest = os.path.join(TEST_DATA_DIR, "inf_nr_dst")
rdst = os.path.join(TEST_DATA_DIR, "inf_nr_rdst")
clean_dir(source)
os.mkfifo(os.path.join(source, "fifo"))
_write(os.path.join(source, "a.txt"), b"a\n")
clean_dir(dest)
clean_dir(rdst)
rsync_result = _rsync(["-rlt", "--info=nonreg", source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
result, _ = run_client(source, dest, flags=["-rlt", "--info=nonreg"],
port=shared_server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:200]
rsync_lines = sorted(l for l in rsync_result.stdout.splitlines()
if l.startswith("skipping non-regular"))
fast_lines = sorted(l for l in result.stdout.splitlines()
if l.startswith("skipping non-regular"))
assert fast_lines == rsync_lines, (rsync_lines, fast_lines)
assert fast_lines, "no non-regular skip line emitted"
@requires_rsync
@pytest.mark.ci
def test_info_del_real_matches_rsync(self, shared_server):
source = os.path.join(TEST_DATA_DIR, "inf_dl_src")
dest = os.path.join(TEST_DATA_DIR, "inf_dl_dst")
rdst = os.path.join(TEST_DATA_DIR, "inf_dl_rdst")
clean_dir(source)
_write(os.path.join(source, "keep.txt"), b"keep\n")
clean_dir(rdst)
_write(os.path.join(rdst, "extra.txt"), b"x\n")
_write(os.path.join(rdst, "extra2.txt"), b"y\n")
rsync_result = _rsync(["-a", "--delete", "--info=del", source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
rsync_lines = sorted(l for l in rsync_result.stdout.splitlines()
if l.startswith("deleting "))
clean_dir(dest)
received = get_dest_received_dir(dest, source)
_write(os.path.join(received, "extra.txt"), b"x\n")
_write(os.path.join(received, "extra2.txt"), b"y\n")
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
result, _ = run_client(source, dest, flags=["-a", "--delete", "--info=del"],
port=server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:200]
fast_lines = sorted(l for l in result.stdout.splitlines()
if l.startswith("deleting "))
assert fast_lines == rsync_lines, (rsync_lines, fast_lines)
assert fast_lines, "no deletion lines emitted"
@requires_rsync
@pytest.mark.ci
def test_info_del_itemize_real_matches_rsync(self, shared_server):
source = os.path.join(TEST_DATA_DIR, "inf_di_src")
dest = os.path.join(TEST_DATA_DIR, "inf_di_dst")
rdst = os.path.join(TEST_DATA_DIR, "inf_di_rdst")
clean_dir(source)
_write(os.path.join(source, "keep.txt"), b"keep\n")
clean_dir(rdst)
_write(os.path.join(rdst, "extra.txt"), b"x\n")
rsync_result = _rsync(["-a", "-i", "--delete", source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
rsync_lines = sorted(l for l in rsync_result.stdout.splitlines()
if l.startswith("*deleting"))
clean_dir(dest)
received = get_dest_received_dir(dest, source)
_write(os.path.join(received, "extra.txt"), b"x\n")
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
result, _ = run_client(source, dest, flags=["-a", "-i", "--delete"],
port=server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:200]
fast_lines = sorted(l for l in result.stdout.splitlines()
if l.startswith("*deleting"))
assert fast_lines == rsync_lines, (rsync_lines, fast_lines)
@requires_rsync
@pytest.mark.ci
def test_info_del_dry_run_matches_rsync(self, shared_server):
source = os.path.join(TEST_DATA_DIR, "inf_dd_src")
dest = os.path.join(TEST_DATA_DIR, "inf_dd_dst")
rdst = os.path.join(TEST_DATA_DIR, "inf_dd_rdst")
clean_dir(source)
_write(os.path.join(source, "keep.txt"), b"keep\n")
clean_dir(rdst)
_write(os.path.join(rdst, "extra.txt"), b"x\n")
rsync_result = _rsync(["-a", "-n", "--delete", "--info=del", source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
rsync_lines = sorted(l for l in rsync_result.stdout.splitlines()
if l.startswith("deleting "))
clean_dir(dest)
received = get_dest_received_dir(dest, source)
_write(os.path.join(received, "extra.txt"), b"x\n")
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
result, _ = run_client(source, dest, flags=["-a", "-n", "--delete", "--info=del"],
port=server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:200]
fast_lines = sorted(l for l in result.stdout.splitlines()
if l.startswith("deleting "))
assert fast_lines == rsync_lines, (rsync_lines, fast_lines)
@requires_rsync
@pytest.mark.ci
def test_info_remove_matches_rsync(self, shared_server):
tag = "inf_rm"
rsync_src = os.path.join(TEST_DATA_DIR, f"{tag}_rsrc")
rsync_dst = os.path.join(TEST_DATA_DIR, f"{tag}_rdst")
fast_src = os.path.join(TEST_DATA_DIR, f"{tag}_fsrc")
fast_dst = os.path.join(TEST_DATA_DIR, f"{tag}_fdst")
for root in (rsync_src, rsync_dst, fast_src, fast_dst):
clean_dir(root)
_write(os.path.join(rsync_src, "a.txt"), b"a\n")
_write(os.path.join(rsync_src, "sub", "b.txt"), b"b\n")
_write(os.path.join(fast_src, "a.txt"), b"a\n")
_write(os.path.join(fast_src, "sub", "b.txt"), b"b\n")
rsync_result = _rsync(["-a", "--remove-source-files", "--info=remove",
rsync_src + "/", rsync_dst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
rsync_lines = sorted(l for l in rsync_result.stdout.splitlines()
if l.startswith("sender removed "))
result, _ = run_client(fast_src, fast_dst,
flags=["-a", "--remove-source-files", "--info=remove"],
port=shared_server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:200]
fast_lines = sorted(l for l in result.stdout.splitlines()
if l.startswith("sender removed "))
assert fast_lines == rsync_lines, (rsync_lines, fast_lines)
assert fast_lines, "no source-removal lines emitted"
class TestIgnoreErrorsParity:
"""--ignore-errors: a source I/O error skips deletion by default; the flag
lets deletion proceed. Both exit 23. Run the client as an unprivileged user
so the mode-000 directory is genuinely unreadable."""
@pytest.mark.setpriv
def test_delete_after_io_error_matches_rsync(self):
if os.geteuid() != 0 or shutil.which("setpriv") is None:
pytest.skip("requires root + setpriv to drop privileges for the client")
tag = f"ie_{os.getpid()}"
source = os.path.join(TEST_DATA_DIR, f"{tag}_src")
rsync_dst = os.path.join(TEST_DATA_DIR, f"{tag}_rdst")
dest = os.path.join(TEST_DATA_DIR, f"{tag}_dst")
clean_dir(source)
clean_dir(rsync_dst)
clean_dir(dest)
_write(os.path.join(source, "top.txt"), b"top\n")
_write(os.path.join(source, "locked", "blocked.txt"), b"blocked\n")
os.chmod(os.path.join(source, "locked"), 0)
os.chmod(TEST_DATA_DIR, 0o777)
os.chmod(source, 0o755)
os.chmod(rsync_dst, 0o777)
os.chmod(dest, 0o777)
try:
for ignore in (False, True):
flags = ["-a", "--delete-after"] + (["--ignore-errors"] if ignore else [])
# rsync side
_write(os.path.join(rsync_dst, "extra.txt"), b"x\n")
os.chmod(os.path.join(rsync_dst, "extra.txt"), 0o666)
rres = _rsync(flags + [source + "/", rsync_dst + "/"], as_nobody=True)
rsync_extra = os.path.exists(os.path.join(rsync_dst, "extra.txt"))
# fastsync side
received = get_dest_received_dir(dest, source)
_write(os.path.join(received, "extra.txt"), b"x\n")
os.chmod(os.path.join(received, "extra.txt"), 0o666)
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
fflags = (["--delete", "--ignore-errors"] if ignore else ["--delete"])
cmd = CLIENT_CMD + ["--source-dir", source, "--dest-dir", dest,
"--save-to-disk", "--server-port", str(server.port)] + fflags
fres = subprocess.run(
["setpriv", "--reuid=65534", "--regid=65534", "--clear-groups"] + cmd,
text=True, capture_output=True)
fast_extra = os.path.exists(os.path.join(received, "extra.txt"))
assert rres.returncode == 23, (ignore, rres.returncode, rres.stderr[:200])
assert fres.returncode == 23, (ignore, fres.returncode, fres.stderr[:200])
assert rsync_extra == fast_extra, (
f"ignore_errors={ignore}: rsync extra={rsync_extra} fastsync extra={fast_extra}"
)
assert fast_extra is (not ignore), (ignore, fast_extra)
finally:
os.chmod(os.path.join(source, "locked"), 0o755)
class TestRemoteOptionDaemon:
"""rsync forwards -M/--remote-option to its remote process over a daemon
connection; FastSync's daemon has no per-connection argv channel and rejects
it. This pins the documented divergence with evidence."""
@requires_rsync
def test_rsync_forwards_M_over_daemon_and_fastsync_rejects(self, tmp_path):
import socket
with socket.socket() as probe:
probe.bind(("127.0.0.1", 0))
port = probe.getsockname()[1]
module_root = tmp_path / "mod"
module_root.mkdir()
os.chmod(module_root, 0o777)
source = tmp_path / "src"
source.mkdir()
(source / "a.txt").write_bytes(b"hello\n")
conf = tmp_path / "rsyncd.conf"
conf.write_text(
f"port = {port}\nuse chroot = no\n[m]\npath = {module_root}\nread only = no\n"
)
daemon = subprocess.Popen(
[RSYNC, "--daemon", "--no-detach", "--port", str(port), "--config", str(conf)],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
try:
deadline = time.monotonic() + 5
while time.monotonic() < deadline:
try:
with socket.create_connection(("127.0.0.1", port), timeout=0.3):
break
except OSError:
time.sleep(0.05)
else:
pytest.skip("rsync daemon did not start")
# A well-formed -M option is forwarded and accepted by the daemon...
ok = _rsync(["-a", "-M--safe-links", source.as_posix() + "/",
f"rsync://127.0.0.1:{port}/m/"])
# ...and a bogus one is rejected ON THE REMOTE with "unknown option",
# which proves the option reached the daemon's parser.
bogus = _rsync(["-a", "-M--totally-bogus", source.as_posix() + "/",
f"rsync://127.0.0.1:{port}/m/"])
assert bogus.returncode != 0
assert "unknown option" in (bogus.stderr + bogus.stdout), bogus.stderr
del ok
finally:
daemon.terminate()
try:
daemon.wait(timeout=5)
except subprocess.TimeoutExpired:
daemon.kill()
# FastSync rejects -M for a non-SSH transport up front.
dest = os.path.join(TEST_DATA_DIR, "ro_dst")
clean_dir(dest)
result, _ = run_client(source.as_posix(), dest, flags=["-a", "-M--safe-links"])
assert result.returncode != 0
assert "remote-option" in (result.stderr + result.stdout)
class TestFilterProtect:
"""Receiver-derived delete protection: a `protect`/`P` rule is compiled by
the sender and sent on the config frame, so the receiver shields a
destination-only entry that never appeared on the sender, matching rsync."""
@requires_rsync
@pytest.mark.ci
def test_protect_dest_only_matches_rsync(self, shared_server):
source = os.path.join(TEST_DATA_DIR, "fpd_src")
dest = os.path.join(TEST_DATA_DIR, "fpd_dst")
rdst = os.path.join(TEST_DATA_DIR, "fpd_rdst")
clean_dir(source)
_write(os.path.join(source, "keep.txt"), b"keep\n")
clean_dir(rdst)
_write(os.path.join(rdst, "extra.log"), b"extra\n")
_write(os.path.join(rdst, "other.txt"), b"other\n")
rsync_result = _rsync(["-a", "--delete", "--filter=P *.log", source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
assert os.path.exists(os.path.join(rdst, "extra.log")), "rsync did not protect extra.log"
assert not os.path.exists(os.path.join(rdst, "other.txt")), "rsync did not delete other.txt"
clean_dir(dest)
received = get_dest_received_dir(dest, source)
_write(os.path.join(received, "extra.log"), b"extra\n")
_write(os.path.join(received, "other.txt"), b"other\n")
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
result, _ = run_client(source, dest,
flags=["-a", "--delete", "--filter=P *.log"],
port=server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:200]
assert os.path.exists(os.path.join(received, "extra.log")), (
"FastSync must protect a destination-only P match like rsync")
assert not os.path.exists(os.path.join(received, "other.txt"))
@pytest.mark.ci
def test_protect_dest_only_dry_run_enumeration(self, shared_server):
source = os.path.join(TEST_DATA_DIR, "fpd_nd_src")
dest = os.path.join(TEST_DATA_DIR, "fpd_nd_dst")
clean_dir(source)
_write(os.path.join(source, "keep.txt"), b"keep\n")
received = get_dest_received_dir(dest, source)
clean_dir(received)
_write(os.path.join(received, "keep.txt"), b"keep\n")
_write(os.path.join(received, "extra.log"), b"extra\n")
_write(os.path.join(received, "other.txt"), b"other\n")
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
result, _ = run_client(source, dest,
flags=["-a", "-n", "--delete", "--out-format=%n",
"--filter=P *.log"],
port=server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
assert "other.txt" in result.stdout, result.stdout
assert "extra.log" not in result.stdout, result.stdout
assert os.path.exists(os.path.join(received, "extra.log"))
assert os.path.exists(os.path.join(received, "other.txt"))
+527 -1
View File
@@ -5,6 +5,7 @@ differential tests run the SAME transfer with real ``rsync 3.4.1`` and with
fastsync and compare stdout, so they are skipped when rsync is unavailable.
"""
import os
import re
import shutil
import subprocess
import sys
@@ -12,7 +13,7 @@ import sys
import pytest
sys.path.insert(0, os.path.dirname(__file__))
from common import TEST_DATA_DIR, run_client, clean_dir, get_dest_received_dir
from common import TEST_DATA_DIR, run_client, clean_dir, get_dest_received_dir, ServerManager
RSYNC = shutil.which("rsync")
requires_rsync = pytest.mark.skipif(RSYNC is None, reason="rsync 3.4.1 not installed")
@@ -280,3 +281,528 @@ class TestListOnlyParity:
assert fast_lines == rsync_lines, (
f"rsync={rsync_lines}\nfastsync={fast_lines}"
)
def _make_one_file(root, name="f.bin", size=100):
clean_dir(root)
with open(os.path.join(root, name), "wb") as fh:
fh.write(bytes((i * 7 + 3) & 0xFF for i in range(size)))
def _make_multidir_tree(root):
"""Multi-directory corpus for the --progress file-list tests: nested files,
a directory-only branch, an empty directory and a symlink."""
clean_dir(root)
for rel, data in (("a.txt", b"alpha\n"), ("b.txt", b"bravo\n"),
("sub1/c.txt", b"charlie\n"), ("sub1/deep/d.txt", b"delta\n"),
("sub2/e.txt", b"echo\n")):
path = os.path.join(root, rel)
os.makedirs(os.path.dirname(path), exist_ok=True)
with open(path, "wb") as fh:
fh.write(data)
os.symlink("a.txt", os.path.join(root, "link1"))
os.makedirs(os.path.join(root, "emptydir"), exist_ok=True)
def _parse_progress(text):
"""Name lines and the `to-chk` denominators from a --progress run."""
names = []
totals = set()
for line in text.splitlines():
line = line.rstrip()
if not line or line == "sending incremental file list":
continue
if "%" in line:
match = re.search(r"to-chk=\d+/(\d+)", line)
if match:
totals.add(int(match.group(1)))
continue
if line == "./": # root-line trigger is a separate documented residual
continue
names.append(line)
return sorted(names), totals
def _pick_stats(text, keys):
out = {}
for line in text.splitlines():
for key in keys:
if line.startswith(key + ":"):
out[key] = line
return out
class TestWireStatsParity:
"""Wire-counter output parity: --out-format %b/%c/%C, --progress and
--stats versus real rsync 3.4.1."""
@requires_rsync
@pytest.mark.ci
def test_out_format_checksum_matches_rsync(self, shared_server):
"""%C (whole-file xxh128, seed 0) is protocol-independent, so the full
`%C %l %n` line must be byte-identical to rsync."""
source = os.path.join(TEST_DATA_DIR, "wire_ck_src")
dest = os.path.join(TEST_DATA_DIR, "wire_ck_dst")
rdst = os.path.join(TEST_DATA_DIR, "wire_ck_rdst")
_make_one_file(source, "f.bin", 200000)
clean_dir(dest)
clean_dir(rdst)
fmt = "%C %l %n"
rsync_result = _rsync(["-a", "--out-format=" + fmt, source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
result, _ = run_client(source, dest, flags=["-a", "--out-format=" + fmt],
port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
def file_lines(text):
# Ignore the root directory entry: fastsync does not transfer the
# source-root dir itself (a separate pre-existing divergence).
return [
line for line in text.splitlines() if not line.rsplit(" ", 1)[-1].endswith("/")
]
assert file_lines(result.stdout) == file_lines(rsync_result.stdout), (
f"rsync={rsync_result.stdout!r} fastsync={result.stdout!r}"
)
@requires_rsync
@pytest.mark.ci
def test_out_format_b_is_wire_bytes(self, shared_server):
"""%b is the bytes actually transferred (wire), not the source length.
A differential run against rsync confirms both implementations report a
framed value greater than %l. The exact numbers are not compared: each
counts its own protocol framing and checksum trailer, so the two are
protocol-specific and cannot be numerically equal (documented
divergence)."""
source = os.path.join(TEST_DATA_DIR, "wire_b_src")
dest = os.path.join(TEST_DATA_DIR, "wire_b_dst")
rdst = os.path.join(TEST_DATA_DIR, "wire_b_rdst")
_make_one_file(source, "f.bin", 5000)
clean_dir(dest)
clean_dir(rdst)
fmt = "%b %l"
rsync_result = _rsync(["-a", "--out-format=" + fmt, source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
result, _ = run_client(source, dest, flags=["-a", "--out-format=" + fmt],
port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
rb, rl = (int(x) for x in rsync_result.stdout.split()[:2])
fb, fl = (int(x) for x in result.stdout.split()[:2])
assert rl == fl == 5000, (rsync_result.stdout, result.stdout)
assert rb > rl, f"rsync %b must include framing: {rsync_result.stdout!r}"
assert fb > fl, f"fastsync %b must include framing: {result.stdout!r}"
@requires_rsync
@pytest.mark.ci
def test_out_format_c_whole_file_matches_rsync(self, shared_server):
"""%c is the block-checksum bytes received. rsync reports its 16-byte
sum header even for a whole-file transfer (no basis), so `%c` must match
rsync exactly for the whole-file case."""
source = os.path.join(TEST_DATA_DIR, "wire_c_src")
dest = os.path.join(TEST_DATA_DIR, "wire_c_dst")
rdst = os.path.join(TEST_DATA_DIR, "wire_c_rdst")
_make_one_file(source, "f.bin", 5000)
clean_dir(dest)
clean_dir(rdst)
fmt = "%c %l %n"
rsync_result = _rsync(["-a", "--out-format=" + fmt, source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
result, _ = run_client(source, dest, flags=["-a", "--out-format=" + fmt],
port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
def file_lines(text):
return [
line for line in text.splitlines()
if line and not line.rsplit(" ", 1)[-1].endswith("/")
]
assert file_lines(result.stdout) == file_lines(rsync_result.stdout), (
f"rsync={rsync_result.stdout!r} fastsync={result.stdout!r}"
)
assert result.stdout.split()[0] == rsync_result.stdout.split()[0] == "16", (
f"%c must be rsync's 16-byte sum header: {result.stdout!r}"
)
@requires_rsync
@pytest.mark.ci
def test_out_format_c_delta_mode_divergence(self, shared_server):
"""Documented residual: with delta enabled, rsync's %c is its 16-byte sum
header plus one checksum entry per block (protocol-specific, so it grows
with the basis size), while FastSync's %c is the bytes of its own delta
handshake. FastSync's delta %c therefore cannot match rsync numerically;
only the whole-file case is aligned. Pinned here so a future change is
noticed."""
source = os.path.join(TEST_DATA_DIR, "wire_cd_src")
dest = os.path.join(TEST_DATA_DIR, "wire_cd_dst")
rdst = os.path.join(TEST_DATA_DIR, "wire_cd_rdst")
_make_one_file(source, "f.bin", 5000)
clean_dir(dest)
clean_dir(rdst)
fmt = "%c %l"
# rsync local default is whole-file; force the block-delta path.
rsync_result = _rsync(["-a", "--no-whole-file", "--out-format=" + fmt,
source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
result, _ = run_client(source, dest,
flags=["-a", "--incremental", "--delta",
"--out-format=" + fmt],
port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
rs_c = int(rsync_result.stdout.split()[0])
fs_c = int(result.stdout.split()[0])
# No basis exists, so rsync still reports only its sum header.
assert rs_c == 16, rsync_result.stdout
# FastSync reports its own handshake bytes and is not aligned.
assert fs_c > 16, (
f"FastSync delta %c changed to {fs_c}; the documented divergence "
"may be closable now"
)
@requires_rsync
@pytest.mark.ci
@pytest.mark.parametrize("mt", [False, True])
@pytest.mark.parametrize("progress_flag", ["--progress", "-P"])
def test_progress_first_frame_matches_rsync(self, shared_server, progress_flag, mt):
"""For a sub-32 KiB file the first --progress/-P frame is deterministic
(0.00 kB/s, 0:00:00) and must be byte-identical to rsync's, in both the
single-threaded and --threads send paths."""
source = os.path.join(TEST_DATA_DIR, "wire_pg_src")
dest = os.path.join(TEST_DATA_DIR, "wire_pg_dst")
rdst = os.path.join(TEST_DATA_DIR, "wire_pg_rdst")
_make_one_file(source, "f.bin", 100)
clean_dir(dest)
clean_dir(rdst)
rsync_result = _rsync(["-a", progress_flag, source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
flags = ["-a", progress_flag] + (["--threads"] if mt else [])
result, _ = run_client(source, dest, flags=flags, port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
def frames(text):
# subprocess text mode normalizes \r to \n (universal newlines).
return [p for p in text.split("\n") if "%" in p]
rsync_frames = frames(rsync_result.stdout)
fast_frames = frames(result.stdout)
assert rsync_frames and fast_frames, (rsync_result.stdout, result.stdout)
assert fast_frames[0] == rsync_frames[0], (rsync_frames[0], fast_frames[0])
assert "(xfr#1," in fast_frames[-1], fast_frames[-1]
@requires_rsync
@pytest.mark.ci
def test_progress_leading_root_line_and_to_chk_match_rsync(self, shared_server):
"""A single-file transfer: rsync emits the transfer-root `./` name line
and a `to-chk=0/2` denominator that counts that root entry. Both must
match FastSync byte-for-byte for the deterministic frames."""
source = os.path.join(TEST_DATA_DIR, "wire_pgroot_src")
dest = os.path.join(TEST_DATA_DIR, "wire_pgroot_dst")
rdst = os.path.join(TEST_DATA_DIR, "wire_pgroot_rdst")
_make_one_file(source, "f.bin", 100)
clean_dir(dest)
# rsync prints the `./` root line only when the transfer root itself is
# created, so make the rsync destination absent. The "created directory"
# line it then emits has no FastSync counterpart (different mirror
# layout), so only the name/frame lines are compared.
shutil.rmtree(rdst, ignore_errors=True)
rsync_result = _rsync(["-a", "--progress", source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
result, _ = run_client(source, dest, flags=["-a", "--progress"],
port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
# subprocess text mode normalizes \r to \n (universal newlines).
def lines_of(text):
return [ln for ln in text.splitlines() if ln and not ln.startswith("created directory")]
rsync_lines = lines_of(rsync_result.stdout)
fast_lines = lines_of(result.stdout)
rsync_names = [ln for ln in rsync_lines if "%" not in ln]
fast_names = [ln for ln in fast_lines if "%" not in ln]
assert rsync_names == ["sending incremental file list", "./", "f.bin"], rsync_names
assert fast_names == rsync_names, (rsync_names, fast_names)
# The final frame's to-chk denominator must include the source-root entry.
assert "to-chk=0/2" in fast_lines[-1], fast_lines[-1]
assert fast_lines[-1] == rsync_lines[-1], (rsync_lines[-1], fast_lines[-1])
@requires_rsync
@pytest.mark.ci
@pytest.mark.parametrize("mt", [False, True])
def test_progress_multidir_file_list_matches_rsync(self, shared_server, mt):
"""A multi-directory tree: the paths-only pre-count must reproduce
rsync's file-list set and `to-chk` denominator. Per-directory name
lines are emitted for directories, symlinks and the empty directory; the
name set and the denominator (every entry plus the transfer root) match
rsync, while the emitted *order* remains a documented residual (rsync
sorts depth-first, FastSync streams in readdir/BFS order)."""
source = os.path.join(TEST_DATA_DIR, "wire_pgmd_src")
dest = os.path.join(TEST_DATA_DIR, "wire_pgmd_dst")
rdst = os.path.join(TEST_DATA_DIR, "wire_pgmd_rdst")
_make_multidir_tree(source)
clean_dir(dest)
clean_dir(rdst)
rsync_result = _rsync(["-a", "--progress", source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
flags = ["-a", "--progress"] + (["--threads"] if mt else [])
result, _ = run_client(source, dest, flags=flags, port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
rsync_names, rsync_totals = _parse_progress(rsync_result.stdout)
fast_names, fast_totals = _parse_progress(result.stdout)
assert sorted(rsync_names) == [
"a.txt", "b.txt", "emptydir/", "link1 -> a.txt", "sub1/",
"sub1/c.txt", "sub1/deep/", "sub1/deep/d.txt", "sub2/", "sub2/e.txt",
], rsync_names
assert fast_names == rsync_names, (rsync_names, fast_names)
# 10 entries + the transfer-root "." counted by rsync's file list.
assert rsync_totals == {11}, rsync_totals
assert fast_totals == rsync_totals, (rsync_totals, fast_totals)
@pytest.mark.ci
def test_progress_delete_during_reuses_pre_scan(self):
"""--delete-during + --progress reuses the keep-set pre-scan instead of
walking the tree a second time: the file-list total and directory name
lines are identical to a plain --progress run."""
source = os.path.join(TEST_DATA_DIR, "wire_pgdel_src")
dest = os.path.join(TEST_DATA_DIR, "wire_pgdel_dst")
_make_multidir_tree(source)
clean_dir(dest)
server = ServerManager()
server.start(extra_args=["--allow-super", "--allow-delete"])
try:
result, _ = run_client(source, dest, flags=["-a", "--progress", "--delete-during"],
port=server.port)
finally:
server.stop()
assert result.returncode == 0, result.stderr[:300]
names, totals = _parse_progress(result.stdout)
assert totals == {11}, totals
assert "sub1/" in names and "sub1/deep/" in names and "emptydir/" in names, names
assert "link1 -> a.txt" in names, names
@requires_rsync
@pytest.mark.ci
@pytest.mark.parametrize("mt", [False, True])
def test_stats_selected_lines_match_rsync(self, shared_server, mt):
"""The protocol-independent --stats lines must match rsync exactly, in
both the single-threaded and --threads (multithreaded) send paths."""
source = os.path.join(TEST_DATA_DIR, "wire_st_src")
dest = os.path.join(TEST_DATA_DIR, "wire_st_dst")
rdst = os.path.join(TEST_DATA_DIR, "wire_st_rdst")
_make_one_file(source, "f.bin", 6000)
clean_dir(dest)
clean_dir(rdst)
# Start both tools from the same state: rsync's destination root exists,
# so pre-create FastSync's mirrored logical root as well.
os.makedirs(get_dest_received_dir(dest, source), exist_ok=True)
rsync_result = _rsync(["-a", "--stats", source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
flags = ["-a", "--stats"] + (["--threads"] if mt else [])
result, _ = run_client(source, dest, flags=flags, port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
keys = (
"Number of regular files transferred",
"Total file size",
"Total transferred file size",
"Literal data",
"Matched data",
"Number of deleted files",
"File list size",
)
def pick(text):
out = {}
for line in text.splitlines():
for key in keys:
if line.startswith(key + ":"):
out[key] = line
return out
assert pick(result.stdout) == pick(rsync_result.stdout), (
f"rsync={pick(rsync_result.stdout)} fastsync={pick(result.stdout)}"
)
@requires_rsync
@pytest.mark.ci
def test_stats_file_count_breakdown_matches_rsync(self, shared_server):
"""`Number of files` and `Number of created files` both carry rsync's
per-type breakdown (protocol 2.28.0 reports the receiver-created
reg/dir/link/special split over STATUS_STATS)."""
source = os.path.join(TEST_DATA_DIR, "wire_stc_src")
dest = os.path.join(TEST_DATA_DIR, "wire_stc_dst")
rdst = os.path.join(TEST_DATA_DIR, "wire_stc_rdst")
_make_one_file(source, "f.bin", 6000)
clean_dir(dest)
clean_dir(rdst)
# Start both tools from the same state: rsync's destination root exists,
# so pre-create FastSync's mirrored logical root as well.
os.makedirs(get_dest_received_dir(dest, source), exist_ok=True)
rsync_result = _rsync(["-a", "--stats", source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
result, _ = run_client(source, dest, flags=["-a", "--stats"],
port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
def stats_line(text, key):
for line in text.splitlines():
if line.startswith(key + ":"):
return line
return None
r_files = stats_line(rsync_result.stdout, "Number of files")
r_created = stats_line(rsync_result.stdout, "Number of created files")
f_files = stats_line(result.stdout, "Number of files")
f_created = stats_line(result.stdout, "Number of created files")
assert re.match(r"Number of files: 2 \(reg: 1, dir: 1\)$", r_files), r_files
assert r_files == f_files, (r_files, f_files)
assert re.match(r"Number of created files: 1 \(reg: 1\)$", r_created), r_created
assert f_created == r_created, (r_created, f_created)
@requires_rsync
@pytest.mark.ci
@pytest.mark.parametrize("mt", [False, True])
def test_stats_created_and_literal_fresh_update_delta(self, shared_server, mt):
"""The receiver-observed counters must match rsync for the three
transfer shapes: a fresh create (created breakdown + whole-file literal),
an update (created == 0, whole-file literal), and a delta update (only
the literal delta fragments are counted, not the whole file)."""
source = os.path.join(TEST_DATA_DIR, "wire_stcd_src")
dest = os.path.join(TEST_DATA_DIR, "wire_stcd_dst")
rdst = os.path.join(TEST_DATA_DIR, "wire_stcd_rdst")
clean_dir(source)
clean_dir(dest)
clean_dir(rdst)
os.makedirs(source, exist_ok=True)
os.makedirs(get_dest_received_dir(dest, source), exist_ok=True)
with open(os.path.join(source, "big.bin"), "wb") as fh:
fh.write(bytes(range(256)) * 4096) # 1 MiB
mt_flag = ["--threads"] if mt else []
def compare(tag):
# Pin the delta block size on both ends: rsync's adaptive block size
# would otherwise make the literal/matched split non-comparable.
rsync_result = _rsync(["-a", "--stats", "--no-whole-file", "-B8192",
source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
result, _ = run_client(
source, dest,
flags=["-a", "--stats", "--incremental", "--delta", "-B8192"] + mt_flag,
port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
keys = ("Number of created files", "Literal data", "Matched data",
"Total transferred file size")
r = _pick_stats(rsync_result.stdout, keys)
f = _pick_stats(result.stdout, keys)
assert r == f, f"{tag}: rsync={r} fastsync={f}"
return r
fresh = compare("fresh")
assert re.match(r"Number of created files: 1 \(reg: 1\)$",
fresh["Number of created files"]), fresh
# Update the source and re-run: the destination already exists.
sleep_mtime = os.path.getmtime(os.path.join(source, "big.bin")) + 2
with open(os.path.join(source, "big.bin"), "r+b") as fh:
fh.seek(100)
fh.write(b"XXXXXXXXXX")
os.utime(os.path.join(source, "big.bin"), (sleep_mtime, sleep_mtime))
update = compare("update")
assert update["Number of created files"] == "Number of created files: 0", update
# Second delta update: change bytes far apart, so rsync ships only the
# literal fragments and FastSync must report the same Literal data.
sleep_mtime = os.path.getmtime(os.path.join(source, "big.bin")) + 2
with open(os.path.join(source, "big.bin"), "r+b") as fh:
fh.seek(500000)
fh.write(b"YYYYYYYYYY")
os.utime(os.path.join(source, "big.bin"), (sleep_mtime, sleep_mtime))
delta = compare("delta")
assert delta["Number of created files"] == "Number of created files: 0", delta
lit = int(delta["Literal data"].split(":", 1)[1].strip().split()[0].replace(",", ""))
assert 0 < lit < 1024 * 1024, delta
@requires_rsync
@pytest.mark.ci
@pytest.mark.parametrize("choice", ["xxh128", "xxh64", "xxh3", "md5", "md4", "sha1", "none"])
def test_out_format_C_selected_algorithm_matches_rsync(self, shared_server, choice):
"""`%C` must use the algorithm selected by --checksum-choice, not always
xxh128, and render it exactly like rsync (big-endian for the 64-bit
hashes, high-then-low for xxh128, standard hex for md5/md4/sha1)."""
source = os.path.join(TEST_DATA_DIR, f"wire_cc_{choice}_src")
dest = os.path.join(TEST_DATA_DIR, f"wire_cc_{choice}_dst")
rdst = os.path.join(TEST_DATA_DIR, f"wire_cc_{choice}_rdst")
_make_one_file(source, "f.bin", 200000)
clean_dir(dest)
clean_dir(rdst)
fmt = "%C %l %n"
rsync_result = _rsync(["-a", "--checksum-choice=" + choice,
"--out-format=" + fmt, source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
result, _ = run_client(source, dest,
flags=["-a", "--checksum-choice=" + choice,
"--out-format=" + fmt],
port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
def file_lines(text):
return [line for line in text.splitlines()
if line and not line.rsplit(" ", 1)[-1].endswith("/")]
assert file_lines(result.stdout) == file_lines(rsync_result.stdout), (
f"choice={choice}: rsync={rsync_result.stdout!r} fastsync={result.stdout!r}"
)
@requires_rsync
@pytest.mark.ci
@pytest.mark.parametrize("mt", [False, True])
def test_dry_run_delete_lines_match_rsync(self, mt):
"""-n --delete emits transfer-relative `*deleting` lines like rsync
(single-threaded and --threads)."""
source = os.path.join(TEST_DATA_DIR, "wire_del_src")
dest = os.path.join(TEST_DATA_DIR, "wire_del_dst")
rdst = os.path.join(TEST_DATA_DIR, "wire_del_rdst")
clean_dir(source)
clean_dir(dest)
clean_dir(rdst)
with open(os.path.join(source, "a.txt"), "wb") as fh:
fh.write(b"a\n")
for root, entries in (
(rdst, {"extra.txt": b"x\n"}),
(rdst, {"sub/y.txt": b"y\n", "extradir/z.txt": b"z\n"}),
):
for rel, data in entries.items():
full = os.path.join(root, rel)
os.makedirs(os.path.dirname(full), exist_ok=True)
with open(full, "wb") as fh:
fh.write(data)
# FastSync mirrors the source's absolute path under dest.
received = get_dest_received_dir(dest, source)
for rel, data in (
("extra.txt", b"x\n"),
("sub/y.txt", b"y\n"),
("extradir/z.txt", b"z\n"),
):
full = os.path.join(received, rel)
os.makedirs(os.path.dirname(full), exist_ok=True)
with open(full, "wb") as fh:
fh.write(data)
rsync_result = _rsync(["-a", "-n", "--delete", "-i", source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
rsync_del = sorted(
line for line in rsync_result.stdout.splitlines() if line.startswith("*deleting")
)
# The shared session server refuses deletion; start one that allows it.
flags = ["-a", "-n", "--delete", "-i"] + (["--threads"] if mt else [])
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
result, _ = run_client(source, dest, flags=flags, port=server.port)
assert result.returncode == 0, result.stderr[:300]
fast_del = sorted(
line for line in result.stdout.splitlines() if line.startswith("*deleting")
)
assert fast_del == rsync_del, f"rsync={rsync_del}\nfastsync={fast_del}"
+299
View File
@@ -0,0 +1,299 @@
"""Differential/regression coverage for the parity-completion review blockers.
Each test pins a fix against real ``rsync 3.4.1`` where a deterministic
comparison exists; the differential tests skip cleanly when rsync is absent.
"""
import os
import shutil
import subprocess
import sys
import pytest
sys.path.insert(0, os.path.dirname(__file__))
from common import ( # noqa: E402
TEST_DATA_DIR,
ServerManager,
clean_dir,
get_dest_received_dir,
run_client,
)
RSYNC = shutil.which("rsync")
requires_rsync = pytest.mark.skipif(RSYNC is None, reason="rsync 3.4.1 not installed")
def _write(path, content):
os.makedirs(os.path.dirname(path), exist_ok=True)
with open(path, "wb") as fh:
fh.write(content)
def _tree(root):
"""Sorted relative paths of every entry below root (files and dirs)."""
out = []
for dirpath, dirs, files in os.walk(root):
for name in dirs:
out.append(os.path.relpath(os.path.join(dirpath, name), root))
for name in files:
out.append(os.path.relpath(os.path.join(dirpath, name), root))
return sorted(out)
def _rsync(args):
env = dict(os.environ, LC_ALL="C")
return subprocess.run([RSYNC] + args, capture_output=True, text=True, env=env, timeout=120)
class TestRelativePerDirDeleteScope:
"""Blocker #1: -R --delete-during/--delete-delay must not delete destination
content outside the transferred prefix (rsync keeps sibling directories)."""
@requires_rsync
@pytest.mark.ci
@pytest.mark.parametrize("mt", [False, True])
@pytest.mark.parametrize("timing", ["--delete-during", "--delete-delay"])
def test_prefix_scoped_delete_matches_rsync(self, timing, mt):
source = os.path.join(TEST_DATA_DIR, f"delblk_src{int(mt)}")
clean_dir(source)
_write(os.path.join(source, "foo", "a.txt"), b"payload\n")
spec = source + "/./foo"
def seed(root):
clean_dir(root)
_write(os.path.join(root, "foo", "extra.txt"), b"stale\n")
_write(os.path.join(root, "unrelated", "keep.txt"), b"keep\n")
rdst = os.path.join(TEST_DATA_DIR, f"delblk_rdst{int(mt)}")
dest = os.path.join(TEST_DATA_DIR, f"delblk_dst{int(mt)}")
seed(rdst)
seed(dest)
r = _rsync(["-aR", timing, spec, rdst + "/"])
assert r.returncode == 0, r.stderr
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
flags = ["-a", "-R", timing] + (["--threads"] if mt else [])
result, _ = run_client(spec, dest, flags=flags, port=server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
#The prefix's parent-directory sibling survives on both sides.
assert os.path.isfile(os.path.join(dest, "unrelated", "keep.txt"))
assert os.path.isfile(os.path.join(rdst, "unrelated", "keep.txt"))
#The in - scope extra is removed on both sides.
assert not os.path.exists(os.path.join(dest, "foo", "extra.txt"))
assert not os.path.exists(os.path.join(rdst, "foo", "extra.txt"))
assert _tree(dest) == _tree(rdst)
def _stats_value(text, label):
for line in text.splitlines():
if line.startswith(label + ":"):
return int(line.split(":", 1)[1].strip().split()[0].replace(",", ""))
return None
def _seed_delta_pair(tag):
"""Source file plus a same-size/basis destination file whose mtime differs,
and an extra destination file to be deleted."""
source = os.path.join(TEST_DATA_DIR, f"stats_{tag}_src")
dest = os.path.join(TEST_DATA_DIR, f"stats_{tag}_dst")
rdst = os.path.join(TEST_DATA_DIR, f"stats_{tag}_rdst")
clean_dir(source)
clean_dir(dest)
clean_dir(rdst)
payload = (b"0123456789abcdef" * 16384)[:200000]
_write(os.path.join(source, "f.bin"), payload)
#Destination basis : same length, one byte changed, deliberately older.
basis = bytearray(payload)
basis[100000] ^= 0xFF
received = get_dest_received_dir(dest, source)
for root in (rdst, received):
_write(os.path.join(root, "f.bin"), bytes(basis))
_write(os.path.join(root, "extra.txt"), b"delete me\n")
old = 1000000
os.utime(os.path.join(root, "f.bin"), (old, old))
return source, dest, rdst
class TestReceiverWireStats:
"""Blocker #3/#4: the receiver must populate the STATUS_STATS counters
(matched data, deleted files) on both the single-threaded and -m paths."""
@requires_rsync
@pytest.mark.ci
@pytest.mark.parametrize("threads", [False, True])
def test_stats_reports_matched_and_deleted(self, threads):
source, dest, rdst = _seed_delta_pair(f"mt{int(threads)}")
rsync_result = _rsync(["-a", "--stats", "--delete", "--no-whole-file", source + "/",
rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
assert _stats_value(rsync_result.stdout, "Matched data") > 0
assert _stats_value(rsync_result.stdout, "Number of deleted files") == 1
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
flags = ["-a", "--stats", "--delete", "--delta", "--incremental"]
if threads:
flags.append("--threads")
result, _ = run_client(source, dest, flags=flags, port=server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
assert _stats_value(result.stdout, "Matched data") > 0, result.stdout
assert _stats_value(result.stdout, "Number of deleted files") == 1, result.stdout
@requires_rsync
@pytest.mark.ci
def test_threads_dry_run_delete_lines_match_rsync(self):
"""-n --delete --threads must emit transfer-relative `*deleting` lines."""
source = os.path.join(TEST_DATA_DIR, "stats_drydel_src")
dest = os.path.join(TEST_DATA_DIR, "stats_drydel_dst")
rdst = os.path.join(TEST_DATA_DIR, "stats_drydel_rdst")
clean_dir(source)
clean_dir(dest)
clean_dir(rdst)
_write(os.path.join(source, "a.txt"), b"a\n")
for root in (rdst, get_dest_received_dir(dest, source)):
_write(os.path.join(root, "extra.txt"), b"x\n")
_write(os.path.join(root, "sub", "y.txt"), b"y\n")
rsync_result = _rsync(["-a", "-n", "--delete", "-i", source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
rsync_del = sorted(
line for line in rsync_result.stdout.splitlines() if line.startswith("*deleting")
)
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
result, _ = run_client(source, dest,
flags=["-a", "-n", "--delete", "-i", "--threads"],
port=server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
fast_del = sorted(
line for line in result.stdout.splitlines() if line.startswith("*deleting")
)
assert fast_del and fast_del == rsync_del, f"rsync={rsync_del}\nfastsync={fast_del}"
class TestRelativeFilesFromProtect:
"""Blocker #9: a -R + --files-from receiver-protect rule must record the bare
relative wire path so the protected destination mirror survives --delete."""
@pytest.mark.ci
@pytest.mark.parametrize("mt", [False, True])
def test_hidden_protected_mirror_survives_delete(self, mt):
source = os.path.join(TEST_DATA_DIR, "rfprot_src")
dest = os.path.join(TEST_DATA_DIR, "rfprot_dst")
clean_dir(source)
clean_dir(dest)
#Root - level entry exercises the parallel root scanner; the nested one
#exercises the sequential worker scanner.
_write(os.path.join(source, "root_secret.tmp"), b"root\n")
_write(os.path.join(source, "sub", "nested_secret.tmp"), b"nested\n")
_write(os.path.join(source, "sub", "keep.txt"), b"keep\n")
listfile = os.path.join(TEST_DATA_DIR, "rfprot.list")
with open(listfile, "w") as fh:
fh.write(".\n")
#H hides from the sender, P protects the receiver mirror from-- delete.
filters = ["--filter=H root_secret.tmp", "--filter=P root_secret.tmp",
"--filter=H sub/nested_secret.tmp", "--filter=P sub/nested_secret.tmp"]
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
seed = ["--files-from", listfile, "-R"] + (["--threads"] if mt else [])
result, _ = run_client(source, dest, flags=seed, port=server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
assert os.path.isfile(os.path.join(dest, "root_secret.tmp"))
assert os.path.isfile(os.path.join(dest, "sub", "nested_secret.tmp"))
_write(os.path.join(dest, "extra.txt"), b"extra\n")
_write(os.path.join(dest, "sub", "extra.txt"), b"extra\n")
flags = seed + ["--delete"] + filters
result, _ = run_client(source, dest, flags=flags, port=server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
assert os.path.isfile(os.path.join(dest, "root_secret.tmp")), \
"root-level protected mirror was deleted"
assert os.path.isfile(os.path.join(dest, "sub", "nested_secret.tmp")), \
"nested protected mirror was deleted"
assert not os.path.exists(os.path.join(dest, "extra.txt"))
assert not os.path.exists(os.path.join(dest, "sub", "extra.txt"))
class TestInvalidPerDirFilter:
"""Blocker #8: a per-directory filter file that fails to parse must fail the
scan even when an earlier merge file in the same directory existed."""
@pytest.mark.ci
@pytest.mark.parametrize("mt", [False, True])
def test_invalid_dir_filter_fails_scan(self, mt):
source = os.path.join(TEST_DATA_DIR, "badfilter_src")
dest = os.path.join(TEST_DATA_DIR, "badfilter_dst")
clean_dir(source)
clean_dir(dest)
#A valid.rsync - filter makes any_exists true for the directory; the
#invalid.rules must not then be silently ignored.
_write(os.path.join(source, ".rsync-filter"), b"- *.bak\n")
_write(os.path.join(source, ".rules"), b"protect\n")
_write(os.path.join(source, "a.txt"), b"a\n")
flags = ["-a", "-F", "--filter=: .rules"]
if mt:
flags.append("--threads")
with ServerManager() as server:
result, _ = run_client(source, dest, flags=flags, port=server.port)
assert result.returncode != 0, "invalid per-directory filter was silently ignored"
assert "invalid per-directory filter" in (result.stderr + result.stdout)
class TestWouldDeleteEscaping:
"""Blocker #5: -n --delete --out-format must escape control bytes in a
peer-supplied would-delete path so it cannot forge output lines."""
@pytest.mark.ci
def test_out_format_escapes_control_chars(self):
source = os.path.join(TEST_DATA_DIR, "esc_src")
dest = os.path.join(TEST_DATA_DIR, "esc_dst")
clean_dir(source)
_write(os.path.join(source, "a.txt"), b"a\n")
received = get_dest_received_dir(dest, source)
clean_dir(received)
_write(os.path.join(received, "a.txt"), b"a\n")
#A newline in a destination filename must not split the printed line.
with open(os.path.join(received, "evil\nname.txt"), "wb") as fh:
fh.write(b"x\n")
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
result, _ = run_client(source, dest,
flags=["-a", "-n", "--delete", "--out-format=%n"],
port=server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
assert "\\#012" in result.stdout, result.stdout
assert "evil\nname.txt" not in result.stdout, result.stdout
class TestEmptySourceDirectoryDelete:
"""Blocker #10: an empty in-scope source directory must survive
--delete-during/--delete-delay (rsync keeps it) while its extras are still
removed."""
@requires_rsync
@pytest.mark.ci
@pytest.mark.parametrize("mt", [False, True])
@pytest.mark.parametrize("timing", ["--delete-during", "--delete-delay"])
def test_empty_source_dir_survives_matches_rsync(self, timing, mt):
source = os.path.join(TEST_DATA_DIR, f"emptydir_src{int(mt)}")
dest = os.path.join(TEST_DATA_DIR, f"emptydir_dst{int(mt)}")
rdst = os.path.join(TEST_DATA_DIR, f"emptydir_rdst{int(mt)}")
clean_dir(source)
os.makedirs(os.path.join(source, "empty"))
_write(os.path.join(source, "keep.txt"), b"keep\n")
received = get_dest_received_dir(dest, source)
for root in (rdst, received):
clean_dir(root)
_write(os.path.join(root, "keep.txt"), b"keep\n")
_write(os.path.join(root, "empty", "extra.txt"), b"extra\n")
rsync_result = _rsync(["-a", timing, source + "/", rdst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr
assert os.path.isdir(os.path.join(rdst, "empty"))
assert not os.path.exists(os.path.join(rdst, "empty", "extra.txt"))
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
flags = ["-a", timing] + (["--threads"] if mt else [])
result, _ = run_client(source, dest, flags=flags, port=server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
assert os.path.isdir(os.path.join(received, "empty")), \
"empty source directory was removed"
assert not os.path.exists(os.path.join(received, "empty", "extra.txt"))
assert _tree(received) == _tree(rdst)
File diff suppressed because it is too large. Load diff
+326
View File
@@ -0,0 +1,326 @@
"""rsync 3.4.1 parity for selection/path semantics and client option aliases.
Each test pins behaviour against real ``rsync 3.4.1``; the differential tests
skip cleanly when rsync is not installed.
"""
import os
import shutil
import subprocess
import sys
import pytest
sys.path.insert(0, os.path.dirname(__file__))
from common import (
TEST_DATA_DIR,
CLIENT_CMD,
ServerManager,
run_client,
clean_dir,
get_dest_received_dir,
)
RSYNC = shutil.which("rsync")
requires_rsync = pytest.mark.skipif(RSYNC is None, reason="rsync 3.4.1 not installed")
def _tree(root):
"""Sorted relative paths of directories (``D ``) and files (``F ``)."""
out = []
for dirpath, dirs, files in os.walk(root):
rel = os.path.relpath(dirpath, root)
for d in dirs:
out.append("D " + (d if rel == "." else os.path.join(rel, d)))
for f in files:
out.append("F " + (f if rel == "." else os.path.join(rel, f)))
return sorted(out)
def _rsync(args):
env = dict(os.environ, LC_ALL="C")
return subprocess.run([RSYNC] + args, capture_output=True, text=True, env=env, timeout=120)
def _make_tree(root):
clean_dir(root)
for rel, content in {
"top.txt": b"top\n",
"foo/bar/baz/f.txt": b"deep\n",
"sub/x.txt": b"x\n",
}.items():
full = os.path.join(root, rel)
os.makedirs(os.path.dirname(full), exist_ok=True)
with open(full, "wb") as fh:
fh.write(content)
return root
class TestRelativeGeneral:
"""#11: -R without --files-from uses rsync's '/./' cut and relative
reconstruction instead of always mirroring the full source path."""
@requires_rsync
@pytest.mark.ci
@pytest.mark.parametrize("suffix", ["", "/./foo", "/./foo/bar", "/./"])
def test_relative_cut_matches_rsync(self, shared_server, suffix):
source = _make_tree(os.path.join(TEST_DATA_DIR, "sel_rel_src"))
dest = os.path.join(TEST_DATA_DIR, "sel_rel_dst")
rdst = os.path.join(TEST_DATA_DIR, "sel_rel_rdst")
clean_dir(dest)
clean_dir(rdst)
spec = source + suffix
r = _rsync(["-aR", spec, rdst + "/"])
assert r.returncode == 0, r.stderr
result, _ = run_client(spec, dest, flags=["-R"], port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
assert _tree(rdst) == _tree(dest), f"layout mismatch for {spec!r}"
@requires_rsync
@pytest.mark.ci
def test_no_implied_dirs_matches_rsync(self, shared_server):
source = _make_tree(os.path.join(TEST_DATA_DIR, "sel_nid_src"))
a = os.path.join(source, "foo")
b = os.path.join(source, "foo", "bar")
os.chmod(a, 0o700)
os.chmod(b, 0o711)
os.utime(a, (978307200, 978307200))
os.utime(b, (978307200, 978307200))
spec = source + "/./foo/bar"
for extra in ([], ["--no-implied-dirs"]):
dest = os.path.join(TEST_DATA_DIR, "sel_nid_dst")
rdst = os.path.join(TEST_DATA_DIR, "sel_nid_rdst")
clean_dir(dest)
clean_dir(rdst)
r = _rsync(["-aR"] + extra + [spec, rdst + "/"])
assert r.returncode == 0, r.stderr
result, _ = run_client(spec, dest, flags=["-a", "-R"] + extra,
port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
for rel in ("foo", "foo/bar"):
rs = os.stat(os.path.join(rdst, rel))
fs = os.stat(os.path.join(dest, rel))
assert (rs.st_mode & 0o7777) == (fs.st_mode & 0o7777), \
f"mode mismatch for {rel} with {extra}"
if extra == ["--no-implied-dirs"]:
# The implied parent directory is created at run time (no
# metadata applied), so rsync's and FastSync's separate runs
# can differ by a second; compare with a tolerance.
assert abs(rs.st_mtime - fs.st_mtime) <= 2, \
f"mtime mismatch for {rel} with {extra}"
else:
assert int(rs.st_mtime) == int(fs.st_mtime), \
f"mtime mismatch for {rel} with {extra}"
@requires_rsync
@pytest.mark.ci
def test_no_implied_dirs_files_from_matches_rsync(self, shared_server):
"""-R --no-implied-dirs --files-from: a listed file whose parent is not
itself listed still transfers; the implied parent is created with
default attributes (rsync 3.4.1 parity)."""
source = _make_tree(os.path.join(TEST_DATA_DIR, "sel_nidff_src"))
# Make the implied parent unmistakably non-default on the source so a
# wrongly-applied attribute would be observable.
os.chmod(os.path.join(source, "foo"), 0o700)
os.chmod(os.path.join(source, "foo", "bar"), 0o711)
os.utime(os.path.join(source, "foo"), (978307200, 978307200))
os.utime(os.path.join(source, "foo", "bar"), (978307200, 978307200))
lst = os.path.join(TEST_DATA_DIR, "sel_nidff_list")
with open(lst, "w") as fh:
fh.write("foo/bar/baz/f.txt\n")
dest = os.path.join(TEST_DATA_DIR, "sel_nidff_dst")
rdst = os.path.join(TEST_DATA_DIR, "sel_nidff_rdst")
clean_dir(dest)
clean_dir(rdst)
r = _rsync(["-rlpt", "-R", "--no-implied-dirs", "--files-from=" + lst,
source + "/", rdst + "/"])
assert r.returncode == 0, r.stderr
result, _ = run_client(source, dest, flags=[
"-rlpt", "-R", "--no-implied-dirs", "--files-from", lst],
port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
assert _tree(rdst) == _tree(dest), "implied-parent layout mismatch"
# The implied parents exist on both sides and carry the run-time default
# attributes, not the source's (non-default) ones.
for rel in ("foo", "foo/bar", "foo/bar/baz"):
rs = os.stat(os.path.join(rdst, rel))
fs = os.stat(os.path.join(dest, rel))
assert (rs.st_mode & 0o7777) == (fs.st_mode & 0o7777), \
f"mode mismatch for implied {rel}"
# The listed file is transferred with its content.
with open(os.path.join(dest, "foo", "bar", "baz", "f.txt"), "rb") as fh:
assert fh.read() == b"deep\n"
class TestDirsOneLevel:
"""#13: -d with a trailing slash (or '.') lists the source's immediate
contents; FastSync mirrors them below the source-root mirror, so compare
rsync's destination tree against that mirror."""
@requires_rsync
@pytest.mark.ci
def test_dirs_trailing_slash_matches_rsync(self, shared_server):
source = _make_tree(os.path.join(TEST_DATA_DIR, "sel_dirs_src"))
os.makedirs(os.path.join(source, "empty"), exist_ok=True)
dest = os.path.join(TEST_DATA_DIR, "sel_dirs_dst")
rdst = os.path.join(TEST_DATA_DIR, "sel_dirs_rdst")
clean_dir(dest)
clean_dir(rdst)
r = _rsync(["-d", source + "/", rdst + "/"])
assert r.returncode == 0, r.stderr
result, _ = run_client(source + "/", dest, flags=["-d"], port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
mirror = get_dest_received_dir(dest, source)
assert _tree(rdst) == _tree(mirror)
@requires_rsync
@pytest.mark.ci
def test_dirs_relative_matches_rsync(self, shared_server):
source = _make_tree(os.path.join(TEST_DATA_DIR, "sel_dirsr_src"))
dest = os.path.join(TEST_DATA_DIR, "sel_dirsr_dst")
rdst = os.path.join(TEST_DATA_DIR, "sel_dirsr_rdst")
clean_dir(dest)
clean_dir(rdst)
spec = source + "/./foo"
r = _rsync(["-d", "-R", spec, rdst + "/"])
assert r.returncode == 0, r.stderr
result, _ = run_client(spec, dest, flags=["-d", "-R"], port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
assert _tree(rdst) == _tree(dest)
@requires_rsync
@pytest.mark.ci
def test_relative_delete_scope_matches_rsync(self):
"""-R --delete must be confined to the transferred prefix subtree so a
sibling destination directory survives (rsync parity)."""
source = _make_tree(os.path.join(TEST_DATA_DIR, "sel_delscope_src"))
dest = os.path.join(TEST_DATA_DIR, "sel_delscope_dst")
rdst = os.path.join(TEST_DATA_DIR, "sel_delscope_rdst")
spec = source + "/./foo"
for root in (dest, rdst):
clean_dir(root)
os.makedirs(os.path.join(root, "foo"))
with open(os.path.join(root, "foo", "extra.txt"), "wb") as fh:
fh.write(b"extra\n")
os.makedirs(os.path.join(root, "unrelated"))
with open(os.path.join(root, "unrelated", "keep.txt"), "wb") as fh:
fh.write(b"keep\n")
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
r = _rsync(["-aR", "--delete", spec, rdst + "/"])
assert r.returncode == 0, r.stderr
result, _ = run_client(spec, dest, flags=["-a", "-R", "--delete"],
port=server.port)
assert result.returncode == 0, result.stderr[:300]
assert (os.path.isfile(os.path.join(dest, "unrelated", "keep.txt"))
== os.path.isfile(os.path.join(rdst, "unrelated", "keep.txt")))
assert _tree(dest) == _tree(rdst)
@requires_rsync
@pytest.mark.ci
@pytest.mark.parametrize("mt", [False, True])
def test_relative_delete_protects_excluded_mirror(self, mt):
"""-R --delete with --exclude must protect the destination mirror of an
excluded source path (recorded as a prefix-relative wire path)."""
source = _make_tree(os.path.join(TEST_DATA_DIR, "sel_delexc_src"))
with open(os.path.join(source, "foo", "secret.tmp"), "wb") as fh:
fh.write(b"secret\n")
dest = os.path.join(TEST_DATA_DIR, "sel_delexc_dst")
rdst = os.path.join(TEST_DATA_DIR, "sel_delexc_rdst")
for root in (dest, rdst):
clean_dir(root)
os.makedirs(os.path.join(root, "foo"))
with open(os.path.join(root, "foo", "secret.tmp"), "wb") as fh:
fh.write(b"secret\n")
with open(os.path.join(root, "foo", "extra.txt"), "wb") as fh:
fh.write(b"extra\n")
spec = source + "/./foo"
with ServerManager() as server:
server.start(extra_args=["--allow-delete"])
r = _rsync(["-aR", "--delete", "--exclude=*.tmp", spec, rdst + "/"])
assert r.returncode == 0, r.stderr
flags = ["-a", "-R", "--delete", "--exclude=*.tmp"] + (["--threads"] if mt else [])
result, _ = run_client(spec, dest, flags=flags, port=server.port)
assert result.returncode == 0, result.stderr[:300]
assert _tree(dest) == _tree(rdst)
assert os.path.isfile(os.path.join(dest, "foo", "secret.tmp"))
assert not os.path.exists(os.path.join(dest, "foo", "extra.txt"))
class TestClientAliases:
"""#5: safe rsync option aliases accepted client-side."""
def _seed(self):
source = _make_tree(os.path.join(TEST_DATA_DIR, "sel_alias_src"))
return source
@pytest.mark.parametrize(
"flag",
[
"--ignore-non-existing",
"--protect-args",
"--msgs2stderr",
"--no-msgs2stderr",
"--no-iconv",
"--iconv=.",
"--iconv=-",
],
)
def test_alias_accepted(self, shared_server, flag):
source = self._seed()
dest = os.path.join(TEST_DATA_DIR, "sel_alias_dst")
clean_dir(dest)
result, _ = run_client(source, dest, flags=[flag], port=shared_server.port)
assert result.returncode == 0, f"{flag} rejected: {result.stderr[:300]}"
def test_lone_h_prints_help(self):
result = subprocess.run([CLIENT_CMD[0], "-h"], capture_output=True, text=True,
timeout=30)
assert result.returncode == 0, result.stderr
assert "Usage" in (result.stdout + result.stderr)
def test_h_with_args_still_human_readable(self, shared_server):
source = self._seed()
dest = os.path.join(TEST_DATA_DIR, "sel_h_dst")
clean_dir(dest)
result, _ = run_client(source, dest, flags=["-h"], port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
received = get_dest_received_dir(dest, source)
assert os.path.isfile(os.path.join(received, "top.txt"))
class TestFilesFromEdges:
"""#8/#58: --files-from empty list succeeds; rsync 3.4.1 rejects the
--no-ignore-missing-args negation, so FastSync must reject it too."""
@requires_rsync
@pytest.mark.ci
def test_empty_files_from_list_succeeds(self, shared_server):
source = _make_tree(os.path.join(TEST_DATA_DIR, "sel_ff_src"))
dest = os.path.join(TEST_DATA_DIR, "sel_ff_dst")
rdst = os.path.join(TEST_DATA_DIR, "sel_ff_rdst")
clean_dir(dest)
clean_dir(rdst)
lst = os.path.join(TEST_DATA_DIR, "sel_ff_empty")
with open(lst, "w") as fh:
fh.write("")
r = _rsync(["-a", "--files-from=" + lst, source + "/", rdst + "/"])
assert r.returncode == 0, r.stderr
result, _ = run_client(source, dest, flags=["--files-from", lst],
port=shared_server.port)
assert result.returncode == 0, result.stderr[:300]
assert _tree(rdst) == []
assert _tree(dest) == []
@requires_rsync
@pytest.mark.ci
def test_no_ignore_missing_args_rejected_like_rsync(self):
source = _make_tree(os.path.join(TEST_DATA_DIR, "sel_nima_src"))
r = _rsync(["-a", "--no-ignore-missing-args", source + "/",
os.path.join(TEST_DATA_DIR, "sel_nima_rdst") + "/"])
assert r.returncode != 0, "rsync unexpectedly accepted --no-ignore-missing-args"
cmd = CLIENT_CMD + ["--source-dir", source, "--dest-dir",
os.path.join(TEST_DATA_DIR, "sel_nima_dst"), "--save-to-disk",
"--no-ignore-missing-args"]
result = subprocess.run(cmd, capture_output=True, text=True, timeout=30)
assert result.returncode != 0, "FastSync unexpectedly accepted the negation"
+4 -4
View File
@@ -94,14 +94,14 @@ def _seed_protocol_source(source):
class TestProtocol:
@pytest.mark.ci
def test_protocol_current_version_accepted(self, shared_server):
"""--protocol=2.23.0 (the current PROTOCOL_VERSION) is accepted and the
"""--protocol=2.28.0 (the current PROTOCOL_VERSION) is accepted and the
transfer completes normally."""
source = os.path.join(TEST_DATA_DIR, "proto_ok_src")
dest = os.path.join(TEST_DATA_DIR, "proto_ok_dst")
shutil.rmtree(dest, ignore_errors=True)
os.makedirs(dest)
_seed_protocol_source(source)
result, _ = run_client(source, dest, flags=["--protocol=2.23.0"],
result, _ = run_client(source, dest, flags=["--protocol=2.28.0"],
port=shared_server.port)
assert result.returncode == 0, \
f"--protocol current run failed: {(result.stderr or result.stdout)[:400]}"
@@ -118,8 +118,8 @@ class TestProtocol:
shutil.rmtree(dest, ignore_errors=True)
os.makedirs(dest)
_seed_protocol_source(source)
for bad in ("2.22.0", "2.21.0", "2.20.0", "2.19.0", "2.18.0", "2.17.0", "2.15.0", "2.16.0",
"216", "31"):
for bad in ("2.27.0", "2.26.0", "2.25.0", "2.24.0", "2.23.0", "2.22.0", "2.21.0", "2.20.0",
"2.19.0", "2.18.0", "2.17.0", "2.15.0", "2.16.0", "216", "31"):
result, _ = run_client(source, dest, flags=[f"--protocol={bad}"],
port=shared_server.port)
assert result.returncode != 0, f"--protocol={bad} should be rejected"
+8
View File
@@ -11,6 +11,7 @@
#include "test_daemon_conf.h"
#include "test_daemon_limits.h"
#include "test_delay_updates.h"
#include "test_delete_plan.h"
#include "test_delta.h"
#include "test_file.h"
#include "test_file_list.h"
@@ -42,6 +43,7 @@
#include "test_utils.h"
#include "test_xattr.h"
#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
// Define global test state variables
@@ -51,6 +53,11 @@ bool current_test_failed = false;
int main() {
signal(SIGPIPE, SIG_IGN);
/* The codec/checksum resolvers consult rsync's preference-list environment
* variables; clear them so a developer's shell cannot change test outcomes.
* The env-specific tests set and restore their own values. */
unsetenv("RSYNC_COMPRESS_LIST");
unsetenv("RSYNC_CHECKSUM_LIST");
printf("\033[1;36m=== RUNNING UNIT TESTS ===\033[0m\n\n");
RUN_TEST(test_queue);
@@ -66,6 +73,7 @@ int main() {
RUN_TEST(test_scanner);
RUN_TEST(test_checksum);
RUN_TEST(test_delta);
RUN_TEST(test_delete_plan);
RUN_TEST(test_data);
RUN_TEST(test_protocol);
RUN_TEST(test_protocol_error);
+35
View File
@@ -177,6 +177,40 @@ static void test_change_list_enabled() {
config_delete(config); /* closes config->log_file */
}
/* %C uses the negotiated TRANSFER checksum's column width (not the pre-transfer
* whole-file digest), and `none` renders as a blank 2-char column, matching
* rsync. A not-yet-filled checksum renders as spaces. */
static void test_format_C_padding_uses_transfer_algo() {
ChangeEvent event = sample_event();
Config* config = config_create();
EXPECT_NOT_NULL(config);
/* Deliberately different pre-transfer algorithm: the transfer one must win. */
config->checksum_algo = (int)CHECKSUM_ALGO_MD4;
struct {
int algo;
int width;
} cases[] = {
{CHECKSUM_ALGO_XXH128, 32}, {CHECKSUM_ALGO_XXH64, 16}, {CHECKSUM_ALGO_XXH3, 16},
{CHECKSUM_ALGO_MD5, 32}, {CHECKSUM_ALGO_MD4, 32}, {CHECKSUM_ALGO_SHA1, 40},
{CHECKSUM_ALGO_NONE, 2},
};
for (size_t i = 0; i < sizeof(cases) / sizeof(cases[0]); i++) {
config->checksum_transfer_algo = cases[i].algo;
char expected[64];
size_t n = 0;
expected[n++] = '[';
for (int j = 0; j < cases[i].width; j++)
expected[n++] = ' ';
expected[n++] = ']';
expected[n] = '\0';
char* line = change_render_format("[%C]", config, &event);
EXPECT_NOT_NULL(line);
EXPECT_EQ_STR(line, expected);
free(line);
}
config_delete(config);
}
void test_change_list() {
test_format_tokens();
test_format_unknown_tokens_preserved();
@@ -188,4 +222,5 @@ void test_change_list() {
test_render_itemize_up_to_date_is_empty();
test_render_list_line();
test_change_list_enabled();
test_format_C_padding_uses_transfer_algo();
}
+135 -5
View File
@@ -1,7 +1,9 @@
#include "test_checksum.h"
#include "checksum.h"
#include "test_utils.h"
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
/* Known xxHash64 vector (seed 0) for the empty string and a literal.
* The md5 vectors are the standard NIST/RFC1321 test strings. These pin the
@@ -91,6 +93,63 @@ static void test_checksum_md5_seed_ignored() {
EXPECT_TRUE(memcmp(a, b, alen) == 0);
}
static void test_checksum_md4_vectors() {
uint8_t out[CHECKSUM_MAX_DIGEST_LEN];
size_t len = 0;
/* RFC 1320 / RFC 1321 test vectors. */
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_MD4, 0, "", 0, out, sizeof(out), &len));
EXPECT_TRUE(len == (size_t)16);
const uint8_t expect_empty[16] = {0x31, 0xd6, 0xcf, 0xe0, 0xd1, 0x6a, 0xe9, 0x31,
0xb7, 0x3c, 0x59, 0xd7, 0xe0, 0xc0, 0x89, 0xc0};
EXPECT_TRUE(memcmp(out, expect_empty, 16) == 0);
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_MD4, 0, "abc", 3, out, sizeof(out), &len));
const uint8_t expect_abc[16] = {0xa4, 0x48, 0x01, 0x7a, 0xaf, 0x21, 0xd8, 0x52,
0x5f, 0xc1, 0x0a, 0xe8, 0x7a, 0xa6, 0x72, 0x9d};
EXPECT_TRUE(memcmp(out, expect_abc, 16) == 0);
/* A longer input exercises the block loop and the padding boundary. */
const char* msg =
"12345678901234567890123456789012345678901234567890123456789012345678901234567890";
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_MD4, 0, msg, strlen(msg), out, sizeof(out), &len));
const uint8_t expect_long[16] = {0xe3, 0x3b, 0x4d, 0xdc, 0x9c, 0x38, 0xf2, 0x19,
0x9c, 0x3e, 0x7b, 0x16, 0x4f, 0xcc, 0x05, 0x36};
EXPECT_TRUE(memcmp(out, expect_long, 16) == 0);
}
static void test_checksum_sha1_vectors() {
uint8_t out[CHECKSUM_MAX_DIGEST_LEN];
size_t len = 0;
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_SHA1, 0, "abc", 3, out, sizeof(out), &len));
EXPECT_TRUE(len == (size_t)20);
const uint8_t expect_abc[20] = {0xa9, 0x99, 0x3e, 0x36, 0x47, 0x06, 0x81, 0x6a, 0xba, 0x3e,
0x25, 0x71, 0x78, 0x50, 0xc2, 0x6c, 0x9c, 0xd0, 0xd8, 0x9d};
EXPECT_TRUE(memcmp(out, expect_abc, 20) == 0);
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_SHA1, 0, "", 0, out, sizeof(out), &len));
EXPECT_TRUE(len == (size_t)20);
const uint8_t expect_empty[20] = {0xda, 0x39, 0xa3, 0xee, 0x5e, 0x6b, 0x4b, 0x0d, 0x32, 0x55,
0xbf, 0xef, 0x95, 0x60, 0x18, 0x90, 0xaf, 0xd8, 0x07, 0x09};
EXPECT_TRUE(memcmp(out, expect_empty, 20) == 0);
/* sha1 has no seed: the digest is seed-independent (documented). */
uint8_t seeded[CHECKSUM_MAX_DIGEST_LEN];
size_t seeded_len = 0;
EXPECT_TRUE(
checksum_digest(CHECKSUM_ALGO_SHA1, 12345, "abc", 3, seeded, sizeof(seeded), &seeded_len));
EXPECT_TRUE(seeded_len == (size_t)20);
EXPECT_TRUE(memcmp(expect_abc, seeded, 20) == 0);
}
/* "none" is a successful no-digest: length 0, nothing written. */
static void test_checksum_none_digest() {
uint8_t out[CHECKSUM_MAX_DIGEST_LEN];
size_t len = 99;
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_NONE, 0, "data", 4, out, sizeof(out), &len));
EXPECT_EQ_INT((int)len, 0);
EXPECT_EQ_INT((int)checksum_digest_len(CHECKSUM_ALGO_NONE), 0);
}
static void test_checksum_algo_name_mapping() {
EXPECT_EQ_INT(checksum_algo_from_name("xxh64"), (int)CHECKSUM_ALGO_XXH64);
EXPECT_EQ_INT(checksum_algo_from_name("XXH64"), (int)CHECKSUM_ALGO_XXH64);
@@ -102,12 +161,14 @@ static void test_checksum_algo_name_mapping() {
EXPECT_EQ_INT(checksum_algo_from_name("XXH3"), (int)CHECKSUM_ALGO_XXH3);
EXPECT_EQ_INT(checksum_algo_from_name("xxh128"), (int)CHECKSUM_ALGO_XXH128);
EXPECT_EQ_INT(checksum_algo_from_name("XXH128"), (int)CHECKSUM_ALGO_XXH128);
/* rsync choices FastSync does not implement are rejected by name. */
EXPECT_TRUE(checksum_algo_from_name("md4") < 0);
EXPECT_TRUE(checksum_algo_from_name("sha1") < 0);
EXPECT_EQ_INT(checksum_algo_from_name("md4"), (int)CHECKSUM_ALGO_MD4);
EXPECT_EQ_INT(checksum_algo_from_name("MD4"), (int)CHECKSUM_ALGO_MD4);
EXPECT_EQ_INT(checksum_algo_from_name("sha1"), (int)CHECKSUM_ALGO_SHA1);
EXPECT_EQ_INT(checksum_algo_from_name("SHA1"), (int)CHECKSUM_ALGO_SHA1);
EXPECT_EQ_INT(checksum_algo_from_name("none"), (int)CHECKSUM_ALGO_NONE);
/* Names rsync does not offer (or FastSync cannot compute) are rejected. */
EXPECT_TRUE(checksum_algo_from_name("sha256") < 0);
EXPECT_TRUE(checksum_algo_from_name("crc32") < 0);
EXPECT_TRUE(checksum_algo_from_name("none") < 0);
EXPECT_TRUE(checksum_algo_from_name("") < 0);
EXPECT_TRUE(checksum_algo_from_name(NULL) < 0);
@@ -115,11 +176,20 @@ static void test_checksum_algo_name_mapping() {
EXPECT_TRUE(checksum_algo_valid((int)CHECKSUM_ALGO_MD5));
EXPECT_TRUE(checksum_algo_valid((int)CHECKSUM_ALGO_XXH3));
EXPECT_TRUE(checksum_algo_valid((int)CHECKSUM_ALGO_XXH128));
EXPECT_TRUE(checksum_algo_valid((int)CHECKSUM_ALGO_MD4));
EXPECT_TRUE(checksum_algo_valid((int)CHECKSUM_ALGO_SHA1));
EXPECT_TRUE(checksum_algo_valid((int)CHECKSUM_ALGO_NONE));
EXPECT_FALSE(checksum_algo_valid(99));
EXPECT_EQ_STR(checksum_algo_name(CHECKSUM_ALGO_XXH64), "xxh64");
EXPECT_EQ_STR(checksum_algo_name(CHECKSUM_ALGO_MD5), "md5");
EXPECT_EQ_STR(checksum_algo_name(CHECKSUM_ALGO_XXH3), "xxh3");
EXPECT_EQ_STR(checksum_algo_name(CHECKSUM_ALGO_XXH128), "xxh128");
EXPECT_EQ_STR(checksum_algo_name(CHECKSUM_ALGO_MD4), "md4");
EXPECT_EQ_STR(checksum_algo_name(CHECKSUM_ALGO_SHA1), "sha1");
EXPECT_EQ_STR(checksum_algo_name(CHECKSUM_ALGO_NONE), "none");
/* rsync 3.4.1 auto-negotiates xxh128 first. */
EXPECT_EQ_INT((int)checksum_negotiate_default(), (int)CHECKSUM_ALGO_XXH128);
}
/* xxh3 is 8 bytes and seed-aware; xxh128 is 16 bytes and differs from both
@@ -162,16 +232,76 @@ static void test_checksum_null_empty_digest() {
EXPECT_TRUE(memcmp(a, b, alen) == 0);
}
/* Every algorithm checksum_digest_file() claims to support must produce the
* SAME digest as the in-memory one-shot, including the newly added md4/sha1/
* none. A 200000-byte payload forces several 64 KiB streaming reads. */
static void test_checksum_digest_file_matches_oneshot(void) {
static const ChecksumAlgo algos[] = {
CHECKSUM_ALGO_XXH64, CHECKSUM_ALGO_XXH3, CHECKSUM_ALGO_XXH128, CHECKSUM_ALGO_MD5,
CHECKSUM_ALGO_MD4, CHECKSUM_ALGO_SHA1, CHECKSUM_ALGO_NONE,
};
enum { SIZE = 200000 };
uint8_t* data = malloc(SIZE);
EXPECT_NOT_NULL(data);
for (int i = 0; i < SIZE; i++)
data[i] = (uint8_t)((i * 7 + 3) & 0xff);
char path[] = "/tmp/fastsync_ck_XXXXXX";
int fd = mkstemp(path);
EXPECT_TRUE(fd >= 0);
ssize_t written = write(fd, data, SIZE);
close(fd);
EXPECT_EQ_INT((int)written, SIZE);
for (size_t a = 0; a < sizeof(algos) / sizeof(algos[0]); a++) {
ChecksumAlgo algo = algos[a];
uint8_t one[CHECKSUM_MAX_DIGEST_LEN];
uint8_t file[CHECKSUM_MAX_DIGEST_LEN];
size_t one_len = 0, file_len = 0;
EXPECT_TRUE(checksum_digest(algo, 0, data, SIZE, one, sizeof(one), &one_len));
EXPECT_TRUE(checksum_digest_file(algo, 0, path, file, sizeof(file), &file_len));
EXPECT_EQ_INT((int)file_len, (int)one_len);
EXPECT_TRUE(memcmp(one, file, one_len) == 0);
}
unlink(path);
free(data);
}
/* RSYNC_CHECKSUM_LIST precedence, syntax and fallback. */
static void test_checksum_choice_env_list() {
unsetenv("RSYNC_CHECKSUM_LIST");
EXPECT_EQ_INT(checksum_choice_resolve(), (int)CHECKSUM_ALGO_XXH128);
EXPECT_EQ_INT((int)checksum_negotiate_default(), (int)CHECKSUM_ALGO_XXH128);
setenv("RSYNC_CHECKSUM_LIST", "bogus md5 xxh3", 1);
EXPECT_EQ_INT(checksum_choice_resolve(), (int)CHECKSUM_ALGO_MD5);
setenv("RSYNC_CHECKSUM_LIST", "SHA1", 1);
EXPECT_EQ_INT(checksum_choice_resolve(), (int)CHECKSUM_ALGO_SHA1);
/* Whitespace-separated only: comma is not a separator in rsync's syntax. */
setenv("RSYNC_CHECKSUM_LIST", "md5,xxh3", 1);
EXPECT_EQ_INT(checksum_choice_resolve(), -1);
setenv("RSYNC_CHECKSUM_LIST", " ", 1);
EXPECT_EQ_INT(checksum_choice_resolve(), (int)CHECKSUM_ALGO_XXH128);
unsetenv("RSYNC_CHECKSUM_LIST");
}
void test_checksum(void) {
test_checksum_xxh64_seed0();
test_checksum_xxh64_empty();
test_checksum_xxh64_seed_changes_digest();
test_checksum_xxh64_seed_deterministic();
test_checksum_md5_vectors();
test_checksum_md4_vectors();
test_checksum_sha1_vectors();
test_checksum_none_digest();
test_checksum_algo_lengths_distinct();
test_checksum_md5_seed_ignored();
test_checksum_algo_name_mapping();
test_checksum_xxh3_xxh128();
test_checksum_truncated_buffer_rejected();
test_checksum_null_empty_digest();
}
test_checksum_digest_file_matches_oneshot();
test_checksum_choice_env_list();
}
+584 -26
View File
@@ -7,6 +7,7 @@
#include "delta.h"
#include "file_list.h"
#include "log.h"
#include "protocol.h"
#include "test_utils.h"
#include "utils.h"
#include <pwd.h>
@@ -317,7 +318,7 @@ static void test_parse_args_protocol_accept_current() {
Config* cfg = valid_client_config();
EXPECT_NOT_NULL(cfg);
char* argv_equals[] = {"fastsync", "--source-dir", "/src",
"--dest-dir", "/dst", "--protocol=2.23.0"};
"--dest-dir", "/dst", "--protocol=2.28.0"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 6, argv_equals, positional_args, &positional_count), 0);
@@ -327,7 +328,7 @@ static void test_parse_args_protocol_accept_current() {
cfg = valid_client_config();
EXPECT_NOT_NULL(cfg);
char* argv_space[] = {"fastsync", "--source-dir", "/src", "--dest-dir",
"/dst", "--protocol", "2.23.0"};
"/dst", "--protocol", "2.28.0"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 7, argv_space, positional_args, &positional_count), 0);
EXPECT_EQ_STR(cfg->version, PROTOCOL_VERSION);
@@ -339,6 +340,7 @@ static void test_parse_args_protocol_accept_current() {
static void test_parse_args_protocol_rejects_other_versions() {
static const char* const bad_versions[] = {"2.17", "2.16", "2.15.0", "2.16.0", "2.17.0",
"2.18.0", "2.19.0", "2.20.0", "2.21.0", "2.22.0",
"2.23.0", "2.24.0", "2.25.0", "2.26.0", "2.27.0",
"216", "31", "abc", ""};
for (size_t i = 0; i < sizeof(bad_versions) / sizeof(bad_versions[0]); i++) {
Config* cfg = valid_client_config();
@@ -776,7 +778,7 @@ static void test_parse_args_debug_help() {
}
static void test_parse_args_debug_flags_validation() {
static const char* const values[] = {"", "io,", ",io", "io,,proto", "acl", "tls", "unknown"};
static const char* const values[] = {"", "io,", ",io", "io,,proto", "tls", "unknown"};
for (size_t i = 0; i < sizeof(values) / sizeof(values[0]); i++) {
Config* cfg = config_create();
char option[64];
@@ -1044,10 +1046,28 @@ static void test_parse_args_basis_dirs() {
config_delete(cfg);
}
/* Absolute, escaping, or degenerate basis-dir values must be rejected up
front: they would resolve outside the destination root on the receiver. */
/* --verify-basis (FastSync-only, long-only): default off; parses on as a plain
boolean and leaves the basis implications intact. */
static void test_parse_args_verify_basis() {
Config* cfg = config_create();
EXPECT_FALSE(cfg->verify_basis);
config_delete(cfg);
cfg = config_create();
int positional_args[2];
int positional_count = 0;
char* argv[] = {"fastsync", "--link-dest=prior", "--verify-basis", "/src", "/dst"};
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->verify_basis);
EXPECT_TRUE(config_has_basis(cfg));
config_delete(cfg);
}
/* Escaping or degenerate basis-dir values must be rejected up front (they would
resolve outside the destination root on the receiver); an absolute path is
accepted (rsync parity) and canonicalized with its leading '/' preserved. */
static void test_parse_args_basis_invalid_paths() {
static const char* const invalid[] = {"/abs", "..", "a/../b", "."};
static const char* const invalid[] = {"..", "a/../b", ".", "/", ""};
for (size_t i = 0; i < sizeof(invalid) / sizeof(invalid[0]); i++) {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--link-dest", (char*)invalid[i], "/src", "/dst"};
@@ -1056,6 +1076,15 @@ static void test_parse_args_basis_invalid_paths() {
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), -1);
config_delete(cfg);
}
Config* cfg = config_create();
char* argv[] = {"fastsync", "--link-dest=/abs/dir", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->basis_count, 1);
EXPECT_EQ_STR(cfg->basis_dirs[0].path, "/abs/dir");
config_delete(cfg);
}
/* Basis dirs require the per-file incremental handshake, which -s disables. */
@@ -1121,6 +1150,63 @@ static void test_parse_args_delete_timing_flags() {
config_delete(cfg);
}
/* Plain --delete with no explicit timing defaults to delete-during, matching
* rsync's --del default (progressive deletion). --delete-commit is the
* FastSync-only long spelling that selects rsync's --delete-after timing (the
* late whole-tree commit), and an explicit timing always wins over the default.
*/
static void test_parse_args_delete_default_timing_and_commit() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--delete", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->use_delete);
EXPECT_TRUE(cfg->delete_during);
EXPECT_FALSE(cfg->delete_before);
EXPECT_FALSE(cfg->delete_delay);
EXPECT_FALSE(cfg->delete_after);
cfg->send_directory = str_dup("/src");
cfg->receive_root_directory = str_dup("/dst");
EXPECT_TRUE(validate_config(cfg));
config_delete(cfg);
/* --delete-commit selects the late whole-tree commit (delete_after) and
implies --delete. */
cfg = config_create();
char* argv_commit[] = {"fastsync", "--delete-commit", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv_commit, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->use_delete);
EXPECT_TRUE(cfg->delete_after);
EXPECT_FALSE(cfg->delete_before);
EXPECT_FALSE(cfg->delete_during);
EXPECT_FALSE(cfg->delete_delay);
cfg->send_directory = str_dup("/src");
cfg->receive_root_directory = str_dup("/dst");
EXPECT_TRUE(validate_config(cfg));
config_delete(cfg);
/* An explicit --delete-after alongside plain --delete keeps the late timing:
the default never overwrites an explicit timing. */
cfg = config_create();
char* argv_after[] = {"fastsync", "--delete", "--delete-after", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv_after, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->use_delete);
EXPECT_TRUE(cfg->delete_after);
EXPECT_FALSE(cfg->delete_during);
config_delete(cfg);
/* --delete-commit conflicts with a different timing. */
cfg = config_create();
char* argv_conflict[] = {"fastsync", "--delete-commit", "--delete-during", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv_conflict, positional_args, &positional_count), 0);
EXPECT_FALSE(validate_config(cfg));
config_delete(cfg);
}
/* Two different delete-timing flags on one command line are a conflict, not a
* silent last-one-wins choice. */
static void test_parse_args_delete_timing_conflict_rejected() {
@@ -1317,7 +1403,110 @@ static void test_parse_args_rejects_invalid_info_flag() {
config_delete(cfg);
}
/* rsync's info "name" category maps to fastsync's per-file name output, and
* --info=help prints the flag list and exits without error. */
static void test_parse_args_info_name_and_help() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--info=name", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->info_level, LOG_INFO_NAME);
config_delete(cfg);
cfg = config_create();
char* help_argv[] = {"fastsync", "--info=help"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 2, help_argv, positional_args, &positional_count), 1);
config_delete(cfg);
}
/* rsync 3.4.1's full --info/--debug vocabulary parses. The info categories
* with a FastSync event set their flag; the remaining rsync-only categories
* (backup/mount/symsafe/syms) parse but stay silent. Every --debug category
* listed here is FastSync-silent, so debug_level stays 0. */
static void test_parse_args_rsync_flag_vocabulary_accepted() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--info=backup,del,flist,mount,nonreg,progress,remove,symsafe,syms",
"--debug=acl,backup,bind,chdir,cmd,connect,del,deltasum,dup,exit,"
"filter,flist,fuzzy,genr,hash,hlink,iconv,nstr,own,recv,send,time,"
"hl,owner",
"/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->info_level, LOG_INFO_DEL | LOG_INFO_FLIST | LOG_INFO_NONREG |
LOG_INFO_PROGRESS | LOG_INFO_REMOVE);
EXPECT_EQ_INT(cfg->debug_level, 0);
config_delete(cfg);
}
/* Test parse_args with --archive flag */
/* rsync accepts a trailing level digit on --debug/--info items (e.g. io2,
* all4); level 0 silences the item. */
static void test_parse_args_debug_info_levels() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--debug=io2,proto0,all", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->debug_level, LOG_DEBUG_ALL);
config_delete(cfg);
cfg = config_create();
char* io0_argv[] = {"fastsync", "--debug=all,io0", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, io0_argv, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->debug_level, LOG_DEBUG_ALL & ~LOG_DEBUG_IO);
config_delete(cfg);
cfg = config_create();
char* info_argv[] = {"fastsync", "--info=stats2", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, info_argv, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->info_level, LOG_INFO_STATS);
config_delete(cfg);
/* --info=name level 2 enables the "is uptodate" marker; a later level-1 or
level-0 token clears it again. */
cfg = config_create();
char* name2_argv[] = {"fastsync", "--info=name2", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, name2_argv, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->info_level, LOG_INFO_NAME | LOG_INFO_NAME_UPTODATE);
config_delete(cfg);
cfg = config_create();
char* name1_argv[] = {"fastsync", "--info=name2,name1", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, name1_argv, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->info_level, LOG_INFO_NAME);
config_delete(cfg);
cfg = config_create();
char* name0_argv[] = {"fastsync", "--info=name2,name0", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, name0_argv, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->info_level, 0);
config_delete(cfg);
cfg = config_create();
char* bad_argv[] = {"fastsync", "--debug=123", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, bad_argv, positional_args, &positional_count), -1);
config_delete(cfg);
/* rsync accepts category names case-insensitively. */
cfg = config_create();
char* upper_argv[] = {"fastsync", "--info=STATS2", "--debug=IO", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, upper_argv, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->info_level, LOG_INFO_STATS);
EXPECT_EQ_INT(cfg->debug_level, LOG_DEBUG_IO);
config_delete(cfg);
}
static void test_parse_args_archive() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--archive", "/src", "/dst"};
@@ -1440,22 +1629,25 @@ static void test_parse_args_checksum_choice_equals_forms() {
config_delete(cfg);
}
/* An algorithm FastSync does not support must be rejected, never a silent
no-op. */
/* An algorithm FastSync does not support, an empty half, a lone/extra comma or
a malformed separator must be rejected, never a silent no-op. A single
md4/sha1/none name and the two-name transfer,pre-transfer form are valid. */
static void test_parse_args_checksum_choice_rejects_unsupported() {
static const char* const bad[] = {"md4", "sha1", "sha256", "crc32",
"none", "bogus", "xxh64,md5", "xxhash:md5"};
static const char* const bad[] = {"sha256", "crc32", "bogus", "xxhash:md5",
"md5,", ",md5", "md5,md4,sha1"};
for (size_t i = 0; i < sizeof(bad) / sizeof(bad[0]); i++) {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--checksum-choice", (char*)bad[i], "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), -1);
EXPECT_EQ_INT(cfg->cli_exit_code, 4);
config_delete(cfg);
}
}
/* xxh3/xxh128 are accepted; "auto" keeps the default algorithm. */
/* xxh3/xxh128/md4/sha1/none and the two-name form are accepted; "auto"
resolves to FastSync's negotiated default xxh128. */
static void test_parse_args_checksum_choice_new_algos() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--checksum-choice=xxh3", "/src", "/dst"};
@@ -1476,7 +1668,69 @@ static void test_parse_args_checksum_choice_new_algos() {
char* argv3[] = {"fastsync", "--checksum-choice=auto", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv3, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->checksum_algo, (int)CHECKSUM_ALGO_XXH64);
EXPECT_EQ_INT(cfg->checksum_algo, (int)CHECKSUM_ALGO_XXH128);
config_delete(cfg);
/* A single md4/sha1 name selects it for both transfer and pre-transfer. */
static const int single[] = {(int)CHECKSUM_ALGO_MD4, (int)CHECKSUM_ALGO_SHA1};
static const char* const single_names[] = {"md4", "sha1"};
for (size_t i = 0; i < 2; i++) {
cfg = config_create();
char* arg = (char*)single_names[i];
char* argv4[] = {"fastsync", "--cc", arg, "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv4, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->checksum_algo, single[i]);
EXPECT_EQ_INT(cfg->checksum_transfer_algo, single[i]);
config_delete(cfg);
}
/* Two-name form: first is the transfer checksum, second the pre-transfer one
that FastSync actually uses. */
cfg = config_create();
char* argv5[] = {"fastsync", "--cc=sha1,md4", "/checksum/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv5, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->checksum_transfer_algo, (int)CHECKSUM_ALGO_SHA1);
EXPECT_EQ_INT(cfg->checksum_algo, (int)CHECKSUM_ALGO_MD4);
config_delete(cfg);
/* "none" is accepted without --checksum but forces --whole-file like rsync. */
cfg = config_create();
char* argv6[] = {"fastsync", "--cc=none", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv6, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->checksum_algo, (int)CHECKSUM_ALGO_NONE);
EXPECT_TRUE(cfg->whole_file);
config_delete(cfg);
}
/* rsync rejects "none" as the pre-transfer checksum with --checksum (exit 4),
regardless of option order. */
static void test_parse_args_checksum_none_with_checksum_rejected() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--checksum", "--cc=none", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), -1);
EXPECT_EQ_INT(cfg->cli_exit_code, 4);
config_delete(cfg);
cfg = config_create();
char* argv2[] = {"fastsync", "--checksum", "--cc=md5,none", "/checksum/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv2, positional_args, &positional_count), -1);
EXPECT_EQ_INT(cfg->cli_exit_code, 4);
config_delete(cfg);
/* "none" as the TRANSFER checksum with a real pre-transfer checksum is
accepted (rsync allows none,md5 with -c). */
cfg = config_create();
char* argv3[] = {"fastsync", "--checksum", "--cc=none,md5", "/checksum/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv3, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->checksum_algo, (int)CHECKSUM_ALGO_MD5);
EXPECT_TRUE(cfg->whole_file);
config_delete(cfg);
}
@@ -1535,28 +1789,40 @@ static void test_parse_args_timeout_zero_and_no_forms() {
config_delete(cfg);
}
/* rsync's --compress-choice choices FastSync does not implement are rejected by
* name; zstd/none/auto are accepted. */
/* Every rsync 3.4.1 --compress-choice name is accepted and mapped to a real
* codec; "auto" resolves to the negotiated default (zstd). An unknown name is
* rejected with rsync's exit code 4. */
static void test_parse_args_compress_choice_parity() {
static const char* const good[] = {"zstd", "none", "auto"};
struct {
const char* name;
CompressionAlgo algo;
bool enabled;
} good[] = {
{"zstd", COMPRESSION_ALGO_ZSTD, true}, {"lz4", COMPRESSION_ALGO_LZ4, true},
{"zlib", COMPRESSION_ALGO_ZLIB, true}, {"zlibx", COMPRESSION_ALGO_ZLIBX, true},
{"none", COMPRESSION_ALGO_NONE, false}, {"auto", COMPRESSION_ALGO_ZSTD, true},
{"ZSTD", COMPRESSION_ALGO_ZSTD, true},
};
for (size_t i = 0; i < sizeof(good) / sizeof(good[0]); i++) {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--compress-choice", (char*)good[i], "/src", "/dst"};
char* argv[] = {"fastsync", "--compress-choice", (char*)good[i].name, "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), 0);
/* "auto" is normalized to the canonical "zstd" the receiver accepts. */
EXPECT_EQ_STR(cfg->compress_choice, strcmp(good[i], "auto") == 0 ? "zstd" : good[i]);
EXPECT_EQ_INT(cfg->use_compression, strcmp(good[i], "none") != 0 ? 1 : 0);
EXPECT_EQ_STR(cfg->compress_choice, compression_algo_name(good[i].algo));
EXPECT_EQ_INT(cfg->compression_algo, (int)good[i].algo);
EXPECT_EQ_INT(cfg->use_compression, good[i].enabled ? 1 : 0);
config_delete(cfg);
}
static const char* const bad[] = {"lz4", "zlib", "zlibx", "bogus"};
static const char* const bad[] = {"bogus", "", "zstd,lz4"};
for (size_t i = 0; i < sizeof(bad) / sizeof(bad[0]); i++) {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--compress-choice", (char*)bad[i], "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), -1);
EXPECT_EQ_INT(cfg->cli_exit_code, 4);
config_delete(cfg);
}
}
@@ -2240,6 +2506,119 @@ static void test_parse_args_rejects_invalid_compression_choice() {
config_delete(cfg);
}
/* rsync gives each codec its own default --compress-level; an omitted level
* resolves to that default and an explicit one is clamped to the codec range. */
static void test_parse_args_per_codec_compression_level_defaults() {
unsetenv("RSYNC_COMPRESS_LIST");
struct {
const char* choice;
int level;
} cases[] = {
{"zstd", 3},
{"zlib", 6},
{"zlibx", 6},
{"lz4", 1},
};
for (size_t i = 0; i < sizeof(cases) / sizeof(cases[0]); i++) {
Config* cfg = config_create();
char* argv[] = {"fastsync", "-z", "--compress-choice", (char*)cases[i].choice, "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 6, argv, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->use_compression);
EXPECT_EQ_INT(cfg->compression_level, cases[i].level);
config_delete(cfg);
}
/* Bare -z resolves to the zstd default. */
Config* cfg = config_create();
char* bare[] = {"fastsync", "-z", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, bare, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->compression_level, 3);
config_delete(cfg);
/* An explicit level wins unchanged for zstd... */
cfg = config_create();
char* zv[] = {"fastsync", "-z", "--compress-level", "10", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, zv, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->compression_level, 10);
config_delete(cfg);
/* ...but zlib clamps an over-range level to 9 like rsync. */
cfg = config_create();
char* zc[] = {"fastsync", "-z", "--compress-choice", "zlib", "--compress-level", "15",
"/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 8, zc, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->compression_level, 9);
config_delete(cfg);
}
/* RSYNC_COMPRESS_LIST drives the bare -z ("auto") resolution. */
static void test_parse_args_compression_env_list() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "-z", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
setenv("RSYNC_COMPRESS_LIST", "zlib lz4", 1);
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->compression_algo, (int)COMPRESSION_ALGO_ZLIB);
EXPECT_EQ_INT(cfg->compression_level, 6);
config_delete(cfg);
/* An explicit --compress-choice beats the env list. */
cfg = config_create();
char* explicit_argv[] = {"fastsync", "-z", "--compress-choice", "zstd", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 6, explicit_argv, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->compression_algo, (int)COMPRESSION_ALGO_ZSTD);
config_delete(cfg);
/* A list with no supported name is rsync's failed negotiation (exit 4). */
setenv("RSYNC_COMPRESS_LIST", "bogus", 1);
cfg = config_create();
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), -1);
EXPECT_EQ_INT(cfg->cli_exit_code, 4);
config_delete(cfg);
unsetenv("RSYNC_COMPRESS_LIST");
}
/* RSYNC_CHECKSUM_LIST drives the default checksum choice. */
static void test_parse_args_checksum_env_list() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--checksum", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
setenv("RSYNC_CHECKSUM_LIST", "md5", 1);
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->checksum_algo, (int)CHECKSUM_ALGO_MD5);
EXPECT_EQ_INT(cfg->checksum_transfer_algo, (int)CHECKSUM_ALGO_MD5);
config_delete(cfg);
/* An explicit --cc wins. */
cfg = config_create();
char* cc_argv[] = {"fastsync", "--checksum", "--cc=sha1", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, cc_argv, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->checksum_algo, (int)CHECKSUM_ALGO_SHA1);
config_delete(cfg);
/* A list with no supported name is rsync's failed negotiation (exit 4). */
setenv("RSYNC_CHECKSUM_LIST", "bogus", 1);
cfg = config_create();
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), -1);
EXPECT_EQ_INT(cfg->cli_exit_code, 4);
config_delete(cfg);
unsetenv("RSYNC_CHECKSUM_LIST");
}
/* Every value-taking table option accepts an inline "--opt=value" form. */
static void test_parse_args_table_equals_size_options() {
Config* cfg = config_create();
@@ -2492,15 +2871,35 @@ static void test_parse_args_filter_rules() {
EXPECT_EQ_INT(parse_args(cfg, 4, missing_argv, positional_args, &positional_count), -1);
config_delete(cfg);
/* rsync shorthands/modifiers we do not support are rejected instead of being
* silently parsed as literal patterns. */
static const char* const unsupported[] = {
": .rsync-filter", ". /tmp/rules", "-s foo", "-p bar", "-C", "-! *.o", "!",
/* Full rsync grammar (rule words, modifiers, clear) is supported. */
cfg = config_create();
positional_count = 0;
char* grammar_argv[] = {"fastsync",
"--filter=hide *.tmp",
"--filter=show *.txt",
"--filter=protect *.bak",
"--filter=risk *.o",
"--filter=-s foo",
"--filter=-p bar",
"--filter=-! *.o",
"--filter=dir-merge .rules",
"--filter=!",
"/src",
"/dst"};
EXPECT_EQ_INT(parse_args(cfg, 11, grammar_argv, positional_args, &positional_count), 0);
config_delete(cfg);
/* Genuinely malformed rules are still rejected. */
static const char* const malformed[] = {
"merge", /* merge requires a filename */
"dir-merge", /* dir-merge requires a filename */
"clear extra", /* clear takes no pattern */
"no-such-rule x", /* unknown rule word */
};
for (size_t i = 0; i < sizeof(unsupported) / sizeof(unsupported[0]); i++) {
for (size_t i = 0; i < sizeof(malformed) / sizeof(malformed[0]); i++) {
cfg = config_create();
positional_count = 0;
char* rule_argv[] = {"fastsync", "--filter", (char*)unsupported[i], "/src", "/dst"};
char* rule_argv[] = {"fastsync", "--filter", (char*)malformed[i], "/src", "/dst"};
EXPECT_EQ_INT(parse_args(cfg, 5, rule_argv, positional_args, &positional_count), -1);
config_delete(cfg);
}
@@ -3087,6 +3486,27 @@ static void test_parse_args_chown() {
EXPECT_TRUE(cfg->chown_gid_set);
EXPECT_EQ_INT(cfg->chown_gid, IDENTITY_CURRENT);
config_delete(cfg);
/* A --chown NAME is converted to the equivalent receiver-resolved map rule
* (rsync implements --chown as --usermap=*:USER --groupmap=*:GROUP), so the
* name is carried on the wire as to_name instead of being resolved on the
* sender. A name that does not exist on the sender is accepted and left for
* the receiver to resolve (or warn about), matching rsync. */
cfg = config_create();
positional_count = 0;
char* argv5[] = {"fastsync", "--chown=no_such_user_zzz:no_such_group_zzz", "/src", "/dst"};
EXPECT_EQ_INT(parse_args(cfg, 4, argv5, positional_args, &positional_count), 0);
EXPECT_EQ_INT(cfg->usermap_count, 1);
EXPECT_NOT_NULL(cfg->usermap[0].to_name);
if (cfg->usermap[0].to_name)
EXPECT_EQ_STR(cfg->usermap[0].to_name, "no_such_user_zzz");
EXPECT_FALSE(cfg->chown_uid_set);
EXPECT_EQ_INT(cfg->groupmap_count, 1);
EXPECT_NOT_NULL(cfg->groupmap[0].to_name);
if (cfg->groupmap[0].to_name)
EXPECT_EQ_STR(cfg->groupmap[0].to_name, "no_such_group_zzz");
EXPECT_FALSE(cfg->chown_gid_set);
config_delete(cfg);
}
/* --copy-as=USER[:GROUP] (P7 Wave E): resolve the user/group against the local
@@ -3161,7 +3581,6 @@ static void test_parse_args_rejects_malformed_identity() {
{"--groupmap", "@1"},
{"--groupmap", "no_such_group_qqq:x"},
{"--chown", "a:b:c"},
{"--chown", "no_such_user_zzz:"},
{"--copy-as", ""},
{"--copy-as", ":"},
{"--copy-as", "a:b:c"},
@@ -3666,6 +4085,113 @@ static void test_parse_args_unsigned_options_reject_sign() {
config_delete(cfg);
}
/* --bwlimit must parse with rsync 3.4.1's units and quantization: a bare value
* is KiB/s, K/M/G/T/P are binary multipliers, KB/MB are decimal, KiB/MiB are
* binary, decimals are rounded to whole KiB like rsync's (size + 512) / 1024,
* and 0 (or an empty value) means "no limit". */
static void test_parse_args_bwlimit_rsync_units() {
struct {
const char* value;
unsigned long long expected; /* bytes/sec */
int ok;
} cases[] = {
{"100", 100ULL * 1024, 1},
{"0", 0, 1},
{"", 0, 1},
{"1.5", 2ULL * 1024, 1},
{"100K", 100ULL * 1024, 1},
{"100KiB", 100ULL * 1024, 1},
{"100KB", (100000ULL + 512) / 1024 * 1024, 1},
{"1M", 1024ULL * 1024, 1},
{"1MB", (1000000ULL + 512) / 1024 * 1024, 1},
{"1.5m", 1536ULL * 1024, 1},
{"1G", 1024ULL * 1024 * 1024, 1},
{"1000B", (1000ULL + 512) / 1024 * 1024, 1},
{"100B", 0, 0}, /* below the 512-byte floor (not 0) */
{"0.4", 0, 0}, /* 409 bytes, below the floor */
{"511", 511ULL * 1024, 1},
{"-1", 0, 0},
{"abc", 0, 0},
{"1x", 0, 0},
};
for (size_t i = 0; i < sizeof(cases) / sizeof(cases[0]); i++) {
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
int positional_args[2];
int positional_count = 0;
char option[32];
snprintf(option, sizeof(option), "--bwlimit=%s", cases[i].value);
char* argv[] = {"fastsync", option, "/src", "/dst"};
int rc = parse_args(cfg, 4, argv, positional_args, &positional_count);
if (cases[i].ok) {
EXPECT_EQ_INT(rc, 0);
EXPECT_TRUE(io_get_bwlimit() == cases[i].expected);
} else {
EXPECT_EQ_INT(rc, -1);
}
config_delete(cfg);
}
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
int positional_args[2];
int positional_count = 0;
char* argv[] = {"fastsync", "--bwlimit", "512", "/src", "/dst"};
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), 0);
EXPECT_TRUE(io_get_bwlimit() == 512ULL * 1024);
config_delete(cfg);
io_set_bwlimit(0);
}
/* Huge/malformed --bwlimit values must be rejected (not accepted or UB) and
* oversized-but-representable ones must still be accepted: the scaling used to
* be done with an unchecked signed double->long long cast, which is undefined
* when the product leaves long long's range. */
static void test_parse_args_bwlimit_huge_and_boundary() {
struct {
const char* value;
unsigned long long expected; /* bytes/sec, ignored when !ok */
int ok;
} cases[] = {
/* Malformed / non-numeric prefixes. */
{"1e300", 0, 0},
{"99999999999999999999999999e3", 0, 0},
/* Products that exceed LLONG_MAX at every suffix. */
{"99999999999999999999999999", 0, 0},
{"99999999999999999999999999K", 0, 0},
{"100000000000000000000P", 0, 0},
{"99999999999999999999999999999999999999999999999999B", 0, 0},
/* `strtod` overflow to +inf must be caught by the isfinite() guard. */
{"9999999999999999999999999999999999999999999999999999999999999999999999"
"9999999999999999999999999999999999999999999999999999999999999999999999"
"99999999999999999999999999999999999999999999999999999999999999999999999",
0, 0},
/* 2^52 KiB/s: the largest power-of-two scaling that still fits. */
{"4503599627370496", 4611686018427387904ULL, 1},
/* The +/-1 suffix forms accepted by rsync. */
{"1+1", 1024ULL, 1},
{"1-1", 1024ULL, 1},
};
for (size_t i = 0; i < sizeof(cases) / sizeof(cases[0]); i++) {
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
int positional_args[2];
int positional_count = 0;
char option[1024];
snprintf(option, sizeof(option), "--bwlimit=%s", cases[i].value);
char* argv[] = {"fastsync", option, "/src", "/dst"};
int rc = parse_args(cfg, 4, argv, positional_args, &positional_count);
if (cases[i].ok) {
EXPECT_EQ_INT(rc, 0);
EXPECT_TRUE(io_get_bwlimit() == cases[i].expected);
} else {
EXPECT_EQ_INT(rc, -1);
}
config_delete(cfg);
io_set_bwlimit(0);
}
}
/* --dry-run must not emit a batch file, so it is rejected alongside
* --read-batch/--only-write-batch. */
static void test_validate_config_dry_run_rejects_write_batch() {
@@ -4190,6 +4716,26 @@ static void test_parse_args_rejects_unsupported_short() {
}
}
/* The --ignore-errors deletion gate, unit-tested without a privileged source
* directory: a clean scan always deletes; an I/O error suppresses deletion
* unless --ignore-errors is set. (The end-to-end mode-000 differential lives in
* the setpriv integration test; this pins the decision itself in the PR gate.) */
static void test_ignore_errors_allows_delete(void) {
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
EXPECT_TRUE(ignore_errors_allows_delete(cfg, false));
EXPECT_FALSE(ignore_errors_allows_delete(cfg, true));
cfg->ignore_errors = true;
EXPECT_TRUE(ignore_errors_allows_delete(cfg, false));
EXPECT_TRUE(ignore_errors_allows_delete(cfg, true));
/* A NULL config cannot opt into --ignore-errors. */
EXPECT_TRUE(ignore_errors_allows_delete(NULL, false));
EXPECT_FALSE(ignore_errors_allows_delete(NULL, true));
config_delete(cfg);
}
void test_client_cli() {
test_validate_config_required_paths();
test_parse_args_numeric_ids();
@@ -4255,6 +4801,8 @@ void test_client_cli() {
test_parse_args_debug_flags();
test_parse_args_debug_help();
test_parse_args_debug_flags_validation();
test_parse_args_rsync_flag_vocabulary_accepted();
test_parse_args_debug_info_levels();
test_parse_args_modify_window();
test_parse_args_rejects_invalid_modify_window();
test_parse_args_skip_compress();
@@ -4267,6 +4815,7 @@ void test_client_cli() {
test_parse_args_relative_no_implied_mkpath();
test_parse_args_delete_during_alias();
test_parse_args_delete_timing_flags();
test_parse_args_delete_default_timing_and_commit();
test_parse_args_delete_timing_conflict_rejected();
test_parse_args_delete_timing_without_delete_rejected();
test_parse_args_rejects_unimplemented_options();
@@ -4278,6 +4827,7 @@ void test_client_cli() {
test_parse_args_info_flags();
test_parse_args_info_verbose_order();
test_parse_args_rejects_invalid_info_flag();
test_parse_args_info_name_and_help();
test_parse_args_archive();
test_parse_args_preserve_attributes_are_independent();
test_parse_args_preserve_long_form();
@@ -4321,6 +4871,9 @@ void test_client_cli() {
test_parse_args_compression_alias_equals();
test_parse_args_rejects_invalid_compression_level_equals();
test_parse_args_rejects_invalid_compression_choice();
test_parse_args_per_codec_compression_level_defaults();
test_parse_args_compression_env_list();
test_parse_args_checksum_env_list();
test_parse_args_table_equals_size_options();
test_parse_args_table_equals_string_and_int_options();
test_parse_args_missing_argument_diagnostic();
@@ -4339,6 +4892,7 @@ void test_client_cli() {
test_parse_args_checksum_choice_equals_forms();
test_parse_args_checksum_choice_rejects_unsupported();
test_parse_args_checksum_choice_new_algos();
test_parse_args_checksum_none_with_checksum_rejected();
test_parse_args_checksum_implies_incremental_only();
test_parse_args_no_whole_file();
test_parse_args_timeout_zero_and_no_forms();
@@ -4353,6 +4907,7 @@ void test_client_cli() {
test_parse_args_filter_rules();
test_parse_args_from0_cvs_filter_file_flags();
test_parse_args_basis_dirs();
test_parse_args_verify_basis();
test_parse_args_basis_invalid_paths();
test_validate_config_basis_rejects_chunk_serialization();
test_parse_args_delete_policy_flags();
@@ -4371,6 +4926,8 @@ void test_client_cli() {
test_parse_args_password_file();
test_parse_args_pattern_file_oversized_rejected();
test_parse_args_unsigned_options_reject_sign();
test_parse_args_bwlimit_rsync_units();
test_parse_args_bwlimit_huge_and_boundary();
test_validate_config_dry_run_rejects_write_batch();
test_parse_args_short_clustering();
test_parse_args_attached_short_values();
@@ -4379,4 +4936,5 @@ void test_client_cli() {
test_parse_args_noop_does_not_consume_argv();
test_parse_args_backup_copy_links_shorts();
test_parse_args_rejects_unsupported_short();
test_ignore_errors_allows_delete();
}
+148 -6
View File
@@ -4,6 +4,7 @@
#include "data.h"
#include "file.h"
#include "utils.h"
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <sys/wait.h>
@@ -157,20 +158,22 @@ static void test_chunk_compress_decompress_roundtrip() {
/* Build a zstd frame whose header omits the content size (the content size
* flag is cleared), which ZSTD_getFrameContentSize reports as
* ZSTD_CONTENTSIZE_UNKNOWN. */
* ZSTD_CONTENTSIZE_UNKNOWN. The frame carries the codec-id prefix the
* decompressor dispatches on. */
static Data* make_unknown_size_frame(const void* src, size_t len) {
ZSTD_CCtx* cctx = ZSTD_createCCtx();
if (!cctx)
return NULL;
ZSTD_CCtx_setParameter(cctx, ZSTD_c_contentSizeFlag, 0);
size_t cap = ZSTD_compressBound(len);
Data* out = data_create_empty(cap);
Data* out = data_create_empty(cap + 1);
if (!out) {
ZSTD_freeCCtx(cctx);
return NULL;
}
((uint8_t*)out->data)[0] = (uint8_t)COMPRESSION_ALGO_ZSTD;
ZSTD_inBuffer in = {src, len, 0};
ZSTD_outBuffer ob = {out->data, cap, 0};
ZSTD_outBuffer ob = {(uint8_t*)out->data + 1, cap, 0};
size_t ret;
do {
ret = ZSTD_compressStream2(cctx, &ob, &in, ZSTD_e_end);
@@ -180,7 +183,7 @@ static Data* make_unknown_size_frame(const void* src, size_t len) {
return NULL;
}
} while (ret > 0);
out->size = ob.pos;
out->size = ob.pos + 1;
ZSTD_freeCCtx(cctx);
return out;
}
@@ -199,8 +202,9 @@ static void test_data_decompress_unknown_size_frame() {
Data* frame = make_unknown_size_frame(buf, len);
free(buf);
EXPECT_NOT_NULL(frame);
/* Guard the premise of the test: the frame really has no stored size. */
EXPECT_EQ_INT((int)ZSTD_getFrameContentSize(frame->data, frame->size),
/* Guard the premise of the test: the frame (after the codec byte) really has
* no stored size. */
EXPECT_EQ_INT((int)ZSTD_getFrameContentSize((uint8_t*)frame->data + 1, frame->size - 1),
(int)ZSTD_CONTENTSIZE_UNKNOWN);
Data* decompressed = data_decompress(frame);
@@ -326,6 +330,139 @@ static void test_data_decompress_truncated_frame_fails() {
data_destroy(input);
}
/* Every codec must round-trip byte-exactly through the self-describing frame,
* including the empty and a highly compressible large payload. */
static void codec_roundtrip(CompressionAlgo algo) {
const char* samples[] = {
"",
"Hello, World! This is test data for compression round-trip!",
"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
};
for (size_t s = 0; s < sizeof(samples) / sizeof(samples[0]); s++) {
size_t len = strlen(samples[s]);
Data* original = data_create_empty(len);
EXPECT_NOT_NULL(original);
if (len > 0)
memcpy(original->data, samples[s], len);
original->size = len;
Data* compressed = data_compress_codec(original, algo, 3, 0);
EXPECT_NOT_NULL(compressed);
EXPECT_EQ_INT((int)((uint8_t*)compressed->data)[0], (int)algo);
Data* decompressed = data_decompress(compressed);
EXPECT_NOT_NULL(decompressed);
EXPECT_EQ_INT((int)decompressed->size, (int)len);
EXPECT_EQ_INT(memcmp(decompressed->data, original->data, len), 0);
data_destroy(decompressed);
data_destroy(compressed);
data_destroy(original);
}
}
static void test_codec_roundtrips() {
codec_roundtrip(COMPRESSION_ALGO_NONE);
codec_roundtrip(COMPRESSION_ALGO_ZSTD);
codec_roundtrip(COMPRESSION_ALGO_LZ4);
codec_roundtrip(COMPRESSION_ALGO_ZLIB);
codec_roundtrip(COMPRESSION_ALGO_ZLIBX);
}
static void test_codec_name_mapping() {
EXPECT_EQ_INT(compression_algo_from_name("zstd"), (int)COMPRESSION_ALGO_ZSTD);
EXPECT_EQ_INT(compression_algo_from_name("ZSTD"), (int)COMPRESSION_ALGO_ZSTD);
EXPECT_EQ_INT(compression_algo_from_name("lz4"), (int)COMPRESSION_ALGO_LZ4);
EXPECT_EQ_INT(compression_algo_from_name("zlib"), (int)COMPRESSION_ALGO_ZLIB);
EXPECT_EQ_INT(compression_algo_from_name("zlibx"), (int)COMPRESSION_ALGO_ZLIBX);
EXPECT_EQ_INT(compression_algo_from_name("none"), (int)COMPRESSION_ALGO_NONE);
EXPECT_TRUE(compression_algo_from_name("bogus") < 0);
EXPECT_TRUE(compression_algo_from_name(NULL) < 0);
EXPECT_TRUE(compression_algo_valid((int)COMPRESSION_ALGO_LZ4));
EXPECT_TRUE(compression_algo_valid((int)COMPRESSION_ALGO_ZLIB));
EXPECT_TRUE(compression_algo_valid((int)COMPRESSION_ALGO_ZLIBX));
EXPECT_FALSE(compression_algo_valid(99));
EXPECT_EQ_STR(compression_algo_name(COMPRESSION_ALGO_ZSTD), "zstd");
EXPECT_EQ_STR(compression_algo_name(COMPRESSION_ALGO_LZ4), "lz4");
EXPECT_EQ_STR(compression_algo_name(COMPRESSION_ALGO_ZLIB), "zlib");
EXPECT_EQ_STR(compression_algo_name(COMPRESSION_ALGO_ZLIBX), "zlibx");
EXPECT_EQ_STR(compression_algo_name(COMPRESSION_ALGO_NONE), "none");
/* rsync 3.4.1 auto-negotiates zstd first. */
EXPECT_EQ_INT((int)compression_negotiate_default(), (int)COMPRESSION_ALGO_ZSTD);
EXPECT_FALSE(compression_algo_enabled(COMPRESSION_ALGO_NONE));
EXPECT_TRUE(compression_algo_enabled(COMPRESSION_ALGO_ZSTD));
}
/* rsync 3.4.1's per-codec default levels and its clamping ranges. */
static void test_codec_level_defaults_and_clamp() {
EXPECT_EQ_INT(compression_default_level(COMPRESSION_ALGO_ZSTD), ZSTD_CLEVEL_DEFAULT);
EXPECT_EQ_INT(compression_default_level(COMPRESSION_ALGO_ZSTD), 3);
EXPECT_EQ_INT(compression_default_level(COMPRESSION_ALGO_ZLIB), 6);
EXPECT_EQ_INT(compression_default_level(COMPRESSION_ALGO_ZLIBX), 6);
/* lz4 has no tunable level; a positive placeholder keeps the codec engaged. */
EXPECT_TRUE(compression_default_level(COMPRESSION_ALGO_LZ4) > 0);
EXPECT_EQ_INT(compression_default_level(COMPRESSION_ALGO_NONE), 0);
EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_ZSTD, 1), 1);
EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_ZSTD, 22), 22);
EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_ZSTD, 23), 22);
EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_ZSTD, 0), 1);
EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_ZLIB, 15), 9);
EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_ZLIBX, 15), 9);
EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_ZLIB, 1), 1);
EXPECT_TRUE(compression_clamp_level(COMPRESSION_ALGO_LZ4, 20) > 0);
EXPECT_EQ_INT(compression_clamp_level(COMPRESSION_ALGO_NONE, 20), 0);
}
/* RSYNC_COMPRESS_LIST precedence, syntax and fallback. */
static void test_codec_choice_env_list() {
unsetenv("RSYNC_COMPRESS_LIST");
EXPECT_EQ_INT(compression_choice_resolve(), (int)COMPRESSION_ALGO_ZSTD);
EXPECT_EQ_INT((int)compression_negotiate_default(), (int)COMPRESSION_ALGO_ZSTD);
/* Unknown entries are skipped; the first supported wins. */
setenv("RSYNC_COMPRESS_LIST", "bogus zlib lz4", 1);
EXPECT_EQ_INT(compression_choice_resolve(), (int)COMPRESSION_ALGO_ZLIB);
/* Case-insensitive. */
setenv("RSYNC_COMPRESS_LIST", "ZSTD", 1);
EXPECT_EQ_INT(compression_choice_resolve(), (int)COMPRESSION_ALGO_ZSTD);
/* Whitespace-separated; the client half ends at '&'. */
setenv("RSYNC_COMPRESS_LIST", "lz4 zlib & zstd", 1);
EXPECT_EQ_INT(compression_choice_resolve(), (int)COMPRESSION_ALGO_LZ4);
/* Blank falls back to the compiled-in order. */
setenv("RSYNC_COMPRESS_LIST", " ", 1);
EXPECT_EQ_INT(compression_choice_resolve(), (int)COMPRESSION_ALGO_ZSTD);
/* rsync's syntax has no comma/colon separator: this is one unknown name. */
setenv("RSYNC_COMPRESS_LIST", "bogus,lz4", 1);
EXPECT_EQ_INT(compression_choice_resolve(), -1);
unsetenv("RSYNC_COMPRESS_LIST");
}
/* The process-global codec selects what the legacy wrappers produce. */
static void test_codec_global_selection() {
Data* original = data_create_empty(64);
EXPECT_NOT_NULL(original);
memset(original->data, 'q', 64);
original->size = 64;
compression_set_algo(COMPRESSION_ALGO_LZ4);
Data* compressed = data_compress(original, 3);
EXPECT_NOT_NULL(compressed);
EXPECT_EQ_INT((int)((uint8_t*)compressed->data)[0], (int)COMPRESSION_ALGO_LZ4);
Data* decompressed = data_decompress(compressed);
EXPECT_NOT_NULL(decompressed);
EXPECT_TRUE(memcmp(decompressed->data, original->data, 64) == 0);
data_destroy(decompressed);
data_destroy(compressed);
/* Restore the default so later tests are unaffected. */
compression_set_algo(COMPRESSION_ALGO_ZSTD);
data_destroy(original);
}
void test_compression() {
test_data_compress_decompress_roundtrip();
test_data_compress_decompress_large();
@@ -335,4 +472,9 @@ void test_compression() {
test_data_compress_with_threads_roundtrip();
test_data_compress_reused_contexts_multithreaded();
test_chunk_compress_decompress_roundtrip();
test_codec_roundtrips();
test_codec_name_mapping();
test_codec_level_defaults_and_clamp();
test_codec_choice_env_list();
test_codec_global_selection();
}
+183 -11
View File
@@ -1,6 +1,7 @@
#include "test_config.h"
#include "config.h"
#include "delta.h"
#include "filter.h"
#include "identity.h"
#include "multiprocessing.h"
#include "protocol.h"
@@ -852,13 +853,15 @@ static void test_config_delete_timing_early_helper() {
cfg->use_delete = true;
cfg->delete_before = true;
EXPECT_TRUE(config_delete_timing_early(cfg));
EXPECT_FALSE(config_delete_timing_per_dir(cfg));
EXPECT_TRUE(config_has_valid_delete_timing(cfg));
config_delete(cfg);
cfg = config_create();
cfg->use_delete = true;
cfg->delete_during = true;
EXPECT_TRUE(config_delete_timing_early(cfg));
EXPECT_FALSE(config_delete_timing_early(cfg));
EXPECT_TRUE(config_delete_timing_per_dir(cfg));
EXPECT_TRUE(config_has_valid_delete_timing(cfg));
config_delete(cfg);
@@ -866,6 +869,7 @@ static void test_config_delete_timing_early_helper() {
cfg->use_delete = true;
cfg->delete_delay = true;
EXPECT_FALSE(config_delete_timing_early(cfg));
EXPECT_TRUE(config_delete_timing_per_dir(cfg));
EXPECT_TRUE(config_has_valid_delete_timing(cfg));
config_delete(cfg);
@@ -873,6 +877,7 @@ static void test_config_delete_timing_early_helper() {
cfg->use_delete = true;
cfg->delete_after = true;
EXPECT_FALSE(config_delete_timing_early(cfg));
EXPECT_FALSE(config_delete_timing_per_dir(cfg));
EXPECT_TRUE(config_has_valid_delete_timing(cfg));
config_delete(cfg);
@@ -1192,7 +1197,10 @@ static void test_config_basis_wire_rejects_escaping() {
c->basis_dirs = calloc(1, sizeof(BasisDest));
c->basis_dirs[0].type = BASIS_DEST_LINK;
c->basis_dirs[0].path = str_dup("/abs");
EXPECT_FALSE(roundtrip_config_ok(c));
/* An absolute basis dir is accepted (rsync parity); it is only usable when it
lies within the receiver's authorized root, which file_open_secure_parent
enforces at lookup time. */
EXPECT_TRUE(roundtrip_config_ok(c));
config_delete(c);
/* A well-formed list still round-trips even with a manually built struct. */
@@ -1225,8 +1233,13 @@ static void test_config_basis_normalization() {
/* Degenerate values that normalize away to nothing stay rejected. */
EXPECT_EQ_INT(config_basis_append(c, BASIS_DEST_LINK, "."), -1);
EXPECT_EQ_INT(config_basis_append(c, BASIS_DEST_LINK, ".."), -1);
EXPECT_EQ_INT(config_basis_append(c, BASIS_DEST_LINK, "/abs"), -1);
/* An absolute path is canonicalized (leading '/' preserved) and accepted. */
EXPECT_EQ_INT(config_basis_append(c, BASIS_DEST_LINK, "/abs"), 0);
EXPECT_EQ_STR(c->basis_dirs[c->basis_count - 1].path, "/abs");
EXPECT_EQ_INT(config_basis_append(c, BASIS_DEST_LINK, "/a//b/"), 0);
EXPECT_EQ_STR(c->basis_dirs[c->basis_count - 1].path, "/a/b");
EXPECT_EQ_INT(config_basis_append(c, BASIS_DEST_LINK, "a/../b"), -1);
EXPECT_EQ_INT(config_basis_append(c, BASIS_DEST_LINK, "/"), -1);
EXPECT_EQ_INT(config_basis_append(c, BASIS_DEST_LINK, ""), -1);
config_delete(c);
}
@@ -1356,6 +1369,61 @@ static void test_config_receive_rejects_invalid_checksum_algo() {
EXPECT_FALSE(roundtrip_config_ok(c));
config_delete(c);
}
/* The negotiated codec id and the human --compress-choice spelling must agree,
* and the id itself must be a known codec. */
static void test_config_receive_rejects_invalid_compression_algo() {
if (is_running_under_valgrind())
return;
Config* c = config_create();
EXPECT_NOT_NULL(c);
c->send_directory = str_dup("/src");
c->receive_root_directory = str_dup("/dst");
c->compression_algo = 99;
EXPECT_FALSE(roundtrip_config_ok(c));
config_delete(c);
}
static void test_config_receive_rejects_codec_mismatch() {
if (is_running_under_valgrind())
return;
Config* c = config_create();
EXPECT_NOT_NULL(c);
c->send_directory = str_dup("/src");
c->receive_root_directory = str_dup("/dst");
free(c->compress_choice);
c->compress_choice = str_dup("lz4");
c->use_compression = true;
c->compression_algo = (int)COMPRESSION_ALGO_ZSTD; /* does not match lz4 */
EXPECT_FALSE(roundtrip_config_ok(c));
config_delete(c);
}
static void test_config_receive_rejects_none_codec_with_compression() {
if (is_running_under_valgrind())
return;
Config* c = config_create();
EXPECT_NOT_NULL(c);
c->send_directory = str_dup("/src");
c->receive_root_directory = str_dup("/dst");
c->use_compression = true;
c->compression_algo = (int)COMPRESSION_ALGO_NONE;
EXPECT_FALSE(roundtrip_config_ok(c));
config_delete(c);
}
static void test_config_receive_rejects_checksum_none_with_checksum() {
if (is_running_under_valgrind())
return;
Config* c = config_create();
EXPECT_NOT_NULL(c);
c->send_directory = str_dup("/src");
c->receive_root_directory = str_dup("/dst");
c->checksum = true;
c->checksum_algo = (int)CHECKSUM_ALGO_NONE;
EXPECT_FALSE(roundtrip_config_ok(c));
config_delete(c);
}
/* The identity-mapping fields (--numeric-ids / --usermap / --groupmap /
--chown) cross the config wire unchanged: the receiver needs them to apply
ownership with the same policy the client requested. */
@@ -2052,6 +2120,48 @@ static void test_config_receive_rejects_oversized_string_budget() {
config_delete(over_bytes);
}
/* Pre-auth bounds for the receiver-side filter rule block (protocol 2.28.0).
A peer may send `protect`/`risk` rules; the receiver must reject an over-cap
count or an over-long pattern before evaluating anything, so a crafted config
cannot drive unbounded glob work or install a rule that silently never
matches. */
static void test_config_receive_rejects_bad_protect_rules() {
if (is_running_under_valgrind())
return;
/* Over-cap rule count: one more than MAX_FILTER_RULES rules. */
Config* c = config_create();
EXPECT_NOT_NULL(c);
c->send_directory = str_dup("/src");
c->receive_root_directory = str_dup("/dst");
c->filters = array_list_create(free);
EXPECT_NOT_NULL(c->filters);
for (int i = 0; i <= MAX_FILTER_RULES; i++)
EXPECT_TRUE(array_list_add(c->filters, str_dup("- *.tmp")));
EXPECT_TRUE(roundtrip_config_rejected(c));
config_delete(c);
/* A pattern longer than the receiver's evaluation bound is rejected by the
sender (mirroring the receiver's guard) instead of being sent as an inert
rule. */
c = config_create();
EXPECT_NOT_NULL(c);
c->send_directory = str_dup("/src");
c->receive_root_directory = str_dup("/dst");
c->filters = array_list_create(free);
EXPECT_NOT_NULL(c->filters);
size_t big = MAX_PROTECT_PATTERN_LEN + 1;
char* long_rule = malloc(big + 3);
EXPECT_NOT_NULL(long_rule);
long_rule[0] = '-';
long_rule[1] = ' ';
memset(long_rule + 2, 'x', big);
long_rule[big + 2] = '\0';
EXPECT_TRUE(array_list_add(c->filters, long_rule));
EXPECT_TRUE(roundtrip_config_rejected(c));
config_delete(c);
}
/* identity_copy_as_refused() is the pure, pre-snapshot refusal predicate: a
--copy-as is refused when the receiver is not root OR the effective super
mode is OFF (an operator veto), and never when --copy-as is unset. */
@@ -2523,6 +2633,46 @@ static bool basis_equal(const Config* a, const Config* b) {
return true;
}
/* The receiver reconstructs its delete-protection list from the sender's
* compiled base rules, so compare the received list against a fresh
* filter_base_build() of the sender's raw --filter texts. */
static bool filter_rules_equal(const Config* a, const FilterRuleList* got) {
int count = a->filters ? a->filters->size : 0;
const char** texts = NULL;
if (count > 0) {
texts = calloc((size_t)count, sizeof(char*));
if (!texts)
return false;
for (int i = 0; i < count; i++)
texts[i] = (const char*)a->filters->items[i];
}
char err[160];
FilterRuleList* expected =
filter_base_build(texts, count, a->cvs_exclude, a->delete_excluded, err, sizeof(err));
free(texts);
if (!expected)
return false;
bool equal = true;
int want = expected->count;
int have = got ? got->count : 0;
if (want != have) {
equal = false;
} else {
for (int i = 0; i < want; i++) {
const FilterRule* x = expected->items[i];
const FilterRule* y = got->items[i];
if (x->action != y->action || x->sides != y->sides || x->anchored != y->anchored ||
x->dir_only != y->dir_only || x->negate != y->negate ||
!str_opt_equal(x->owner, y->owner) || !str_opt_equal(x->pattern, y->pattern)) {
equal = false;
break;
}
}
}
filter_rule_list_free(expected);
return equal;
}
#define CONFIG_CMP_BOOL(a, b, name) ((a)->name == (b)->name)
#define CONFIG_CMP_INT(a, b, name) ((a)->name == (b)->name)
#define CONFIG_CMP_RAW(a, b, name) ((a)->name == (b)->name)
@@ -2537,6 +2687,7 @@ static bool basis_equal(const Config* a, const Config* b) {
#define CONFIG_CMP_STR_MODULE(a, b, name) str_opt_equal((a)->name, (b)->name)
#define CONFIG_CMP_STR_REDACTED_AUTH(a, b, name) str_opt_equal((a)->name, (b)->name)
#define CONFIG_CMP_INT_CHECKSUM_ALGO(a, b, name) ((a)->name == (b)->name)
#define CONFIG_CMP_INT_COMPRESSION_ALGO(a, b, name) ((a)->name == (b)->name)
#define CONFIG_CMP_SUPERMODE(a, b, name) ((a)->name == (b)->name)
#define CONFIG_CMP_INT_IDENTITY(a, b, name) ((a)->name == (b)->name)
#define CONFIG_CMP_INT_SKIPCOUNT(a, b, name) ((a)->name == (b)->name)
@@ -2547,6 +2698,7 @@ static bool basis_equal(const Config* a, const Config* b) {
#define CONFIG_CMP_COPY_AS_ID(a, b, name) (!(a)->copy_as_set || (a)->name == (b)->name)
#define CONFIG_CMP_BLOCK_SKIP_SUFFIXES(a, b, name) skip_suffixes_equal((a), (b))
#define CONFIG_CMP_BLOCK_BASIS(a, b, name) basis_equal((a), (b))
#define CONFIG_CMP_BLOCK_PROTECT_RULES(a, b, name) filter_rules_equal((a), (b)->name)
#define CONFIG_CMP_BLOCK_IDMAP(a, b, name) \
idmap_equal((a)->name, (a)->name##_count, (b)->name, (b)->name##_count)
@@ -2716,6 +2868,7 @@ static void golden_config_populate(Config* c) {
c->skip_compress_suffixes[1] = str_dup(".xz");
EXPECT_EQ_INT(config_basis_append(c, BASIS_DEST_COMPARE, "compare"), 0);
EXPECT_EQ_INT(config_basis_append(c, BASIS_DEST_LINK, "link"), 0);
c->verify_basis = true;
c->fuzzy = true;
c->checksum_algo = CHECKSUM_ALGO_MD5;
c->checksum_seed = 0x1122334455667788ULL;
@@ -2761,16 +2914,29 @@ static void golden_config_populate(Config* c) {
c->copy_as_set = true;
c->copy_as_uid = 111;
c->copy_as_gid = 222;
/* Compile-through delete-protection rules (protocol 2.28.0). The golden
* sender serializes its compiled base rules, so populate a diverse set that
* exercises both sides, negate, anchoring and dir-only. */
c->filters = array_list_create(free);
array_list_add(c->filters, str_dup("P *.log"));
array_list_add(c->filters, str_dup("+r **/*.txt"));
array_list_add(c->filters, str_dup("H,!secret"));
array_list_add(c->filters, str_dup("- /sub/dir/"));
}
/* The pinned golden frame (protocol 2.23.0). The values below are the only
/* The pinned golden frame (protocol 2.28.0). The values below are the only
* thing that ties the generated table to the historical wire format; update
* them ONLY with a PROTOCOL_VERSION bump and a documented reason. The 2.23.0
* rsync-parity wave changes the config-frame layout (map-entry range + TO name,
* one report_dest_info bool, and other wire changes landing in this version);
* the byte-exact values are recomputed for the merged layout. */
#define GOLDEN_WIRE_LEN 697
#define GOLDEN_WIRE_HASH 7835017034643051109ULL
* them ONLY with a PROTOCOL_VERSION bump and a documented reason. The 2.24.0
* delete-plan wave changed only the version string; 2.25.0 appended the
* report_stats bool, 2.26.0 appended the compression_algo int, 2.27.0 appended
* the report_deletes bool, and 2.28.0 changed only the version string and
* appended the receiver-side delete-protection rule block (the STATUS_STATS
* body also grew, but that is not part of this frame). Track 5a appends the
* FastSync-only verify_basis bool to the basis block WITHOUT a version bump
* (project decision), so the frame grew by one int to 886 bytes. The
* byte-exact values are recomputed for the merged layout. */
#define GOLDEN_WIRE_LEN 886
#define GOLDEN_WIRE_HASH 5809509022716816757ULL
static unsigned long long fnv1a_64(const unsigned char* buf, size_t len) {
unsigned long long h = 1469598103934665603ULL;
@@ -2852,7 +3018,7 @@ static unsigned long long capture_wire_hash(const Config* cfg, size_t* out_len)
return h;
}
/* Byte-for-byte wire compatibility guard (protocol 2.23.0). The expected hash
/* Byte-for-byte wire compatibility guard (protocol 2.28.0). The expected hash
* pins the pre-X-macro byte stream; the refactor MUST NOT change it. */
static void test_config_wire_golden() {
if (is_running_under_valgrind())
@@ -2919,6 +3085,7 @@ static void test_config_wire_golden_receive() {
recv->groupmap[0].to_name != NULL && strcmp(recv->groupmap[0].to_name, "root") == 0;
ok = ok && recv->basis_count == 2 && recv->basis_dirs[0].type == BASIS_DEST_COMPARE &&
recv->basis_dirs[1].type == BASIS_DEST_LINK;
ok = ok && recv->verify_basis;
ok = ok && recv->module != NULL && strcmp(recv->module, "goldenmod") == 0;
ok = ok && recv->copy_as_set && recv->copy_as_uid == 111 && recv->copy_as_gid == 222;
}
@@ -3147,6 +3314,10 @@ void test_config() {
test_config_basis_normalization();
test_config_checksum_options_wire_roundtrip();
test_config_receive_rejects_invalid_checksum_algo();
test_config_receive_rejects_invalid_compression_algo();
test_config_receive_rejects_codec_mismatch();
test_config_receive_rejects_none_codec_with_compression();
test_config_receive_rejects_checksum_none_with_checksum();
test_config_identity_wire_roundtrip();
test_config_receive_rejects_invalid_identity();
test_config_metadata_times_wire_roundtrip();
@@ -3175,6 +3346,7 @@ void test_config() {
test_config_wire_golden_receive();
test_config_wire_receive_bounds();
test_config_receive_rejects_overcap_counts();
test_config_receive_rejects_bad_protect_rules();
test_config_wire_roundtrip_all_fields();
test_config_preserve_attribute_wire_roundtrip();
}
+274
View File
@@ -0,0 +1,274 @@
#include "test_delete_plan.h"
#include "charset.h"
#include "config.h"
#include "delete_plan.h"
#include "protocol.h"
#include "test_utils.h"
#include "utils.h"
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/socket.h>
#include <sys/stat.h>
#include <unistd.h>
/* Send one STATUS_DELETE_PLAN body (the leading status is consumed by the
* caller/receiver entry point) describing `dir` with no kept children. */
static void send_plan_frame(int fd, const char* dir) {
EXPECT_TRUE(send_int(fd, 0)); /* has_config */
EXPECT_TRUE(send_int(fd, 1)); /* apply: a real plan */
EXPECT_TRUE(send_wire_str(fd, dir));
EXPECT_TRUE(send_int(fd, 0)); /* kept child dirs */
EXPECT_TRUE(send_int(fd, 0)); /* kept child files */
}
/* --delete-delay: a directory snapshotted into the plan that is refilled before
* the commit is re-scanned and removed recursively (rsync parity). Regression
* for the old single-unlink ENOTEMPTY path that left the directory behind. */
static void test_delete_delay_refilled_dir_removed_recursively(void) {
char root[] = "/tmp/fastsync_dp_refill_XXXXXX";
EXPECT_TRUE(mkdtemp(root) != NULL);
char extra[1024];
snprintf(extra, sizeof(extra), "%s/extra", root);
EXPECT_EQ_INT(mkdir(extra, 0700), 0);
Config* config = config_create();
EXPECT_NOT_NULL(config);
config->receive_root_directory = str_dup(root);
config->use_delete = true;
config->delete_delay = true;
int p[2];
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
DeletePlanSession* session = delete_plan_session_create(config);
EXPECT_NOT_NULL(session);
send_plan_frame(p[1], ".");
EXPECT_EQ_INT(delete_plan_session_receive(session, config, p[0]), 0);
/* The empty extra directory was snapshotted, not removed yet. */
EXPECT_EQ_INT((int)delete_plan_session_deleted(session), 0);
/* Refill the directory while the deferred commit is pending. */
char refill[1200];
snprintf(refill, sizeof(refill), "%s/new.txt", extra);
int fd = open(refill, O_WRONLY | O_CREAT | O_TRUNC, 0600);
EXPECT_TRUE(fd >= 0);
close(fd);
EXPECT_EQ_INT(delete_plan_session_commit(session, config), DELETE_COMMIT_OK);
/* The late content and the directory itself are both removed. */
EXPECT_EQ_INT((int)delete_plan_session_deleted(session), 2);
struct stat st;
EXPECT_TRUE(lstat(extra, &st) != 0);
delete_plan_session_destroy(session);
close(p[0]);
close(p[1]);
rmdir(root);
config_delete(config);
}
/* The complement: a deferred regular extra that DOES get removed is counted. */
static void test_delete_delay_removed_file_counted(void) {
char root[] = "/tmp/fastsync_dp_file_XXXXXX";
EXPECT_TRUE(mkdtemp(root) != NULL);
char extra[1024];
snprintf(extra, sizeof(extra), "%s/extra.txt", root);
int fd = open(extra, O_WRONLY | O_CREAT | O_TRUNC, 0600);
EXPECT_TRUE(fd >= 0);
close(fd);
Config* config = config_create();
EXPECT_NOT_NULL(config);
config->receive_root_directory = str_dup(root);
config->use_delete = true;
config->delete_delay = true;
int p[2];
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
DeletePlanSession* session = delete_plan_session_create(config);
EXPECT_NOT_NULL(session);
send_plan_frame(p[1], ".");
EXPECT_EQ_INT(delete_plan_session_receive(session, config, p[0]), 0);
EXPECT_EQ_INT((int)delete_plan_session_deleted(session), 0);
EXPECT_EQ_INT(delete_plan_session_commit(session, config), DELETE_COMMIT_OK);
EXPECT_EQ_INT((int)delete_plan_session_deleted(session), 1);
EXPECT_TRUE(lstat(extra, &(struct stat){0}) != 0);
delete_plan_session_destroy(session);
close(p[0]);
close(p[1]);
rmdir(root);
config_delete(config);
}
/* --max-delete is charged on actual removals, not at plan/snapshot time: after
* receiving the plans the budget is untouched, and only the commit removes up to
* the limit. */
static void test_delete_delay_max_delete_bounds_actual(void) {
char root[] = "/tmp/fastsync_dp_max_XXXXXX";
EXPECT_TRUE(mkdtemp(root) != NULL);
for (int i = 0; i < 3; i++) {
char path[1024];
snprintf(path, sizeof(path), "%s/e%d.txt", root, i);
int fd = open(path, O_WRONLY | O_CREAT | O_TRUNC, 0600);
EXPECT_TRUE(fd >= 0);
close(fd);
}
Config* config = config_create();
EXPECT_NOT_NULL(config);
config->receive_root_directory = str_dup(root);
config->use_delete = true;
config->delete_delay = true;
config->max_delete = 1;
int p[2];
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
DeletePlanSession* session = delete_plan_session_create(config);
EXPECT_NOT_NULL(session);
send_plan_frame(p[1], ".");
EXPECT_EQ_INT(delete_plan_session_receive(session, config, p[0]), 0);
/* Nothing removed yet, so the budget is not consumed at snapshot time. */
EXPECT_FALSE(delete_plan_session_limit_reached(session));
EXPECT_EQ_INT((int)delete_plan_session_deleted(session), 0);
EXPECT_EQ_INT(delete_plan_session_commit(session, config), DELETE_COMMIT_LIMIT_REACHED);
EXPECT_TRUE(delete_plan_session_limit_reached(session));
EXPECT_EQ_INT((int)delete_plan_session_deleted(session), 1);
delete_plan_session_destroy(session);
close(p[0]);
close(p[1]);
for (int i = 0; i < 3; i++) {
char path[1024];
snprintf(path, sizeof(path), "%s/e%d.txt", root, i);
unlink(path);
}
rmdir(root);
config_delete(config);
}
/* --max-delete is charged on ACTUAL removals: the refilled directory's late
* content is removed first (consuming the single budget slot), so the directory
* itself and the later extra are skipped, matching rsync. The two plans are
* sent as separate frames for "a" then "b", so the ordering that decides which
* entry gets the budget is deterministic (unlike readdir order). */
static void test_delete_delay_actual_removal_charges_budget(void) {
char root[] = "/tmp/fastsync_dp_planbudget_XXXXXX";
EXPECT_TRUE(mkdtemp(root) != NULL);
char adir[1024], bdir[1024], xdir[1024], ydir[1024];
snprintf(adir, sizeof(adir), "%s/a", root);
snprintf(bdir, sizeof(bdir), "%s/b", root);
snprintf(xdir, sizeof(xdir), "%s/a/x", root);
snprintf(ydir, sizeof(ydir), "%s/b/y", root);
EXPECT_EQ_INT(mkdir(adir, 0700), 0);
EXPECT_EQ_INT(mkdir(bdir, 0700), 0);
EXPECT_EQ_INT(mkdir(xdir, 0700), 0);
EXPECT_EQ_INT(mkdir(ydir, 0700), 0);
Config* config = config_create();
EXPECT_NOT_NULL(config);
config->receive_root_directory = str_dup(root);
config->use_delete = true;
config->delete_delay = true;
config->max_delete = 1;
int p[2];
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
DeletePlanSession* session = delete_plan_session_create(config);
EXPECT_NOT_NULL(session);
/* Both plan snapshots are taken; neither consumes budget yet. */
send_plan_frame(p[1], "a");
EXPECT_EQ_INT(delete_plan_session_receive(session, config, p[0]), 0);
EXPECT_FALSE(delete_plan_session_limit_reached(session));
send_plan_frame(p[1], "b");
EXPECT_EQ_INT(delete_plan_session_receive(session, config, p[0]), 0);
EXPECT_FALSE(delete_plan_session_limit_reached(session));
EXPECT_EQ_INT((int)delete_plan_session_deleted(session), 0);
/* Refill a/x after its plan: the recursive commit must remove this content. */
char refill[1200];
snprintf(refill, sizeof(refill), "%s/new.txt", xdir);
int fd = open(refill, O_WRONLY | O_CREAT | O_TRUNC, 0600);
EXPECT_TRUE(fd >= 0);
close(fd);
EXPECT_EQ_INT(delete_plan_session_commit(session, config), DELETE_COMMIT_LIMIT_REACHED);
/* The one budget slot removed the late content; the two directories survive. */
EXPECT_EQ_INT((int)delete_plan_session_deleted(session), 1);
EXPECT_TRUE(delete_plan_session_limit_reached(session));
struct stat st;
EXPECT_TRUE(lstat(refill, &st) != 0);
EXPECT_EQ_INT(lstat(xdir, &st), 0);
EXPECT_EQ_INT(lstat(ydir, &st), 0);
delete_plan_session_destroy(session);
close(p[0]);
close(p[1]);
rmdir(xdir);
rmdir(ydir);
rmdir(adir);
rmdir(bdir);
rmdir(root);
config_delete(config);
}
/* Send a config-only carrier frame (apply=false): the per-run config block with
* one --delete-missing-args exact path, and no directory walk. */
static void send_config_only_frame(int fd, const char* missing_path) {
EXPECT_TRUE(send_int(fd, 1)); /* has_config */
EXPECT_TRUE(send_int(fd, 0)); /* protected prefixes */
EXPECT_TRUE(send_int(fd, 0)); /* size-skipped */
EXPECT_TRUE(send_int(fd, 1)); /* missing args */
EXPECT_TRUE(send_wire_str(fd, missing_path));
EXPECT_TRUE(send_int(fd, 0)); /* apply = false */
EXPECT_TRUE(send_wire_str(fd, "."));
EXPECT_TRUE(send_int(fd, 0));
EXPECT_TRUE(send_int(fd, 0));
}
/* The config-only carrier frame (apply=false) still applies the
* --delete-missing-args exact deletions even though it walks no directory. This
* is the fix for a --files-from list that synchronizes no directory. */
static void test_config_only_frame_applies_missing_args(void) {
char root[] = "/tmp/fastsync_dp_cfgonly_XXXXXX";
EXPECT_TRUE(mkdtemp(root) != NULL);
char gone[1024];
snprintf(gone, sizeof(gone), "%s/gone.txt", root);
int fd = open(gone, O_WRONLY | O_CREAT | O_TRUNC, 0600);
EXPECT_TRUE(fd >= 0);
close(fd);
Config* config = config_create();
EXPECT_NOT_NULL(config);
config->receive_root_directory = str_dup(root);
config->delete_missing_args = true;
int p[2];
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
DeletePlanSession* session = delete_plan_session_create(config);
EXPECT_NOT_NULL(session);
send_config_only_frame(p[1], "gone.txt");
EXPECT_EQ_INT(delete_plan_session_receive(session, config, p[0]), 0);
EXPECT_EQ_INT((int)delete_plan_session_deleted(session), 1);
EXPECT_TRUE(lstat(gone, &(struct stat){0}) != 0);
delete_plan_session_destroy(session);
close(p[0]);
close(p[1]);
rmdir(root);
config_delete(config);
}
void test_delete_plan(void) {
test_delete_delay_refilled_dir_removed_recursively();
test_delete_delay_removed_file_counted();
test_delete_delay_max_delete_bounds_actual();
test_delete_delay_actual_removal_charges_budget();
test_config_only_frame_applies_missing_args();
}
+6
View File
@@ -0,0 +1,6 @@
#ifndef TEST_DELETE_PLAN_H
#define TEST_DELETE_PLAN_H
void test_delete_plan(void);
#endif
+252
View File
@@ -6,6 +6,7 @@
#include "file_receive.h"
#include "data.h"
#include "config.h"
#include "charset.h"
#include "utils.h"
#include "protocol.h"
#include "test_utils.h"
@@ -1807,6 +1808,159 @@ static void test_receive_incremental_check_empty_path() {
config_delete(cfg);
}
/* Pure policy helpers behind the basis quick-check / --verify-basis decision. */
static void test_file_basis_quick_match_decision() {
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
EXPECT_FALSE(file_basis_content_required(cfg));
cfg->verify_basis = true;
EXPECT_TRUE(file_basis_content_required(cfg));
cfg->verify_basis = false;
struct stat st;
memset(&st, 0, sizeof(st));
st.st_mtime = 1500000000;
#ifdef __linux__
st.st_mtim.tv_nsec = 500;
#endif
/* Equal size is required by the caller; this leg is the mtime / --size-only
rule. Equal mtime matches, a different mtime misses by default. */
EXPECT_TRUE(file_basis_quick_match(cfg, &st, 1500000000, 500));
EXPECT_FALSE(file_basis_quick_match(cfg, &st, 1500000001, 500));
cfg->size_only = true;
EXPECT_TRUE(file_basis_quick_match(cfg, &st, 1500000001, 500));
cfg->size_only = false;
cfg->modify_window = 2;
EXPECT_TRUE(file_basis_quick_match(cfg, &st, 1500000002, 500));
config_delete(cfg);
}
/* End-to-end handshake decision for a same-size, same-mtime, DIFFERENT-content
basis. Default (rsync parity): the metadata quick-check is trusted, the
receiver answers STATUS_OK and materializes the basis bytes. --verify-basis:
the whole-file digest is required, the basis is rejected and the receiver
asks for the source (STATUS_NEXT + full transfer). */
static void test_receive_incremental_check_basis_quick_check_and_verify() {
const char* root = "test_basis_quick_root";
const char* basis_dir = "test_basis_quick_root/basis";
const char* basis_file = "test_basis_quick_root/basis/f.txt";
unlink(basis_file);
unlink("test_basis_quick_root/f.txt");
rmdir(basis_dir);
rmdir(root);
EXPECT_EQ_INT(mkdir(root, 0755), 0);
EXPECT_EQ_INT(mkdir(basis_dir, 0755), 0);
const char* src_bytes = "AAAA";
const unsigned long long size = 4;
const time_t mtime = 1500000000;
{
FILE* fh = fopen(basis_file, "wb");
EXPECT_NOT_NULL(fh);
// cppcheck-suppress knownConditionTrueFalse
if (fh) {
EXPECT_EQ_INT((int)fwrite("BBBB", 1, (size_t)size, fh), (int)size);
fclose(fh);
}
}
struct timespec ts[2] = {{mtime, 0}, {mtime, 0}};
EXPECT_EQ_INT(utimensat(AT_FDCWD, basis_file, ts, 0), 0);
char root_abs[PATH_MAX];
EXPECT_NOT_NULL(realpath(root, root_abs));
int root_fd = open(root_abs, O_RDONLY | O_DIRECTORY | O_CLOEXEC);
EXPECT_TRUE(root_fd >= 0);
// cppcheck-suppress knownConditionTrueFalse
if (root_fd < 0) {
unlink(basis_file);
rmdir(basis_dir);
rmdir(root);
return;
}
EXPECT_TRUE(utils_set_authorized_root(root_fd, root_abs));
uint8_t digest[CHECKSUM_MAX_DIGEST_LEN];
size_t digest_len = 0;
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_XXH64, 0, src_bytes, size, digest, sizeof(digest),
&digest_len));
/* Route protocol I/O through the explicit descriptors (a previous test group
may have left io_set_fds() bound to its own pipe). */
io_set_fds(-1, -1);
io_set_bwlimit(0);
for (int verify = 0; verify <= 1; verify++) {
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
cfg->receive_root_directory = str_dup(root_abs);
cfg->checksum = false;
cfg->checksum_algo = CHECKSUM_ALGO_XXH64;
cfg->checksum_seed = 0;
cfg->use_incremental = true;
cfg->use_delta = false;
cfg->use_metadata = false;
cfg->verify_basis = (verify != 0);
EXPECT_EQ_INT(config_basis_append(cfg, BASIS_DEST_LINK, "basis"), 0);
int p[2];
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
EXPECT_TRUE(send_wire_str(p[1], "f.txt"));
unsigned long long check_size = size;
long long check_mtime = (long long)mtime;
long long check_mtime_nsec = 0;
EXPECT_TRUE(send_n_data(p[1], &check_size, sizeof(check_size)));
EXPECT_TRUE(send_n_data(p[1], &check_mtime, sizeof(check_mtime)));
EXPECT_TRUE(send_n_data(p[1], &check_mtime_nsec, sizeof(check_mtime_nsec)));
/* Only --verify-basis needs the digest (cfg->checksum is false) and the
pre-staged fallback full transfer the receiver will request. */
if (verify) {
uint8_t wire_len = (uint8_t)digest_len;
EXPECT_TRUE(send_n_data(p[1], &wire_len, sizeof(wire_len)));
EXPECT_TRUE(send_n_data(p[1], digest, digest_len));
char* payload_bytes = str_dup(src_bytes);
EXPECT_NOT_NULL(payload_bytes);
Data* payload = data_create(payload_bytes, size);
EXPECT_NOT_NULL(payload);
// cppcheck-suppress knownConditionTrueFalse
if (payload)
EXPECT_TRUE(send_data(p[1], payload));
data_destroy(payload);
}
bool skipped = false;
File* file = receive_incremental_check(p[0], cfg, &skipped);
EXPECT_NOT_NULL(file);
// cppcheck-suppress knownConditionTrueFalse
if (file) {
EXPECT_FALSE(skipped);
Status reply = STATUS_ERROR;
EXPECT_TRUE(receive_status(p[1], &reply));
if (verify) {
EXPECT_EQ_INT((int)reply, (int)STATUS_NEXT);
EXPECT_NOT_NULL(file->data->data);
EXPECT_TRUE(file->data->data != NULL && memcmp(file->data->data, src_bytes, size) == 0);
} else {
EXPECT_EQ_INT((int)reply, (int)STATUS_OK);
EXPECT_TRUE(file->skip);
/* The default quick-check hit materializes from the basis PATH at
install time (streaming), so no content is buffered on the File. */
EXPECT_NOT_NULL(file->basis_link);
EXPECT_NULL(file->data->data);
}
file_destroy(file);
}
close(p[0]);
close(p[1]);
config_delete(cfg);
}
utils_set_authorized_root(-1, NULL);
close(root_fd);
unlink(basis_file);
rmdir(basis_dir);
rmdir(root);
}
/* -K/--keep-dirlinks secure open: with an authorized root, a destination path
* component that is a symlink to an IN-ROOT directory is used as that directory
* (its referent is opened through a relative O_NOFOLLOW walk from the root fd,
@@ -2002,6 +2156,100 @@ static void test_manifest_delete_missing_dir_budget_double_count() {
rmdir(root);
}
/* Blocker #7: when the receive root is "/", every absolute basis path is below
it and its child relative form must drop only the single leading slash. */
static void test_basis_delete_relative_root_slash() {
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
cfg->receive_root_directory = str_dup("/");
char* rel = file_receive_basis_delete_relative(cfg, "/a");
EXPECT_NOT_NULL(rel);
EXPECT_EQ_STR(rel, "a");
free(rel);
rel = file_receive_basis_delete_relative(cfg, "/a/b");
EXPECT_NOT_NULL(rel);
EXPECT_EQ_STR(rel, "a/b");
free(rel);
/* The root itself is not a child. */
EXPECT_NULL(file_receive_basis_delete_relative(cfg, "/"));
/* A relative entry is already root-relative. */
rel = file_receive_basis_delete_relative(cfg, "x/y");
EXPECT_NOT_NULL(rel);
EXPECT_EQ_STR(rel, "x/y");
free(rel);
/* An absolute path outside a non-"/" root is unreachable. */
free(cfg->receive_root_directory);
cfg->receive_root_directory = str_dup("/root");
EXPECT_NULL(file_receive_basis_delete_relative(cfg, "/other/a"));
rel = file_receive_basis_delete_relative(cfg, "/root/a");
EXPECT_NOT_NULL(rel);
EXPECT_EQ_STR(rel, "a");
free(rel);
config_delete(cfg);
}
/* Blocker #6: -n --delete would-delete enumeration must normalize an absolute
basis directory under the receive root exactly like the real commit path, so
the basis snapshot is protected rather than reported as a deletable extra. */
static void test_manifest_would_delete_protects_absolute_basis() {
char root[PATH_MAX];
snprintf(root, sizeof(root), "/tmp/fastsync_wdbasis_%d", (int)getpid());
char* basis = path_cat(root, "basis");
char* basis_file = path_cat(basis, "snapshot.bin");
char* extra = path_cat(root, "extra.txt");
EXPECT_NOT_NULL(basis);
EXPECT_NOT_NULL(basis_file);
EXPECT_NOT_NULL(extra);
mkdir(root, 0755);
mkdir(basis, 0755);
EXPECT_TRUE(file_write_to_disk(basis_file, "x", 1, false, false));
EXPECT_TRUE(file_write_to_disk(extra, "e", 1, false, false));
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
cfg->receive_root_directory = str_dup(root);
cfg->use_delete = true;
EXPECT_EQ_INT(config_basis_append(cfg, BASIS_DEST_COMPARE, basis), 0);
const char* synced[] = {"."};
DeleteManifest manifest = {0};
manifest.keeps = make_manifest_string_list(NULL, 0);
manifest.protected = make_manifest_string_list(NULL, 0);
manifest.dirs = make_manifest_string_list(synced, 1);
EXPECT_NOT_NULL(manifest.keeps);
EXPECT_NOT_NULL(manifest.protected);
EXPECT_NOT_NULL(manifest.dirs);
ArrayList* out = array_list_create(free);
EXPECT_NOT_NULL(out);
size_t count = 0;
EXPECT_TRUE(manifest_would_delete_list(cfg, &manifest, out, &count));
bool saw_basis = false;
bool saw_extra = false;
for (int i = 0; i < out->size; i++) {
const char* p = (const char*)out->items[i];
if (strcmp(p, "basis") == 0 || strncmp(p, "basis/", 6) == 0)
saw_basis = true;
if (strcmp(p, "extra.txt") == 0)
saw_extra = true;
}
EXPECT_FALSE(saw_basis);
EXPECT_TRUE(saw_extra);
array_list_delete(out);
array_list_delete(manifest.keeps);
array_list_delete(manifest.protected);
array_list_delete(manifest.dirs);
config_delete(cfg);
unlink(basis_file);
rmdir(basis);
unlink(extra);
rmdir(root);
free(basis);
free(basis_file);
free(extra);
}
void test_file() {
test_file_create();
test_file_special_rdev_valid();
@@ -2046,6 +2294,8 @@ void test_file() {
test_dir_time_list();
test_dir_time_list_cap();
test_receive_incremental_check_empty_path();
test_file_basis_quick_match_decision();
test_receive_incremental_check_basis_quick_check_and_verify();
test_keep_dirlinks_secure_open();
test_inplace_overwrite_clears_special_mode_bits();
test_inplace_overwrite_metadata_strips_special_bits();
@@ -2057,4 +2307,6 @@ void test_file() {
test_inplace_refuses_fifo_destination();
test_inplace_refuses_device_destination();
test_manifest_delete_missing_dir_budget_double_count();
test_basis_delete_relative_root_slash();
test_manifest_would_delete_protects_absolute_basis();
}
+34
View File
@@ -86,9 +86,43 @@ static void test_dest_state_roundtrip() {
close(fds[1]);
}
static void test_stats_roundtrip() {
/* STATUS_STATS grew from three counters (2.25.0) to eight (2.28.0); the codec
* must carry every field, including the receiver-observed literal/created
* counters, across the wire in order. */
int fds[2];
if (socketpair(AF_UNIX, SOCK_STREAM, 0, fds) != 0)
return;
ReceiverStats out;
memset(&out, 0, sizeof(out));
out.matched_data = 111111111ULL;
out.deleted_files = 7;
out.would_delete_count = 3;
out.literal_bytes = 222222222ULL;
out.created_reg = 5;
out.created_dir = 4;
out.created_link = 2;
out.created_special = 1;
ReceiverStats in;
memset(&in, 0, sizeof(in));
EXPECT_TRUE(format_stats_send(fds[0], &out));
EXPECT_TRUE(format_stats_receive(fds[1], &in));
EXPECT_TRUE(in.matched_data == out.matched_data);
EXPECT_TRUE(in.deleted_files == out.deleted_files);
EXPECT_TRUE(in.would_delete_count == out.would_delete_count);
EXPECT_TRUE(in.literal_bytes == out.literal_bytes);
EXPECT_TRUE(in.created_reg == out.created_reg);
EXPECT_TRUE(in.created_dir == out.created_dir);
EXPECT_TRUE(in.created_link == out.created_link);
EXPECT_TRUE(in.created_special == out.created_special);
close(fds[0]);
close(fds[1]);
}
void test_format(void) {
test_human_size_decimal();
test_big_num_grouping();
test_datetime_format();
test_dest_state_roundtrip();
test_stats_roundtrip();
}
+17 -15
View File
@@ -18,9 +18,11 @@
/* P8 config-frame tail: super_mode (4) + copy-as presence (4) + uid (4) + gid (4). */
#define P8_TAIL_BYTES 16
/* Protocol 2.23.0 appends one trailing bool (report_dest_info) AFTER the P8
* tail, so the P8 fields sit this many bytes before the end of the frame. */
#define OUTPUT_TAIL_BYTES 4
/* Bytes after the P8 tail: report_dest_info (4), report_stats (4),
* report_deletes (4, --info=del wave), compression_algo (4, codec wave) and the
* receiver delete-protection count (4, protocol 2.28.0). The P8 fields sit
* this many bytes before the end of the frame. */
#define POST_P8_TAIL_BYTES 20
/* Smoke test for chunk_deserialize fuzz target */
static void test_fuzz_chunk_deserialize() {
@@ -323,7 +325,7 @@ static void test_fuzz_config_receive_p8_tail() {
size_t len = 0;
bool captured = capture_config_frame(c, &frame, &len);
config_delete(c);
if (!captured || len <= P8_TAIL_BYTES) {
if (!captured || len <= P8_TAIL_BYTES + POST_P8_TAIL_BYTES) {
free(frame);
EXPECT_TRUE(false);
return;
@@ -337,31 +339,31 @@ static void test_fuzz_config_receive_p8_tail() {
/* super_mode outside the 0..2 tri-state is refused. */
memcpy(mut, frame, len);
put_i32(mut, len - OUTPUT_TAIL_BYTES - P8_TAIL_BYTES, 99);
put_i32(mut, len - POST_P8_TAIL_BYTES - P8_TAIL_BYTES, 99);
EXPECT_FALSE(receive_config_frame(mut, len));
put_i32(mut, len - OUTPUT_TAIL_BYTES - P8_TAIL_BYTES, -1);
put_i32(mut, len - POST_P8_TAIL_BYTES - P8_TAIL_BYTES, -1);
EXPECT_FALSE(receive_config_frame(mut, len));
/* A negative (sentinel) and an extreme copy-as uid/gid are refused. */
memcpy(mut, frame, len);
put_i32(mut, len - OUTPUT_TAIL_BYTES - P8_TAIL_BYTES, SUPER_MODE_AUTO);
put_i32(mut, len - OUTPUT_TAIL_BYTES - P8_TAIL_BYTES + 4, 1);
put_i32(mut, len - OUTPUT_TAIL_BYTES - P8_TAIL_BYTES + 8, -1);
put_i32(mut, len - OUTPUT_TAIL_BYTES - P8_TAIL_BYTES + 12, 0);
put_i32(mut, len - POST_P8_TAIL_BYTES - P8_TAIL_BYTES, SUPER_MODE_AUTO);
put_i32(mut, len - POST_P8_TAIL_BYTES - P8_TAIL_BYTES + 4, 1);
put_i32(mut, len - POST_P8_TAIL_BYTES - P8_TAIL_BYTES + 8, -1);
put_i32(mut, len - POST_P8_TAIL_BYTES - P8_TAIL_BYTES + 12, 0);
EXPECT_FALSE(receive_config_frame(mut, len));
put_i32(mut, len - OUTPUT_TAIL_BYTES - P8_TAIL_BYTES + 8, 0);
put_i32(mut, len - OUTPUT_TAIL_BYTES - P8_TAIL_BYTES + 12, INT32_MIN);
put_i32(mut, len - POST_P8_TAIL_BYTES - P8_TAIL_BYTES + 8, 0);
put_i32(mut, len - POST_P8_TAIL_BYTES - P8_TAIL_BYTES + 12, INT32_MIN);
EXPECT_FALSE(receive_config_frame(mut, len));
/* A presence int that is not a wire bool is refused. */
memcpy(mut, frame, len);
put_i32(mut, len - OUTPUT_TAIL_BYTES - P8_TAIL_BYTES, SUPER_MODE_AUTO);
put_i32(mut, len - OUTPUT_TAIL_BYTES - P8_TAIL_BYTES + 4, 2);
put_i32(mut, len - POST_P8_TAIL_BYTES - P8_TAIL_BYTES, SUPER_MODE_AUTO);
put_i32(mut, len - POST_P8_TAIL_BYTES - P8_TAIL_BYTES + 4, 2);
EXPECT_FALSE(receive_config_frame(mut, len));
/* Truncating anywhere inside the P8 tail is refused. */
EXPECT_FALSE(receive_config_frame(frame, len - 2));
EXPECT_FALSE(receive_config_frame(frame, len - OUTPUT_TAIL_BYTES - P8_TAIL_BYTES));
EXPECT_FALSE(receive_config_frame(frame, len - POST_P8_TAIL_BYTES - P8_TAIL_BYTES));
free(mut);
free(frame);
+21 -4
View File
@@ -148,10 +148,23 @@ static void test_iconv_wire_sender_converts_local_to_remote() {
charset_wire_free();
}
static void test_iconv_wire_receiver_converts_remote_to_local() {
/* rsync push parity: with no server --iconv the destination charset is the
* client spec's REMOTE half, so the receiver writes the wire bytes verbatim. */
static void test_iconv_wire_receiver_default_writes_remote() {
EXPECT_TRUE(charset_wire_init_receiver("utf-8,iso-8859-1", NULL));
char* local = charset_wire_apply("caf\xe9");
EXPECT_NOT_NULL(local);
EXPECT_EQ_INT(strcmp(local, "caf\xe9"), 0);
free(local);
charset_wire_free();
}
/* A server that declares its own --iconv LOCAL converts wire(REMOTE) into that
* declared charset (the daemon "charset" analog). */
static void test_iconv_wire_receiver_server_local_override() {
EXPECT_TRUE(charset_wire_init_receiver("utf-8,iso-8859-1", "utf-8"));
char* local = charset_wire_apply("caf\xe9");
EXPECT_NOT_NULL(local);
EXPECT_EQ_INT(strcmp(local, "caf\xc3\xa9"), 0);
free(local);
charset_wire_free();
@@ -179,7 +192,7 @@ static void test_iconv_wire_str_roundtrip() {
close(p[1]);
io_set_fds(p[0], p[0]);
charset_wire_free();
charset_wire_init_receiver("utf-8,iso-8859-1", NULL);
charset_wire_init_receiver("utf-8,iso-8859-1", "utf-8");
char* got = receive_wire_str(p[0]);
bool ok = got != NULL && strcmp(got, "caf\xc3\xa9") == 0;
free(got);
@@ -211,7 +224,11 @@ void test_iconv() {
test_iconv_exact_fill_no_overflow();
test_iconv_growth_expanding_name();
test_iconv_wire_sender_converts_local_to_remote();
test_iconv_wire_receiver_converts_remote_to_local();
test_iconv_wire_receiver_default_writes_remote();
test_iconv_wire_receiver_server_local_override();
test_iconv_wire_disabled_passthrough();
test_iconv_wire_str_roundtrip();
// This subtest forks to exercise the wire string handshake; the instrumented
// parent is too slow under valgrind for the child's blocking reads.
if (!is_running_under_valgrind())
test_iconv_wire_str_roundtrip();
}
+62
View File
@@ -103,6 +103,62 @@ static void test_sender_zero_capacity() {
config_delete(cfg);
}
/* Regression (blocker): PipelineContextSender.delete_suppressed must be
initialized false. A garbage true silently suppresses the --delete keep-set
manifest under -m/--threads, so destination extras would never be removed. */
static void test_sender_delete_suppressed_initialized() {
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
free(cfg->version);
cfg->version = str_dup(PROTOCOL_VERSION);
cfg->send_directory = str_dup("/src");
cfg->receive_root_directory = str_dup("/dst");
Queue* q1 = queue_create(5, NULL);
Queue* q2 = queue_create(5, NULL);
EXPECT_NOT_NULL(q1);
EXPECT_NOT_NULL(q2);
PipelineContextSender* ctx = pipeline_context_sender_create(cfg, q1, q2);
EXPECT_NOT_NULL(ctx);
EXPECT_FALSE(ctx->delete_suppressed);
pipeline_context_sender_destroy(ctx);
config_delete(cfg);
}
/* --info=del (report_deletes) is the only reason the receiver retains the
actually-removed paths: a plain --delete receiver must not allocate the list,
and with report_deletes armed the observer records into it. */
static void test_receiver_deleted_paths_gated_by_report_deletes() {
Config* plain = config_create();
EXPECT_NOT_NULL(plain);
free(plain->version);
plain->version = str_dup(PROTOCOL_VERSION);
plain->receive_root_directory = str_dup("/dst");
plain->report_deletes = false;
Queue* q_plain = queue_create(5, file_destroy);
EXPECT_NOT_NULL(q_plain);
PipelineContextReceiver* ctx_plain = pipeline_context_receiver_create(plain, q_plain, -1, NULL);
EXPECT_NOT_NULL(ctx_plain);
EXPECT_NULL(ctx_plain->deleted_paths);
pipeline_context_receiver_destroy(ctx_plain);
Config* info = config_create();
EXPECT_NOT_NULL(info);
free(info->version);
info->version = str_dup(PROTOCOL_VERSION);
info->receive_root_directory = str_dup("/dst");
info->report_deletes = true;
Queue* q_info = queue_create(5, file_destroy);
EXPECT_NOT_NULL(q_info);
PipelineContextReceiver* ctx_info = pipeline_context_receiver_create(info, q_info, -1, NULL);
EXPECT_NOT_NULL(ctx_info);
EXPECT_NOT_NULL(ctx_info->deleted_paths);
receiver_record_deleted_path(ctx_info->deleted_paths, "d/old_extra");
EXPECT_EQ_INT(ctx_info->deleted_paths->size, 1);
EXPECT_EQ_STR((const char*)ctx_info->deleted_paths->items[0], "d/old_extra");
pipeline_context_receiver_destroy(ctx_info);
}
/* Test receiver with zero file_descriptor */
static void test_receiver_fd_zero() {
Config* cfg = config_create();
@@ -243,6 +299,8 @@ static void test_write_thread_done() {
* However, pipeline_context_receiver_destroy will call config_delete
* and queue_destroy which would double-free since we created them
* in this test. Let me just free the context directly. */
array_list_delete(ctx->would_delete);
array_list_delete(ctx->deleted_paths);
mtx_destroy(&ctx->mutex);
cnd_destroy(&ctx->condition_not_full);
cnd_destroy(&ctx->condition_not_empty);
@@ -327,6 +385,8 @@ static void test_receiver_enqueue_byte_budget() {
EXPECT_EQ_INT((int)ctx->queued_bytes, 2000); /* second payload now in flight */
/* Tear down: the second file is still queued and is freed by queue_destroy. */
array_list_delete(ctx->would_delete);
array_list_delete(ctx->deleted_paths);
mtx_destroy(&ctx->mutex);
cnd_destroy(&ctx->condition_not_full);
cnd_destroy(&ctx->condition_not_empty);
@@ -458,6 +518,8 @@ void test_multiprocessing() {
test_receiver_create_destroy();
test_sender_queue_capacities();
test_sender_zero_capacity();
test_sender_delete_suppressed_initialized();
test_receiver_deleted_paths_gated_by_report_deletes();
test_receiver_fd_zero();
if (!is_running_under_valgrind()) {
test_receive_thread_finished();
+1 -1
View File
@@ -65,7 +65,7 @@ static void test_receiver_aborts_idle_keepalive() {
ssize_t wrote = write(sv[0], &keepalive, sizeof(keepalive));
int result = -2;
if (wrote == (ssize_t)sizeof(keepalive))
result = receiver_process_pending(config, sv[1], &sink, NULL);
result = receiver_process_pending(config, sv[1], &sink, NULL, NULL);
Status reply = STATUS_OK;
ssize_t got = -1;
if (result == -1)
+156 -6
View File
@@ -855,7 +855,7 @@ static void test_filter_rules(bool parallel) {
/* - *.tmp excludes only the tmp file; other files remain (default include). */
const char* exclude_only[] = {"- *.tmp"};
char err[160];
FilterRuleList* base = filter_base_build(exclude_only, 1, false, err, sizeof(err));
FilterRuleList* base = filter_base_build(exclude_only, 1, false, false, err, sizeof(err));
EXPECT_NOT_NULL(base);
ScannerOptions options = {0};
options.base_filters = base;
@@ -875,7 +875,7 @@ static void test_filter_rules(bool parallel) {
/* Anchored include then exclude-all: only root-level keep* survives. */
const char* anchored[] = {"+ /a.txt", "- *"};
base = filter_base_build(anchored, 2, false, err, sizeof(err));
base = filter_base_build(anchored, 2, false, false, err, sizeof(err));
EXPECT_NOT_NULL(base);
options.base_filters = base;
rc = parallel ? collect_files_parallel(root, &options, &paths, &count)
@@ -889,7 +889,7 @@ static void test_filter_rules(bool parallel) {
/* The common include idiom (the exact rule order the CLI compiles from
* --include='*.txt' --exclude='*'): only .txt files survive. */
const char* idiom[] = {"+ *.txt", "- *"};
base = filter_base_build(idiom, 2, false, err, sizeof(err));
base = filter_base_build(idiom, 2, false, false, err, sizeof(err));
EXPECT_NOT_NULL(base);
options.base_filters = base;
rc = parallel ? collect_files_parallel(root, &options, &paths, &count)
@@ -905,7 +905,7 @@ static void test_filter_rules(bool parallel) {
/* An include rule alone is NOT a mandatory whitelist (rsync semantics): only
* the matching file is affected, everything else is still transferred. */
const char* include_alone[] = {"+ *.txt"};
base = filter_base_build(include_alone, 1, false, err, sizeof(err));
base = filter_base_build(include_alone, 1, false, false, err, sizeof(err));
EXPECT_NOT_NULL(base);
options.base_filters = base;
rc = parallel ? collect_files_parallel(root, &options, &paths, &count)
@@ -932,7 +932,7 @@ static void test_filter_dir_only_and_anchored(bool parallel) {
const char* rules[] = {"- /sub/"};
char err[160];
FilterRuleList* base = filter_base_build(rules, 1, false, err, sizeof(err));
FilterRuleList* base = filter_base_build(rules, 1, false, false, err, sizeof(err));
EXPECT_NOT_NULL(base);
ScannerOptions options = {0};
options.base_filters = base;
@@ -966,7 +966,7 @@ static void test_cvs_defaults(bool parallel) {
create_test_file("test_scan_cvs/keep.txt", "keep");
char err[160];
FilterRuleList* base = filter_base_build(NULL, 0, true, err, sizeof(err));
FilterRuleList* base = filter_base_build(NULL, 0, true, false, err, sizeof(err));
EXPECT_NOT_NULL(base);
ScannerOptions options = {0};
options.base_filters = base;
@@ -1005,6 +1005,7 @@ static void test_per_dir_filter(bool parallel) {
ScannerOptions options = {0};
options.per_dir_filters = true;
options.exclude_per_dir_filter_files = true; /* -FF */
if (parallel)
options.num_threads = 2;
char** paths = NULL;
@@ -1069,6 +1070,59 @@ static void test_scanner_path_relative() {
EXPECT_NULL(scanner_path_relative("/tmp/foo", "/tmp/foobar"));
}
/* -R/--relative destination prefix: the '/./' cut point and normalization. */
static void test_scanner_relative_prefix() {
char* p = NULL;
/* No cut: the whole spec with leading/trailing slashes removed. */
p = scanner_relative_prefix("/tmp/src/foo/");
EXPECT_NOT_NULL(p);
EXPECT_EQ_STR(p, "tmp/src/foo");
free(p);
p = scanner_relative_prefix("src/foo");
EXPECT_NOT_NULL(p);
EXPECT_EQ_STR(p, "src/foo");
free(p);
/* Trailing "/." is the directory itself, not a cut. */
p = scanner_relative_prefix("src/foo/.");
EXPECT_NOT_NULL(p);
EXPECT_EQ_STR(p, "src/foo");
free(p);
/* The first "/./" cuts everything before it. */
p = scanner_relative_prefix("/a/./b/c");
EXPECT_NOT_NULL(p);
EXPECT_EQ_STR(p, "b/c");
free(p);
p = scanner_relative_prefix("src/./");
EXPECT_NOT_NULL(p);
EXPECT_EQ_STR(p, "");
free(p);
/* A later "." component is normalized away. */
p = scanner_relative_prefix("a/./b/./c");
EXPECT_NOT_NULL(p);
EXPECT_EQ_STR(p, "b/c");
free(p);
/* A leading "./" is the cut at the start. */
p = scanner_relative_prefix("./s2");
EXPECT_NOT_NULL(p);
EXPECT_EQ_STR(p, "s2");
free(p);
p = scanner_relative_prefix(".");
EXPECT_NOT_NULL(p);
EXPECT_EQ_STR(p, "");
free(p);
EXPECT_NULL(scanner_relative_prefix(NULL));
EXPECT_NULL(scanner_relative_prefix(""));
}
/* rsync precedence: a deeper .rsync-filter overrides a shallower one, so an
* inner "+ *.tmp" re-includes what the outer "- *.tmp" excluded. */
static void test_per_dir_filter_override(bool parallel) {
@@ -1084,6 +1138,7 @@ static void test_per_dir_filter_override(bool parallel) {
ScannerOptions options = {0};
options.per_dir_filters = true;
options.exclude_per_dir_filter_files = true; /* -FF */
if (parallel)
options.num_threads = 2;
char** paths = NULL;
@@ -1185,6 +1240,55 @@ static int collect_scan_info_parallel(ParallelScanner* scanner, const char* root
return failed ? -1 : count;
}
/* The --progress paths-only pre-count relies on `list_dirs` emitting every
* directory (including empty ones) exactly once, alongside the files and
* symlinks the streaming scanner already emits. */
static void test_scanner_list_dirs_counts_every_entry() {
const char* root = "test_scan_listdirs";
EXPECT_EQ_INT(mkdir(root, 0755), 0);
EXPECT_EQ_INT(mkdir("test_scan_listdirs/sub1", 0755), 0);
EXPECT_EQ_INT(mkdir("test_scan_listdirs/sub1/deep", 0755), 0);
EXPECT_EQ_INT(mkdir("test_scan_listdirs/sub2", 0755), 0);
EXPECT_EQ_INT(mkdir("test_scan_listdirs/emptydir", 0755), 0);
create_test_file("test_scan_listdirs/a.txt", "a");
create_test_file("test_scan_listdirs/sub1/c.txt", "c");
create_test_file("test_scan_listdirs/sub1/deep/d.txt", "d");
create_test_file("test_scan_listdirs/sub2/e.txt", "e");
EXPECT_EQ_INT(symlink("a.txt", "test_scan_listdirs/link1"), 0);
ScannerOptions options = {0};
options.list_dirs = true;
options.emit_empty_dirs = true;
options.follow_symlinks = true; /* -l/--links: carry symlinks, don't skip */
ScanInfo infos[16];
int count = collect_scan_info(root, &options, infos, 16);
EXPECT_EQ_INT(count, 9);
int dirs = 0;
for (int i = 0; i < count; i++) {
if (infos[i].is_dir)
dirs++;
}
EXPECT_EQ_INT(dirs, 4);
EXPECT_TRUE(scan_info_present(infos, count, "sub1", true, NULL));
EXPECT_TRUE(scan_info_present(infos, count, "sub1/deep", true, NULL));
EXPECT_TRUE(scan_info_present(infos, count, "sub2", true, NULL));
EXPECT_TRUE(scan_info_present(infos, count, "emptydir", true, NULL));
EXPECT_TRUE(scan_info_present(infos, count, "a.txt", false, ""));
EXPECT_TRUE(scan_info_present(infos, count, "sub1/c.txt", false, ""));
EXPECT_TRUE(scan_info_present(infos, count, "link1", false, ""));
unlink("test_scan_listdirs/a.txt");
unlink("test_scan_listdirs/sub1/c.txt");
unlink("test_scan_listdirs/sub1/deep/d.txt");
unlink("test_scan_listdirs/sub2/e.txt");
unlink("test_scan_listdirs/link1");
rmdir("test_scan_listdirs/sub1/deep");
rmdir("test_scan_listdirs/sub1");
rmdir("test_scan_listdirs/sub2");
rmdir("test_scan_listdirs/emptydir");
rmdir(root);
}
/* -d without --files-from emits exactly the source-root directory (empty) and
* never descends. */
static void test_dirs_no_descent() {
@@ -1533,6 +1637,49 @@ static void test_scanner_entry_classification() {
rmdir(root);
}
/* A dereferenced symlink with no referent (broken/unreadable) must record a
* non-fatal I/O error so the run can exit 23 like rsync, without aborting the
* scan or treating the condition as a fatal failure. */
static void test_scanner_broken_referent_io_error(void) {
const char* root = "test_scan_broken_ref";
const char* good = "test_scan_broken_ref/good.txt";
const char* broken = "test_scan_broken_ref/broken";
EXPECT_EQ_INT(mkdir(root, 0755), 0);
create_test_file(good, "hello");
EXPECT_EQ_INT(symlink("/nonexistent/quickwins/target", broken), 0);
{
ScannerOptions options = {0};
options.copy_links = true;
DirectoryScanner* scanner = directory_scanner_create_with_options(root, &options);
EXPECT_NOT_NULL(scanner);
Chunk* chunk;
while ((chunk = directory_scanner_next(scanner)) != NULL)
chunk_destroy(chunk);
EXPECT_FALSE(directory_scanner_failed(scanner));
EXPECT_TRUE(directory_scanner_had_io_error(scanner));
directory_scanner_destroy(scanner);
}
{
ScannerOptions options = {0};
options.copy_links = true;
ParallelScanner* scanner = parallel_scanner_create_with_options(root, &options, NULL);
EXPECT_NOT_NULL(scanner);
Chunk* chunk;
while ((chunk = parallel_scanner_next(scanner)) != NULL)
chunk_destroy(chunk);
EXPECT_FALSE(parallel_scanner_failed(scanner));
EXPECT_TRUE(parallel_scanner_had_io_error(scanner));
parallel_scanner_destroy(scanner);
}
unlink(broken);
unlink(good);
rmdir(root);
}
void test_scanner() {
test_scanner_single_file();
test_scanner_multiple_files();
@@ -1551,6 +1698,7 @@ void test_scanner() {
test_scanner_one_file_system_decision();
test_scanner_one_file_system_same_device();
test_parallel_scanner_one_file_system_same_device();
test_scanner_broken_referent_io_error();
test_scanner_one_file_system_cross_device();
test_files_from_subset(false);
test_files_from_subset(true);
@@ -1563,10 +1711,12 @@ void test_scanner() {
test_per_dir_filter(false);
test_per_dir_filter(true);
test_scanner_path_relative();
test_scanner_relative_prefix();
test_per_dir_filter_override(false);
test_per_dir_filter_override(true);
test_dirs_no_descent();
test_dirs_files_from();
test_scanner_list_dirs_counts_every_entry();
test_files_from_relative_send_path();
test_scanner_captures_directory_times();
test_scanner_chunk_ownership();
+204 -18
View File
@@ -3,6 +3,8 @@
#include "config.h"
#include "delta.h"
#include "file.h"
#include "file_receive.h"
#include "format.h"
#include "log.h"
#include "protocol.h"
#include "test_utils.h"
@@ -134,6 +136,124 @@ static void test_receive_files_single_file() {
}
}
/* Protocol 2.28.0: a fresh single-file transfer over the wire reports the
* receiver-observed literal bytes and the created-regular counter through the
* terminal STATUS_STATS frame, and an update reports created_reg == 0. */
static void test_receive_stats_frame_created_and_literal() {
const char* content = "stats frame content";
size_t len = strlen(content);
char root_template[] = "/tmp/fastsync_stats_XXXXXX";
char* root = mkdtemp(root_template);
EXPECT_NOT_NULL(root);
int p[2];
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
io_set_fds(p[0], p[1]);
io_set_bwlimit(0);
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
free(cfg->version);
cfg->version = str_dup(PROTOCOL_VERSION);
cfg->send_directory = str_dup("/src");
cfg->receive_root_directory = str_dup(root);
cfg->save_to_disk = true;
cfg->report_stats = true;
pid_t pid = fork();
if (pid == 0) {
close(p[1]);
io_set_fds(p[0], p[0]);
int ret = receiver_receive_files(cfg, p[0]);
close(p[0]);
config_delete(cfg);
_exit(ret == 0 ? 0 : 1);
}
close(p[0]);
io_set_fds(p[1], p[1]);
send_status(p[1], STATUS_NEXT);
File* file = file_create("created.bin");
EXPECT_NOT_NULL(file);
file->data->data = malloc(len);
EXPECT_NOT_NULL(file->data->data);
memcpy(file->data->data, content, len);
file->data->size = len;
send_str(p[1], file->path);
send_data(p[1], file->data);
file_destroy(file);
send_status(p[1], STATUS_FINISHED);
Status status;
EXPECT_TRUE(receive_status(p[1], &status));
EXPECT_EQ_INT(status, STATUS_STATS);
ReceiverStats stats;
EXPECT_TRUE(format_stats_receive(p[1], &stats));
int would = 0;
EXPECT_TRUE(receive_int(p[1], &would));
EXPECT_EQ_INT(would, 0);
EXPECT_TRUE(stats.created_reg == 1);
EXPECT_TRUE(stats.created_dir == 0);
EXPECT_TRUE(stats.created_link == 0);
EXPECT_TRUE(stats.created_special == 0);
EXPECT_TRUE(stats.literal_bytes == (unsigned long long)len);
EXPECT_TRUE(stats.matched_data == 0);
Status final;
EXPECT_TRUE(receive_status(p[1], &final));
EXPECT_EQ_INT(final, STATUS_OK);
int wstatus;
waitpid(pid, &wstatus, 0);
close(p[1]);
config_delete(cfg);
EXPECT_TRUE(WIFEXITED(wstatus) && WEXITSTATUS(wstatus) == 0);
/* Second run against the now-existing destination: no created file. */
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
io_set_fds(p[0], p[1]);
io_set_bwlimit(0);
cfg = config_create();
EXPECT_NOT_NULL(cfg);
free(cfg->version);
cfg->version = str_dup(PROTOCOL_VERSION);
cfg->send_directory = str_dup("/src");
cfg->receive_root_directory = str_dup(root);
cfg->save_to_disk = true;
cfg->report_stats = true;
pid = fork();
if (pid == 0) {
close(p[1]);
io_set_fds(p[0], p[0]);
int ret = receiver_receive_files(cfg, p[0]);
close(p[0]);
config_delete(cfg);
_exit(ret == 0 ? 0 : 1);
}
close(p[0]);
io_set_fds(p[1], p[1]);
send_status(p[1], STATUS_NEXT);
file = file_create("created.bin");
EXPECT_NOT_NULL(file);
file->data->data = malloc(len);
EXPECT_NOT_NULL(file->data->data);
memcpy(file->data->data, content, len);
file->data->size = len;
send_str(p[1], file->path);
send_data(p[1], file->data);
file_destroy(file);
send_status(p[1], STATUS_FINISHED);
EXPECT_TRUE(receive_status(p[1], &status));
EXPECT_EQ_INT(status, STATUS_STATS);
EXPECT_TRUE(format_stats_receive(p[1], &stats));
EXPECT_TRUE(receive_int(p[1], &would));
EXPECT_TRUE(stats.created_reg == 0);
EXPECT_TRUE(stats.literal_bytes == (unsigned long long)len);
EXPECT_TRUE(receive_status(p[1], &final));
waitpid(pid, &wstatus, 0);
close(p[1]);
config_delete(cfg);
EXPECT_TRUE(WIFEXITED(wstatus) && WEXITSTATUS(wstatus) == 0);
}
/* Test receive_files with STATUS_ABORT */
static void test_receive_files_abort() {
Config* cfg = config_create();
@@ -566,7 +686,7 @@ static Config* make_late_delete_config(const char* root) {
static int run_pending_receiver(Config* cfg, int fd, DeleteManifest** pending) {
ReceiverSink sink = {0};
return receiver_process_pending(cfg, fd, &sink, pending);
return receiver_process_pending(cfg, fd, &sink, pending, NULL);
}
static void test_late_manifest_abort_frees_keepset() {
@@ -797,7 +917,7 @@ static void test_receiver_pending_commits_missing_args() {
/* NULL pending: the single-threaded commit path deletes at FINISHED. The
sink sends the terminal STATUS_OK success frame. */
ReceiverSink sink = {.send_success = true};
EXPECT_EQ_INT(receiver_process_pending(cfg, p[0], &sink, NULL), 0);
EXPECT_EQ_INT(receiver_process_pending(cfg, p[0], &sink, NULL, NULL), 0);
Status ack;
EXPECT_TRUE(receive_status(p[1], &ack));
EXPECT_EQ_INT(ack, STATUS_OK);
@@ -943,14 +1063,18 @@ static void test_incremental_check_fifo_destination_does_not_hang() {
}
/* A server-contacting --dry-run with an alternate basis dir must never read or
hash the basis file. An exact (size+mtime+content) basis match would
otherwise let a client probe the basis bytes against its own supplied digest
(a 1-bit content oracle). The dry-run decision is metadata-only, so even a
byte-identical basis is reported as would-transfer, not a compare-dest skip. */
static void test_incremental_check_dry_run_basis_does_not_read_content() {
hash the basis file. Under the default metadata quick-check a hit needs no
basis bytes, so a compare-dest match is reported as a skip (STATUS_OK) just
like a real run -- and still no content is read. Under --verify-basis a hit
would require hashing the basis against the client-supplied digest (a 1-bit
content oracle), which a dry-run must never do, so even a byte-identical
basis is reported as would-transfer. The destination is never materialized
in either arm. */
static void run_dry_run_basis_check(bool verify, Status expected) {
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
cfg->dry_run = true;
cfg->verify_basis = verify;
char* root = make_check_root("dryb");
EXPECT_NOT_NULL(root);
cfg->receive_root_directory = str_dup(root);
@@ -966,8 +1090,8 @@ static void test_incremental_check_dry_run_basis_does_not_read_content() {
EXPECT_EQ_INT(stat(basis_path, &bst), 0);
EXPECT_EQ_INT(config_basis_append(cfg, BASIS_DEST_COMPARE, "basis"), 0);
/* The (correct) source digest for the basis bytes: an unfixed dry-run would
read+hash the basis and treat this as an exact compare-dest hit. */
/* The (correct) source digest for the basis bytes: a buggy dry-run that read
and hashed the basis would treat this as an exact compare-dest hit. */
uint8_t digest[CHECKSUM_MAX_DIGEST_LEN];
size_t digest_len = 0;
EXPECT_TRUE(checksum_digest((ChecksumAlgo)cfg->checksum_algo, cfg->checksum_seed, content,
@@ -986,7 +1110,8 @@ static void test_incremental_check_dry_run_basis_does_not_read_content() {
bool skipped = false;
bool would_transfer = false;
File* file = receive_incremental_check_ex(p[0], cfg, &skipped, &would_transfer);
bool ok = file == NULL && !skipped && would_transfer;
bool ok = file == NULL && skipped == (expected == STATUS_OK) &&
would_transfer == (expected != STATUS_OK);
file_destroy(file);
config_delete(cfg);
close(p[0]);
@@ -1004,13 +1129,16 @@ static void test_incremental_check_dry_run_basis_does_not_read_content() {
EXPECT_TRUE(send_n_data(p[1], &size, sizeof(size)));
EXPECT_TRUE(send_n_data(p[1], &mtime, sizeof(mtime)));
EXPECT_TRUE(send_n_data(p[1], &mtime_nsec, sizeof(mtime_nsec)));
uint8_t wire_len = (uint8_t)digest_len;
EXPECT_TRUE(send_n_data(p[1], &wire_len, sizeof(wire_len)));
EXPECT_TRUE(send_n_data(p[1], digest, digest_len));
/* The digest is only on the wire when --checksum or --verify-basis needs it
(cfg->checksum is false here); the default quick-check arm sends none. */
if (verify) {
uint8_t wire_len = (uint8_t)digest_len;
EXPECT_TRUE(send_n_data(p[1], &wire_len, sizeof(wire_len)));
EXPECT_TRUE(send_n_data(p[1], digest, digest_len));
}
Status s;
EXPECT_TRUE(receive_status(p[1], &s));
/* A skip here would mean the receiver read+hashed the basis file. */
EXPECT_EQ_INT(s, STATUS_DRY_RUN_TRANSFER);
EXPECT_EQ_INT(s, expected);
int status;
waitpid(pid, &status, 0);
@@ -1028,6 +1156,11 @@ static void test_incremental_check_dry_run_basis_does_not_read_content() {
}
}
static void test_incremental_check_dry_run_basis_does_not_read_content() {
run_dry_run_basis_check(false, STATUS_OK);
run_dry_run_basis_check(true, STATUS_DRY_RUN_TRANSFER);
}
/* B1: a FIFO planted in a --link-dest basis directory must not block
* basis_open_regular() either; the basis match is simply declined. */
static void test_incremental_check_basis_fifo_does_not_hang() {
@@ -1072,10 +1205,10 @@ static void test_incremental_check_basis_fifo_does_not_hang() {
EXPECT_TRUE(send_n_data(p[1], &mtime, sizeof(mtime)));
EXPECT_TRUE(send_n_data(p[1], &mtime_nsec, sizeof(mtime_nsec)));
/* config_has_basis() makes the request carry the source digest. */
uint8_t wire_len = 8;
uint8_t digest[8] = {0};
uint8_t wire_len = checksum_digest_len((ChecksumAlgo)cfg->checksum_algo);
uint8_t digest[CHECKSUM_MAX_DIGEST_LEN] = {0};
EXPECT_TRUE(send_n_data(p[1], &wire_len, sizeof(wire_len)));
EXPECT_TRUE(send_n_data(p[1], digest, sizeof(digest)));
EXPECT_TRUE(send_n_data(p[1], digest, wire_len));
Status s;
EXPECT_TRUE(receive_status(p[1], &s));
EXPECT_EQ_INT(s, STATUS_NEXT);
@@ -1128,11 +1261,63 @@ static void test_receive_manifest_total_entry_cap() {
config_delete(cfg);
}
/* A server-contacting --dry-run must never delete, even on the per-directory
(--delete-during/--delete-delay) commit path. The receive path already skips
plan application under -n, but a plan frame carrying --delete-missing-args
exact deletions used to be honored by delete_plan_session_commit(). Seed a
destination mirror, stream a plan naming it, and prove it survives. */
static void test_dry_run_delete_plan_commit_does_not_delete() {
char* root = make_check_root("drydelplan");
EXPECT_NOT_NULL(root);
write_check_file(root, "victim.txt", "must survive");
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
cfg->send_directory = str_dup("/src");
cfg->receive_root_directory = str_dup(root);
cfg->use_delete = true;
cfg->delete_during = true;
cfg->delete_missing_args = true;
cfg->dry_run = true;
int p[2];
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
io_set_fds(p[0], p[1]);
io_set_bwlimit(0);
EXPECT_TRUE(send_status(p[1], STATUS_DELETE_PLAN));
EXPECT_TRUE(send_int(p[1], 1)); /* first frame carries the config sections */
EXPECT_TRUE(send_int(p[1], 0)); /* protected prefixes */
EXPECT_TRUE(send_int(p[1], 0)); /* size-skipped prefixes */
EXPECT_TRUE(send_int(p[1], 1)); /* missing-args exact deletions */
EXPECT_TRUE(send_str(p[1], "victim.txt"));
EXPECT_TRUE(send_int(p[1], 1)); /* apply: a real plan */
EXPECT_TRUE(send_str(p[1], ".")); /* receive root plan */
EXPECT_TRUE(send_int(p[1], 0)); /* kept child directories */
EXPECT_TRUE(send_int(p[1], 0)); /* kept child files */
EXPECT_TRUE(send_status(p[1], STATUS_FINISHED));
ReceiverSink sink = {.send_success = true};
EXPECT_EQ_INT(receiver_process_pending(cfg, p[0], &sink, NULL, NULL), 0);
char path[1024];
snprintf(path, sizeof(path), "%s/victim.txt", root);
EXPECT_EQ_INT(access(path, F_OK), 0);
close(p[0]);
close(p[1]);
config_delete(cfg);
remove(path);
rmdir(root);
free(root);
}
void test_server() {
test_special_socket_path_log_escaped();
if (!is_running_under_valgrind()) {
test_receive_files_finished();
test_receive_files_single_file();
test_receive_stats_frame_created_and_literal();
test_receive_files_abort();
test_receive_manifest_rejects_traversal();
test_receive_incremental_check_rejects_invalid_nanoseconds();
@@ -1150,5 +1335,6 @@ void test_server() {
test_receive_manifest_three_sections();
test_manifest_delete_missing_args();
test_receiver_pending_commits_missing_args();
test_dry_run_delete_plan_commit_does_not_delete();
}
}
+84 -6
View File
@@ -10,6 +10,7 @@
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <strings.h>
#include <sys/socket.h>
#include <sys/stat.h>
#include <threads.h>
@@ -137,7 +138,7 @@ static void test_walker_removes_extras_keeps_manifest_and_protected() {
DeleteSkipEntry skip = {"prot", false};
size_t deleted = 0;
DeleteWalkResult result =
delete_extras_limited(root, manifest, NULL, 100000, &skip, 1, &deleted, NULL);
delete_extras_limited(root, manifest, NULL, 100000, &skip, 1, NULL, &deleted, NULL);
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_OK);
EXPECT_FALSE(file_exists(root, "a.txt"));
EXPECT_TRUE(file_exists(root, "keep.txt"));
@@ -172,7 +173,7 @@ static void test_walker_keeps_nested_manifest_dirs() {
EXPECT_NOT_NULL(manifest);
size_t deleted = 0;
DeleteWalkResult result =
delete_extras_limited(root, manifest, NULL, 100000, NULL, 0, &deleted, NULL);
delete_extras_limited(root, manifest, NULL, 100000, NULL, 0, NULL, &deleted, NULL);
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_OK);
EXPECT_FALSE(file_exists(root, "extra.txt"));
EXPECT_TRUE(file_exists(root, "keepdir/deep/keep.txt"));
@@ -203,7 +204,7 @@ static void test_walker_max_delete_partial_deletes_up_to_cap() {
size_t deleted = 999;
size_t skipped = 0;
DeleteWalkResult result =
delete_extras_limited(root, manifest, NULL, 2, NULL, 0, &deleted, &skipped);
delete_extras_limited(root, manifest, NULL, 2, NULL, 0, NULL, &deleted, &skipped);
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_LIMIT_REACHED);
EXPECT_EQ_INT((int)deleted, 2);
EXPECT_EQ_INT((int)skipped, 1);
@@ -224,7 +225,8 @@ static void test_walker_max_delete_exact_bound_deletes() {
ArrayList* manifest = make_manifest_strings(keeps, 0);
EXPECT_NOT_NULL(manifest);
size_t deleted = 0;
DeleteWalkResult result = delete_extras_limited(root, manifest, NULL, 2, NULL, 0, &deleted, NULL);
DeleteWalkResult result =
delete_extras_limited(root, manifest, NULL, 2, NULL, 0, NULL, &deleted, NULL);
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_OK);
EXPECT_EQ_INT((int)deleted, 2);
EXPECT_FALSE(file_exists(root, "a.txt"));
@@ -257,7 +259,7 @@ static void test_walker_removes_extraneous_symlinks() {
EXPECT_NOT_NULL(manifest);
size_t deleted = 0;
DeleteWalkResult result =
delete_extras_limited(root, manifest, NULL, 100000, NULL, 0, &deleted, NULL);
delete_extras_limited(root, manifest, NULL, 100000, NULL, 0, NULL, &deleted, NULL);
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_OK);
EXPECT_FALSE(file_exists(root, "link_file"));
EXPECT_FALSE(file_exists(root, "link_dir"));
@@ -293,7 +295,7 @@ static void test_walker_confines_deletion_to_synced_dirs() {
EXPECT_TRUE(array_list_add(dirs, str_dup("inscope")));
size_t deleted = 0;
DeleteWalkResult result =
delete_extras_limited(root, manifest, dirs, 100000, NULL, 0, &deleted, NULL);
delete_extras_limited(root, manifest, dirs, 100000, NULL, 0, NULL, &deleted, NULL);
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_OK);
EXPECT_TRUE(file_exists(root, "rootextra.txt"));
EXPECT_FALSE(file_exists(root, "inscope/extra.txt"));
@@ -323,6 +325,45 @@ static void test_walker_unlimited_deletes_all() {
free(root);
}
/* Receiver-side filter protection (protocol 2.28.0): a compiled protect rule
shields a DESTINATION-ONLY extra that never appeared on the sender, a risk
rule cancels an earlier/later protect (first match wins), and a dir-only
protect rule shields the whole subtree. */
static void test_walker_protect_rules_shield_dest_only() {
char* root = make_walk_root("protectrules");
EXPECT_NOT_NULL(root);
EXPECT_TRUE(write_file_at(root, "keep.txt", "kept"));
EXPECT_TRUE(write_file_at(root, "extra.log", "risk cancels protect"));
EXPECT_TRUE(write_file_at(root, "safe.log", "protected"));
EXPECT_TRUE(write_file_at(root, "other.txt", "deleted"));
EXPECT_EQ_INT(make_subdir(root, "prot"), 0);
EXPECT_TRUE(write_file_at(root, "prot/inside.txt", "shielded subtree"));
EXPECT_TRUE(write_file_at(root, "prot/deep.log", "shielded subtree"));
const char* keeps[] = {"keep.txt"};
ArrayList* manifest = make_manifest_strings(keeps, 1);
EXPECT_NOT_NULL(manifest);
const char* rule_text[] = {"R extra.log", "P *.log", "P prot/"};
char err[160];
FilterRuleList* rules = filter_base_build(rule_text, 3, false, false, err, sizeof(err));
EXPECT_NOT_NULL(rules);
size_t deleted = 0;
DeleteWalkResult result =
delete_extras_limited(root, manifest, NULL, 100000, NULL, 0, rules, &deleted, NULL);
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_OK);
EXPECT_TRUE(file_exists(root, "keep.txt"));
EXPECT_FALSE(file_exists(root, "extra.log")); /* risk wins the first match */
EXPECT_TRUE(file_exists(root, "safe.log")); /* protect shields the extra */
EXPECT_FALSE(file_exists(root, "other.txt"));
EXPECT_TRUE(dir_exists(root, "prot"));
EXPECT_TRUE(file_exists(root, "prot/inside.txt"));
EXPECT_TRUE(file_exists(root, "prot/deep.log"));
filter_rule_list_free(rules);
array_list_delete(manifest);
remove_walk_tree(root);
free(root);
}
typedef struct {
bool eight_bit_output;
const char* expected;
@@ -578,7 +619,43 @@ static void test_getdelim_bounded() {
fclose(fp);
}
static int test_env_resolver(const char* name) {
if (strcasecmp(name, "alpha") == 0)
return 10;
if (strcasecmp(name, "beta") == 0)
return 20;
return -1;
}
static void test_env_choice_first_parsing() {
const char* var = "FASTSYNC_TEST_CHOICE_LIST";
bool specified = true;
unsetenv(var);
EXPECT_EQ_INT(env_choice_first(var, test_env_resolver, &specified), -1);
EXPECT_FALSE(specified);
/* Unknown entries are skipped, case-insensitive, first supported wins. */
setenv(var, "bogus BETA alpha", 1);
EXPECT_EQ_INT(env_choice_first(var, test_env_resolver, &specified), 20);
EXPECT_TRUE(specified);
/* The client half ends at '&'. */
setenv(var, "alpha & beta", 1);
EXPECT_EQ_INT(env_choice_first(var, test_env_resolver, &specified), 10);
/* Blank means "unspecified"; all-unknown means "specified but no match". */
setenv(var, " ", 1);
EXPECT_EQ_INT(env_choice_first(var, test_env_resolver, &specified), -1);
EXPECT_FALSE(specified);
setenv(var, "nope,alpha", 1);
EXPECT_EQ_INT(env_choice_first(var, test_env_resolver, &specified), -1);
EXPECT_TRUE(specified);
unsetenv(var);
}
void test_shared_utils() {
test_env_choice_first_parsing();
test_path_index_bounded();
test_path_index_semantics();
test_getdelim_bounded();
@@ -589,6 +666,7 @@ void test_shared_utils() {
test_walker_removes_extraneous_symlinks();
test_walker_confines_deletion_to_synced_dirs();
test_walker_unlimited_deletes_all();
test_walker_protect_rules_shield_dest_only();
test_loopback_helpers();
test_fd_peer_ip();
+16 -3
View File
@@ -6,10 +6,22 @@
#include <string.h>
#include <stdbool.h>
// Detect if running under valgrind by checking /proc/self/maps for vgpreload.
// This is used to skip fork-based tests that are incompatible with valgrind
// (the instrumented parent runs too slowly, causing pipe timeouts).
// Reset the thread-local protocol descriptor redirection installed by
// io_set_fds(), so a suite that leaks a test pipe's fds cannot redirect a later
// suite's raw send_n_data()/receive_n_data() to the wrong descriptor.
void io_set_fds(int read_fd, int write_fd);
// Detect if running under valgrind. The CI valgrind step exports
// FASTSYNC_UNDER_VALGRIND=1; a /proc/self/maps scan is the fallback for a local
// valgrind run (newer valgrind versions can hide their own mappings from the
// guest, so the "vgpreload" match is not reliable on every version -- set
// FASTSYNC_UNDER_VALGRIND=1 when invoking valgrind by hand). Used to skip
// fork-based tests incompatible with valgrind, whose instrumented parent runs
// too slowly and causes pipe timeouts.
static inline bool is_running_under_valgrind(void) {
const char* env = getenv("FASTSYNC_UNDER_VALGRIND");
if (env && env[0] != '\0' && strcmp(env, "0") != 0)
return true;
FILE* f = fopen("/proc/self/maps", "r");
if (!f)
return false;
@@ -31,6 +43,7 @@ extern bool current_test_failed;
printf("Running %s...\n", #test_func); \
tests_run++; \
current_test_failed = false; \
io_set_fds(-1, -1); \
test_func(); \
if (current_test_failed) { \
tests_failed++; \
+2 -1
View File
@@ -210,7 +210,8 @@ static void test_xattr_receive_drops_acl_without_preserve_acls() {
/* MINOR-2: a --link-dest / -H copy fallback (linkat refused) must still apply
* the per-file xattrs and --fake-super stat. A DIRECTORY basis forces linkat
* to fail with EPERM, exercising the byte-copy fallback deterministically.
* to fail with EPERM, exercising the byte-copy fallback deterministically (the
* basis is not a regular file, so the fallback uses the caller's bytes).
* Guarded on filesystem xattr support. */
static void test_link_copy_fallback_preserves_xattrs() {
const char* dest = "test_link_xattr_dest.txt";