Three receiver security fixes from the audit:
1. --temp-dir symlink escape (High): file_open_temp_dir() opened the
client-controlled scratch dir with a bare open(), so a symlink planted
under the receive root let a peer redirect receiver scratch files
outside the authorized root. The opened dir is now judged by the REAL
path of its fd (via /proc/self/fd), and any target outside the
authorized receive root is refused with a logged error (EACCES). An
in-root symlink (the EXDEV cross-filesystem fallback case) still works,
and the no-root local batch path is unchanged.
2. setuid/setgid/sticky under SUPER_MODE_OFF (High): the special bits were
applied under --perms (and via --chmod) even when the connection forbade
super-user activities. FileAttrPolicy gains super_permitted, set by
file_attr_policy_from_config() from privilege_super_mode_permitted();
metadata_mode_for_policy(), the symlink path, the special-node creation
path, and the deferred directory-mode apply now strip the special bits
when it is false. Exact rsync semantics are preserved when permitted.
3. daemon umask (Low): daemonize() forced umask(0), so implied parent
directories created without -p were world-writable 0777. Set the
conventional daemon umask 022 instead (rsync never forces 0); -p/-a mode
preservation is unaffected because it restores modes via fchmod.
Tests: new unit tests for file_open_temp_dir confinement and the
masked/unmasked special-bit policy (incl. the --chmod path), a daemon
world-writable-dir regression test, an integration escape test, and a
root-only integration test asserting special bits are masked without
--allow-super. The old cross-filesystem test encoded the vulnerable
behavior (symlink target outside the root) and is replaced by the escape
test; the EXDEV fallback code is retained for in-root links.
--info=mount now prints rsync's mount-point skip line (matching rsync
3.4.1, which emits it for repeated -xx and drops the mount-point dir);
--info=stats enables the same block as --stats; -x is repeatable.
--stats counts traversed directories for the Number of files breakdown
even when no directory metadata is captured (-r without -t/-p).
--debug enables real output for flist/del/hash/deltasum/recv/filter/send
at their natural FastSync events (synthetic categories stay inert).
--stats and --debug rows keep their documented residual status.
Transmit the complete --delete-during/--delete-delay per-directory plan
set before the first data frame, so a mid-transfer abort has already
applied every planned removal like rsync's generator; completed runs are
unchanged. Route -d/--dirs through the same per-directory plans: the
generator records only directories whose direct children it enumerated,
so extras directly inside a listed directory are removed while an
untraversed subdirectory's mirror is shielded (rsync's -d DIR/ --delete).
Also shields a -x mount point's untraversed destination content.
Buffer and sort each directory's inspected entries (non-directories
ascending, then directories ascending) and walk them depth-first via a
LIFO directory stack, so the sequential scanner's stream matches rsync
3.4.1's flist order. This makes the --info=name transfer order and the
--delete-during/--delete-delay deletion sequence byte-identical to rsync
(differential tests in test_parity_order.py); --threads stays unordered
(no rsync analogue) and is documented as such.
Adds LIFO queue_push/queue_pop over the existing ring buffer.
Resolve a relative --compare-dest/--copy-dest/--link-dest DIR against the
destination directory and append the file's transfer-relative name, as
rsync 3.4.1 does, instead of appending FastSync's source-mirrored wire
path (the historical spelling stays as a fallback for existing layouts).
Stop inheriting the ordinary delta engine's 16 KiB minimum and 10x size
ratio in the -y/--fuzzy candidate search: rsync's find_fuzzy has no
delta-size gate, so an oversized or sub-16-KiB sibling is now reused.
The ordinary delta path's bounds are unchanged.
- receiver stats: exclude basis-dir materializations (--link-dest/--copy-dest)
from created/literal tallies; rsync reports 0 for a basis hit, so a fresh
--link-dest --stats run now matches (differential test_link_dest_stats_matches_rsync)
- filter wire block: reject a pattern above the glob evaluation bound and lower
MAX_FILTER_RULES to 1024, so a crafted rule list cannot amplify delete-walk
glob work or install a rule that silently never protects
- negative tests for over-cap count and over-long pattern
- README: correct --delete default, --stats/--progress description, add
--delete-commit; RSYNC_COMPAT stale version labels/overclaim fixed
- plain --delete with no timing flag now selects delete-during (progressive
deletion, matching rsync and avoiding the full old+new tree peak)
- new long-only FastSync --delete-commit restores the old atomic behavior
(delete only after the whole transfer succeeds); timing-identical to
--delete-after, implemented via the same wire bool
- timing flags are mutually exclusive; --delete-commit conflicts with other
timings; --delete-before/--delete-during rows reworded per Phase-0 probes
- CHANGELOG migration note; tally unchanged 116/14/27
Probe shows the candidate choice is observable only as --stats bandwidth
counters (tree/exit always identical). FastSync already uses rsync's
fuzzy_distance/find_filename_suffix name heuristic; the residual is the
narrower delta eligibility window (>=16 KiB, <=10x) vs rsync's wider one.
Row -> caveat; tally 116/14/27.
Adds the exclude_protect_dest_only differential and flips --delete-excluded
to parity now that receiver-side rules protect a destination-only excluded
entry like rsync; tally 116/10/31.
- new bounded config-wire block (BLOCK_PROTECT_RULES) serializes the sender's
compiled filter rules to the receiver (bounded count + 256 KiB patterns;
strict action/sides validation)
- receiver evaluates protect/risk in the whole-tree extras walk and the
per-directory delete plans, so a dest-only entry matching 'P' is kept like
rsync; dry-run would-delete enumeration also honours it
- --filter flips to parity (115/11/31); per-dir merge receiver re-derivation
remains the documented residual
Probe shows the block-checksum choice is not observable in the parity surface:
%c, Matched/Literal data and the destination tree are invariant across
xxh64/xxh128/xxh3/md5/md4/sha1 and the transfer,pre-transfer form; only %C
changes, and it is byte-identical to rsync. FastSync's fixed xxHash32 block
strong sum is collision-safe within the payload cap. Row -> parity (114/11/32).
- when --progress/--info=progress is requested, a metadata-only pre-scan
builds the full file-list total and directory names so the to-chk
denominator counts every regular/dir/link/special entry like rsync
- per-directory/symlink/special name lines emitted; sequential and --threads
- reuses the --delete-during/delay pre-scan when present; non-progress runs
take no extra pass
- --progress stays a caveat (emission order still differs); single-file output
remains byte-identical
- PROTOCOL_VERSION 2.27.0 -> 2.28.0; STATUS_STATS gains literal_bytes and
the created reg/dir/link/special counters (golden wire updated)
- receiver reports which destination entries it newly created, including
implicitly-created parent directories below the logical transfer root, so
Number of created files carries rsync's per-type breakdown
- Literal data is now exact for a delta transfer (receiver counts the literal
fragments it stored)
- differential-tested vs rsync 3.4.1 for fresh-create, update and delta
- -n/--delete sends the same protected/size-skipped/scope as a real run, so
the read-only would-delete walk no longer over-reports (row -> caveat)
- --delete-delay charges --max-delete on actual removals and recursively
re-scans a refilled deferred directory at commit; independent deferred cap
- --info=name emits the leading ./ root line and name2 'is uptodate' lines
- differential tests promoted from residual pins to rsync parity assertions
The max_delete case allowlisted the tree aspect because the surviving extras after a partial --max-delete abort are deletion-order dependent. Under FASTSYNC_PARITY_STRICT a run where the orders coincide was reported as a stale entry (CI failure), while removing the entry made the order-dependent tree mismatch fail. Add a per-case 'compare_tree' flag: max_delete now asserts rc=25 plus the survivor count via extra_check instead of exact tree identity, so the allowlist can be empty. Burn-down reached zero.
- --delete-delay: state that the reported count advances on actual removal
while --max-delete is charged at plan/snapshot time (defer_add/planned).
A new differential shows rsync instead charges on actual removals and
recursively removes a queued directory, so the row moves to Caveat
(matrix 111/13/33) and the residual is pinned by tests.
- Add a deterministic unit test (plan-time budget charge), a FastSync
integration test (byte-barrier refill + --max-delete), and an rsync
differential for a refilled deferred directory.
- Soften the --fuzzy summary: the tree is byte-exact by design, so it is
pinned by the threshold suite, not a byte-level differential.
- README: describe what -m parity actually selects; fix the allowlist
example to the real max_delete entry.
- xdist-safe delete-timing fixture names (timing and --threads mode).
- Renumber the duplicate HANDOFF item 9 to 10; add the missing final
newline to test_checksum.c.
- Expose ignore_errors_allows_delete and unit-test the deletion gate
without a privileged source directory; update the stale deleted-count
doc comment.
No production behavior changes.
- cli: remove UB in --bwlimit scaling (range-check the double product before
casting, drop atoi for the +/-1 form) and add huge/boundary unit tests
- test: widen the CI throttle wall-clock band to [1.5, 4.5]s with a 2s
cross-tolerance so a loaded runner cannot flake it
- log: drop the unused LOG_INFO_BACKUP bit; --info=backup is accepted-but-
silent like the other rsync-only categories
- client_send: remove the duplicate delete_display_path forward declaration
- utils: add non-allocating utils_strip_transfer_root and use it from
scanner_note_nonreg and delete_display_path (was duplicated logic)
- scanner: lstat() instead of stat() when re-reading an empty dir's metadata
- file: drop the no-op else-if and the redundant ELOOP arm in
file_ensure_directory_secure (symlinks are refused anyway)
- format/stats: document literal_data as whole-file accurate (delta upper
bound) instead of claiming literal bytes sent
- docs: refresh stale protocol 2.26.0 labels to 2.27.0
- Initialize PipelineContextSender.delete_suppressed=false: an uninitialized
true silently skipped the --delete keep-set manifest under -m/--threads,
so destination extras were never removed.
- Allocate/install the receiver deleted-path observer only when
report_deletes is set (--info=del / -i / --out-format under --delete), cap
the retained list at MAX_MANIFEST_ENTRIES, and free already-created lists
on the receiver-pipeline create failure path.
- Validate report_deletes/report_stats/report_dest_info on receive.
- Correct stale comments (config.h report_deletes, delete_plan.h deleted
count, multiprocessing.h stats locking, utils.h observer placement).
- Tests: sender delete_suppressed init, report_deletes gating (unit), and
-m/--delete default delete-after keep-set removal (integration).
- Wire: config frame gains report_deletes (protocol 2.26.0 -> 2.27.0); the
receiver lists actually-removed paths in the STATUS_STATS path list, so the
sender prints rsync's `deleting PATH` / `*deleting PATH` lines for a real
--delete run (and -i/out-format). Observers threaded through the manifest,
missing-args and per-directory delete engines; golden wire len/hash updated.
- bwlimit: throttle now paces like rsync 3.4.1 -- ~100ms burst capacity and the
sleep is no longer credited as refill, so 4 MiB at 1024/2048 KiB/s matches
rsync within ~4% (was ~2x too fast).
Each row's exact residual reproduced against rsync 3.4.1:
- basis dirs (--compare/copy/link-dest): FastSync xxHash-verifies a basis hit
while rsync --size-only installs the wrong same-size basis content.
- --delay-updates: the fixed .fastsync-stage name wipes an unrelated
destination entry of that name even without --delete; rsync leaves it.
- --fuzzy: deterministic name/size heuristic (10x window), not rsync's matcher.
- --dry-run: would-delete report over-reports the updated file and an
excluded-but-protected extra, and ordering differs.
Docs: RSYNC_COMPAT tally 109/16/31; HANDOFF item 9. clang-format + cppcheck +
ASan + full suite clean.
- --bwlimit: faithful port of rsync 3.4.1 parse_size_arg (default KiB/s,
binary K/M/G/T/P, decimal KB/MB, KiB/MiB, decimals, 0 = unlimited, 512-byte
floor, (size+512)/1024 quantization). Unit tests + docs.
- --info: wire del/remove/name/flist/nonreg/progress to real FastSync events in
rsync's line format (deleting PATH, sender removed NAME, name lines,
'sending incremental file list', skipping non-regular file "NAME"); name no
longer aliases copy; --info=progress drives the progress path and report_stats.
- --ignore-errors: match rsync's default -- a source I/O error skips deletion
unless --ignore-errors, while the readable tree still transfers and the run
exits 23. Covers all delete timings and both send paths.
- --iconv now matches rsync's push direction: the destination charset is the
client spec's REMOTE half, so a default receiver writes wire names verbatim;
a server's own --iconv LOCAL overrides it (daemon charset analog). Updated
unit + integration tests and added a default-server differential gate case.
- Empty-directory emission is gated behind a new ScannerOptions.emit_empty_dirs
set only by the real sender, so low-level scanner helpers keep the historical
file-only list.
- --temp-dir reclassified to Divergent: relative dirs match rsync exactly
(resolved under the destination), but an absolute path is deliberately
rejected by the confined receiver; differential test added.
- Docs/tally: 109 Parity / 22 Caveat / 25 Divergent.
- Recursive scans emit a directory entry for every traversed directory that
produced no transferred child, so empty source dirs (and dirs emptied by
filtering) are recreated like rsync; -m prunes them, --files-from/--list-only
never emit implicit dirs.
- A directory entry now replaces a destination regular file (rsync removes the
non-directory) instead of aborting; confined to the secure parent fd.
- -R --no-implied-dirs --files-from: stop refusing a listed file whose parent
is not listed; create the implied parent with default attributes (no source
metadata is captured for it), matching rsync 3.4.1.
- Differential gate: drop min_size/empty_dirs_recursive/dirs_plain allowlist
entries (dirs_plain now hands FastSync the same trailing-slash source as
rsync); add differential test for the files-from implied parent.
- Docs: -d row -> Parity, tally 108/24/24.
No wire change (PROTOCOL_VERSION stays 2.26.0).
- --delete-delay: count/track only entries actually removed; a directory
refilled before commit (ENOTEMPTY) no longer inflates Number of deleted
files or the --max-delete budget (unit + integration + rsync differential).
- --stats: per-type Number of files breakdown; only stored regular files
count as transferred; transferred/literal byte totals and Total file size
(symlink target lengths) now match rsync for whole-file transfers.
- --progress: print the leading ./ root line and include the root entry in
the to-chk denominator (single-file output byte-identical to rsync).
- --out-format %C: use the selected transfer checksum and render every
algorithm exactly like rsync; checksum_digest_file gains md4/sha1/none.
Reclassify --out-format to Divergent (protocol-specific %b/delta-%c).
- Docs: RSYNC_COMPAT tally 107/25/24, HANDOFF update. No wire change.
Add tests/integration/test_differential_parity.py plus a shared
parity_harness.py and a data-driven parity_caveats.py allowlist. The gate
runs real rsync 3.4.1 and FastSync over the same corpora, compares the
destination trees (paths, hashes, symlink targets, modes, hard-link
grouping) and the normalized -i/--stats/--out-format output, and fails on
any difference not listed in the allowlist. Stale allowlist entries warn
(or fail under FASTSYNC_PARITY_STRICT=1) so the residual list shrinks.
Register parity/parity_ci markers and wire a fast PR job (parity_ci) plus a
push-only full job (parity, strict) into .gitea/workflows/ci.yaml.
Document the gate and the allowlist workflow in tests/integration/README.md.
Add test_out_format_c_delta_mode_divergence documenting that FastSync's
delta %c (its own handshake bytes) cannot match rsync's (16-byte sum header
plus per-block checksums); only the whole-file case is aligned. The test
pins both values so a future parity improvement is noticed.
- Output parity: add `File list size` to the strict --stats differential and
document the row-#3 residual with test_stats_file_count_breakdown_residual
(rsync's `Number of files`/`Number of created files` type breakdown is not
reproducible from what the sender knows: no directory accounting and no
per-entry destination-created state). The row stays a caveat.
- Add --threads variants for the --stats and --progress/-P differentials.
The `-n --delete --threads` variant is a documented xfail: the threaded
dry-run path does not consume the receiver's STATUS_STATS delete list yet.
- Replace the 0.2s sleep flake in the delete-timing proxy with a socket
barrier: the hook now fires only after the server sends a reply (proving it
processed the preceding per-directory delete plan), using --incremental +
--ignore-times to guarantee a mid-transfer handshake reply.
Blockers addressed together (shared scanner/delete-plan plumbing):
* #10: an empty in-scope source directory produced no plan keep entry, so the
receiver deleted the destination directory itself. The scanner now records
every traversed directory into a delete-plan sink, the plan sender keeps them,
and any directory whose plan the data stream never triggered is emitted after
the data so its extras are still removed. Differential tests cover
--delete-during and --delete-delay.
* #8: an invalid per-directory filter file was silently ignored when an earlier
merge file in the same directory existed; key the failure off the error text
(both sequential and parallel scanners) and fail the scan.
* #9: -R + --files-from receiver-protect rules recorded the source-relative
path; record the bare relative wire path in both scanners so the protected
destination mirror survives --delete.
* #5: the STATUS_STATS would-delete parser now validates each retained path and
enforces the shared MAX_MANIFEST_BYTES budget, and the --out-format dry-run
delete line is escaped like the itemize line.
* #11: drop the unused DELETE_PLAN_MAX_NAMES macro, log the delete-limit
warning once per session, roll back dir-merge names from a per-directory file
that fails to parse, and guard every filter error snprintf against err==NULL.
#10 leaves the empty directory itself kept and its extras removed, matching
rsync's final state on both per-directory timings.
rsync's %c counts the block-checksum bytes received: even a whole-file
transfer with no basis receives rsync's 16-byte sum header (append and
inplace included), while a dry run receives nothing. FastSync's
whole-file path has no equivalent header, so report 16 for parity when
delta is inactive, keep 0 for dry runs, and keep the real received bytes
when delta is active (FastSync's signature framing differs, so delta %c
stays divergent). Add a strict %c/%l/%n differential against rsync and
turn the %b check into a real rsync differential (semantics: both count
wire bytes and exceed %l; the exact values are protocol-specific).
Accept the full rsync 3.4.1 --info (backup, del, flist, mount, nonreg,
progress, remove, symsafe) and --debug (acl, backup, bind, chdir, connect,
cmd, del, deltasum, dup, exit, filter, flist, fuzzy, genr, hash, hlink,
iconv, nstr, own, recv, send, time) vocabularies, plus the historical
syms/hl/owner aliases, with level suffixes. Categories FastSync already
emits (copy/name/misc/skip/stats and io/proto/pack/util) still set their
log flags; the rest are accepted but silent. Unknown names remain
rejected by name, matching rsync. Update the CLI unit tests and the
rsync-parity integration tests (previously they required del/filter to be
rejected).
- Assert FastSync's sparse/preallocate block accounting matches rsync 3.4.1
for a hole file (preallocate wins over sparse, exactly like rsync).
- Assert --ignore-existing is answered by the receiver before the sender
transmits the payload (CountingProxy wire volume near-zero).
- CountingProxy.run gains a bounded join_timeout so tests that only need the
client->server count do not wait for the server's idle socket.
- Fix the stale protocol 2.24.0 comment in test_config.c (golden is 2.26.0).
The single-threaded and -m receivers never populated ReceiverStats.matched_data
or .deleted_files, so --stats always printed 0 for both even when rsync
reported nonzero. Track the bytes reconstructed from the basis file while
applying a delta, and tally the delete-commit counts (manifest and
per-directory sessions) into the receiver stats. The -m pipeline now carries
its own stats/would-delete fields and emits the STATUS_STATS frame before the
terminal success, so --threads finally reports the counters and renders
-n --delete lines.
Also normalize the -n --delete would-delete enumeration's absolute basis
prefixes exactly like the real commit path (fixing an over-report) and fix the
basis_delete_relative off-by-one when the receive root is '/'. Unit tests
cover the root mapping and the basis protection; integration tests cover
matched/deleted stats for both receivers and the --threads dry-run delete
lines.
The -R prefix marker installed in synced_dirs was discarded when finalizing
the per-directory delete sender (--delete-during/--delete-delay), so the
up-front root plan was the receive root '.', whose keep list only held the
first prefix component. The receiver then deleted destination content
outside the transferred prefix (e.g. unrelated/keep.txt), a data-loss bug;
rsync keeps it.
Confine the walk to the -R prefix: send that prefix's plan as the root plan,
only transmit plans at or below it, and never emit the receive root plan for
a scoped run. Add a differential test covering both --delete-during and
--delete-delay.
TapTap