Pure structural split of src/shared/file_receive.c into focused translation
units behind the unchanged file_receive.h facade:
- file_save.c : save-to-disk, special nodes, --delay-updates staging
- incremental_check.c : xattr/delta/basis/fuzzy receive + check state machine
- delete_commit.c : manifest receive + delete budget walkers
- file_receive.c : wire receive dispatch + deferred dir metadata
The shared receive_file_xattrs helper and MAX_FILE_DATA_SIZE are declared in
incremental_check.h. file_save_to_disk_full_ex is decomposed into static
helpers (validation, special dispatch, dir/symlink creation, path resolution,
pre-write policies, data install) routed through one cleanup epilogue.
No behavior change.
Move the filter rule-tree/context helpers and entry inspection into
scanner_filter.c, the parallel scanner into scanner_parallel.c, and keep
the sequential scanner in scanner.c. Shared internal declarations live in
the new scanner_internal.h; scanner.h stays the public façade.
Decompose directory_scanner_next into static helpers (skipped-entry,
selection-protection, mount/-x, directory-finish and per-entry handlers)
with no semantic change.
Move the stats/progress reporting, scanner-preparation/scan helpers and
manifest/list/dry-run senders out of the ~3.9k-line client_send.c into
client_report.c, client_scan.c and client_manifest.c, sharing declarations
through the new internal client_send_internal.h. client_send.c keeps the
transfer orchestration and is now ~2.1k lines.
Decompose the monolithic send_files into static phase helpers
(send_files_prepare/_prepare_delete/_run/_finalize/_cleanup) driven by a
single SendFilesState; ownership, ordering and exit codes are unchanged.
No behavior change.
Update the docs for the audit follow-up fixes:
- --filter merge modifiers e/n/w/- are now accepted-and-consumed on
merge/dir-merge rules (rejected on non-merge, x rejected everywhere);
their semantics stay unimplemented, so the --filter row moves to Caveat
and the tally becomes 119/11/27 = 157.
- --inplace + --partial-dir is rejected with rsync's message.
- secret_file_open() O_NOFOLLOW (symlinked credential paths fail closed;
fd-backed paths exempt) and ~3 s bound-wait on FIFO reads.
- AGENTS setpriv wording corrected to the collected instance count.
- CHANGELOG [Unreleased] audit section extended with the follow-ups.
The earlier modifier-rejection change rejected e/n/w on all rules, but rsync
3.4.1 accepts them (plus the '-' merge-only modifier) on merge and dir-merge
rules. Restrict the rejection to non-merge rules and consume the merge-file
modifiers (e/n/w/-) so they no longer leak into the merge filename.
- is_merge_rule()/is_merge_modifier_char() gate the merge-only modifiers.
- scan vs consume sets: e/n/w still count as modifier-run chars on every rule
(pure tokens like -new/-press stay rejected), but are only consumed on merge
rules, preserving mixed-token parsing such as H,!secret -> ecret.
- '-' is accepted/consumed only on merge/dir-merge (e.g. dir-merge,- .rules).
- x remains rejected everywhere with its dedicated message.
- e/n/w/- semantics remain unimplemented and are documented as accepted-but-
ignored in filter.h.
Tests: split the merge forms out of the rejection test into a new acceptance
test asserting the merge file is read and dir_merge_names keeps the modifier-
free basename; non-merge pure-modifier forms still rejected.
secret_file_open() opened secret files with O_NONBLOCK and only cleared it
for S_ISREG, so on a FIFO/process-substitution source (--password-file
<(...), --early-input <(...)) fgets() failed immediately with EAGAIN when
the writer had not yet produced data, breaking slow producers.
Keep O_NONBLOCK at open() (a writer-less FIFO must not block the open) and
route all three readers through a new secret_read_line() helper. It
accumulates a line across reads and, on EAGAIN/EWOULDBLOCK (or a partial
line) with no newline and no EOF, clearerr()s and polls for readability
against one overall CLOCK_MONOTONIC deadline of
CREDENTIAL_FIFO_READ_TIMEOUT_MS (3000 ms); on timeout or a real read error
it fails with a clear message. EOF finishes normally. Regular files are
left blocking and read exactly as before.
Handles a line split across several write()s and keeps the owner/mode
fstat gate, O_NOFOLLOW and the /dev/fd/N exception unchanged.
secret_file_open() previously opened --password-file/--early-input with
plain O_RDONLY, so a symlinked path was followed before the owner/mode
fstat gate ran, and an empty/planted FIFO could block fgets forever.
Open with O_NOFOLLOW|O_NONBLOCK|O_CLOEXEC (mirroring the dummy-key
sidecar): ELOOP now fails closed, and a writer-less FIFO yields EOF/EAGAIN
instead of hanging. Clear O_NONBLOCK again for regular files, where it is
a no-op, so their stdio read path is unchanged.
file.c: drop the duplicate <fcntl.h>/<unistd.h> includes (kept the first
occurrences).
Tests: a symlinked password file is rejected, and a writer-less named
FIFO fails cleanly without hanging.
Client: replace the non-async-signal-safe signal(3) call inside
client_signal_handler() with a precomputed SIG_DFL sigaction(2), which is
on the POSIX async-signal-safe list. The handler stays installed while a
transfer is armed so a repeated Ctrl-C still leads to a graceful abort
rather than a hard kill mid-cleanup.
Server: cleanup() now only calls _exit(2) (async-signal-safe). The former
server_delete()/daemon_conf_free()/credentials_free() teardown called
free()/close()/SSL_CTX_free() from signal context, which can deadlock or
corrupt the heap if the signal lands inside malloc/free. Handlers are
installed with sigaction(2) instead of signal(3). The normal shutdown
path in main() still performs the full teardown; the signal path relies on
process exit to reclaim the parent daemon's socket, anonymous shared
mapping and heap (no named/persistent parent resource is left behind).
The standalone filter_rule_parse() already rejected the rsync xattr-name
'x' modifier, but the list parser used by --filter/-f silently dropped the
flag for merge/dir-merge rules (and relied on a second parse for plain
rules). Reject it explicitly in filter_list_parse_append_depth() with the
same diagnostic, so '-x', 'merge,x' and 'dir-merge,x' all fail cleanly.
Also reject the unimplemented rsync merge modifiers 'e', 'n' and 'w'
instead of folding them into the pattern, which previously produced
misleading errors such as "could not read merge file 'n file'". Only a
token made up solely of modifier characters is treated as a modifier run,
so glued patterns ('-newfile', '-e2e') and mixed tokens ("H,!secret")
keep their historical parsing.
Adds tests/test_filter.c with focused rejection and supported-syntax
cases.
- multiprocessing: destroy mutex_progress on the dir_entries_mutex
init-failure path (init >= 7); drop bogus log_perror
- delete_plan: capture errno before free() in apply_deferred_path
- queue/array_list: log_message instead of log_perror for non-errno
conditions
- protocol: reject unknown wire Status values via status_is_valid() in
receive_status, receive_status_timed and the keepalive reader; declare
protocol_receive_status_timed in protocol.h
- protocol: %llu for unsigned long long debug counters
- tests: out-of-range status rejection test
Three receiver security fixes from the audit:
1. --temp-dir symlink escape (High): file_open_temp_dir() opened the
client-controlled scratch dir with a bare open(), so a symlink planted
under the receive root let a peer redirect receiver scratch files
outside the authorized root. The opened dir is now judged by the REAL
path of its fd (via /proc/self/fd), and any target outside the
authorized receive root is refused with a logged error (EACCES). An
in-root symlink (the EXDEV cross-filesystem fallback case) still works,
and the no-root local batch path is unchanged.
2. setuid/setgid/sticky under SUPER_MODE_OFF (High): the special bits were
applied under --perms (and via --chmod) even when the connection forbade
super-user activities. FileAttrPolicy gains super_permitted, set by
file_attr_policy_from_config() from privilege_super_mode_permitted();
metadata_mode_for_policy(), the symlink path, the special-node creation
path, and the deferred directory-mode apply now strip the special bits
when it is false. Exact rsync semantics are preserved when permitted.
3. daemon umask (Low): daemonize() forced umask(0), so implied parent
directories created without -p were world-writable 0777. Set the
conventional daemon umask 022 instead (rsync never forces 0); -p/-a mode
preservation is unaffected because it restores modes via fchmod.
Tests: new unit tests for file_open_temp_dir confinement and the
masked/unmasked special-bit policy (incl. the --chmod path), a daemon
world-writable-dir regression test, an integration escape test, and a
root-only integration test asserting special bits are masked without
--allow-super. The old cross-filesystem test encoded the vulnerable
behavior (symlink target outside the root) and is replaced by the escape
test; the EXDEV fallback code is retained for in-root links.
rsync 3.4.1 resolves --partial-dir after option parsing and sets keep_partial,
so --partial-dir=DIR alone retains an interrupted transfer's partial file.
FastSync only used the partial dir when --partial was also given, silently
discarding it otherwise.
Set Config->partial in cli_finalize_config whenever partial_dir is set.
Following rsync, an explicit --no-partial does NOT win (verified on rsync
3.4.1 in either option order); --inplace is guarded because it writes the
destination in place with no partial staging.
Tests: CLI unit coverage for the implication/precedence/inplace guard, and a
deterministic integration case that blocks the final install (non-empty
directory at the destination) and asserts the staged partial survives under
--partial-dir alone.
MAX_DECOMPRESSED_SIZE was 100 MiB while the receiver advertises and the
sender compresses whole files up to MAX_RECEIVE_WHOLE_FILE_SIZE (256 MiB),
so -z on a 100-256 MiB regular file failed with 'Declared decompressed
size exceeds 104857600 bytes'. Define the internal bomb-guard ceiling in
terms of the protocol constant so the two bounds cannot drift, and add
unit coverage for a 130 MiB payload (accepted) and an over-ceiling
declared size (still rejected).
--info=mount now prints rsync's mount-point skip line (matching rsync
3.4.1, which emits it for repeated -xx and drops the mount-point dir);
--info=stats enables the same block as --stats; -x is repeatable.
--stats counts traversed directories for the Number of files breakdown
even when no directory metadata is captured (-r without -t/-p).
--debug enables real output for flist/del/hash/deltasum/recv/filter/send
at their natural FastSync events (synthetic categories stay inert).
--stats and --debug rows keep their documented residual status.
Transmit the complete --delete-during/--delete-delay per-directory plan
set before the first data frame, so a mid-transfer abort has already
applied every planned removal like rsync's generator; completed runs are
unchanged. Route -d/--dirs through the same per-directory plans: the
generator records only directories whose direct children it enumerated,
so extras directly inside a listed directory are removed while an
untraversed subdirectory's mirror is shielded (rsync's -d DIR/ --delete).
Also shields a -x mount point's untraversed destination content.
Buffer and sort each directory's inspected entries (non-directories
ascending, then directories ascending) and walk them depth-first via a
LIFO directory stack, so the sequential scanner's stream matches rsync
3.4.1's flist order. This makes the --info=name transfer order and the
--delete-during/--delete-delay deletion sequence byte-identical to rsync
(differential tests in test_parity_order.py); --threads stays unordered
(no rsync analogue) and is documented as such.
Adds LIFO queue_push/queue_pop over the existing ring buffer.
Resolve a relative --compare-dest/--copy-dest/--link-dest DIR against the
destination directory and append the file's transfer-relative name, as
rsync 3.4.1 does, instead of appending FastSync's source-mirrored wire
path (the historical spelling stays as a fallback for existing layouts).
Stop inheriting the ordinary delta engine's 16 KiB minimum and 10x size
ratio in the -y/--fuzzy candidate search: rsync's find_fuzzy has no
delta-size gate, so an oversized or sub-16-KiB sibling is now reused.
The ordinary delta path's bounds are unchanged.
Collect each directory's entries up front and process extraneous
subdirectories first (descending name, depth-first), then extraneous
files (descending name), then descend into kept subdirectories in
ascending order. Emit a trailing slash for deleted directories in
observers/dry-run output. This matches rsync's delete order for
--delete-before/--delete-after/--delete-delay and for dry-run listings.
The parent sends the file data body after the receiver's STATUS_NEXT, but
the forked child exited as soon as receive_incremental_check returned. The
parent's send_data could then race the child's exit into a spurious EPIPE
(seen in the coverage job as test_server.c:1222), or the reverse: the parent
could be descheduled past the child's exit.
Keep the child alive until the parent closes its write end (drain to EOF),
and close the parent's write end before waitpid so the child can observe EOF.
Applied to the three tests sharing the pattern: size-mismatch, FIFO
destination, and FIFO basis. Child exit status remains the authoritative
assertion.
Adds a unit test proving the config-only STATUS_DELETE_PLAN frame applies
--delete-missing-args exact deletions while walking no directory (the
--files-from-with-no-synced-dir fix).
- receiver stats: exclude basis-dir materializations (--link-dest/--copy-dest)
from created/literal tallies; rsync reports 0 for a basis hit, so a fresh
--link-dest --stats run now matches (differential test_link_dest_stats_matches_rsync)
- filter wire block: reject a pattern above the glob evaluation bound and lower
MAX_FILTER_RULES to 1024, so a crafted rule list cannot amplify delete-walk
glob work or install a rule that silently never protects
- negative tests for over-cap count and over-long pattern
- README: correct --delete default, --stats/--progress description, add
--delete-commit; RSYNC_COMPAT stale version labels/overclaim fixed
- plain --delete with no timing flag now selects delete-during (progressive
deletion, matching rsync and avoiding the full old+new tree peak)
- new long-only FastSync --delete-commit restores the old atomic behavior
(delete only after the whole transfer succeeds); timing-identical to
--delete-after, implemented via the same wire bool
- timing flags are mutually exclusive; --delete-commit conflicts with other
timings; --delete-before/--delete-during rows reworded per Phase-0 probes
- CHANGELOG migration note; tally unchanged 116/14/27
Probe shows the candidate choice is observable only as --stats bandwidth
counters (tree/exit always identical). FastSync already uses rsync's
fuzzy_distance/find_filename_suffix name heuristic; the residual is the
narrower delta eligibility window (>=16 KiB, <=10x) vs rsync's wider one.
Row -> caveat; tally 116/14/27.
Adds the exclude_protect_dest_only differential and flips --delete-excluded
to parity now that receiver-side rules protect a destination-only excluded
entry like rsync; tally 116/10/31.
- new bounded config-wire block (BLOCK_PROTECT_RULES) serializes the sender's
compiled filter rules to the receiver (bounded count + 256 KiB patterns;
strict action/sides validation)
- receiver evaluates protect/risk in the whole-tree extras walk and the
per-directory delete plans, so a dest-only entry matching 'P' is kept like
rsync; dry-run would-delete enumeration also honours it
- --filter flips to parity (115/11/31); per-dir merge receiver re-derivation
remains the documented residual
Probe shows the block-checksum choice is not observable in the parity surface:
%c, Matched/Literal data and the destination tree are invariant across
xxh64/xxh128/xxh3/md5/md4/sha1 and the transfer,pre-transfer form; only %C
changes, and it is byte-identical to rsync. FastSync's fixed xxHash32 block
strong sum is collision-safe within the payload cap. Row -> parity (114/11/32).
- when --progress/--info=progress is requested, a metadata-only pre-scan
builds the full file-list total and directory names so the to-chk
denominator counts every regular/dir/link/special entry like rsync
- per-directory/symlink/special name lines emitted; sequential and --threads
- reuses the --delete-during/delay pre-scan when present; non-progress runs
take no extra pass
- --progress stays a caveat (emission order still differs); single-file output
remains byte-identical
- PROTOCOL_VERSION 2.27.0 -> 2.28.0; STATUS_STATS gains literal_bytes and
the created reg/dir/link/special counters (golden wire updated)
- receiver reports which destination entries it newly created, including
implicitly-created parent directories below the logical transfer root, so
Number of created files carries rsync's per-type breakdown
- Literal data is now exact for a delta transfer (receiver counts the literal
fragments it stored)
- differential-tested vs rsync 3.4.1 for fresh-create, update and delta
- -n/--delete sends the same protected/size-skipped/scope as a real run, so
the read-only would-delete walk no longer over-reports (row -> caveat)
- --delete-delay charges --max-delete on actual removals and recursively
re-scans a refilled deferred directory at commit; independent deferred cap
- --info=name emits the leading ./ root line and name2 'is uptodate' lines
- differential tests promoted from residual pins to rsync parity assertions
The max_delete case allowlisted the tree aspect because the surviving extras after a partial --max-delete abort are deletion-order dependent. Under FASTSYNC_PARITY_STRICT a run where the orders coincide was reported as a stale entry (CI failure), while removing the entry made the order-dependent tree mismatch fail. Add a per-case 'compare_tree' flag: max_delete now asserts rc=25 plus the survivor count via extra_check instead of exact tree identity, so the allowlist can be empty. Burn-down reached zero.
- --delete-delay: state that the reported count advances on actual removal
while --max-delete is charged at plan/snapshot time (defer_add/planned).
A new differential shows rsync instead charges on actual removals and
recursively removes a queued directory, so the row moves to Caveat
(matrix 111/13/33) and the residual is pinned by tests.
- Add a deterministic unit test (plan-time budget charge), a FastSync
integration test (byte-barrier refill + --max-delete), and an rsync
differential for a refilled deferred directory.
- Soften the --fuzzy summary: the tree is byte-exact by design, so it is
pinned by the threshold suite, not a byte-level differential.
- README: describe what -m parity actually selects; fix the allowlist
example to the real max_delete entry.
- xdist-safe delete-timing fixture names (timing and --threads mode).
- Renumber the duplicate HANDOFF item 9 to 10; add the missing final
newline to test_checksum.c.
- Expose ignore_errors_allows_delete and unit-test the deletion gate
without a privileged source directory; update the stale deleted-count
doc comment.
No production behavior changes.
- cli: remove UB in --bwlimit scaling (range-check the double product before
casting, drop atoi for the +/-1 form) and add huge/boundary unit tests
- test: widen the CI throttle wall-clock band to [1.5, 4.5]s with a 2s
cross-tolerance so a loaded runner cannot flake it
- log: drop the unused LOG_INFO_BACKUP bit; --info=backup is accepted-but-
silent like the other rsync-only categories
- client_send: remove the duplicate delete_display_path forward declaration
- utils: add non-allocating utils_strip_transfer_root and use it from
scanner_note_nonreg and delete_display_path (was duplicated logic)
- scanner: lstat() instead of stat() when re-reading an empty dir's metadata
- file: drop the no-op else-if and the redundant ELOOP arm in
file_ensure_directory_secure (symlinks are refused anyway)
- format/stats: document literal_data as whole-file accurate (delta upper
bound) instead of claiming literal bytes sent
- docs: refresh stale protocol 2.26.0 labels to 2.27.0
- Initialize PipelineContextSender.delete_suppressed=false: an uninitialized
true silently skipped the --delete keep-set manifest under -m/--threads,
so destination extras were never removed.
- Allocate/install the receiver deleted-path observer only when
report_deletes is set (--info=del / -i / --out-format under --delete), cap
the retained list at MAX_MANIFEST_ENTRIES, and free already-created lists
on the receiver-pipeline create failure path.
- Validate report_deletes/report_stats/report_dest_info on receive.
- Correct stale comments (config.h report_deletes, delete_plan.h deleted
count, multiprocessing.h stats locking, utils.h observer placement).
- Tests: sender delete_suppressed init, report_deletes gating (unit), and
-m/--delete default delete-after keep-set removal (integration).
- Wire: config frame gains report_deletes (protocol 2.26.0 -> 2.27.0); the
receiver lists actually-removed paths in the STATUS_STATS path list, so the
sender prints rsync's `deleting PATH` / `*deleting PATH` lines for a real
--delete run (and -i/out-format). Observers threaded through the manifest,
missing-args and per-directory delete engines; golden wire len/hash updated.
- bwlimit: throttle now paces like rsync 3.4.1 -- ~100ms burst capacity and the
sleep is no longer credited as refill, so 4 MiB at 1024/2048 KiB/s matches
rsync within ~4% (was ~2x too fast).
Each row's exact residual reproduced against rsync 3.4.1:
- basis dirs (--compare/copy/link-dest): FastSync xxHash-verifies a basis hit
while rsync --size-only installs the wrong same-size basis content.
- --delay-updates: the fixed .fastsync-stage name wipes an unrelated
destination entry of that name even without --delete; rsync leaves it.
- --fuzzy: deterministic name/size heuristic (10x window), not rsync's matcher.
- --dry-run: would-delete report over-reports the updated file and an
excluded-but-protected extra, and ordering differs.
Docs: RSYNC_COMPAT tally 109/16/31; HANDOFF item 9. clang-format + cppcheck +
ASan + full suite clean.
- --bwlimit: faithful port of rsync 3.4.1 parse_size_arg (default KiB/s,
binary K/M/G/T/P, decimal KB/MB, KiB/MiB, decimals, 0 = unlimited, 512-byte
floor, (size+512)/1024 quantization). Unit tests + docs.
- --info: wire del/remove/name/flist/nonreg/progress to real FastSync events in
rsync's line format (deleting PATH, sender removed NAME, name lines,
'sending incremental file list', skipping non-regular file "NAME"); name no
longer aliases copy; --info=progress drives the progress path and report_stats.
- --ignore-errors: match rsync's default -- a source I/O error skips deletion
unless --ignore-errors, while the readable tree still transfers and the run
exits 23. Covers all delete timings and both send paths.
TapTap