Extend _snapshot_tree to record mode, inode, xattrs, directories and
special nodes, and add coverage proving a server-contacting --dry-run
leaves the destination structurally identical for --delay-updates,
--backup, symlinks, hardlinks, FIFOs, and daemon modules (including a
read-only module). Add a regression test for the --read-batch --dry-run
refusal and for a missing/non-directory receive root failing a dry-run
exactly like a real run.
Fix stale version comments (2.20.0/633 -> 2.21.0/637) and RSYNC_COMPAT's
current --protocol value, and add a unit assertion that
--server-port/--port (and --server-host) set the dry-run routing bit.
Today a server rejection sends a bare STATUS_ERROR and the reason only
reaches the server log, so the client cannot say why a transfer was
refused. Add an optional, bounded server->client error-detail frame:
- Status gains STATUS_ERROR_DETAIL appended LAST so existing wire
values are unchanged.
- send_error_detail(fd, msg) sends STATUS_ERROR_DETAIL followed by the
existing length-prefixed string primitive, slicing over-long messages
to MAX_ERROR_DETAIL_BYTES (4096).
- receive_status() (and the timed/keepalive status readers) always
consume the detail body and map the status back to STATUS_ERROR,
capturing the text into a thread-local buffer exposed by
protocol_last_error(); a bare STATUS_ERROR leaves it cleared. Every
existing call site keeps working and the stream cannot desync.
- Upgrade the daemon module gate / config validation (config.c), the
final transfer failure (server.c) and receiver-side path/node
validation (file_receive.c) to send a concrete reason; surface it on
the client in client_send.c/config.c.
- Bump PROTOCOL_VERSION to 2.21.0 (CMake VERSION, CHANGELOG, docs) and
update the pinned config wire golden hash / CLI-version tests.
- Add tests/test_protocol_error.c covering mapping+capture, the
over-long bound, bare-error clearing, and thread-locality.
--dry-run now handshakes with a remote/daemon receiver and reports what
WOULD transfer/skip based on receiver state, mutating nothing on either
side.
- Serialize Config.dry_run into the wire config frame and append
STATUS_DRY_RUN_TRANSFER to the status enum (no renumbering); bump
PROTOCOL_VERSION/CMake VERSION/CHANGELOG/golden wire to 2.21.0.
- Receiver: receive_incremental_check_ex runs the normal read-only
decision and answers STATUS_OK (skip) or STATUS_DRY_RUN_TRANSFER
(would transfer) with no basis materialization/append/delta/full
transfer. All mutation sites are guarded by !dry_run: file store,
manifest deletes, --mkpath root creation, --delay-updates staging,
publication, directory-time application, and outcome acks.
- Client: send_dry_run_remote connects, sends the config, checks each
regular file and prints the would-transfer set + trailer; no file data
or delete manifest is sent. Plain local destinations keep the
client-side manifest.
The config_receive_{basis,skip,idmap}_count helpers wrote the
peer-controlled int through the Config member before range-checking it.
An over-cap basis_count therefore left config->basis_count huge while
config->basis_dirs was still NULL; config_receive()'s error path then
called config_delete(), whose basis loop dereferenced NULL and crashed
the daemon before authentication.
Read each count into a local, validate, and only then assign, leaving the
member untouched on failure. config_delete() also guards the basis loop
with the array pointer as defense in depth.
Add a regression test that feeds over-cap basis/idmap/skip counts and
asserts rejection without crashing, plus a direct config_delete() check
on the partial (count set, array NULL) state.
Address low-severity review findings on the X-macro config refactor:
1. The golden test only hashed config_send_wire_block(), so a
receive-side KIND that reads a different width/order could still
round-trip symmetrically. Add test_config_wire_golden_receive():
capture the same hash-pinned 633-byte frame and feed it through
config_receive(), asserting every field (config_wire_equal) plus the
derived use_delta/use_xattrs bits and representative bounded kinds.
Add test_config_wire_receive_bounds() for bounds the symmetric
round-trip cannot reach: an out-of-range BOOL (hand-built frame),
RAW_MAXALLOC zero, a malformed STR_MODULE, an over-cap
INT_IDMAPCOUNT, and an out-of-range INT_IDENTITY chown_uid.
2. golden_config_populate() set long runs of booleans to all-1, so an
adjacent swap within a run produced identical bytes. Alternate the
boolean values and make the fixture receiver-valid (chmod grammar
"u=rwx,go=rx" is the same 11 bytes; delta_max_file_size inside the
bound). Re-pin the golden: len stays 633, hash is now
9160991280011164139 (computed, not guessed).
3. Document in config.h and client_cli.c that the CLI option tables
remain hand-maintained and are deliberately not generated from the
wire-field X-macro (client-only fields, flag/alias/negation
semantics). No CLI-table rewrite.
PROTOCOL_VERSION stays "2.20.0"; src/shared/config.c is untouched and
the wire bytes are unchanged apart from the fixture's own new values.
test_config_wire_golden() serializes a fully-populated Config through
config_send_wire_block() and pins the exact frame to len=633 and FNV-1a
hash 6163263374908258816, captured from the pre-X-macro implementation.
Any field reorder, resize or codec change fails the test.
test_config_wire_roundtrip_all_fields() serializes/deserializes a defaults
Config and a fully-populated Config over a socketpair and compares every
serialized field. The comparison is itself generated from
CONFIG_WIRE_FIELDS (one CONFIG_CMP_<KIND> per table entry), so a new table
entry automatically extends coverage; it cannot fall out of sync. It
normalizes the receiver's NULL/"" canonicalization, the max_alloc server
clamp and the derived use_delta/use_xattrs bits.
Type Config.super_mode as SuperMode (a proper C enum) instead of a bare
int. The wire boundary still carries the mode as an int: send casts the
enum explicitly and receive reads a temporary int, validates the
AUTO..OFF range, then casts. Emitted bytes and accepted values are
unchanged. ModuleGateContext.super_mode_override keeps its -1 sentinel
as int with an explicit cast at the apply site.
Behavior preserved.
- A6: escape attacker-controlled file paths and the receive root in log
lines (file_receive, server, protocol DEBUG) with output_escape()
- A8: identity_set_active() returns bool and fails closed when a requested
usermap/groupmap cannot be deep-copied; handler refuses the connection
- remove the const cast and duplicate super_mode clamp from
server_module_gate via an explicit override the handler applies once
- release the identity snapshot on the queue_create failure path
- refactor identity_parse_copy_as to a single cleanup tail and drop the
duplicated group error format specifier
A1: daemon refuses every client-chosen ownership/super-user request
(--numeric-ids/--chown/--usermap/--groupmap/--fake-super/--copy-as/--super)
unless the selected module opts in with 'client owner = yes'.
A2: fake-super owner replay requires an explicit ownership identity policy.
A3: --super no longer implies --numeric-ids (ownership stays opt-in).
A5: a failed --copy-as chown marks the entry failed instead of reporting
success with the wrong owner.
Add the receiver-side --super / --no-super tri-state (Config->super_mode)
under the safe-subset + clear-refusal privilege model: FastSync never
elevates privileges, it only permits super-user attempts that are already
confined fd-relative below the authorized receive root.
- identity: privilege_super_permitted() gate (OFF=false, ON=true, AUTO follows
geteuid()==0); identity_apply_ownership/_link become no-ops when not
permitted; --super with no explicit identity policy implies raw numeric-id
preservation (explicit usermap/groupmap/chown/numeric-ids still win); warn
exactly once when --super is requested by a non-root receiver.
- file_receive: gate char/block device-node creation on the gate; FIFO/socket
handling is unchanged.
- wire: trailing super_mode int after the --iconv spec, validated 0..2 in
receive_privilege_options and validate_received_config; PROTOCOL_VERSION
2.17.0 -> 2.18.0; version-sensitive tests and docs updated.
- CLI: --super/--no-super parsed explicitly before the generic --no-* branch
(malformed --super=x rejected); usage text added.
- tests: config wire round-trip + invalid-value rejection, privilege-gate mode
unit test, CLI parse test, integration transfer + root-gated ownership
suppression/appliance tests.
- docs: RSYNC_COMPAT --super row + Wave E note, protocol mentions, README.
Force the receiver to apply the requested owner/group to every written
entry through the confined fd-relative identity path instead of switching
the process credentials (unsafe for the multithreaded receiver). An
unprivileged receiver refuses the transfer up front in server_module_gate,
before STATUS_OK, so no data is written with the wrong ownership.
- new Config fields copy_as_set/copy_as_uid/copy_as_gid + defaults
- identity_parse_copy_as (name/@N/* resolution, primary-gid default,
gid==uid fallback for numeric ids with no passwd entry); implies -M
- identity snapshot + highest-priority forcing in identity_resolve_targets
- identity_copy_as_refused() helper
- trailing config-frame block (presence int + two int32 ids, >=0 checked)
- PROTOCOL_VERSION 2.17.0 -> 2.18.0; version-sensitive tests updated
- unit tests for parse + wire round-trip/negative-id rejection
- integration TestCopyAs: unprivileged refusal + root chown assertion
- RSYNC_COMPAT.md --copy-as row updated (safe subset + divergence); README
protocol version refreshed
Capture+transmit source atime (pre-read stat; O_NOATIME sender guard) and birth
time (statx STATX_BTIME); receiver restores atime with mtime (crtime not settable
portably -> transmitted, explicitly not applied). --open-noatime is client-only.
-O/-J documented as accepted no-ops (FastSync never preserves dir/symlink times).
Wire: metadata frame gains atime/crtime val+sec+nsec; PROTOCOL_VERSION
2.11.0->2.12.0. Review fixes: gate atime capture to Linux (no epoch clobber on
non-Linux), close fd on fdopen failure, honest -O/-J status (Compat no-op).
Source files sharing (st_dev,st_ino) are recreated as hard links on the
destination; only the first member's data crosses the wire (siblings ride a
payload-less STATUS_HARDLINK frame). Ordering requires the single-FIFO-writer
receiver + forced sequential scan (documented). link()-failure falls back to a
byte-identical local copy. Rejects -s/--append. PROTOCOL_VERSION 2.11.0->2.12.0.
Review fixes: delete the dead HardLinkRegistry (ordering holds by FIFO writer),
and --existing no longer aborts when the first member is absent but the sibling
exists (leaves the sibling in place).
Receiver allocates the destination file's full size before streaming data
(posix_fallocate preferred, ftruncate fallback on EOPNOTSUPP/ENOSYS) so an
out-of-space transfer fails fast instead of partway. Additive config bool
crossing the wire; PROTOCOL_VERSION 2.10.0 -> 2.11.0. Threaded through all
store paths (atomic, inplace, partial/delay-updates staging, link-dest copy
fallback). Review hardening: explicit lseek(0) before the data write so
correctness does not depend on posix_fallocate leaving the fd offset unchanged.
Receiver-side ownership application, opt-in and privilege-gated:
- OFF for every existing transfer (plain -M/--preserve still never applies
ownership); only triggers on an explicit identity flag + receiver permission.
- EPERM/EACCES warn-and-continue (never aborts); other fchown errors escalate.
- fd-relative fchown after the file is written (symlink-safe, confined).
- New src/shared/identity.{c,h}; config fields numeric_ids / chown uid/gid /
usermap + groupmap id-pair tables cross the wire; PROTOCOL_VERSION 2.10.0
-> 2.11.0. CLI in client_cli.c; per-connection snapshot in server.c.
- Review fixes: EPERM/EACCES-only warn-and-continue, prominent root-receiver
notice, identity_clear_active on early server error paths, --numeric-ids
kept inert standalone (removed from activation trigger set).
- CLI: --append/--append-verify acceptance (parse + imply --incremental,
validate) and incompatibility rejection with -s and --whole-file; both
removed from the unimplemented reject list.
- Config: on-the-wire append/append_verify round-trip.
- Unit: append_resume_eligible / append_tail_length pure resume math.
- Integration (test_append.py): matching-prefix resume is byte-identical and
tail-only (wire bytes << source size); --append with a wrong prefix keeps
prefix+tail (rsync parity) while --append-verify detects the mismatch and
falls back to a byte-exact full transfer; --append with --inplace and -m.
Unit: CLI parse + imply relationships (delete-missing implies ignore, not
delete; valid with --delete and delete timing); delete_missing_args config wire
round-trip (ignore_missing_args confirmed client-only); three-section manifest
round-trip and third-section traversal rejection; manifest_delete_missing_args
exact-path semantics; commit-time parking. Existing two-section manifest frames
updated to the three-section format. Integration: default missing entry is a
hard pre-transfer error; --ignore-missing-args transfers the rest and succeeds
(all-missing transfers nothing; empty list still errors); --delete-missing-args
removes exactly the missing mirror and leaves unrelated extras unless --delete is
also present; filter-exclusion protection never blocks the explicit deletion;
--delete-before early timing composes; all parametrized single-threaded vs -m.
Unit: walker all-or-nothing bounds (exceeded -> nothing removed + distinct
result; exact bound -> deletes), protected-prefix skipping, config wire
round-trip for force_delete/delete_excluded/prune_empty_dirs/max_delete, CLI
parse/validation for the new flags. Integration (TestDeletePolicy): default
delete-excluded protection and --delete-excluded opt-out (single-thread, -m,
early --delete-before, excluded-dir subtrees), --max-delete all-or-nothing over
and at the limit, --force file-over-nonempty-dir replacement, --prune-empty-dirs
(--dirs mode + recursion-mode parity), and --ignore-errors keeping deletion
active across a genuine scan I/O error (run as an unprivileged user).
Unit: parse -y/--fuzzy and --no-fuzzy negation (order-independent), -W and
--no-delta leave fuzzy inert, --fuzzy implies --incremental/--delta, and
validate_config rejects fuzzy with -s (chunk serialization) and -f
(sendfile). Config: fuzzy survives the socketpair wire round-trip.
Integration (TestFuzzy, byte counts via a new CountingProxy helper): the
rename case transfers a 2 MiB file in a few percent of its size with byte-
exact output under --fuzzy, while the no-fuzzy, no-candidate, dissimilar-
sibling and --whole-file runs send the whole file; -y alias; -m parity;
dest-holds-unsuitable-file falls back to the sibling; --delay-updates and
--remove-source-files keep their semantics on fuzzy transfers.
- unit: config basis-path normalization (trailing slash, a//b, ./x/./y collapse;
degenerate inputs rejected).
- integration: same-size/same-mtime/different-content fixtures prove the xxHash
gate -- the basis changed.txt now has the SAME byte size as the source so the
size short-circuit can no longer mask the hash comparison, plus a dedicated
parametrized same-size mismatch test asserting link/copy never use a
content-mismatched basis and compare-dest transfers.
- basis-dir priority is first-match-wins: two link-dest dirs (inode of the
first), and compare-dest before link-dest stays sparse while the reverse
order hard-links.
- --size-only links a same-content basis file with a different mtime;
--ignore-times never links even an exact match.
- --delay-updates + --delete removes extras inside a nested .fastsync-stage
dir (regression guard) while keeping the real staging dir and basis tree.
- a basis run containing a file above the whole-file limit fails up front with
a clear error and transfers nothing.
- CLI: each timing flag (+ --del alias) accepted and implies --delete;
conflicting timings and a timing with --no-delete are rejected.
- Config: delete_during/delete_delay survive config_send/config_receive;
two simultaneous timings are rejected by the receiver-side validation;
config_delete_timing_early() mapping is unit-tested.
- Integration (TCP, single- and multithreaded): every flag removes extras on
a successful transfer; early modes (--delete-before/--delete-during/--del)
delete before data is applied so a destination file blocking a nested
write is removed and the transfer succeeds, while plain --delete /
--delete-after / --delete-delay keep it and fail with every extra intact
(commit-style). Early timing also completes (without deleting) when the
server refuses deletion.
- parse_args accepts each flag in both forms, keeps repetition order/types,
implies --incremental + metadata, and rejects absolute/escaping/degenerate
paths; validate_config rejects basis dirs combined with -s.
- config wire round-trips a mixed basis list and rejects escaping/absolute
paths on the receiver side.
RSYNC_COMPAT Phase-2 row M: path-list construction and destination
directory creation while preserving traversal safety.
- -R/--relative with --files-from: transmit each listed entry under its
bare relative destination path (no source-root mirror). Files keep an
absolute local read path plus a separate wire/dest path (File.send_path);
manifest, incremental quick-check and change output follow the wire path,
so --delete and --remove-source-files stay consistent. -R without
--files-from is unchanged (full mirror).
- --no-implied-dirs: client-only, only meaningful with -R + --files-from.
A listed file whose parent dir is not itself (or via an ancestor)
explicitly listed cannot be placed; the run fails up front with a clear
error. No effect otherwise.
- --dirs/-d + --old-dirs/--old-d aliases: -d <dir> transmits the source
root as an explicit empty directory entry (STATUS_MKDIR frame); with
--files-from listed dirs are created empty and listed files transferred,
never descending. Works single-threaded, -m (sequential scanner in the
-m scan thread) and chunk-serialization (per-file type marker).
Directory entries appear in the delete manifest.
- --mkpath: new wire bool; server creates the destination root (and missing
leading components under its authorized root) at connection start. A
missing destination root is now rejected by default.
- Protocol bumped to 2.7.0 (mkpath wire field + STATUS_MKDIR + chunk type
marker). All receive paths funnel through file_save_to_disk_full which
creates directories via the secure confined mkdir engine; dir entries are
excluded from --remove-source-files outcome acknowledgements on both ends.
- Unit coverage: CLI parse (relative/dirs aliases/mkpath/no-implied-dirs),
config round-trip (relative + mkpath), scanner --dirs non-recursion and
-R send_path (sequential + parallel), receiver dir-entry save.
- Integration coverage: TestRelativeFilesFrom, TestNoImpliedDirs, TestDirs,
TestMkpath (single and -m).
- RSYNC_COMPAT: 4 rows move to Implemented (Summary 67/3/5/1/71 = 147).
Review fixes for --delay-updates:
- --delete no longer deletes the staged files: the delete walker gains a
skip_root_child parameter and receive_manifest passes DELAY_UPDATES_STAGING_DIR
when delay_updates is active, so deletion removes genuine extras while the
staging dir (a direct child of the receive root) is left for publication in
both single and -m modes.
- --backup-dir is rejected when it collides with the reserved internal staging
name .fastsync-stage (trailing slash normalized), in client validation and in
the received-config wire validation, preventing old backups from being
silently installed as new files.
- Staging dir is now held under an exclusive advisory flock for the whole
transfer (context lifetime): two simultaneous delayed transfers to one
destination root no longer share/destroy each other's staged data - the
second fails cleanly. Cleanup only touches the staging dir when this context
owns the lock, so a lock-contention failure cannot wipe a live session.
- Post-publish staging cleanup now returns/logs instead of discarding failures
(warning when the staging dir cannot be fully removed).
- Reworked the publish-failure integration test to exercise real mid-publish
semantics (top-level file published, nested rename fails, no rollback,
sources retained under --remove-source-files) and added integration tests for
--delete + --delay-updates ordering and reserved --backup-dir rejection.
- RSYNC_COMPAT note documents delete ordering, the reserved-name hazard, and
the concurrency guard.
Stage every successfully written file under a private 0700 .fastsync-stage
directory inside the receive root and atomically publish all staged files
only after the whole protocol stream (manifest/delete handling included)
has completed, immediately before the success/outcome frame. On any
abort/error before publication nothing is installed and staging is removed;
a publish failure aborts the transfer with best-effort cleanup of the
remainder (already-published files are not rolled back). Crash leftovers
are wiped when the next delayed transfer starts.
Wire: new delay_updates config flag (selection-options block), protocol
version bumped to 2.6.0, client/server validation rejects --inplace.
CLI/usage/validation updated. Works in single-threaded and -m modes
(exactly one write_thread stages files; the staged-file registry is
mutex-protected; publication runs once after both threads join).
--existing/--ignore-existing/--update decide against the final destination
at stage time; --backup is deferred to publication. remove_source_files
outcomes are only sent after publication so skipped/unpublished sources are
never deleted. Default (no flag) behavior is unchanged.
Tests: config wire round-trip, CLI parse, --inplace rejection, new
test_delay_updates unit suite (27 suites total), and integration
TestDelayUpdates covering single/-m parity, incremental reruns, remove
source files, receiver-skip ordering, and a deterministic publish-failure
abort path.
TapTap