#289 checksum/compression:
- -c/--checksum now implies the incremental content quick-check (without
implying -t), so an unchanged file is skipped like rsync.
- --checksum-choice/--cc accepts xxh64/xxhash, xxh3, xxh128, md5 and auto;
md4/sha1/none and the two-name form are rejected by name.
- --compress-choice/--zc rejects lz4/zlib/zlibx by name (zstd/none/auto kept).
- --checksum-seed=0 is randomized per transfer and sent on the wire.
- --skip-compress uses rsync 3.4.1's default suffix list; slash separators and
dot-less suffixes are accepted.
- add --no-whole-file.
#295 timeouts/alloc/temp-dir:
- --timeout default 0 (disabled), --contimeout default 60; 0 disables both,
plus --no-timeout/--no-contimeout.
- --max-alloc=0 means no allocation limit (was rejected).
- --temp-dir accepts any dir, requires it to exist, and falls back to a
non-atomic copy on EXDEV instead of aborting.
#296 connectivity/daemon:
- -M/--remote-option is rejected for daemon/TCP destinations (SSH-only).
- --trust-sender clarified as receiver-local; server-path tests added.
- --stop-at accepts rsync's full date form (y-m-dTh:m etc.).
Adds unit and integration coverage; no wire-field change, PROTOCOL_VERSION stays
2.22.0.
Split FastSync's single use_metadata bundle into four independent rsync-parity attributes: preserve_perms, preserve_times, preserve_owner, preserve_group. use_metadata is now a derived transport bit (config_derived_use_metadata).
CLI: real -p/--perms, -t/--times, -o/--owner, -g/--group plus --no-perms/--no-times/--no-owner/--no-group (short and long) and --no-preserve; -a is now rsync -rlptgoD; --preserve = -pt; -A implies -p; -X does not; --chmod implies -p; --usermap/--groupmap/--chown imply owner/group per side; --incremental/--delta still auto-preserve unless negated.
Receiver: per-attribute FileAttrPolicy gating for files, dirs (modes applied at end of transfer), symlinks and specials; rsync -E read-bit rule; new files get source_mode & ~umask sanitized (no group/other write); per-side identity resolution; deferred directory metadata; batch dir-metadata replay; daemon modules without 'client owner = yes' no longer refuse plain -a but force super off (no ownership) with a warning.
Wire: PROTOCOL_VERSION 2.21.0 -> 2.22.0 (four appended config bools, golden 653 / 95530566005420798). FileMetadata/chunk/batch framing unchanged. Docs/CHANGELOG/CMake updated to 2.22.0.
Follow-up to a237043 addressing three security/correctness re-review findings.
(1) MEDIUM: a server-contacting --dry-run with --compare-dest/--copy-dest/
--link-dest still read and hashed the basis file and compared it with the
client-supplied digest, a 1-bit content oracle. basis_match_find() gains a
hash_content parameter; the dry-run shortcut passes false and returns no
match without touching basis bytes, so an otherwise-matching entry is
reported as would-transfer. The real (non-dry-run) path is unchanged.
(2) LOW: xattr_capture_path() hardcoded preserve_acls=true, so the receiver's
hard-link copy fallback re-applied system.posix_acl_* even when -A was not
negotiated. The function now takes preserve_acls and members.* is
unaffected; scanner and receiver callers thread the negotiated flag.
(3) INFO: the --fsync --link-dest temp reopen now uses O_NONBLOCK and treats
a raced-in FIFO's ENXIO as a benign fsync-skip instead of blocking.
Tests: dry-run + basis unit test (asserts would-transfer, no content read) and
integration test; xattr capture ACL-filter test. Verified strict build, ASan,
clang-format, cppcheck, and the CI integration subset.
Address confirmed receiver security findings B1-B6:
B1 (HIGH): add O_NONBLOCK to the three receiver read-opens that opened an
existing destination/basis entry before the S_ISREG gate
(incremental_check_open_destination, basis_open_regular, hardlink_read_source)
so a client-planted FIFO can no longer block the receive thread forever while
the post-open type gate still rejects it.
B2 (HIGH/MED): --inplace now fstatat(AT_SYMLINK_NOFOLLOW)-probes the target and
refuses any existing non-regular entry, opens with O_NONBLOCK, and re-checks
S_ISREG on the opened fd. This stops a FIFO from hanging the open and stops a
char/block device from being written directly (bypassing --write-devices).
B3 (MED): under --dry-run the incremental quick-skip no longer reads/hashes the
destination file for --checksum/--delta; it decides from metadata only and
reports would-transfer when the comparison is inconclusive, closing the
read-only-module content-hash oracle.
B4 (LOW): xattr_name_appliable() now gates the two system.posix_acl_* names on
preserve_acls (--acls), not the derived use_xattrs (--xattrs OR --acls). The
receiver drops (never applies) ACL entries when -A was not negotiated while
keeping user.* working for -X.
B5 (INFO): receive_manifest_section() charges a per-entry overhead against
MAX_MANIFEST_BYTES and the aggregate entry count across all three sections is
capped at MAX_MANIFEST_ENTRIES.
B6 (MED): data_charge_session() reserves decompressed/chunk-copy bytes against
the owning ProtocolSession (MAX_CONNECTION_MEMORY) and records them on the Data
so data_destroy() releases them via the Data.owner path. Applied to the
whole-file/append/delta decompression sites and chunk_deserialize() per-file
copies; a missing session owner degrades to the previous uncharged behavior.
Tests: FIFO destination/basis non-hang (with alarm), --inplace FIFO/device
refusal, dry-run no-read oracle test plus updated metadata-only dry-run tests,
ACL-without--acls drop, manifest total-entry cap, and chunk session charging.
- server_module_gate: refuse client-chosen ownership against the ORIGINAL config
so an explicit --super is still refused under an operator --no-super veto
(the veto must not turn a refusal into an accept).
- credentials: open-then-fstat the exact secret inode, require current-user
ownership and no group/other bits, but continue to allow process-substitution
FIFOs; removes the stat->fopen TOCTOU.
- file.c preallocate + protocol.c send-string debug logs escape attacker paths.
- usage/RSYNC_COMPAT updated for --old-args no-op and secret-file rules.
- file: open -K dirlink referents via a race-safe relative O_NOFOLLOW walk
from the authorized-root fd instead of re-opening an absolute realpath()
result (removes the intermediate-symlink swap TOCTOU).
- transport_ssh: always single-quote the server path, including --old-args,
so no mode can inject shell metacharacters.
- transport_tls: set SSL_OP_NO_COMPRESSION and (guarded) SSL_OP_NO_RENEGOTIATION.
- credentials: reject --password-file/--early-input with any group/other
permission bit; chmod 0600 the affected test fixtures.
- file_store: export file_store_write_sparse() and remove the verbatim
file.c duplicate.
- file_ensure_directory_secure() now chowns a final directory it creates under
--copy-as and fails on error; the symlink parent-creation call site propagates
it. The is_dir branch fails when the confined parent cannot be opened under
--copy-as. Closes the residual wrong-owner gap for synthesized/symlink
parent directories.
- file_restore_symlink_metadata() early NULL return is copy-as-aware.
- Preserve errno across the implicit-parent failure cleanup.
- Neutral skip messages (the clamp, not --no-super, may be responsible).
- Daemon copy-as test tolerates the non-root privilege refusal; usage text lists
--copy-as.
- H3: a daemon module without 'client owner = yes' now also has super-user
device activity forced off (char/block mknod, --write-devices), so a root
daemon can no longer be made to create/write raw devices under AUTO. The
entries are skipped, preserving ordinary -a pushes.
- H1/H2: propagate a failed required --copy-as chown from symlink metadata
restore and implicitly-created parent directories, so the entry (and run)
reports failure instead of a wrong-owner success.
- Docs/help/headers updated for A2/A3 and the device clamp; startup warning
spells out the client-owner risk.
- Tests: daemon device clamp (skipped without opt-in, created with opt-in),
updated --super/--fake-super expectations.
Review fixes for Phase 7 Wave D.
#1 (HIGH): STATUS_DIR_TIMES entries no longer create directories. A new
receiver-only File.dir_time_only flag marks dir-time entries; file_save_to_disk_full
short-circuits them as FILE_SAVE_SKIPPED before any device/dir branch, so the sink
still accumulates metadata into the deferred DirTimeList but creates nothing. Empty
source dirs stay untransferred (-a), -m/--prune-empty-dirs semantics are preserved,
and a pre-existing regular file/symlink at an empty-dir mirror path no longer aborts
the transfer. dir_time_list_apply fstatat()s the leaf (AT_SYMLINK_NOFOLLOW) and skips
absent/non-directory paths QUIETLY; only a real existing directory is stamped.
Also initialize File.dir_time_only in file_create() (uninitialised garbage otherwise).
#2 (MED): send_dir_times() chunks entries into repeated STATUS_DIR_TIMES frames of at
most MAX_MANIFEST_ENTRIES, matching the receiver's per-frame bound; the tautological
> INT_MAX check is gone.
#3 (LOW): dir_time_list_add() assigns each grown array right after its realloc (no
dangling) and advances capacity only after both succeed.
#4 (LOW): RSYNC_COMPAT.md -- STATUS_MKDIR carries metadata, dir times are transmitted
via STATUS_DIR_TIMES and applied at the end, empty dirs are still never created; -m
rationale, -O row and Wave D notes updated. Summary counts untouched.
#5 (LOW): integration tests for the three #1 scenarios (empty-dir non-creation under
-a and -a -m, collision non-abort), scanner test now covers empty-dir capture, and
test_file_restore_symlink_metadata asserts the positive apply path when supported.
PROTOCOL_VERSION stays 2.17.0; config-frame layout unchanged.
- fake_super_restore_fd now sanitizes mode like metadata_mode (never grants
S_IWGRP|S_IWOTH; 0666 -> 0644), fixing a privilege regression
- --sparse takes precedence over --preallocate (skip posix_fallocate when
sparse) so holes are not re-allocated; docs corrected
- --partial retention disabled under --no_replace (ignore/existing) and only
marks write_attempted after the write begins (no empty-temp retention)
- accept --block-size=SIZE / --delta-block=SIZE inline forms; neutral messages
- fake-super EPERM/EACCES skipped silently (docs aligned); EINVAL still logged
- sparse unit test now memcmp's the full buffer; TestBlockSize integration keeps
the destination basis so delta is genuinely exercised
- unit 37/37, cppcheck 0, clang-format 0
- write_all_sparse: skips all-zero runs >= 4096 bytes via lseek(SEEK_CUR) and
ftruncates the final size, wired into the atomic temp+rename and --inplace
paths with no wire change (full image already in memory).
- --partial retention: on a save failure after the temp held data, rename the
already-written temp to the destination path (best-effort; falls through to
unlink; never retains when --partial is off) so --append/--append-verify can
resume; tested by forcing futimens EINVAL with an out-of-range nsec.
- --block-size aliases --delta-block; verified config->delta_block_size is
honored by the delta engine end-to-end (unit + integration tests).
- fake_super_restore_fd: parses and re-applies user.fastsync.stat fd-relative
(fchown best-effort/non-root skipped, fchmod, futimens); a save under
--fake-super now both records and re-applies.
- -N/--crtimes and --stderr=client promoted to a new 'Impossible/Divergence'
status bucket (Summary: 136/2/4/3/2 = 147).
- cppcheck/clang-format clean; unit 37/37; integration 407 passed.
Capture+transmit source atime (pre-read stat; O_NOATIME sender guard) and birth
time (statx STATX_BTIME); receiver restores atime with mtime (crtime not settable
portably -> transmitted, explicitly not applied). --open-noatime is client-only.
-O/-J documented as accepted no-ops (FastSync never preserves dir/symlink times).
Wire: metadata frame gains atime/crtime val+sec+nsec; PROTOCOL_VERSION
2.11.0->2.12.0. Review fixes: gate atime capture to Linux (no epoch clobber on
non-Linux), close fd on fdopen failure, honest -O/-J status (Compat no-op).
Source files sharing (st_dev,st_ino) are recreated as hard links on the
destination; only the first member's data crosses the wire (siblings ride a
payload-less STATUS_HARDLINK frame). Ordering requires the single-FIFO-writer
receiver + forced sequential scan (documented). link()-failure falls back to a
byte-identical local copy. Rejects -s/--append. PROTOCOL_VERSION 2.11.0->2.12.0.
Review fixes: delete the dead HardLinkRegistry (ordering holds by FIFO writer),
and --existing no longer aborts when the first member is absent but the sibling
exists (leaves the sibling in place).
Receiver allocates the destination file's full size before streaming data
(posix_fallocate preferred, ftruncate fallback on EOPNOTSUPP/ENOSYS) so an
out-of-space transfer fails fast instead of partway. Additive config bool
crossing the wire; PROTOCOL_VERSION 2.10.0 -> 2.11.0. Threaded through all
store paths (atomic, inplace, partial/delay-updates staging, link-dest copy
fallback). Review hardening: explicit lseek(0) before the data write so
correctness does not depend on posix_fallocate leaving the fd offset unchanged.
Adds real algorithm selection (xxh64 default, plus md5 via OpenSSL EVP) and a
64-bit seed for the per-file whole-file digest used by the --incremental/
--checksum handshake and basis-dir content verification. The seed also feeds
the delta path's per-block xxHash32 strong checksum (low 32 bits) so an
explicit seed deterministically changes those digests too. Sender and receiver
hash identically: the algorithm id and seed cross the config wire frame and the
STATUS_CHECK handshake now carries a length-prefixed, bounded digest instead of
a fixed 64-bit value. Unsupported algorithm names are rejected at parse time
(never a silent no-op). PROTOCOL_VERSION bumped 2.9.0 -> 2.10.0; defaults
(xxh64, seed 0) preserve prior byte-for-byte behavior.
file_remove_tree_secure() opens the final component O_NOFOLLOW below the
authorized root and recursively wipes it with an fd-relative walk (symlinks are
removed by name, never followed); --force uses it to clear a destination
directory that blocks an incoming regular file.
The receiver's per-file incremental check now consults the ordered basis-dir
list whenever the destination is not already up to date. An exact basis match
requires equal size, equal mtime (unless --size-only; --ignore-times disables
basis matching like rsync), and an equal content xxHash64 -- the sender sends
its xxHash for every file whenever basis dirs are configured (not only under
--checksum), so a hard link or local copy is only ever made from byte-identical
content.
On a match:
- compare-dest: reply STATUS_OK and skip data only when the destination does
not already hold the file (sparse, rsync parity). A destination that holds
a DIFFERENT version falls back to a normal transfer instead of rsync's
delete, keeping the mirror complete.
- copy-dest: reply STATUS_OK and hand a synthetic File (bytes read from the
basis file, basis metadata) to the normal store sink, so the file is
installed as a real local copy through the existing atomic temp+rename
engine and honors --existing/--ignore-existing/--update/--backup/
--delay-updates/--partial-dir unchanged.
- link-dest: same, but File.basis_link records the basis path and the store
engine calls the new file_to_disk_secure_link(): an atomic temp hard link +
rename. Cross-filesystem/refused links fall back to a byte-identical local
copy (never a corrupt or partial file); the copy fallback applies metadata,
while a successful link keeps the basis inode's own attributes so the basis
file is never mutated.
Basis-materialized files carry File.skip so they are not acknowledged to a
--remove-source-files sender (the sender already saw STATUS_OK and keeps the
source). The no-match path is byte-for-byte identical to the existing delta /
full-data transfer.
B1: destination-root existence/creation mishandled two legitimate forms.
file_directory_exists_secure/file_ensure_directory_secure now normalize a
trailing-slash destination (so the last component is never an empty leaf)
and treat a destination equal to the already-open authorized root as present
(no spurious <root>/<basename> nested dir with --mkpath). Confinement and
O_NOFOLLOW probing are unchanged. Regression integration tests: existing
dest with trailing slash works with and without --mkpath; dest == authorized
root works and creates no stray nested dir.
W1: chunk serialize/deserialize round-trip unit test for a directory entry
mixed with a regular file, with and without metadata; --dirs coverage under
-s and -s -m.
W2: --list-only and --dry-run now print file_wire_path() so all outputs show
the -R transformed relative name, matching -i/--out-format.
W3: RSYNC_COMPAT -d/--dirs note: dirs are created immediately under
--delay-updates (only regular files are staged).
W4: --dirs generator flushes chunks by element count too, so a long
--files-from list of empty directories cannot exceed the per-chunk file cap.
N1: STATUS_MKDIR added to status_to_string.
N2: removed dead TestMkpath._transfer.
N3: removed redundant --no-implied-dirs OPTION_TABLE row.
N4: integration tests: --dirs --delete keeps the just-created empty dir (and
deletes extras); a listed dir colliding with a regular file at the dest fails
cleanly.
RSYNC_COMPAT Phase-2 row M: path-list construction and destination
directory creation while preserving traversal safety.
- -R/--relative with --files-from: transmit each listed entry under its
bare relative destination path (no source-root mirror). Files keep an
absolute local read path plus a separate wire/dest path (File.send_path);
manifest, incremental quick-check and change output follow the wire path,
so --delete and --remove-source-files stay consistent. -R without
--files-from is unchanged (full mirror).
- --no-implied-dirs: client-only, only meaningful with -R + --files-from.
A listed file whose parent dir is not itself (or via an ancestor)
explicitly listed cannot be placed; the run fails up front with a clear
error. No effect otherwise.
- --dirs/-d + --old-dirs/--old-d aliases: -d <dir> transmits the source
root as an explicit empty directory entry (STATUS_MKDIR frame); with
--files-from listed dirs are created empty and listed files transferred,
never descending. Works single-threaded, -m (sequential scanner in the
-m scan thread) and chunk-serialization (per-file type marker).
Directory entries appear in the delete manifest.
- --mkpath: new wire bool; server creates the destination root (and missing
leading components under its authorized root) at connection start. A
missing destination root is now rejected by default.
- Protocol bumped to 2.7.0 (mkpath wire field + STATUS_MKDIR + chunk type
marker). All receive paths funnel through file_save_to_disk_full which
creates directories via the secure confined mkdir engine; dir entries are
excluded from --remove-source-files outcome acknowledgements on both ends.
- Unit coverage: CLI parse (relative/dirs aliases/mkpath/no-implied-dirs),
config round-trip (relative + mkpath), scanner --dirs non-recursion and
-R send_path (sequential + parallel), receiver dir-entry save.
- Integration coverage: TestRelativeFilesFrom, TestNoImpliedDirs, TestDirs,
TestMkpath (single and -m).
- RSYNC_COMPAT: 4 rows move to Implemented (Summary 67/3/5/1/71 = 147).
Stage every successfully written file under a private 0700 .fastsync-stage
directory inside the receive root and atomically publish all staged files
only after the whole protocol stream (manifest/delete handling included)
has completed, immediately before the success/outcome frame. On any
abort/error before publication nothing is installed and staging is removed;
a publish failure aborts the transfer with best-effort cleanup of the
remainder (already-published files are not rolled back). Crash leftovers
are wiped when the next delayed transfer starts.
Wire: new delay_updates config flag (selection-options block), protocol
version bumped to 2.6.0, client/server validation rejects --inplace.
CLI/usage/validation updated. Works in single-threaded and -m modes
(exactly one write_thread stages files; the staged-file registry is
mutex-protected; publication runs once after both threads join).
--existing/--ignore-existing/--update decide against the final destination
at stage time; --backup is deferred to publication. remove_source_files
outcomes are only sent after publication so skipped/unpublished sources are
never deleted. Default (no flag) behavior is unchanged.
Tests: config wire round-trip, CLI parse, --inplace rejection, new
test_delay_updates unit suite (27 suites total), and integration
TestDelayUpdates covering single/-m parity, incremental reruns, remove
source files, receiver-skip ordering, and a deterministic publish-failure
abort path.
The --inplace branch opened the destination with O_WRONLY|O_CREAT (no
O_TRUNC) and only restored metadata when the sender supplied it. Two
flaws resulted:
1. An existing destination file kept its original mode when no metadata
was sent, so setuid/setgid/sticky bits survived an overwrite (a root
sync could leave a root-owned setuid binary controlled by a client).
2. A shorter payload left stale trailing bytes from the previous version
because the file was never truncated to the new length.
In the inplace branch of file_to_disk_secure_impl:
- Always trim the file to the new payload length (ftruncate after the
write) so stale trailing bytes can never survive; sparse targets keep
their pre-size ftruncate.
- Always normalize the mode after a successful overwrite: apply the
metadata-derived safe mode when metadata is present (as before), else
fchmod to a safe default 0644, so setuid/setgid/sticky are cleared in
both cases.
- The --update newer-destination check still runs before any truncation
or chmod, preserving the skip semantics.
Adds unit tests in test_file.c: (a) setuid/sticky bits on an existing
destination are cleared after an inplace write with and without metadata,
(b) a shorter inplace payload leaves no trailing stale bytes.
The Phase 1 merge conflict resolutions introduced formatting that failed
the CI lint job (clang-format 18.1.3). Reformatted with the exact CI
version; no functional changes.
- file.c split layout retained; security-hardened secure-fs helpers
(open_secure_parent/to_disk_secure/rename_secure/stat_secure) now live in
file.c with file_ prefix and are shared with file_receive.c
- file_receive.c takes the security branch's bounded allocations
(receive_data_limited, data_decompress_limited, size checks) and
STATUS_ERROR signaling
- file_send.c gains the data consistency check on file->data
- client_validation.c: stricter --tls requiring --ca, log_message style
- utils.c: hardened openat/mkdirat mkdir_r from security branch
- Add log_perror() helper (context + strerror(errno)) to the log module
- Replace all bare perror() calls with log_perror() so errors are routed
through the unified logger (stderr sink + optional --log-file sink)
- Convert fprintf(stderr, "Error:/Warning: ...") in client code to
log_message(); raw fprintf kept only for progress/stats output
- file.c: File/FileMetadata lifecycle and local disk helpers (~110 lines)
- file_send.c: client-side send path (file_send_single_calls, file_send_sendfile)
- file_receive.c: server-side receive/save path (file_receive, receive_incremental_check,
receive_manifest, file_save_to_disk)
- file_types.h holds shared struct definitions; file.h remains an umbrella header
so existing includes are unaffected
Completes the transfer/protocol separation started in PR #212
TapTap