Add POSIX ACL/xattr tooling (acl, attr), zstd/lz4/xxhash dev libs and build rsync 3.4.1 from source so drop-in parity tests can run inside CI. Bump all workflow/agent image references v10 -> v11.
The captured_config fixture assumed fault_dst already existed, relying on earlier tests in the same xdist worker creating it via _recover. Under --dist=load a worker can receive the capture test first, so the receiver rejected a missing destination root and the capture run failed. Create DEST_DIR in the autouse seeding fixture so test order/distribution cannot matter.
Split FastSync's single use_metadata bundle into four independent rsync-parity attributes: preserve_perms, preserve_times, preserve_owner, preserve_group. use_metadata is now a derived transport bit (config_derived_use_metadata).
CLI: real -p/--perms, -t/--times, -o/--owner, -g/--group plus --no-perms/--no-times/--no-owner/--no-group (short and long) and --no-preserve; -a is now rsync -rlptgoD; --preserve = -pt; -A implies -p; -X does not; --chmod implies -p; --usermap/--groupmap/--chown imply owner/group per side; --incremental/--delta still auto-preserve unless negated.
Receiver: per-attribute FileAttrPolicy gating for files, dirs (modes applied at end of transfer), symlinks and specials; rsync -E read-bit rule; new files get source_mode & ~umask sanitized (no group/other write); per-side identity resolution; deferred directory metadata; batch dir-metadata replay; daemon modules without 'client owner = yes' no longer refuse plain -a but force super off (no ownership) with a warning.
Wire: PROTOCOL_VERSION 2.21.0 -> 2.22.0 (four appended config bools, golden 653 / 95530566005420798). FileMetadata/chunk/batch framing unchanged. Docs/CHANGELOG/CMake updated to 2.22.0.
Bring README.md and RSYNC_COMPAT.md in line with the actual code/CLI and add
an automated guard so they cannot silently drift again.
Waves A-E:
- Correct stale compatibility claims: archive is `-rlptD` (owner/group are
opt-in via identity flags, not implied), and symlinks, hard links, xattrs,
ACLs and `--dirs` are implemented.
- Remove documented-but-nonexistent features: the six unread FASTSYNC_* env
vars, and `--client-cn` (server-only) from the client table.
- Repair the corrupted "Implementation Details" section (broken list numbering
and emphasis) and correct it against the source.
- Sync the client and server option tables with usage.c / server_cli.c, and
document server-contacting `--dry-run` (protocol 2.21.0).
- Hygiene: `# FastSync` heading, real build commands, consistent binary names,
runnable TLS examples, daemon module keys.
Also align the client `--help` / archive log wording and the RSYNC_COMPAT
archive rows with the opt-in ownership model, and add
tests/integration/test_readme_consistency.py (marked `ci`) asserting every
documented FASTSYNC_* var is read in src/ and every documented client/server
flag appears in the corresponding `--help`.
- generate_bench_data now writes exactly (1-random_ratio)*target bytes of
genuinely compressible repeated content instead of only the small fixed
STRUCTURED_FILES set; measured composition is reported and --dry-run prints
it for scaling checks
- verify each transfer against the source (paths/sizes/byte compare) before
recording timing; add --no-verify; failed runs are counted as invalid
- correct p50/p95 with linear-interpolation percentile (was int(len*0.95))
- tc/netem: run tc directly as root, else sudo; clear error when tc/iproute2
is missing or qdisc setup fails; netem_reset is always safe
- build into dedicated build-bench/ via --build-dir (Release), never reconfigure
the user's build/
- parse --configs with shlex.split
- add MB/s throughput column and throughput_mbps JSON field
- add --warm incremental mode: untimed full seed then measure add/change deltas
Follow-up to a237043 addressing three security/correctness re-review findings.
(1) MEDIUM: a server-contacting --dry-run with --compare-dest/--copy-dest/
--link-dest still read and hashed the basis file and compared it with the
client-supplied digest, a 1-bit content oracle. basis_match_find() gains a
hash_content parameter; the dry-run shortcut passes false and returns no
match without touching basis bytes, so an otherwise-matching entry is
reported as would-transfer. The real (non-dry-run) path is unchanged.
(2) LOW: xattr_capture_path() hardcoded preserve_acls=true, so the receiver's
hard-link copy fallback re-applied system.posix_acl_* even when -A was not
negotiated. The function now takes preserve_acls and members.* is
unaffected; scanner and receiver callers thread the negotiated flag.
(3) INFO: the --fsync --link-dest temp reopen now uses O_NONBLOCK and treats
a raced-in FIFO's ENXIO as a benign fsync-skip instead of blocking.
Tests: dry-run + basis unit test (asserts would-transfer, no content read) and
integration test; xattr capture ACL-filter test. Verified strict build, ASan,
clang-format, cppcheck, and the CI integration subset.
Re-review findings on the C3/C4 hardening branch:
- --stdio is the SSH transport whose remote argv is composed by the client
(including via --remote-option), so accepting --allow-super there let a
client defeat the C3 secure default for a root receiver. Reject it at CLI
parse time (standalone TCP only) and force the process-global flag off for
--stdio as defense in depth. Correct the help text and README/RSYNC_COMPAT:
the --stdio argv is client-composed, super stays off, and a forced command is
needed if the default must hold.
- daemon_conf: the per-module 'hosts allow'/'hosts deny' call sites passed
module_name and replace in the wrong order, so multiple lines replaced
instead of appended and the empty-value error omitted the module name. Pass
(module->name, false) like the global keys; add a unit test for two
per-module allow/deny lines appending.
- tls: read the client CN via ASN1_STRING_to_UTF8 so an exactly-required-length
name is accepted and only actual over-length CNs are rejected.
- client_cli: capture errno before output_escape() in
read_patterns_from_file() so an over-long line is still reported as
EFBIG instead of the (possibly malloc-clobbered) errno.
- file_list: guard string_list_add() capacity doubling against
overflow (capacity > INT_MAX / 2), matching filter_rule_list_add();
callers already surface the false as a memory-allocation error.
- compression: ZSTD_isError() is true for ZSTD_CONTENTSIZE_UNKNOWN,
which made the 3x unknown-size fallback dead code. Test the
CONTENTSIZE_ERROR/UNKNOWN sentinels explicitly so unknown-size frames
reach the estimate path (still bounded by the existing hard limit)
while invalid frames are rejected. Known-size frames and the 100 MB
ceiling/overflow checks are unchanged.
- tests: add an unknown-content-size-frame decompression test.
Tests: ./build/tests and ./build-asan/tests all pass (42/42);
clang-format + cppcheck clean.
- parse_ull_arg() rejects a leading '-'/'+' (strtoull would silently wrap
-1 to ULLONG_MAX) and --chunk-size/--delta-max enforce their upper bounds.
- Escape local untrusted paths before logging (client_send, scanner,
--filter rule, pattern-file reads) with output_escape(..., 8-bit mode).
- Read --exclude-from/--include-from through the bounded line reader.
- Open --log-file with O_NOFOLLOW|O_CLOEXEC, mode 0600, via open+fdopen;
create --write-batch with O_NOFOLLOW|O_CLOEXEC, mode 0600.
- Reject --dry-run together with --write-batch (dry-run must not write the
batch file), alongside the existing --read-batch/--only-write-batch rules.
Tests: signed/oversized numeric rejection, over-long pattern file, dry-run +
write-batch unit and integration coverage.
protocol_receive_n_data_until() aborted on a signal-interrupted plaintext
read (and on SSL_ERROR_SYSCALL with errno==EINTR); retry both, matching the
send path and protocol_read_status_until(). Also clamp each SSL_write() to
INT_MAX so a >INT_MAX size_t request can never truncate into a partial write.
Read per-directory filter files through the bounded reader, guard the rule
list's capacity doubling against INT_MAX/2 overflow, and escape the local
directory path before logging a read failure.
Read list files through utils_getdelim_bounded() so a single multi-gigabyte
line can no longer force unbounded allocation; over-long entries fail with a
clear error. Also add the documented memchr() NUL-byte check (excluding the
NUL delimiter in NUL-separated mode).
Tests: an over-long entry is rejected with an 'exceeds' diagnostic.
Replace the recursive glob matcher with an iterative O(pattern*string)
dynamic program. The old recursion explored exponentially many paths for
overlapping '*'/'**' wildcards (e.g. '*a*a*...*b' against a long run of
'a'), a CPU DoS reachable from --exclude/--include patterns and
.rsync-filter. A differential fuzz against the original matcher confirms
identical results. Doc: has_path_traversal() is a lexical '..' check only.
Add utils_getdelim_bounded(): a getdelim-style reader that never allocates
beyond UTILS_MAX_LINE_LEN, used to cap untrusted list/filter line reads.
Tests: pathological glob completes quickly; bounded reader returns EFBIG on
an over-long record.
data_decompress_limited() looped while ZSTD_decompressStream() returned a
positive hint. A truncated frame keeps returning that hint with all input
consumed, so a malformed/truncated payload spun forever (CPU DoS). Detect
input exhaustion with an incomplete frame and fail via the existing cleanup,
skipping the check when the output buffer merely needs to grow first.
Add a fork+alarm regression test that truncates a valid frame and asserts
decompression returns NULL promptly.
Address confirmed receiver security findings B1-B6:
B1 (HIGH): add O_NONBLOCK to the three receiver read-opens that opened an
existing destination/basis entry before the S_ISREG gate
(incremental_check_open_destination, basis_open_regular, hardlink_read_source)
so a client-planted FIFO can no longer block the receive thread forever while
the post-open type gate still rejects it.
B2 (HIGH/MED): --inplace now fstatat(AT_SYMLINK_NOFOLLOW)-probes the target and
refuses any existing non-regular entry, opens with O_NONBLOCK, and re-checks
S_ISREG on the opened fd. This stops a FIFO from hanging the open and stops a
char/block device from being written directly (bypassing --write-devices).
B3 (MED): under --dry-run the incremental quick-skip no longer reads/hashes the
destination file for --checksum/--delta; it decides from metadata only and
reports would-transfer when the comparison is inconclusive, closing the
read-only-module content-hash oracle.
B4 (LOW): xattr_name_appliable() now gates the two system.posix_acl_* names on
preserve_acls (--acls), not the derived use_xattrs (--xattrs OR --acls). The
receiver drops (never applies) ACL entries when -A was not negotiated while
keeping user.* working for -X.
B5 (INFO): receive_manifest_section() charges a per-entry overhead against
MAX_MANIFEST_BYTES and the aggregate entry count across all three sections is
capped at MAX_MANIFEST_ENTRIES.
B6 (MED): data_charge_session() reserves decompressed/chunk-copy bytes against
the owning ProtocolSession (MAX_CONNECTION_MEMORY) and records them on the Data
so data_destroy() releases them via the Data.owner path. Applied to the
whole-file/append/delta decompression sites and chunk_deserialize() per-file
copies; a missing session owner degrades to the previous uncharged behavior.
Tests: FIFO destination/basis non-hang (with alarm), --inplace FIFO/device
refusal, dry-run no-read oracle test plus updated metadata-only dry-run tests,
ACL-without--acls drop, manifest total-entry cap, and chunk session charging.
C2: --force is deletion authority (an incoming regular file may remove a
non-empty destination directory tree, and --delete-missing-args may
remove a non-empty directory mirror), but it was not masked by the
operator --allow-delete policy. The handler now clears
config->force_delete unless --allow-delete was given, exactly like
--delete and --delete-missing-args.
C3: a standalone TCP / --stdio server running as root defaulted to
SUPER_MODE_AUTO, so an untrusted client --devices/--write-devices/
--super could make it create device nodes, write raw devices, or apply
client-chosen ownership. A privileged standalone receiver now forces
SUPER_MODE_OFF unless the operator opts in with the new server-only
--allow-super flag. Non-root receivers are unchanged, and the daemon
path keeps its per-module `client owner = yes` gate. --allow-super is
rejected with --no-super or --daemon.
C6: tls_client_identity_allowed now rejects a CN whose reported length
reached the buffer bound, so a truncated over-long CN cannot be matched
by a required --client-cn prefix.
Tests: an integration regression proving --force cannot replace a
destination directory without --allow-delete; standalone-default tests
for --copy-as refusal and (root-only) skipped device creation; a CLI
unit test for the new flag. The integration shared_server fixture opts
in with --allow-super so the existing root-only ownership/device/copy-as
tests continue to exercise the opted-in configuration. README and
RSYNC_COMPAT document the flag and the force/delete gating.
secret_is_legacy_hex indexed s[0..63] without first checking the string
length, reading out of bounds for a shorter secret. Require
strlen(s) == 64 before scanning, and add a unit test that short and
63-hex-digit secrets are rejected as ordinary malformed verifiers (never
misreported as legacy).
C5: restrict the TLS 1.2 and below cipher list to ECDHE AEAD suites
(ECDHE+AESGCM:ECDHE+CHACHA20, minus NULL/eNULL/MD5/RC4/3DES) instead of
HIGH (which includes CBC), and set SSL_OP_CIPHER_SERVER_PREFERENCE so the
server's order decides the negotiated cipher. Client and server share
create_ssl_ctx, so both are updated.
C7: load the private key through an O_RDONLY|O_NOFOLLOW|O_CLOEXEC fd,
fstat that fd and validate owner/mode (now also rejecting group/other
execute bits), then load from the fd via BIO_new_fd. This removes the
stat-to-load TOCTOU race while keeping the exact-owner/0600 policy.
C8: verify an IP-literal client hostname against the certificate IP SAN
with X509_VERIFY_PARAM_set1_ip_asc instead of SSL_set1_host (a DNS
check), falling back to SSL_set1_host for real names.
Unit tests assert the server-preference option, the absence of CBC/RC4/
3DES suites, and that context creation still succeeds.
A present hosts allow/hosts deny/auth users key with an empty or
separator-only value produced a zero-length list, silently meaning no
ACL / no auth and contradicting the strict-parse contract.
store_host_list and the auth users parser now track how many entries a
present key actually added and fail the load with a clear error when it
is zero, so a restrictive directive can never silently become open.
Unit tests cover empty, whitespace-only and comma-only values.
A remote destination's user@host token is passed to ssh in option
position, so a host beginning with '-' (e.g. -oProxyCommand=...) was
parsed by ssh as an option, allowing arbitrary command execution.
- config_parse_ssh_dest now validates the user@host prefix and returns
-1 (with a clear logged error) for an empty host or a user/host that
starts with '-'; config_parse_transport_dest propagates the failure.
- transport_ssh.c's parse_remote_dest applies the same validation as
defense-in-depth, and ssh_build_client_argv inserts a '--'
end-of-options marker before the destination token.
- Unit tests cover -oProxyCommand=... / -prefixed hosts / empty host
rejection and the argv shape.
Extend _snapshot_tree to record mode, inode, xattrs, directories and
special nodes, and add coverage proving a server-contacting --dry-run
leaves the destination structurally identical for --delay-updates,
--backup, symlinks, hardlinks, FIFOs, and daemon modules (including a
read-only module). Add a regression test for the --read-batch --dry-run
refusal and for a missing/non-directory receive root failing a dry-run
exactly like a real run.
Fix stale version comments (2.20.0/633 -> 2.21.0/637) and RSYNC_COMPAT's
current --protocol value, and add a unit assertion that
--server-port/--port (and --server-host) set the dry-run routing bit.
receiver_save_file appended to context->outcomes for --remove-source-files
without the !dry_run guard the multithreaded pipeline has, so a hostile
dry-run client could grow outcomes unbounded (raw, uncharged realloc) and
force a per-frame ack. Guard the append on !dry_run.
--dry-run --read-batch=FILE still wrote to the destination because
batch_read_apply -> file_save_to_disk_full bypassed the per-caller
!dry_run guards. Guard file_save_to_disk_full and manifest_delete_all
directly (return SKIPPED/no-op) so every save/delete path is mutation-free
in dry-run, and keep the per-caller guards. Reject --dry-run combined with
--read-batch/--only-write-batch at CLI validation with a clear error (a
dry-run of a local batch apply is not meaningful).
--dry-run --server-host=H (or TLS / source-bind --address) silently ran the
client-side manifest even though a real run contacts the server. Add a
client-only, never-serialized server_host_set bit (alongside the existing
server_port_set) and extend dry_run_targets_server so every explicit remote
target contacts the receiver.
Also make incremental_check return the dry-run code (4) only when the
session actually requested dry-run; a stray STATUS_DRY_RUN_TRANSFER from a
hostile/buggy peer is now a logged protocol error (STATUS_ERROR) instead of
falling through to send file data and desync. Both normal send_single_file
callers handle rc == 4 explicitly as an abort.
A wire dry_run bit must not relax the destination-root precondition:
previously handlers skipped ensure_receive_root entirely in dry-run, so a
client could dry-run against a nonexistent/regular-file root a real session
rejects. Split the existence check (receive_root_exists, never creates)
from the create path and apply the precondition unconditionally: dry-run
runs the existence/directory check only, reports the failure, and creates
nothing (no --mkpath).
Also allow a `read only = yes` daemon module for a dry-run session (a
server-contacting dry-run IS a read-only wire operation) while still
refusing it for real writes, and update the stale read-only comments.
Address review/security findings in the 2.21.0 error-detail feature:
- Keepalive drain no longer erases the terminal detail: capture/clear is
skipped for STATUS_KEEPALIVE so the reason the peer just sent survives the
owed keepalive replies.
- Replace the capture path with a dedicated protocol_receive_error_detail:
the declared length is validated against MAX_ERROR_DETAIL_BYTES before any
allocation, over-cap bodies are drained through a fixed scratch buffer (so
the stream never desyncs), in-cap bodies read straight into the thread-local
detail buffer, and session->max_alloc is never raised. Lengths beyond
MAX_STRING_SIZE are treated as a fatal framing error.
- The detail body now honors the caller's deadline (timed/keepalive paths) and
polls the abort callback between drain chunks.
- Escape peer-controlled detail text with output_escape before logging it in
client_send.c and config.c.
- Clear io_error_detail in io_set_fds so a new connection on the same thread
cannot inherit a stale reason.
- Add unit tests for the keepalive-survival, over-cap drain, absurd-length
fatal framing, and deadline-clamped body read cases.
Today a server rejection sends a bare STATUS_ERROR and the reason only
reaches the server log, so the client cannot say why a transfer was
refused. Add an optional, bounded server->client error-detail frame:
- Status gains STATUS_ERROR_DETAIL appended LAST so existing wire
values are unchanged.
- send_error_detail(fd, msg) sends STATUS_ERROR_DETAIL followed by the
existing length-prefixed string primitive, slicing over-long messages
to MAX_ERROR_DETAIL_BYTES (4096).
- receive_status() (and the timed/keepalive status readers) always
consume the detail body and map the status back to STATUS_ERROR,
capturing the text into a thread-local buffer exposed by
protocol_last_error(); a bare STATUS_ERROR leaves it cleared. Every
existing call site keeps working and the stream cannot desync.
- Upgrade the daemon module gate / config validation (config.c), the
final transfer failure (server.c) and receiver-side path/node
validation (file_receive.c) to send a concrete reason; surface it on
the client in client_send.c/config.c.
- Bump PROTOCOL_VERSION to 2.21.0 (CMake VERSION, CHANGELOG, docs) and
update the pinned config wire golden hash / CLI-version tests.
- Add tests/test_protocol_error.c covering mapping+capture, the
over-long bound, bare-error clearing, and thread-locality.
The per-file STATUS_CHECK fast path was a single 534-line function that
was hard to review. Extract it into small static helpers called in order
by a short linear orchestrator:
- incremental_check_receive_request (receive/validate request frame)
- incremental_check_open_destination (secure open + stat)
- incremental_check_quick_skip (metadata/content skip decision)
- incremental_check_try_basis (compare/copy/link-dest)
- incremental_check_try_append_resume(--append tail resume)
- incremental_check_try_delta (block delta)
- incremental_check_try_fuzzy (--fuzzy basis)
- incremental_check_receive_full (STATUS_NEXT + whole file)
Pure refactor: the ordered sequence of wire operations
(send_status/send_n_data/receive_n_data/receive_status/receive_wire_str)
is byte-for-byte identical to the original, and every resource cleanup
is preserved (a unified idempotent cleanup replaces the duplicated
per-path close/free blocks). No functional changes.
--dry-run now handshakes with a remote/daemon receiver and reports what
WOULD transfer/skip based on receiver state, mutating nothing on either
side.
- Serialize Config.dry_run into the wire config frame and append
STATUS_DRY_RUN_TRANSFER to the status enum (no renumbering); bump
PROTOCOL_VERSION/CMake VERSION/CHANGELOG/golden wire to 2.21.0.
- Receiver: receive_incremental_check_ex runs the normal read-only
decision and answers STATUS_OK (skip) or STATUS_DRY_RUN_TRANSFER
(would transfer) with no basis materialization/append/delta/full
transfer. All mutation sites are guarded by !dry_run: file store,
manifest deletes, --mkpath root creation, --delay-updates staging,
publication, directory-time application, and outcome acks.
- Client: send_dry_run_remote connects, sends the config, checks each
regular file and prints the would-transfer set + trailer; no file data
or delete manifest is sent. Plain local destinations keep the
client-side manifest.
TapTap