Reclassify every rsync-compatibility row as parity / caveat / divergent
(replacing the misleading 143-OK / 0-divergence summary), and document the
protocol 2.23.0 behavior:
- Split the conflated `-M, --preserve` row: `-M` is `--remote-option`,
`--preserve` is the FastSync `-p`+`-t` alias.
- Fix `MAX_CONNECTION_MEMORY` (256 MiB, not 1 GB), `--rsync-path`
(client-only, never crosses the wire), and the `-p` mode behavior
(strict rsync parity; no masking).
- `--specials` now recreates sockets, so `-D` is real parity; fake-super
records the resolved owner and replays mode/time (never real-chowns).
- Document short options/clustering, checksum/compression choices, seed
randomization, timeout/max-alloc defaults, temp-dir confinement + EXDEV,
identity/map parity, verbatim symlinks, delete scoping, `--max-delete`
partial + exit 25, `--chmod`, output caveats, and server `--port`.
- Bump version refs to 2.23.0 and add the 2.23.0 CHANGELOG entry.
Docs-only; no source changes.
- --chmod no longer implies --preserve-perms; repeated --chmod options
accumulate, and D/F/X selectors plus s/t special bits are supported with
rsync's exact parse_chmod/tweak_mode semantics.
- Stop masking group/other write and setuid/setgid/sticky: -p copies the
source mode exactly, no-p new entries use source&~umask, directories keep
setgid/sticky, and special nodes follow the same rules.
- Apply ownership before mode on the fd path so a chown cannot clear the
setuid/setgid bits -p just restored (rsync order).
- Update unit and integration tests, including differential checks against
rsync 3.4.1.
Address review findings on feat/rsync-parity:
- confine --temp-dir below the receive root (reject absolute/.. like
backup-dir/partial-dir); keep EXDEV non-atomic fallback
- floor server session I/O deadlines at SERVER_IO_TIMEOUT_SEC (60s) and
install it on the socket layer at startup (slow-loris)
- charge each --delete-missing-args directory removal once and clamp the
extras-walk remaining budget so it can never underflow past --max-delete
- normalize --compress-choice=auto to zstd client-side and accept it on
receive so auto transfers no longer fail
- map received --max-alloc=0 to MAX_SERVER_ALLOC (receive path only)
- zero File.dest_state; include log-file-format in report_dest_info;
add STATUS_DELETE_LIMIT name; recognize --skip-compress as a
separate-value option; OOM-guard send_list_only root entry; drop the
dead -M= branch; record the bare relative protected prefix for -R
size-prunes in both scanners; refresh delete-manifest comment
- pin the rsync tarball sha256 and bump integrator image to v11
Tests: temp-dir rejection/relative/cross-device, server timeout floor,
delete-missing dir budget regression, compress-choice=auto e2e,
max-alloc=0 receive mapping, dest_state, report_dest_info modes,
skip-compress dash value, -M short forms, -R root size-prune mirror
protection (rsync 3.4.1 confirmed).
A non-empty --delete-missing-args directory removed under --force/--delete
now has its contents deleted entry-by-entry through the budgeted walker, so
every deleted file/dir counts toward --max-delete exactly like rsync (a
capped run leaves the remaining entries and exits 25).
- Scope the --delete extras walk to directories synchronized by the
transfer: add a synchronized-directory section to the delete manifest
(protocol 2.23.0) so --files-from subsets no longer delete untransmitted
paths outside listed directory subtrees (data-loss fix).
- Separate --max-size/--min-size prune protection from --delete-excluded so
size-pruned source mirrors survive (rsync parity).
- Unlink extraneous destination symlinks instead of skipping them.
- Make --max-delete partial (delete up to N, skip the rest) and exit 25;
accept negative values as unlimited.
- Draw --delete-missing-args deletions from the shared --max-delete budget.
- Honor --force during --delay-updates publication.
Add unit and integration regression tests; update the pinned config wire
golden and version strings for the 2.23.0 manifest/status additions.
#289 checksum/compression:
- -c/--checksum now implies the incremental content quick-check (without
implying -t), so an unchanged file is skipped like rsync.
- --checksum-choice/--cc accepts xxh64/xxhash, xxh3, xxh128, md5 and auto;
md4/sha1/none and the two-name form are rejected by name.
- --compress-choice/--zc rejects lz4/zlib/zlibx by name (zstd/none/auto kept).
- --checksum-seed=0 is randomized per transfer and sent on the wire.
- --skip-compress uses rsync 3.4.1's default suffix list; slash separators and
dot-less suffixes are accepted.
- add --no-whole-file.
#295 timeouts/alloc/temp-dir:
- --timeout default 0 (disabled), --contimeout default 60; 0 disables both,
plus --no-timeout/--no-contimeout.
- --max-alloc=0 means no allocation limit (was rejected).
- --temp-dir accepts any dir, requires it to exist, and falls back to a
non-atomic copy on EXDEV instead of aborting.
#296 connectivity/daemon:
- -M/--remote-option is rejected for daemon/TCP destinations (SSH-only).
- --trust-sender clarified as receiver-local; server-path tests added.
- --stop-at accepts rsync's full date form (y-m-dTh:m etc.).
Adds unit and integration coverage; no wire-field change, PROTOCOL_VERSION stays
2.22.0.
- #286: --numeric-ids is a mapping modifier only; it no longer activates
chown by itself (identity_active_enabled/owner/group predicates), and
--fake-super stores the resolved mapping instead of real-chowning.
- #286: apply owner/group to directories via the deferred directory
metadata path; capture+transmit+apply directory xattrs/ACLs (-aX/-aA),
including default ACLs, in STATUS_MKDIR/STATUS_DIR_TIMES.
- #294: --usermap/--groupmap support inclusive ranges, '*', empty FROM
(unnamed ids), and receiver-side TO name resolution; --chown mixing with
a same-side map is rejected like rsync.
- Protocol 2.22.0 -> 2.23.0 (map wire entry gains from_hi + to_name;
dir frames gain a bounded xattr block).
#287:
- --safe-links: keep safe in-tree links AS symlinks and skip unsafe
(absolute or ".."-escaping) ones, mirroring rsync's unsafe_symlink().
Skipped links are recorded as delete-protected so --delete does not
remove their destination mirror (no silent data loss).
- --copy-unsafe-links: preserve safe links as symlinks and dereference
only unsafe ones.
- --munge-links: receiver-side rewrite storing /rsyncd-munged/-prefixed
targets (rsync parity), replacing the no-op #SYMLINK sender prefix.
- -l: store the target verbatim, including absolute and ".." targets
(rsync -l parity); the old receiver containment silently dropped them.
#288:
- --specials: recreate unix-domain sockets via mknod(S_IFSOCK), which
Linux permits unprivileged; keep EEXIST/EPERM skip behavior.
- --copy-devices: copy a device's content into a regular file when
requested; skip unrequested non-regular entries like rsync's default.
#291:
- Compile --exclude/--include/--exclude-from/--include-from into the SAME
ordered rule list as --filter/-f (first match wins), so the common
`--include='*.txt' --exclude='*'` idiom and include-alone semantics match
rsync. The legacy per-kind scanner arrays are no longer applied.
- -x/--one-file-system emits the cross-device mount-point directory entry
(empty) instead of dropping it, in both the sequential and parallel scanners.
- Stop passing the legacy arrays to the scanner; document -f is --filter.
#292:
- New src/shared/format.c/.h: rsync "big_num" (comma-grouped integers) and
decimal -h human sizes, %M/%t timestamp, and the STATUS_DEST_INFO codec.
- Receiver answers each STATUS_CHECK with a pre-transfer destination snapshot
(new report_dest_info wire field + STATUS_DEST_INFO, PROTOCOL_VERSION
2.23.0) so the sender can render true itemize columns.
- Itemize now emits rsync-correct update/type chars and c/s/t/p/o/g columns
for files, dirs, symlinks and hard links, comparing size/time/perms/owner/
group against the reported destination.
- --out-format gains %i %n %f %l %b %M %t %o %p %B %U %G %L; %f is the
relative display path, %M the YYYY/MM/DD-HH:MM:SS form, %b the literal
bytes sent.
- --list-only prints transfer-relative names, directory entries and ls-style
grouped sizes.
- --stats prints rsync's multi-line block on stdout; -h uses decimal units.
Tests: unit tests for the filter ordering, format primitives, itemize
columns; integration + differential tests against real rsync 3.4.1 for
itemize/out-format/list-only/selection and -x. Golden wire len/hash and
protocol version strings updated for 2.23.0.
Implement rsync 3.4.1 client-CLI parity:
- cluster boolean shorts (-av, -aAX, -rlpt) and accept attached values
(-B1048576, -essh, -Mfoo); add the -r, -b, -L and -B short aliases
(-r is a faithful no-op since FastSync is always recursive)
- stop OPT_NOOP (-s/--secluded-args, -r/--recursive) from swallowing the
next argv
- add inline --opt=value for every value-taking long option, including
--exclude/--include/--exclude-from/--include-from/--log-file (#291)
- accept --port on the server CLI in addition to -p (#296)
- reject unknown flags naming the flag and stating it is unsupported
Unit tests cover clustering, attached/inline values, the OPT_NOOP
argument-consumption fix and rejected shorts.
Add POSIX ACL/xattr tooling (acl, attr), zstd/lz4/xxhash dev libs and build rsync 3.4.1 from source so drop-in parity tests can run inside CI. Bump all workflow/agent image references v10 -> v11.
The captured_config fixture assumed fault_dst already existed, relying on earlier tests in the same xdist worker creating it via _recover. Under --dist=load a worker can receive the capture test first, so the receiver rejected a missing destination root and the capture run failed. Create DEST_DIR in the autouse seeding fixture so test order/distribution cannot matter.
Split FastSync's single use_metadata bundle into four independent rsync-parity attributes: preserve_perms, preserve_times, preserve_owner, preserve_group. use_metadata is now a derived transport bit (config_derived_use_metadata).
CLI: real -p/--perms, -t/--times, -o/--owner, -g/--group plus --no-perms/--no-times/--no-owner/--no-group (short and long) and --no-preserve; -a is now rsync -rlptgoD; --preserve = -pt; -A implies -p; -X does not; --chmod implies -p; --usermap/--groupmap/--chown imply owner/group per side; --incremental/--delta still auto-preserve unless negated.
Receiver: per-attribute FileAttrPolicy gating for files, dirs (modes applied at end of transfer), symlinks and specials; rsync -E read-bit rule; new files get source_mode & ~umask sanitized (no group/other write); per-side identity resolution; deferred directory metadata; batch dir-metadata replay; daemon modules without 'client owner = yes' no longer refuse plain -a but force super off (no ownership) with a warning.
Wire: PROTOCOL_VERSION 2.21.0 -> 2.22.0 (four appended config bools, golden 653 / 95530566005420798). FileMetadata/chunk/batch framing unchanged. Docs/CHANGELOG/CMake updated to 2.22.0.
Bring README.md and RSYNC_COMPAT.md in line with the actual code/CLI and add
an automated guard so they cannot silently drift again.
Waves A-E:
- Correct stale compatibility claims: archive is `-rlptD` (owner/group are
opt-in via identity flags, not implied), and symlinks, hard links, xattrs,
ACLs and `--dirs` are implemented.
- Remove documented-but-nonexistent features: the six unread FASTSYNC_* env
vars, and `--client-cn` (server-only) from the client table.
- Repair the corrupted "Implementation Details" section (broken list numbering
and emphasis) and correct it against the source.
- Sync the client and server option tables with usage.c / server_cli.c, and
document server-contacting `--dry-run` (protocol 2.21.0).
- Hygiene: `# FastSync` heading, real build commands, consistent binary names,
runnable TLS examples, daemon module keys.
Also align the client `--help` / archive log wording and the RSYNC_COMPAT
archive rows with the opt-in ownership model, and add
tests/integration/test_readme_consistency.py (marked `ci`) asserting every
documented FASTSYNC_* var is read in src/ and every documented client/server
flag appears in the corresponding `--help`.
- generate_bench_data now writes exactly (1-random_ratio)*target bytes of
genuinely compressible repeated content instead of only the small fixed
STRUCTURED_FILES set; measured composition is reported and --dry-run prints
it for scaling checks
- verify each transfer against the source (paths/sizes/byte compare) before
recording timing; add --no-verify; failed runs are counted as invalid
- correct p50/p95 with linear-interpolation percentile (was int(len*0.95))
- tc/netem: run tc directly as root, else sudo; clear error when tc/iproute2
is missing or qdisc setup fails; netem_reset is always safe
- build into dedicated build-bench/ via --build-dir (Release), never reconfigure
the user's build/
- parse --configs with shlex.split
- add MB/s throughput column and throughput_mbps JSON field
- add --warm incremental mode: untimed full seed then measure add/change deltas
Follow-up to a237043 addressing three security/correctness re-review findings.
(1) MEDIUM: a server-contacting --dry-run with --compare-dest/--copy-dest/
--link-dest still read and hashed the basis file and compared it with the
client-supplied digest, a 1-bit content oracle. basis_match_find() gains a
hash_content parameter; the dry-run shortcut passes false and returns no
match without touching basis bytes, so an otherwise-matching entry is
reported as would-transfer. The real (non-dry-run) path is unchanged.
(2) LOW: xattr_capture_path() hardcoded preserve_acls=true, so the receiver's
hard-link copy fallback re-applied system.posix_acl_* even when -A was not
negotiated. The function now takes preserve_acls and members.* is
unaffected; scanner and receiver callers thread the negotiated flag.
(3) INFO: the --fsync --link-dest temp reopen now uses O_NONBLOCK and treats
a raced-in FIFO's ENXIO as a benign fsync-skip instead of blocking.
Tests: dry-run + basis unit test (asserts would-transfer, no content read) and
integration test; xattr capture ACL-filter test. Verified strict build, ASan,
clang-format, cppcheck, and the CI integration subset.
Re-review findings on the C3/C4 hardening branch:
- --stdio is the SSH transport whose remote argv is composed by the client
(including via --remote-option), so accepting --allow-super there let a
client defeat the C3 secure default for a root receiver. Reject it at CLI
parse time (standalone TCP only) and force the process-global flag off for
--stdio as defense in depth. Correct the help text and README/RSYNC_COMPAT:
the --stdio argv is client-composed, super stays off, and a forced command is
needed if the default must hold.
- daemon_conf: the per-module 'hosts allow'/'hosts deny' call sites passed
module_name and replace in the wrong order, so multiple lines replaced
instead of appended and the empty-value error omitted the module name. Pass
(module->name, false) like the global keys; add a unit test for two
per-module allow/deny lines appending.
- tls: read the client CN via ASN1_STRING_to_UTF8 so an exactly-required-length
name is accepted and only actual over-length CNs are rejected.
- client_cli: capture errno before output_escape() in
read_patterns_from_file() so an over-long line is still reported as
EFBIG instead of the (possibly malloc-clobbered) errno.
- file_list: guard string_list_add() capacity doubling against
overflow (capacity > INT_MAX / 2), matching filter_rule_list_add();
callers already surface the false as a memory-allocation error.
- compression: ZSTD_isError() is true for ZSTD_CONTENTSIZE_UNKNOWN,
which made the 3x unknown-size fallback dead code. Test the
CONTENTSIZE_ERROR/UNKNOWN sentinels explicitly so unknown-size frames
reach the estimate path (still bounded by the existing hard limit)
while invalid frames are rejected. Known-size frames and the 100 MB
ceiling/overflow checks are unchanged.
- tests: add an unknown-content-size-frame decompression test.
Tests: ./build/tests and ./build-asan/tests all pass (42/42);
clang-format + cppcheck clean.
- parse_ull_arg() rejects a leading '-'/'+' (strtoull would silently wrap
-1 to ULLONG_MAX) and --chunk-size/--delta-max enforce their upper bounds.
- Escape local untrusted paths before logging (client_send, scanner,
--filter rule, pattern-file reads) with output_escape(..., 8-bit mode).
- Read --exclude-from/--include-from through the bounded line reader.
- Open --log-file with O_NOFOLLOW|O_CLOEXEC, mode 0600, via open+fdopen;
create --write-batch with O_NOFOLLOW|O_CLOEXEC, mode 0600.
- Reject --dry-run together with --write-batch (dry-run must not write the
batch file), alongside the existing --read-batch/--only-write-batch rules.
Tests: signed/oversized numeric rejection, over-long pattern file, dry-run +
write-batch unit and integration coverage.
protocol_receive_n_data_until() aborted on a signal-interrupted plaintext
read (and on SSL_ERROR_SYSCALL with errno==EINTR); retry both, matching the
send path and protocol_read_status_until(). Also clamp each SSL_write() to
INT_MAX so a >INT_MAX size_t request can never truncate into a partial write.
opencode
TapTap