- plain --delete with no timing flag now selects delete-during (progressive
deletion, matching rsync and avoiding the full old+new tree peak)
- new long-only FastSync --delete-commit restores the old atomic behavior
(delete only after the whole transfer succeeds); timing-identical to
--delete-after, implemented via the same wire bool
- timing flags are mutually exclusive; --delete-commit conflicts with other
timings; --delete-before/--delete-during rows reworded per Phase-0 probes
- CHANGELOG migration note; tally unchanged 116/14/27
- PROTOCOL_VERSION 2.27.0 -> 2.28.0; STATUS_STATS gains literal_bytes and
the created reg/dir/link/special counters (golden wire updated)
- receiver reports which destination entries it newly created, including
implicitly-created parent directories below the logical transfer root, so
Number of created files carries rsync's per-type breakdown
- Literal data is now exact for a delta transfer (receiver counts the literal
fragments it stored)
- differential-tested vs rsync 3.4.1 for fresh-create, update and delta
- Initialize PipelineContextSender.delete_suppressed=false: an uninitialized
true silently skipped the --delete keep-set manifest under -m/--threads,
so destination extras were never removed.
- Allocate/install the receiver deleted-path observer only when
report_deletes is set (--info=del / -i / --out-format under --delete), cap
the retained list at MAX_MANIFEST_ENTRIES, and free already-created lists
on the receiver-pipeline create failure path.
- Validate report_deletes/report_stats/report_dest_info on receive.
- Correct stale comments (config.h report_deletes, delete_plan.h deleted
count, multiprocessing.h stats locking, utils.h observer placement).
- Tests: sender delete_suppressed init, report_deletes gating (unit), and
-m/--delete default delete-after keep-set removal (integration).
- Wire: config frame gains report_deletes (protocol 2.26.0 -> 2.27.0); the
receiver lists actually-removed paths in the STATUS_STATS path list, so the
sender prints rsync's `deleting PATH` / `*deleting PATH` lines for a real
--delete run (and -i/out-format). Observers threaded through the manifest,
missing-args and per-directory delete engines; golden wire len/hash updated.
- bwlimit: throttle now paces like rsync 3.4.1 -- ~100ms burst capacity and the
sleep is no longer credited as refill, so 4 MiB at 1024/2048 KiB/s matches
rsync within ~4% (was ~2x too fast).
The single-threaded and -m receivers never populated ReceiverStats.matched_data
or .deleted_files, so --stats always printed 0 for both even when rsync
reported nonzero. Track the bytes reconstructed from the basis file while
applying a delta, and tally the delete-commit counts (manifest and
per-directory sessions) into the receiver stats. The -m pipeline now carries
its own stats/would-delete fields and emits the STATUS_STATS frame before the
terminal success, so --threads finally reports the counters and renders
-n --delete lines.
Also normalize the -n --delete would-delete enumeration's absolute basis
prefixes exactly like the real commit path (fixing an over-report) and fix the
basis_delete_relative off-by-one when the receive root is '/'. Unit tests
cover the root mapping and the basis protection; integration tests cover
matched/deleted stats for both receivers and the --threads dry-run delete
lines.
delete_plan_session_commit() lacked the central no-mutation guard that
manifest_delete_all() has, so a server-contacting -n run (or a hostile plan
frame) could still remove --delete-missing-args mirrors on the per-directory
timing path. Return DELETE_COMMIT_OK immediately when the session is a
dry-run, and gate the receiver/server commit call sites too. Add a unit test
that streams a plan naming an existing destination file and asserts it
survives.
Accept the full rsync 3.4.1 --compress-choice set (zstd/lz4/zlib/zlibx/
none/auto) and the two-name --checksum-choice TRANSFER,PRE-TRANSFER form,
including rsync's 'none' rules (rejected with --checksum at exit 4, and
forcing --whole-file as the transfer half) and unknown names at exit 4.
The checksum default becomes the auto-negotiated xxh128.
Negotiation is deterministic and symmetric: both peers run the same
preference resolver (rsync's --version order). The resolved
compression_algo crosses the wire as a new trailing config-frame int so
the receiver validates and installs the exact codec; an unsupported
choice is refused before STATUS_OK like rsync's failed negotiation.
Bump PROTOCOL_VERSION to 2.26.0.
Add the STATUS_STATS end-of-transfer receiver report (matched/deleted
counters plus a would-delete path list) behind the report_stats wire
bool, and a read-only delete_extras_list walker. --stats now renders
true wire byte totals and the receiver-reported deleted count; a
server-contacting -n --delete prints transfer-relative '*deleting' lines
matching rsync's itemize layout.
- Compare --delete-during/--delete-delay final state against rsync 3.4.1.
- Force a mid-transfer failure through a byte-slicing proxy: --delete-during has
removed the processed directory's extra, --delete-delay has not.
- Create a destination entry while the transfer is in flight: it survives
--delete-delay's snapshot but is removed by --delete-after's fresh end scan.
- Cover the --delete-delay type-conflict case now matching rsync.
Stream one delete plan per source directory from sender to receiver instead of
a single whole-tree keep-set manifest:
- --delete-during applies each directory's extras as its plan arrives, before
that directory's data (rsync's generator-order deletion).
- --delete-delay snapshots each directory's extras while the plan arrives and
commits the removals only after a fully-successful transfer, so files created
after the scan survive (matching rsync's delete-delay, not delete-after).
- Type conflicts (a destination file blocking a source directory, or vice
versa) are cleared immediately in both modes, so the nested write succeeds.
The plan carries the destination-relative directory, its kept child directory
names and its kept child file names; the first frame also carries the global
protected prefixes, size-skipped prefixes and --delete-missing-args paths.
--delete-before keeps the existing whole-tree early manifest; plain --delete and
--delete-after keep the end-of-transfer manifest commit.
Preserves the existing safety surface: protected/size-skipped prefixes and the
--delay-updates/basis skips are honored at any depth, deletion is scoped to the
synchronized directories (--files-from), MAX_SERVER_DELETE_COUNT and
--max-delete (partial + exit 25) are shared across plans, symlinks are never
followed, and paths are confined to the receive root.
Address review findings on feat/rsync-parity:
- confine --temp-dir below the receive root (reject absolute/.. like
backup-dir/partial-dir); keep EXDEV non-atomic fallback
- floor server session I/O deadlines at SERVER_IO_TIMEOUT_SEC (60s) and
install it on the socket layer at startup (slow-loris)
- charge each --delete-missing-args directory removal once and clamp the
extras-walk remaining budget so it can never underflow past --max-delete
- normalize --compress-choice=auto to zstd client-side and accept it on
receive so auto transfers no longer fail
- map received --max-alloc=0 to MAX_SERVER_ALLOC (receive path only)
- zero File.dest_state; include log-file-format in report_dest_info;
add STATUS_DELETE_LIMIT name; recognize --skip-compress as a
separate-value option; OOM-guard send_list_only root entry; drop the
dead -M= branch; record the bare relative protected prefix for -R
size-prunes in both scanners; refresh delete-manifest comment
- pin the rsync tarball sha256 and bump integrator image to v11
Tests: temp-dir rejection/relative/cross-device, server timeout floor,
delete-missing dir budget regression, compress-choice=auto e2e,
max-alloc=0 receive mapping, dest_state, report_dest_info modes,
skip-compress dash value, -M short forms, -R root size-prune mirror
protection (rsync 3.4.1 confirmed).
- Scope the --delete extras walk to directories synchronized by the
transfer: add a synchronized-directory section to the delete manifest
(protocol 2.23.0) so --files-from subsets no longer delete untransmitted
paths outside listed directory subtrees (data-loss fix).
- Separate --max-size/--min-size prune protection from --delete-excluded so
size-pruned source mirrors survive (rsync parity).
- Unlink extraneous destination symlinks instead of skipping them.
- Make --max-delete partial (delete up to N, skip the rest) and exit 25;
accept negative values as unlimited.
- Draw --delete-missing-args deletions from the shared --max-delete budget.
- Honor --force during --delay-updates publication.
Add unit and integration regression tests; update the pinned config wire
golden and version strings for the 2.23.0 manifest/status additions.
#289 checksum/compression:
- -c/--checksum now implies the incremental content quick-check (without
implying -t), so an unchanged file is skipped like rsync.
- --checksum-choice/--cc accepts xxh64/xxhash, xxh3, xxh128, md5 and auto;
md4/sha1/none and the two-name form are rejected by name.
- --compress-choice/--zc rejects lz4/zlib/zlibx by name (zstd/none/auto kept).
- --checksum-seed=0 is randomized per transfer and sent on the wire.
- --skip-compress uses rsync 3.4.1's default suffix list; slash separators and
dot-less suffixes are accepted.
- add --no-whole-file.
#295 timeouts/alloc/temp-dir:
- --timeout default 0 (disabled), --contimeout default 60; 0 disables both,
plus --no-timeout/--no-contimeout.
- --max-alloc=0 means no allocation limit (was rejected).
- --temp-dir accepts any dir, requires it to exist, and falls back to a
non-atomic copy on EXDEV instead of aborting.
#296 connectivity/daemon:
- -M/--remote-option is rejected for daemon/TCP destinations (SSH-only).
- --trust-sender clarified as receiver-local; server-path tests added.
- --stop-at accepts rsync's full date form (y-m-dTh:m etc.).
Adds unit and integration coverage; no wire-field change, PROTOCOL_VERSION stays
2.22.0.
- #286: --numeric-ids is a mapping modifier only; it no longer activates
chown by itself (identity_active_enabled/owner/group predicates), and
--fake-super stores the resolved mapping instead of real-chowning.
- #286: apply owner/group to directories via the deferred directory
metadata path; capture+transmit+apply directory xattrs/ACLs (-aX/-aA),
including default ACLs, in STATUS_MKDIR/STATUS_DIR_TIMES.
- #294: --usermap/--groupmap support inclusive ranges, '*', empty FROM
(unnamed ids), and receiver-side TO name resolution; --chown mixing with
a same-side map is rejected like rsync.
- Protocol 2.22.0 -> 2.23.0 (map wire entry gains from_hi + to_name;
dir frames gain a bounded xattr block).
Implement rsync 3.4.1 client-CLI parity:
- cluster boolean shorts (-av, -aAX, -rlpt) and accept attached values
(-B1048576, -essh, -Mfoo); add the -r, -b, -L and -B short aliases
(-r is a faithful no-op since FastSync is always recursive)
- stop OPT_NOOP (-s/--secluded-args, -r/--recursive) from swallowing the
next argv
- add inline --opt=value for every value-taking long option, including
--exclude/--include/--exclude-from/--include-from/--log-file (#291)
- accept --port on the server CLI in addition to -p (#296)
- reject unknown flags naming the flag and stating it is unsupported
Unit tests cover clustering, attached/inline values, the OPT_NOOP
argument-consumption fix and rejected shorts.
Split FastSync's single use_metadata bundle into four independent rsync-parity attributes: preserve_perms, preserve_times, preserve_owner, preserve_group. use_metadata is now a derived transport bit (config_derived_use_metadata).
CLI: real -p/--perms, -t/--times, -o/--owner, -g/--group plus --no-perms/--no-times/--no-owner/--no-group (short and long) and --no-preserve; -a is now rsync -rlptgoD; --preserve = -pt; -A implies -p; -X does not; --chmod implies -p; --usermap/--groupmap/--chown imply owner/group per side; --incremental/--delta still auto-preserve unless negated.
Receiver: per-attribute FileAttrPolicy gating for files, dirs (modes applied at end of transfer), symlinks and specials; rsync -E read-bit rule; new files get source_mode & ~umask sanitized (no group/other write); per-side identity resolution; deferred directory metadata; batch dir-metadata replay; daemon modules without 'client owner = yes' no longer refuse plain -a but force super off (no ownership) with a warning.
Wire: PROTOCOL_VERSION 2.21.0 -> 2.22.0 (four appended config bools, golden 653 / 95530566005420798). FileMetadata/chunk/batch framing unchanged. Docs/CHANGELOG/CMake updated to 2.22.0.
Re-review findings on the C3/C4 hardening branch:
- --stdio is the SSH transport whose remote argv is composed by the client
(including via --remote-option), so accepting --allow-super there let a
client defeat the C3 secure default for a root receiver. Reject it at CLI
parse time (standalone TCP only) and force the process-global flag off for
--stdio as defense in depth. Correct the help text and README/RSYNC_COMPAT:
the --stdio argv is client-composed, super stays off, and a forced command is
needed if the default must hold.
- daemon_conf: the per-module 'hosts allow'/'hosts deny' call sites passed
module_name and replace in the wrong order, so multiple lines replaced
instead of appended and the empty-value error omitted the module name. Pass
(module->name, false) like the global keys; add a unit test for two
per-module allow/deny lines appending.
- tls: read the client CN via ASN1_STRING_to_UTF8 so an exactly-required-length
name is accepted and only actual over-length CNs are rejected.
C2: --force is deletion authority (an incoming regular file may remove a
non-empty destination directory tree, and --delete-missing-args may
remove a non-empty directory mirror), but it was not masked by the
operator --allow-delete policy. The handler now clears
config->force_delete unless --allow-delete was given, exactly like
--delete and --delete-missing-args.
C3: a standalone TCP / --stdio server running as root defaulted to
SUPER_MODE_AUTO, so an untrusted client --devices/--write-devices/
--super could make it create device nodes, write raw devices, or apply
client-chosen ownership. A privileged standalone receiver now forces
SUPER_MODE_OFF unless the operator opts in with the new server-only
--allow-super flag. Non-root receivers are unchanged, and the daemon
path keeps its per-module `client owner = yes` gate. --allow-super is
rejected with --no-super or --daemon.
C6: tls_client_identity_allowed now rejects a CN whose reported length
reached the buffer bound, so a truncated over-long CN cannot be matched
by a required --client-cn prefix.
Tests: an integration regression proving --force cannot replace a
destination directory without --allow-delete; standalone-default tests
for --copy-as refusal and (root-only) skipped device creation; a CLI
unit test for the new flag. The integration shared_server fixture opts
in with --allow-super so the existing root-only ownership/device/copy-as
tests continue to exercise the opted-in configuration. README and
RSYNC_COMPAT document the flag and the force/delete gating.
receiver_save_file appended to context->outcomes for --remove-source-files
without the !dry_run guard the multithreaded pipeline has, so a hostile
dry-run client could grow outcomes unbounded (raw, uncharged realloc) and
force a per-frame ack. Guard the append on !dry_run.
A wire dry_run bit must not relax the destination-root precondition:
previously handlers skipped ensure_receive_root entirely in dry-run, so a
client could dry-run against a nonexistent/regular-file root a real session
rejects. Split the existence check (receive_root_exists, never creates)
from the create path and apply the precondition unconditionally: dry-run
runs the existence/directory check only, reports the failure, and creates
nothing (no --mkpath).
Also allow a `read only = yes` daemon module for a dry-run session (a
server-contacting dry-run IS a read-only wire operation) while still
refusing it for real writes, and update the stale read-only comments.
Today a server rejection sends a bare STATUS_ERROR and the reason only
reaches the server log, so the client cannot say why a transfer was
refused. Add an optional, bounded server->client error-detail frame:
- Status gains STATUS_ERROR_DETAIL appended LAST so existing wire
values are unchanged.
- send_error_detail(fd, msg) sends STATUS_ERROR_DETAIL followed by the
existing length-prefixed string primitive, slicing over-long messages
to MAX_ERROR_DETAIL_BYTES (4096).
- receive_status() (and the timed/keepalive status readers) always
consume the detail body and map the status back to STATUS_ERROR,
capturing the text into a thread-local buffer exposed by
protocol_last_error(); a bare STATUS_ERROR leaves it cleared. Every
existing call site keeps working and the stream cannot desync.
- Upgrade the daemon module gate / config validation (config.c), the
final transfer failure (server.c) and receiver-side path/node
validation (file_receive.c) to send a concrete reason; surface it on
the client in client_send.c/config.c.
- Bump PROTOCOL_VERSION to 2.21.0 (CMake VERSION, CHANGELOG, docs) and
update the pinned config wire golden hash / CLI-version tests.
- Add tests/test_protocol_error.c covering mapping+capture, the
over-long bound, bare-error clearing, and thread-locality.
--dry-run now handshakes with a remote/daemon receiver and reports what
WOULD transfer/skip based on receiver state, mutating nothing on either
side.
- Serialize Config.dry_run into the wire config frame and append
STATUS_DRY_RUN_TRANSFER to the status enum (no renumbering); bump
PROTOCOL_VERSION/CMake VERSION/CHANGELOG/golden wire to 2.21.0.
- Receiver: receive_incremental_check_ex runs the normal read-only
decision and answers STATUS_OK (skip) or STATUS_DRY_RUN_TRANSFER
(would transfer) with no basis materialization/append/delta/full
transfer. All mutation sites are guarded by !dry_run: file store,
manifest deletes, --mkpath root creation, --delay-updates staging,
publication, directory-time application, and outcome acks.
- Client: send_dry_run_remote connects, sends the config, checks each
regular file and prints the would-transfer set + trailer; no file data
or delete manifest is sent. Plain local destinations keep the
client-side manifest.
Every client on loopback shares the 127.0.0.1 identity, so counting them
against 'max connections per host' or the default-on auth lockout lets one
local client deny service to all the others (and makes a shared-NAT/proxy
address a natural DoS vector for remote clients). Use
utils_fd_peer_is_local (fail-closed) in the daemon gate to exempt a
provably local peer from the per-source cap and the auth lockout while
keeping the per-module and global caps. Remote peers are unchanged.
Document the shared-NAT/proxy identity limitation and the loopback
exemption in README/RSYNC_COMPAT/CHANGELOG, update the integration test to
assert the exemption, and fix the README 'auth failure delay' cap (5000,
not 60000).
Document on utils_get_authorized_root_path() that the returned pointer is
borrowed and invalidated by the next authorized-root setter, that the fd
and path are not read atomically (non-reentrant), and that the fd remains
caller-owned. Add a matching single-threaded/set-before-threads note at
the accessor definitions in utils.c.
In server.c, drop the redundant utils_set_authorized_root(-1, NULL) after
a failed utils_set_authorized_root(): the setter already fail-closes the
state on allocation failure. The following close(root_fd) is unchanged.
Wire the shared registry into the accept loop (parent claims a slot before
fork, blocks SIGCHLD across fork+pid publication, and reclaims the dead
child's slot from the SIGCHLD handler so per-module/per-source counts are
released even on SIGKILL). The connection child records the selected module
and normalized peer IP once the config frame names them: an over-cap module
or source is refused at the config gate with an audit log, and a source
that exceeded the auth-failure threshold is refused before a SCRAM
challenge (the counter is shared across children and cleared on success).
The existing global cap and host ACLs are untouched.
Type Config.super_mode as SuperMode (a proper C enum) instead of a bare
int. The wire boundary still carries the mode as an int: send casts the
enum explicitly and receive reads a temporary int, validates the
AUTO..OFF range, then casts. Emitted bytes and accepted values are
unchanged. ModuleGateContext.super_mode_override keeps its -1 sentinel
as int with an explicit cast at the apply site.
Behavior preserved.
Deduplicate the repeated directory-time capture gate
(`config->use_metadata && !config->omit_dir_times`) used by the
sender-side (multiprocessing.c) and receiver-side (receiver.c) sinks
into a single predicate declared next to the DirTimeList machinery in
file_receive.h and defined in file_receive.c.
Behavior preserved: identical short-circuit condition and semantics,
no signature or protocol changes.
- server gate: the --allow-unauthenticated loopback allowance now requires
an actual plaintext connection (!gate_ctx->ssl), so a loopback TLS client
whose cert fails the --client-cn check is refused before any SCRAM
challenge instead of falling through the plaintext opt-in. Keep the
invalid-fd guard as belt-and-braces (unreachable after the policy check).
- test: rewrote test_wrong_client_cn_refused_before_auth_challenge to run
deterministically over 127.0.0.1 with --tls + --allow-unauthenticated and
a CA-valid wrong-CN client cert, asserting the gate refusal log and an
unchanged module tree (no skip).
- docs: --client-cn is mandatory with --tls; dummykey sidecar is secret
material; document all transient-fallback reasons; qualify
--allow-unauthenticated in README and --help so it cannot read as
permitting remote plaintext auth.
- credentials.h: drop stale restrictive-umask claim (fchmod forces exact
0600; only create/write/fsync/link/fchmod failure degrades to ephemeral).
utils_fd_peer_is_local now returns true only when getpeername SUCCEEDS and the
peer address classifies as loopback. A non-socket descriptor (pipe/socketpair)
or any getpeername error is NOT local, so the daemon auth gate fails closed
instead of treating an untestable --stdio pipe as trusted (daemon auth modules
are --daemon-only and the stdio path never loads a daemon config).
server_module_gate now requires --allow-unauthenticated for the loopback
plaintext auth path: a plaintext loopback connection without the operator
opt-in is refused at the config gate BEFORE server_auth_handshake, so no SCRAM
challenge is sent. Remote peers still require verified TLS regardless of the
flag; the handler keeps its defense-in-depth checks.
Docs state the exact policy (verified TLS with matching --client-cn, or
operator-opted-in loopback plaintext), drop the SSH/stdio auth-transport claim
(they are daemon-only), and add the loopback trust-boundary relay caveat and
the CN-only (no SAN) residual. Adds a unit-test negative for pipe/socketpair
and an integration test where a relay observes no challenge when the flag is
absent.
Daemon modules that declare 'auth users' no longer accept credentials over a
remote plaintext connection: server_module_gate refuses at the config gate,
before any SCRAM challenge is sent, unless the connection is verified TLS with
a client certificate matching --client-cn, or a local/SSH transport (loopback
TCP peer or the --stdio pipe). --allow-unauthenticated does not relax this.
The TLS client-CN comparison now uses credentials_secure_equal (S2). Clients
sending --password-file to a non-loopback daemon must use --tls; validate_config
rejects the plaintext case before any network I/O.
Adds utils_sockaddr_is_loopback / utils_fd_peer_is_local / utils_host_is_loopback
helpers with unit tests, a client validation unit test, and integration tests
for the client-side plaintext rejection and the wrong-CN gate refusal.
- burn the store-wide dummy_key in credentials_free()
- burn the local mac on hmac_sha256 failure in credentials_get_verifier()
- always run the O(store) constant-time scan, even for off-list users, to
close the pre-existing off-list timing channel; select the real verifier
only when on_list && match
- clarify the server_auth_handshake STATUS_AUTH_FAILED comment (failure
before success vs. a dropped broken connection while writing the signature)
- document accepted anti-enumeration residuals (restart-gated dummy salt;
pre-auth-observable iteration count)
TapTap