Break the ~700-line parse_args god function into cohesive static helpers
grouped by concern: output controls, pre-negation, range/time options, the
OPTION_TABLE dispatcher, flag/meta handlers, IO/network options, filter and
logging options, checksum/socket options, remote/basis/identity options,
positional handling, and a final lowering step.
A file-local CliParseCtx carries the config, cursor, positional buffers and
the mutable parse flags, so each handler stays focused. The dispatcher calls
the handlers in the original recognition order and preserves the exact
return contract (0/1/negative), error messages, log levels and control flow.
Behavior preserved; no functional changes.
Type Config.super_mode as SuperMode (a proper C enum) instead of a bare
int. The wire boundary still carries the mode as an int: send casts the
enum explicitly and receive reads a temporary int, validates the
AUTO..OFF range, then casts. Emitted bytes and accepted values are
unchanged. ModuleGateContext.super_mode_override keeps its -1 sentinel
as int with an explicit cast at the apply site.
Behavior preserved.
Deduplicate the repeated directory-time capture gate
(`config->use_metadata && !config->omit_dir_times`) used by the
sender-side (multiprocessing.c) and receiver-side (receiver.c) sinks
into a single predicate declared next to the DirTimeList machinery in
file_receive.h and defined in file_receive.c.
Behavior preserved: identical short-circuit condition and semantics,
no signature or protocol changes.
- Protocol version 2.19.0 (SCRAM-SHA-256 daemon auth replacing the replayable digest)
- Salted PBKDF2 verifier store + --hash-credentials; legacy store hard-rejected
- Persistent anti-enumeration dummy key (<store>.dummykey)
- Verified TLS / opted-in loopback transport required for auth modules
- Secret wiping; carried-over hardening from the security phases
- Add CHANGELOG.md and set the CMake project version
- server gate: the --allow-unauthenticated loopback allowance now requires
an actual plaintext connection (!gate_ctx->ssl), so a loopback TLS client
whose cert fails the --client-cn check is refused before any SCRAM
challenge instead of falling through the plaintext opt-in. Keep the
invalid-fd guard as belt-and-braces (unreachable after the policy check).
- test: rewrote test_wrong_client_cn_refused_before_auth_challenge to run
deterministically over 127.0.0.1 with --tls + --allow-unauthenticated and
a CA-valid wrong-CN client cert, asserting the gate refusal log and an
unchanged module tree (no skip).
- docs: --client-cn is mandatory with --tls; dummykey sidecar is secret
material; document all transient-fallback reasons; qualify
--allow-unauthenticated in README and --help so it cannot read as
permitting remote plaintext auth.
- credentials.h: drop stale restrictive-umask claim (fchmod forces exact
0600; only create/write/fsync/link/fchmod failure degrades to ephemeral).
- Make the atomic-publish temp name unpredictable by appending 16 random
hex chars to the pid, so a leftover/planted temp cannot be targeted.
- On EEXIST, unlink the stale temp and retry the O_EXCL create once
(bounded), so a crash leftover or reused pid cannot silently defeat
sidecar persistence.
- fchmod the temp fd to 0600 after creation (umask can clear owner bits)
and treat failure as a create failure, so the published sidecar is
always exactly 0600.
- Clarify comments: the sidecar requires exact 0600 while the store and
password files only reject group/other bits.
- Add a unit test that a restrictive umask still yields an exact 0600
sidecar; clean random-suffixed temps in tests.
utils_fd_peer_is_local now returns true only when getpeername SUCCEEDS and the
peer address classifies as loopback. A non-socket descriptor (pipe/socketpair)
or any getpeername error is NOT local, so the daemon auth gate fails closed
instead of treating an untestable --stdio pipe as trusted (daemon auth modules
are --daemon-only and the stdio path never loads a daemon config).
server_module_gate now requires --allow-unauthenticated for the loopback
plaintext auth path: a plaintext loopback connection without the operator
opt-in is refused at the config gate BEFORE server_auth_handshake, so no SCRAM
challenge is sent. Remote peers still require verified TLS regardless of the
flag; the handler keeps its defense-in-depth checks.
Docs state the exact policy (verified TLS with matching --client-cn, or
operator-opted-in loopback plaintext), drop the SSH/stdio auth-transport claim
(they are daemon-only), and add the loopback trust-boundary relay caveat and
the CN-only (no SAN) residual. Adds a unit-test negative for pipe/socketpair
and an integration test where a relay observes no challenge when the flag is
absent.
Address review findings on the persistent dummy-key sidecar:
- Publish atomically: write a private same-directory temp file
(<store>.dummykey.tmp.<pid>, 0600), fsync, then link(2) into place;
fsync the containing directory and drop the temp name. A concurrent
starter can no longer observe a zero/partial sidecar and fail closed.
On EEXIST adopt the winner's sidecar; otherwise warn and use a
transient ephemeral key.
- Harden the read path (initial and EEXIST-adopt) with
O_RDONLY|O_NOFOLLOW|O_NONBLOCK|O_CLOEXEC: reject planted symlinks
(ELOOP fails closed) and never block on a planted FIFO.
- Require the exact owner-only mode (st_mode & 07777) == 0600 and make
the rejection message truthful.
- Report a clear "short write" instead of a stale strerror(errno) when
write() returns 0.
- Document the artifact and its creation-failure caveat (FIFO store
path, read-only filesystem, missing directory) in README.md and
RSYNC_COMPAT.md.
- Tests: known-key sidecar adoption (dummy salt KAT + reload), symlink
rejection, and the exact-0600 rule (0400 now rejected).
Daemon modules that declare 'auth users' no longer accept credentials over a
remote plaintext connection: server_module_gate refuses at the config gate,
before any SCRAM challenge is sent, unless the connection is verified TLS with
a client certificate matching --client-cn, or a local/SSH transport (loopback
TCP peer or the --stdio pipe). --allow-unauthenticated does not relax this.
The TLS client-CN comparison now uses credentials_secure_equal (S2). Clients
sending --password-file to a non-loopback daemon must use --tls; validate_config
rejects the plaintext case before any network I/O.
Adds utils_sockaddr_is_loopback / utils_fd_peer_is_local / utils_host_is_loopback
helpers with unit tests, a client validation unit test, and integration tests
for the client-side plaintext rejection and the wrong-CN gate refusal.
The store-wide dummy key was regenerated on every credentials_load, so an
unknown user's dummy salt changed across daemon restarts while a real user's
stored salt stayed stable -- a restart-gated username-enumeration oracle.
Persist the 32-byte key in a 0600 <store>.dummykey sidecar next to the
credential store. An absent sidecar is created with O_EXCL and fsynced; a
present sidecar is read only when it is an owner-only regular file of exactly
32 bytes (otherwise the load fails closed). If the sidecar cannot be created
(read-only mount, missing directory) fall back to a transient per-run key with
a warning. A NULL store path keeps the key ephemeral.
- burn the store-wide dummy_key in credentials_free()
- burn the local mac on hmac_sha256 failure in credentials_get_verifier()
- always run the O(store) constant-time scan, even for off-list users, to
close the pre-existing off-list timing channel; select the real verifier
only when on_list && match
- clarify the server_auth_handshake STATUS_AUTH_FAILED comment (failure
before success vs. a dropped broken connection while writing the signature)
- document accepted anti-enumeration residuals (restart-gated dummy salt;
pre-auth-observable iteration count)
- tests: pass CREDENTIAL_KEY_LEN to unhex for the 32-byte KAT proof/sig
(sizeof(expect) is 348, over-reading the 65-byte hex literal under ASan)
- credentials: close the username-enumeration oracle with a store-wide
dummy_key and a deterministic per-username dummy salt; make the store's
iteration count uniform (reject intra-file and layered disagreements) and
answer a miss with the store-wide count; run the constant-time key compare
even when found=false and fold the decision with bitwise AND
- credentials_compute_keys: enforce [CREDENTIAL_MIN_ITERS, CREDENTIAL_MAX_ITERS]
- tests: recompute the whole KAT independently at CREDENTIAL_DEFAULT_ITERS
(600000) and pin the golden store line; add non-uniform-store rejection,
bound and deterministic-dummy-salt assertions
- server: send exactly one generic STATUS_AUTH_FAILED on every failure path
(including credentials_get_verifier failure); route all handshake exits
through one burn path
- credentials/server: burn the base64 decoders' scratch on error, the
hash_store_line base64/line buffers on failure, and all handshake key/proof
material
- fuzz: guard the auth-offset scan against size_t underflow and use a found flag
- docs: drop stale digest wording, use CREDENTIAL_MIN_ITERS as the --iterations
bound, document 0600 output for --hash-credentials (plus a stderr warning on
a group/other-accessible stdout file), and describe the deterministic dummy
salt in the no-oracle claims
output_escape() was called on every send_str/receive_str even when
LOG_DEBUG_PROTO logging was disabled, allocating and scanning the whole
payload for a line that log_debug_message() then discarded. Add a
log_debug_enabled(flag) gate mirroring log_debug_message()'s own filter and
check it before escaping. Redacted (secret) strings still log the same
<redacted> marker; no observable log output changes.
- server_module_gate: refuse client-chosen ownership against the ORIGINAL config
so an explicit --super is still refused under an operator --no-super veto
(the veto must not turn a refusal into an accept).
- credentials: open-then-fstat the exact secret inode, require current-user
ownership and no group/other bits, but continue to allow process-substitution
FIFOs; removes the stat->fopen TOCTOU.
- file.c preallocate + protocol.c send-string debug logs escape attacker paths.
- usage/RSYNC_COMPAT updated for --old-args no-op and secret-file rules.
Add two libFuzzer harnesses (GLOBbed from tests/fuzz/*.c) and deterministic
P8 config-frame receive tests:
- fuzz_config_receive.c drives config_receive() from arbitrary bytes. It
captures one canonical valid frame with the production sender and feeds the
receiver four shapes: raw bytes, valid-version-prefix + fuzz bytes, valid
frame minus the P8 tail (super_mode + copy-as) + fuzz bytes, and valid frame
minus the usermap count + fuzz bytes. This reaches the --super/--copy-as and
huge/negative map-count paths that random bytes cannot get through the
preceding wire-bool gate.
- fuzz_identity_parse.c fuzzes identity_parse_copy_as/map/chown plus the
identity_wire_valid/identity_ownership_requested predicates on a fresh
config per input.
- test_fuzz_smoke.c gains deterministic malformed-frame cases: out-of-range
super_mode, negative/extreme copy-as ids, non-bool copy-as presence, tail
truncation, huge/negative usermap counts, version mismatch and a
wrong-order field after the version gate.
Unit build (STRICT_WARNINGS) and the fuzz build are clean; both targets run
3000+ iterations with no crash. No production code changed.
- A6: escape attacker-controlled file paths and the receive root in log
lines (file_receive, server, protocol DEBUG) with output_escape()
- A8: identity_set_active() returns bool and fails closed when a requested
usermap/groupmap cannot be deep-copied; handler refuses the connection
- remove the const cast and duplicate super_mode clamp from
server_module_gate via an explicit override the handler applies once
- release the identity snapshot on the queue_create failure path
- refactor identity_parse_copy_as to a single cleanup tail and drop the
duplicated group error format specifier
- file: open -K dirlink referents via a race-safe relative O_NOFOLLOW walk
from the authorized-root fd instead of re-opening an absolute realpath()
result (removes the intermediate-symlink swap TOCTOU).
- transport_ssh: always single-quote the server path, including --old-args,
so no mode can inject shell metacharacters.
- transport_tls: set SSL_OP_NO_COMPRESSION and (guarded) SSL_OP_NO_RENEGOTIATION.
- credentials: reject --password-file/--early-input with any group/other
permission bit; chmod 0600 the affected test fixtures.
- file_store: export file_store_write_sparse() and remove the verbatim
file.c duplicate.
- server_cli: handle --password-file/--early-input/--iconv via arg_has_value
in one place, removing the unreachable duplicate separate-form arms while
keeping both --opt VALUE and --opt=VALUE working
- client_cli: factor the triplicated --delta-block/--block-size range check
into set_delta_block_size(); drop the redundant use_metadata assignment
after identity_parse_copy_as (the parser already forces it)
- tests: cover both spellings of --iconv/--delta-block, make the archive
short-form test actually call parse_args, add delta-block invalid cases
- file_ensure_directory_secure() now chowns a final directory it creates under
--copy-as and fails on error; the symlink parent-creation call site propagates
it. The is_dir branch fails when the confined parent cannot be opened under
--copy-as. Closes the residual wrong-owner gap for synthesized/symlink
parent directories.
- file_restore_symlink_metadata() early NULL return is copy-as-aware.
- Preserve errno across the implicit-parent failure cleanup.
- Neutral skip messages (the clamp, not --no-super, may be responsible).
- Daemon copy-as test tolerates the non-root privilege refusal; usage text lists
--copy-as.
- H3: a daemon module without 'client owner = yes' now also has super-user
device activity forced off (char/block mknod, --write-devices), so a root
daemon can no longer be made to create/write raw devices under AUTO. The
entries are skipped, preserving ordinary -a pushes.
- H1/H2: propagate a failed required --copy-as chown from symlink metadata
restore and implicitly-created parent directories, so the entry (and run)
reports failure instead of a wrong-owner success.
- Docs/help/headers updated for A2/A3 and the device clamp; startup warning
spells out the client-owner risk.
- Tests: daemon device clamp (skipped without opt-in, created with opt-in),
updated --super/--fake-super expectations.
A1: daemon refuses every client-chosen ownership/super-user request
(--numeric-ids/--chown/--usermap/--groupmap/--fake-super/--copy-as/--super)
unless the selected module opts in with 'client owner = yes'.
A2: fake-super owner replay requires an explicit ownership identity policy.
A3: --super no longer implies --numeric-ids (ownership stays opt-in).
A5: a failed --copy-as chown marks the entry failed instead of reporting
success with the wrong owner.
Add the receiver-side --super / --no-super tri-state (Config->super_mode)
under the safe-subset + clear-refusal privilege model: FastSync never
elevates privileges, it only permits super-user attempts that are already
confined fd-relative below the authorized receive root.
- identity: privilege_super_permitted() gate (OFF=false, ON=true, AUTO follows
geteuid()==0); identity_apply_ownership/_link become no-ops when not
permitted; --super with no explicit identity policy implies raw numeric-id
preservation (explicit usermap/groupmap/chown/numeric-ids still win); warn
exactly once when --super is requested by a non-root receiver.
- file_receive: gate char/block device-node creation on the gate; FIFO/socket
handling is unchanged.
- wire: trailing super_mode int after the --iconv spec, validated 0..2 in
receive_privilege_options and validate_received_config; PROTOCOL_VERSION
2.17.0 -> 2.18.0; version-sensitive tests and docs updated.
- CLI: --super/--no-super parsed explicitly before the generic --no-* branch
(malformed --super=x rejected); usage text added.
- tests: config wire round-trip + invalid-value rejection, privilege-gate mode
unit test, CLI parse test, integration transfer + root-gated ownership
suppression/appliance tests.
- docs: RSYNC_COMPAT --super row + Wave E note, protocol mentions, README.
Force the receiver to apply the requested owner/group to every written
entry through the confined fd-relative identity path instead of switching
the process credentials (unsafe for the multithreaded receiver). An
unprivileged receiver refuses the transfer up front in server_module_gate,
before STATUS_OK, so no data is written with the wrong ownership.
- new Config fields copy_as_set/copy_as_uid/copy_as_gid + defaults
- identity_parse_copy_as (name/@N/* resolution, primary-gid default,
gid==uid fallback for numeric ids with no passwd entry); implies -M
- identity snapshot + highest-priority forcing in identity_resolve_targets
- identity_copy_as_refused() helper
- trailing config-frame block (presence int + two int32 ids, >=0 checked)
- PROTOCOL_VERSION 2.17.0 -> 2.18.0; version-sensitive tests updated
- unit tests for parse + wire round-trip/negative-id rejection
- integration TestCopyAs: unprivileged refusal + root chown assertion
- RSYNC_COMPAT.md --copy-as row updated (safe subset + divergence); README
protocol version refreshed
Review fixes for Phase 7 Wave D.
#1 (HIGH): STATUS_DIR_TIMES entries no longer create directories. A new
receiver-only File.dir_time_only flag marks dir-time entries; file_save_to_disk_full
short-circuits them as FILE_SAVE_SKIPPED before any device/dir branch, so the sink
still accumulates metadata into the deferred DirTimeList but creates nothing. Empty
source dirs stay untransferred (-a), -m/--prune-empty-dirs semantics are preserved,
and a pre-existing regular file/symlink at an empty-dir mirror path no longer aborts
the transfer. dir_time_list_apply fstatat()s the leaf (AT_SYMLINK_NOFOLLOW) and skips
absent/non-directory paths QUIETLY; only a real existing directory is stamped.
Also initialize File.dir_time_only in file_create() (uninitialised garbage otherwise).
#2 (MED): send_dir_times() chunks entries into repeated STATUS_DIR_TIMES frames of at
most MAX_MANIFEST_ENTRIES, matching the receiver's per-frame bound; the tautological
> INT_MAX check is gone.
#3 (LOW): dir_time_list_add() assigns each grown array right after its realloc (no
dangling) and advances capacity only after both succeed.
#4 (LOW): RSYNC_COMPAT.md -- STATUS_MKDIR carries metadata, dir times are transmitted
via STATUS_DIR_TIMES and applied at the end, empty dirs are still never created; -m
rationale, -O row and Wave D notes updated. Summary counts untouched.
#5 (LOW): integration tests for the three #1 scenarios (empty-dir non-creation under
-a and -a -m, collision non-abort), scanner test now covers empty-dir capture, and
test_file_restore_symlink_metadata asserts the positive apply path when supported.
PROTOCOL_VERSION stays 2.17.0; config-frame layout unchanged.
Wave D of Phase 7. Make -O/--omit-dir-times and -J/--omit-link-times real by
preserving directory and symlink times, and mark --secluded-args as an explicit
Impossible/Divergence no-op.
Wire: PROTOCOL_VERSION 2.16.0 -> 2.17.0. Adds a terminal STATUS_DIR_TIMES frame
(int count + (wire path, metadata) pairs) sent after all file data and the
optional delete manifest. STATUS_MKDIR also carries metadata for --dirs entries.
Config-frame layout is unchanged.
Sender: the recursive scanner captures every traversed source directory (both
DirectoryScanner and the parallel scanner root + workers, appends mutex-guarded)
into a shared list; the single-threaded and -m paths transmit it last.
Receiver: a DirTimeList accumulates received directory metadata and applies it
with fd-relative no-follow utimensat only at the very end -- after all children,
after the commit-style --delete, and after --delay-updates publication -- in the
single-threaded success frame and in server.c after the -m threads join. -O skips
the application. Symlink metadata is applied at link creation with
utimensat/fchownat/fchmodat AT_SYMLINK_NOFOLLOW; -J suppresses only link times.
identity_apply_ownership_link shares the identity resolver with the fd path.
Docs: -O/-J rows -> Implemented; --secluded-args -> Impossible/Divergence;
--protocol accepted/rejected values and Phase-6/7 notes updated.
Tests: unit (scanner dir capture, DirTimeList apply, symlink metadata, protocol
version values) and integration (dir mtime round-trip + -O, symlink mtime
round-trip + -J, independent suppression), parameterized over single/multithread.
- fake_super_restore_fd now sanitizes mode like metadata_mode (never grants
S_IWGRP|S_IWOTH; 0666 -> 0644), fixing a privilege regression
- --sparse takes precedence over --preallocate (skip posix_fallocate when
sparse) so holes are not re-allocated; docs corrected
- --partial retention disabled under --no_replace (ignore/existing) and only
marks write_attempted after the write begins (no empty-temp retention)
- accept --block-size=SIZE / --delta-block=SIZE inline forms; neutral messages
- fake-super EPERM/EACCES skipped silently (docs aligned); EINVAL still logged
- sparse unit test now memcmp's the full buffer; TestBlockSize integration keeps
the destination basis so delta is genuinely exercised
- unit 37/37, cppcheck 0, clang-format 0
- write_all_sparse: skips all-zero runs >= 4096 bytes via lseek(SEEK_CUR) and
ftruncates the final size, wired into the atomic temp+rename and --inplace
paths with no wire change (full image already in memory).
- --partial retention: on a save failure after the temp held data, rename the
already-written temp to the destination path (best-effort; falls through to
unlink; never retains when --partial is off) so --append/--append-verify can
resume; tested by forcing futimens EINVAL with an out-of-range nsec.
- --block-size aliases --delta-block; verified config->delta_block_size is
honored by the delta engine end-to-end (unit + integration tests).
- fake_super_restore_fd: parses and re-applies user.fastsync.stat fd-relative
(fchown best-effort/non-root skipped, fchmod, futimens); a save under
--fake-super now both records and re-applies.
- -N/--crtimes and --stderr=client promoted to a new 'Impossible/Divergence'
status bucket (Summary: 136/2/4/3/2 = 147).
- cppcheck/clang-format clean; unit 37/37; integration 407 passed.
- revert over-eager replacement of setfacl -m in test_features.py
- use --chunk-serialization (+ set dirs) in the two append/append-verify
chunk-serialization rejection unit tests so they exercise the real check
- rename archive-negation integration test (--no-preserve is inert under
archive because devices/specials force metadata)
- update stale (-c)/(-m)/(-s)/(-f) display labels and RSYNC_COMPAT -c/-m refs
- document the --no-perms negation limitation in the Wave A note
-c -> --checksum, -m -> --prune-empty-dirs, -M -> --remote-option,
-f -> --filter, -s -> --secluded-args, -p -> --perms, -T -> --temp-dir;
-a/--archive is now real rsync -rlptgoD (links+metadata+devices+specials).
FastSync's own flags moved to long-form-only or new shorts:
-j/--threads (multithreading), --preserve (metadata), --sendfile,
--chunk-serialization, --timeout, --ssh-port. Client-side only; the
wire config fields are unchanged (no PROTOCOL_VERSION bump). The server
keeps -p as its port. Docs (README, RSYNC_COMPAT summary 129->132) and
unit/integration tests updated. 37/37 unit, 400-pass integration.
Implements the client-only residual-batch feature end-to-end:
- src/shared/batch.{c,h}: self-contained single-file batch codec using the
existing chunk_serialize/chunk_deserialize codec (byte-identical by
construction). Magic+format-version header (metadata mode is persisted into
the header so a batch is self-describing across machines), length-prefixed
chunk records, bounded reads that reject malformed/truncated/oversized
records cleanly.
- src/client/client_send.c: write_batch_from_source (deterministic separate
scan pass, loads every chunk's file images, emits header+records) and
apply_batch_to_dest (local apply to a destination root via
file_save_to_disk_full). No wire change, no server involved.
- src/client/client_validation.c: --write-batch XOR --only-write-batch;
--read-batch exclusive with both; --read-batch needs only a DEST,
--only-write-batch only a SOURCE.
- src/client/client_cli.c: main() drives the three batch modes without
connecting/transferring for read/only-write; --write-batch runs the live
transfer (single-threaded so the config survives) then emits the batch.
- tests/test_batch.{c,h} (unit: byte-identical roundtrip with and without
metadata; bad-magic/truncated/oversized rejection) + tests/integration/
test_batch.py (only-write no-server, read-batch no-source roundtrip,
--write-batch with a live transfer, conflict rejections).
- clang-format: realign PART-1 config.h comment block.
No PROTOCOL_VERSION bump, no config-frame field, no server flag.
- test_credentials.c: NUL-terminate the overlong-line stack buffer before
make_tmp_file's strlen() (was a stack-buffer-overflow READ under ASan);
still exercises the overlong-rejection path.
- Add redacted protocol string variants (protocol_send_str_redacted /
receive + fd send_str_redacted/receive_str_redacted) and use them for the
daemon auth username/digest so --verbose / LOG_DEBUG_ALL never logs a
replayable credential while other protocol strings keep their debug trace.
- credentials_verify/gate: replace byte-wise-short-circuiting strcmp with a
fixed-length constant-time username compare (closes user-enumeration oracle);
update doc comment to match.
- read_secret_file: preserve password exact bytes (only strip trailing CR/LF)
and burn the stack line buffer; document the whitespace behavior.
- test_server_cli.c: note the parser zero-inits opts on failure.
- Add debug-level daemon test asserting the digest never appears under --verbose.
PROTOCOL_VERSION stays 2.15.0.
TapTap