262 Commits
Author SHA1 Message Date
TapTap 919a729206 Release v2.21.0
CI / lint (push) Successful in 1m25s
CI / lint (pull_request) Successful in 1m25s
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
CI / sanitizers (address) (push) Successful in 1m6s
CI / sanitizers (undefined) (push) Successful in 1m0s
CI / fuzz-build (push) Successful in 34s
CI / coverage (push) Successful in 55s
CI / build-and-test (pull_request) Successful in 1m49s
CI / valgrind (push) Successful in 3m19s
CI / build-and-test (push) Successful in 5m23s
- Protocol 2.21.0: STATUS_ERROR_DETAIL rejection reasons and server-contacting --dry-run
- Daemon per-module/per-host caps and cross-process auth lockout
- Config X-macro serialization, authorized_root single-owner, Data charge ownership, receiver pipeline move
- Security audit hardening (SSH injection, FIFO/inplace, zstd DoS, TLS, dry-run oracle, bounds)
- Pre-auth basis_count NULL-deref fix; benchmark and nix-shell improvements
- Tested: unit, integration, ASan/UBSan, valgrind, fuzz, coverage (CI green)
2026-09-14 18:16:42 +02:00
TapTap 8cd2b550d9 Merge dev environment fix and push-only documentation
CI / lint (push) Successful in 1m25s
CI / sanitizers (undefined) (push) Successful in 1m2s
CI / sanitizers (address) (push) Successful in 1m9s
CI / fuzz-build (push) Successful in 36s
CI / coverage (push) Successful in 56s
CI / valgrind (push) Successful in 3m18s
CI / build-and-test (push) Successful in 5m25s
2026-09-14 18:08:51 +02:00
TapTap 99c0fd8016 Merge benchmark improvements: accurate data mix, transfer verification, warm mode 2026-09-14 18:08:51 +02:00
TapTap a2200f039a chore(dev): fix nix-shell environment; document push-only direction 2026-09-14 18:08:46 +02:00
TapTap 1437c6dc6b bench: fix data mix, verify transfers, robust netem, warm mode
- generate_bench_data now writes exactly (1-random_ratio)*target bytes of
  genuinely compressible repeated content instead of only the small fixed
  STRUCTURED_FILES set; measured composition is reported and --dry-run prints
  it for scaling checks
- verify each transfer against the source (paths/sizes/byte compare) before
  recording timing; add --no-verify; failed runs are counted as invalid
- correct p50/p95 with linear-interpolation percentile (was int(len*0.95))
- tc/netem: run tc directly as root, else sudo; clear error when tc/iproute2
  is missing or qdisc setup fails; netem_reset is always safe
- build into dedicated build-bench/ via --build-dir (Release), never reconfigure
  the user's build/
- parse --configs with shlex.split
- add MB/s throughput column and throughput_mbps JSON field
- add --warm incremental mode: untimed full seed then measure add/change deltas
2026-09-14 18:07:37 +02:00
TapTap 38356ecc1e test: fix valgrind definite leak in forked compression truncation test
CI / lint (push) Successful in 1m24s
CI / sanitizers (undefined) (push) Successful in 1m1s
CI / sanitizers (address) (push) Successful in 1m6s
CI / fuzz-build (push) Successful in 35s
CI / coverage (push) Successful in 55s
CI / valgrind (push) Successful in 3m18s
CI / build-and-test (push) Successful in 5m25s
2026-09-14 17:54:20 +02:00
TapTap 7badac7f97 test(compression): free inherited Data in forked truncation test (valgrind) 2026-09-14 17:54:20 +02:00
TapTap 6ccf16b650 Merge security hardening wave: parser/compression, receiver confinement, server/transport/TLS
CI / lint (push) Successful in 1m26s
CI / sanitizers (undefined) (push) Successful in 1m2s
CI / sanitizers (address) (push) Successful in 1m7s
CI / fuzz-build (push) Successful in 35s
CI / coverage (push) Successful in 56s
CI / valgrind (push) Failing after 3m19s
CI / build-and-test (push) Successful in 5m26s
2026-09-14 17:40:29 +02:00
TapTap 1174993d6b Merge branch 'fix/sec-server' into fix/sec-integration 2026-09-14 17:38:23 +02:00
TapTap d5fcfa2c5c style(ssh): drop redundant condition flagged by cppcheck 2026-09-14 17:38:23 +02:00
TapTap e79d2b47b0 Merge branch 'fix/sec-server' into fix/sec-integration 2026-09-14 17:27:38 +02:00
TapTap 7c24a365cf Merge branch 'fix/sec-receiver' into fix/sec-integration 2026-09-14 17:27:38 +02:00
TapTap 5893de4a34 fix(receiver): close re-review findings — dry-run basis oracle, ACL capture, fsync reopen
Follow-up to a237043 addressing three security/correctness re-review findings.

(1) MEDIUM: a server-contacting --dry-run with --compare-dest/--copy-dest/
    --link-dest still read and hashed the basis file and compared it with the
    client-supplied digest, a 1-bit content oracle. basis_match_find() gains a
    hash_content parameter; the dry-run shortcut passes false and returns no
    match without touching basis bytes, so an otherwise-matching entry is
    reported as would-transfer. The real (non-dry-run) path is unchanged.

(2) LOW: xattr_capture_path() hardcoded preserve_acls=true, so the receiver's
    hard-link copy fallback re-applied system.posix_acl_* even when -A was not
    negotiated. The function now takes preserve_acls and members.* is
    unaffected; scanner and receiver callers thread the negotiated flag.

(3) INFO: the --fsync --link-dest temp reopen now uses O_NONBLOCK and treats
    a raced-in FIFO's ENXIO as a benign fsync-skip instead of blocking.

Tests: dry-run + basis unit test (asserts would-transfer, no content read) and
integration test; xattr capture ACL-filter test. Verified strict build, ASan,
clang-format, cppcheck, and the CI integration subset.
2026-09-14 17:19:27 +02:00
TapTap 825ba69753 fix(server): reject --allow-super with --stdio, fix module host-list append
Re-review findings on the C3/C4 hardening branch:

- --stdio is the SSH transport whose remote argv is composed by the client
  (including via --remote-option), so accepting --allow-super there let a
  client defeat the C3 secure default for a root receiver.  Reject it at CLI
  parse time (standalone TCP only) and force the process-global flag off for
  --stdio as defense in depth.  Correct the help text and README/RSYNC_COMPAT:
  the --stdio argv is client-composed, super stays off, and a forced command is
  needed if the default must hold.
- daemon_conf: the per-module 'hosts allow'/'hosts deny' call sites passed
  module_name and replace in the wrong order, so multiple lines replaced
  instead of appended and the empty-value error omitted the module name.  Pass
  (module->name, false) like the global keys; add a unit test for two
  per-module allow/deny lines appending.
- tls: read the client CN via ASN1_STRING_to_UTF8 so an exactly-required-length
  name is accepted and only actual over-length CNs are rejected.
2026-09-14 17:16:01 +02:00
TapTap 34abaadb9a fix: address low/informational sec-parser follow-ups
- client_cli: capture errno before output_escape() in
  read_patterns_from_file() so an over-long line is still reported as
  EFBIG instead of the (possibly malloc-clobbered) errno.
- file_list: guard string_list_add() capacity doubling against
  overflow (capacity > INT_MAX / 2), matching filter_rule_list_add();
  callers already surface the false as a memory-allocation error.
- compression: ZSTD_isError() is true for ZSTD_CONTENTSIZE_UNKNOWN,
  which made the 3x unknown-size fallback dead code.  Test the
  CONTENTSIZE_ERROR/UNKNOWN sentinels explicitly so unknown-size frames
  reach the estimate path (still bounded by the existing hard limit)
  while invalid frames are rejected.  Known-size frames and the 100 MB
  ceiling/overflow checks are unchanged.
- tests: add an unknown-content-size-frame decompression test.

Tests: ./build/tests and ./build-asan/tests all pass (42/42);
clang-format + cppcheck clean.
2026-09-14 17:11:02 +02:00
TapTap 10c4ffebdf fix(client): harden CLI args, log escaping, and local artifact opens
- parse_ull_arg() rejects a leading '-'/'+' (strtoull would silently wrap
  -1 to ULLONG_MAX) and --chunk-size/--delta-max enforce their upper bounds.
- Escape local untrusted paths before logging (client_send, scanner,
  --filter rule, pattern-file reads) with output_escape(..., 8-bit mode).
- Read --exclude-from/--include-from through the bounded line reader.
- Open --log-file with O_NOFOLLOW|O_CLOEXEC, mode 0600, via open+fdopen;
  create --write-batch with O_NOFOLLOW|O_CLOEXEC, mode 0600.
- Reject --dry-run together with --write-batch (dry-run must not write the
  batch file), alongside the existing --read-batch/--only-write-batch rules.

Tests: signed/oversized numeric rejection, over-long pattern file, dry-run +
write-batch unit and integration coverage.
2026-09-14 16:39:19 +02:00
TapTap dfa2a42028 fix(protocol): retry EINTR on receive and clamp SSL_write length
protocol_receive_n_data_until() aborted on a signal-interrupted plaintext
read (and on SSL_ERROR_SYSCALL with errno==EINTR); retry both, matching the
send path and protocol_read_status_until().  Also clamp each SSL_write() to
INT_MAX so a >INT_MAX size_t request can never truncate into a partial write.
2026-09-14 16:39:19 +02:00
TapTap 5b0ec5880d fix(filter): bound .rsync-filter lines and guard capacity growth
Read per-directory filter files through the bounded reader, guard the rule
list's capacity doubling against INT_MAX/2 overflow, and escape the local
directory path before logging a read failure.
2026-09-14 16:39:14 +02:00
TapTap 1a26bde2d4 fix(file_list): bound entry length and reject embedded NUL bytes
Read list files through utils_getdelim_bounded() so a single multi-gigabyte
line can no longer force unbounded allocation; over-long entries fail with a
clear error.  Also add the documented memchr() NUL-byte check (excluding the
NUL delimiter in NUL-separated mode).

Tests: an over-long entry is rejected with an 'exceeds' diagnostic.
2026-09-14 16:39:14 +02:00
TapTap 58a28334b8 fix(utils): bound glob matching and line reads
Replace the recursive glob matcher with an iterative O(pattern*string)
dynamic program.  The old recursion explored exponentially many paths for
overlapping '*'/'**' wildcards (e.g. '*a*a*...*b' against a long run of
'a'), a CPU DoS reachable from --exclude/--include patterns and
.rsync-filter.  A differential fuzz against the original matcher confirms
identical results.  Doc: has_path_traversal() is a lexical '..' check only.

Add utils_getdelim_bounded(): a getdelim-style reader that never allocates
beyond UTILS_MAX_LINE_LEN, used to cap untrusted list/filter line reads.

Tests: pathological glob completes quickly; bounded reader returns EFBIG on
an over-long record.
2026-09-14 16:39:10 +02:00
TapTap e48f19ee2b fix(compression): fail truncated zstd frames instead of spinning
data_decompress_limited() looped while ZSTD_decompressStream() returned a
positive hint.  A truncated frame keeps returning that hint with all input
consumed, so a malformed/truncated payload spun forever (CPU DoS).  Detect
input exhaustion with an incomplete frame and fail via the existing cleanup,
skipping the check when the output buffer merely needs to grow first.

Add a fork+alarm regression test that truncates a valid frame and asserts
decompression returns NULL promptly.
2026-09-14 16:39:05 +02:00
TapTap a2370433b2 fix(receiver): non-blocking receiver opens, inplace type gate, dry-run/B4/B5/B6
Address confirmed receiver security findings B1-B6:

B1 (HIGH): add O_NONBLOCK to the three receiver read-opens that opened an
existing destination/basis entry before the S_ISREG gate
(incremental_check_open_destination, basis_open_regular, hardlink_read_source)
so a client-planted FIFO can no longer block the receive thread forever while
the post-open type gate still rejects it.

B2 (HIGH/MED): --inplace now fstatat(AT_SYMLINK_NOFOLLOW)-probes the target and
refuses any existing non-regular entry, opens with O_NONBLOCK, and re-checks
S_ISREG on the opened fd.  This stops a FIFO from hanging the open and stops a
char/block device from being written directly (bypassing --write-devices).

B3 (MED): under --dry-run the incremental quick-skip no longer reads/hashes the
destination file for --checksum/--delta; it decides from metadata only and
reports would-transfer when the comparison is inconclusive, closing the
read-only-module content-hash oracle.

B4 (LOW): xattr_name_appliable() now gates the two system.posix_acl_* names on
preserve_acls (--acls), not the derived use_xattrs (--xattrs OR --acls).  The
receiver drops (never applies) ACL entries when -A was not negotiated while
keeping user.* working for -X.

B5 (INFO): receive_manifest_section() charges a per-entry overhead against
MAX_MANIFEST_BYTES and the aggregate entry count across all three sections is
capped at MAX_MANIFEST_ENTRIES.

B6 (MED): data_charge_session() reserves decompressed/chunk-copy bytes against
the owning ProtocolSession (MAX_CONNECTION_MEMORY) and records them on the Data
so data_destroy() releases them via the Data.owner path.  Applied to the
whole-file/append/delta decompression sites and chunk_deserialize() per-file
copies; a missing session owner degrades to the previous uncharged behavior.

Tests: FIFO destination/basis non-hang (with alarm), --inplace FIFO/device
refusal, dry-run no-read oracle test plus updated metadata-only dry-run tests,
ACL-without--acls drop, manifest total-entry cap, and chunk session charging.
2026-09-14 16:19:26 +02:00
TapTap 9da5a0a9ed fix(server): gate --force by --allow-delete and secure root super default
C2: --force is deletion authority (an incoming regular file may remove a
non-empty destination directory tree, and --delete-missing-args may
remove a non-empty directory mirror), but it was not masked by the
operator --allow-delete policy.  The handler now clears
config->force_delete unless --allow-delete was given, exactly like
--delete and --delete-missing-args.

C3: a standalone TCP / --stdio server running as root defaulted to
SUPER_MODE_AUTO, so an untrusted client --devices/--write-devices/
--super could make it create device nodes, write raw devices, or apply
client-chosen ownership.  A privileged standalone receiver now forces
SUPER_MODE_OFF unless the operator opts in with the new server-only
--allow-super flag.  Non-root receivers are unchanged, and the daemon
path keeps its per-module `client owner = yes` gate.  --allow-super is
rejected with --no-super or --daemon.

C6: tls_client_identity_allowed now rejects a CN whose reported length
reached the buffer bound, so a truncated over-long CN cannot be matched
by a required --client-cn prefix.

Tests: an integration regression proving --force cannot replace a
destination directory without --allow-delete; standalone-default tests
for --copy-as refusal and (root-only) skipped device creation; a CLI
unit test for the new flag.  The integration shared_server fixture opts
in with --allow-super so the existing root-only ownership/device/copy-as
tests continue to exercise the opted-in configuration.  README and
RSYNC_COMPAT document the flag and the force/delete gating.
2026-09-14 16:09:44 +02:00
TapTap 80c1ff321c fix(credentials): length-check before legacy-hex scan (C9)
secret_is_legacy_hex indexed s[0..63] without first checking the string
length, reading out of bounds for a shorter secret.  Require
strlen(s) == 64 before scanning, and add a unit test that short and
63-hex-digit secrets are rejected as ordinary malformed verifiers (never
misreported as legacy).
2026-09-14 16:09:25 +02:00
TapTap 551c187005 fix(tls): AEAD-only 1.2 suites, server preference, TOCTOU key load, IP SAN
C5: restrict the TLS 1.2 and below cipher list to ECDHE AEAD suites
(ECDHE+AESGCM:ECDHE+CHACHA20, minus NULL/eNULL/MD5/RC4/3DES) instead of
HIGH (which includes CBC), and set SSL_OP_CIPHER_SERVER_PREFERENCE so the
server's order decides the negotiated cipher.  Client and server share
create_ssl_ctx, so both are updated.

C7: load the private key through an O_RDONLY|O_NOFOLLOW|O_CLOEXEC fd,
fstat that fd and validate owner/mode (now also rejecting group/other
execute bits), then load from the fd via BIO_new_fd.  This removes the
stat-to-load TOCTOU race while keeping the exact-owner/0600 policy.

C8: verify an IP-literal client hostname against the certificate IP SAN
with X509_VERIFY_PARAM_set1_ip_asc instead of SSL_set1_host (a DNS
check), falling back to SSL_set1_host for real names.

Unit tests assert the server-preference option, the absence of CBC/RC4/
3DES suites, and that context creation still succeeds.
2026-09-14 16:09:21 +02:00
TapTap 0d6c1f784f fix(daemon-conf): reject empty hosts/auth allow-lists (C4)
A present hosts allow/hosts deny/auth users key with an empty or
separator-only value produced a zero-length list, silently meaning no
ACL / no auth and contradicting the strict-parse contract.

store_host_list and the auth users parser now track how many entries a
present key actually added and fail the load with a clear error when it
is zero, so a restrictive directive can never silently become open.
Unit tests cover empty, whitespace-only and comma-only values.
2026-09-14 16:09:16 +02:00
TapTap f75a69f96a fix(ssh): reject option-injection destinations (C1)
A remote destination's user@host token is passed to ssh in option
position, so a host beginning with '-' (e.g. -oProxyCommand=...) was
parsed by ssh as an option, allowing arbitrary command execution.

- config_parse_ssh_dest now validates the user@host prefix and returns
  -1 (with a clear logged error) for an empty host or a user/host that
  starts with '-'; config_parse_transport_dest propagates the failure.
- transport_ssh.c's parse_remote_dest applies the same validation as
  defense-in-depth, and ssh_build_client_argv inserts a '--'
  end-of-options marker before the destination token.
- Unit tests cover -oProxyCommand=... / -prefixed hosts / empty host
  rejection and the argv shape.
2026-09-14 16:08:56 +02:00
TapTap df887c73b1 Merge Wave 9: error-detail frame and server-contacting dry-run (protocol 2.21.0)
CI / lint (push) Successful in 1m21s
CI / sanitizers (undefined) (push) Successful in 1m1s
CI / sanitizers (address) (push) Successful in 1m7s
CI / fuzz-build (push) Successful in 36s
CI / coverage (push) Successful in 56s
CI / valgrind (push) Successful in 3m17s
CI / build-and-test (push) Successful in 5m16s
2026-09-13 13:39:21 +02:00
TapTap 5d39619a8a Merge branch 'feat/w9-dryrun' into fix/w9-integration 2026-09-13 13:27:32 +02:00
TapTap 6269ae54e5 test(dry-run): strengthen no-mutation coverage and refresh docs
Extend _snapshot_tree to record mode, inode, xattrs, directories and
special nodes, and add coverage proving a server-contacting --dry-run
leaves the destination structurally identical for --delay-updates,
--backup, symlinks, hardlinks, FIFOs, and daemon modules (including a
read-only module).  Add a regression test for the --read-batch --dry-run
refusal and for a missing/non-directory receive root failing a dry-run
exactly like a real run.

Fix stale version comments (2.20.0/633 -> 2.21.0/637) and RSYNC_COMPAT's
current --protocol value, and add a unit assertion that
--server-port/--port (and --server-host) set the dry-run routing bit.
2026-09-13 12:57:37 +02:00
TapTap 07f7555c1d fix(server): skip dry-run per-file outcome bookkeeping
receiver_save_file appended to context->outcomes for --remove-source-files
without the !dry_run guard the multithreaded pipeline has, so a hostile
dry-run client could grow outcomes unbounded (raw, uncharged realloc) and
force a per-frame ack.  Guard the append on !dry_run.
2026-09-13 12:57:33 +02:00
TapTap 5b8aca5799 fix(receive): enforce dry-run no-mutation centrally
--dry-run --read-batch=FILE still wrote to the destination because
batch_read_apply -> file_save_to_disk_full bypassed the per-caller
!dry_run guards.  Guard file_save_to_disk_full and manifest_delete_all
directly (return SKIPPED/no-op) so every save/delete path is mutation-free
in dry-run, and keep the per-caller guards.  Reject --dry-run combined with
--read-batch/--only-write-batch at CLI validation with a clear error (a
dry-run of a local batch apply is not meaningful).
2026-09-13 12:57:30 +02:00
TapTap a1eaa93357 fix(client): route explicit remote dry-run targets, fail closed on stray status
--dry-run --server-host=H (or TLS / source-bind --address) silently ran the
client-side manifest even though a real run contacts the server.  Add a
client-only, never-serialized server_host_set bit (alongside the existing
server_port_set) and extend dry_run_targets_server so every explicit remote
target contacts the receiver.

Also make incremental_check return the dry-run code (4) only when the
session actually requested dry-run; a stray STATUS_DRY_RUN_TRANSFER from a
hostile/buggy peer is now a logged protocol error (STATUS_ERROR) instead of
falling through to send file data and desync.  Both normal send_single_file
callers handle rc == 4 explicitly as an abort.
2026-09-13 12:57:26 +02:00
TapTap 99df0a8a6d fix(server): fail-closed dry-run destination-root precondition
A wire dry_run bit must not relax the destination-root precondition:
previously handlers skipped ensure_receive_root entirely in dry-run, so a
client could dry-run against a nonexistent/regular-file root a real session
rejects.  Split the existence check (receive_root_exists, never creates)
from the create path and apply the precondition unconditionally: dry-run
runs the existence/directory check only, reports the failure, and creates
nothing (no --mkpath).

Also allow a `read only = yes` daemon module for a dry-run session (a
server-contacting dry-run IS a read-only wire operation) while still
refusing it for real writes, and update the stale read-only comments.
2026-09-13 12:57:23 +02:00
TapTap 334fc5b3e8 fix(protocol): harden STATUS_ERROR_DETAIL receive path
Address review/security findings in the 2.21.0 error-detail feature:

- Keepalive drain no longer erases the terminal detail: capture/clear is
  skipped for STATUS_KEEPALIVE so the reason the peer just sent survives the
  owed keepalive replies.
- Replace the capture path with a dedicated protocol_receive_error_detail:
  the declared length is validated against MAX_ERROR_DETAIL_BYTES before any
  allocation, over-cap bodies are drained through a fixed scratch buffer (so
  the stream never desyncs), in-cap bodies read straight into the thread-local
  detail buffer, and session->max_alloc is never raised.  Lengths beyond
  MAX_STRING_SIZE are treated as a fatal framing error.
- The detail body now honors the caller's deadline (timed/keepalive paths) and
  polls the abort callback between drain chunks.
- Escape peer-controlled detail text with output_escape before logging it in
  client_send.c and config.c.
- Clear io_error_detail in io_set_fds so a new connection on the same thread
  cannot inherit a stale reason.
- Add unit tests for the keepalive-survival, over-cap drain, absurd-length
  fatal framing, and deadline-clamped body read cases.
2026-09-13 12:40:03 +02:00
TapTap 88aee6ce94 feat(protocol): add optional STATUS_ERROR_DETAIL rejection reason (2.21.0)
Today a server rejection sends a bare STATUS_ERROR and the reason only
reaches the server log, so the client cannot say why a transfer was
refused.  Add an optional, bounded server->client error-detail frame:

  - Status gains STATUS_ERROR_DETAIL appended LAST so existing wire
    values are unchanged.
  - send_error_detail(fd, msg) sends STATUS_ERROR_DETAIL followed by the
    existing length-prefixed string primitive, slicing over-long messages
    to MAX_ERROR_DETAIL_BYTES (4096).
  - receive_status() (and the timed/keepalive status readers) always
    consume the detail body and map the status back to STATUS_ERROR,
    capturing the text into a thread-local buffer exposed by
    protocol_last_error(); a bare STATUS_ERROR leaves it cleared.  Every
    existing call site keeps working and the stream cannot desync.
  - Upgrade the daemon module gate / config validation (config.c), the
    final transfer failure (server.c) and receiver-side path/node
    validation (file_receive.c) to send a concrete reason; surface it on
    the client in client_send.c/config.c.
  - Bump PROTOCOL_VERSION to 2.21.0 (CMake VERSION, CHANGELOG, docs) and
    update the pinned config wire golden hash / CLI-version tests.
  - Add tests/test_protocol_error.c covering mapping+capture, the
    over-long bound, bare-error clearing, and thread-locality.
2026-09-13 12:19:46 +02:00
TapTap f6e8b6ddc4 refactor(file_receive): split receive_incremental_check into helpers
The per-file STATUS_CHECK fast path was a single 534-line function that
was hard to review.  Extract it into small static helpers called in order
by a short linear orchestrator:

  - incremental_check_receive_request  (receive/validate request frame)
  - incremental_check_open_destination (secure open + stat)
  - incremental_check_quick_skip       (metadata/content skip decision)
  - incremental_check_try_basis        (compare/copy/link-dest)
  - incremental_check_try_append_resume(--append tail resume)
  - incremental_check_try_delta        (block delta)
  - incremental_check_try_fuzzy        (--fuzzy basis)
  - incremental_check_receive_full     (STATUS_NEXT + whole file)

Pure refactor: the ordered sequence of wire operations
(send_status/send_n_data/receive_n_data/receive_status/receive_wire_str)
is byte-for-byte identical to the original, and every resource cleanup
is preserved (a unified idempotent cleanup replaces the duplicated
per-path close/free blocks).  No functional changes.
2026-09-13 12:04:31 +02:00
TapTap 6f974eff19 feat(dry-run): server-contacting --dry-run (protocol 2.21.0)
--dry-run now handshakes with a remote/daemon receiver and reports what
WOULD transfer/skip based on receiver state, mutating nothing on either
side.

- Serialize Config.dry_run into the wire config frame and append
  STATUS_DRY_RUN_TRANSFER to the status enum (no renumbering); bump
  PROTOCOL_VERSION/CMake VERSION/CHANGELOG/golden wire to 2.21.0.
- Receiver: receive_incremental_check_ex runs the normal read-only
  decision and answers STATUS_OK (skip) or STATUS_DRY_RUN_TRANSFER
  (would transfer) with no basis materialization/append/delta/full
  transfer.  All mutation sites are guarded by !dry_run: file store,
  manifest deletes, --mkpath root creation, --delay-updates staging,
  publication, directory-time application, and outcome acks.
- Client: send_dry_run_remote connects, sends the config, checks each
  regular file and prints the would-transfer set + trailer; no file data
  or delete manifest is sent.  Plain local destinations keep the
  client-side manifest.
2026-09-13 11:56:05 +02:00
TapTap 72cccaa256 Merge Wave 8: config X-macro, authorized_root dedup, daemon limits, protocol_charge ownership
CI / lint (push) Successful in 1m31s
CI / sanitizers (undefined) (push) Successful in 1m0s
CI / sanitizers (address) (push) Successful in 1m6s
CI / fuzz-build (push) Successful in 34s
CI / coverage (push) Successful in 53s
CI / valgrind (push) Successful in 3m14s
CI / build-and-test (push) Successful in 5m33s
2026-09-13 11:19:55 +02:00
TapTap eb71b29d1c Merge branch 'fix/w8-authroot' into fix/w8-integration 2026-09-13 11:13:40 +02:00
TapTap 4d5befedfe Merge branch 'fix/w8-charge' into fix/w8-integration 2026-09-13 11:13:40 +02:00
TapTap f00844cf9a Merge branch 'fix/w8-daemonlim' into fix/w8-integration 2026-09-13 11:13:40 +02:00
TapTap 2ec17e821c fix(daemon): harden bounded per-source registry races
Stamp host_last_use before publishing a bucket key and treat an unstamped
(last_use == 0) bucket as live, so a just-claimed bucket can no longer be
stolen by a concurrent reclaimer.

After a successful eviction CAS, re-scan for the interned key and, when an
earlier bucket already holds it, zero the duplicate's active count and
return the canonical bucket, preventing orphaned per-host counts and cap
overshoot under full-table concurrency.

Add a message-carrying EXPECT_FAIL primitive and use it for the daemon-conf
buffer-overflow guard, and add a fork-based test that records auth failures
from forked children and asserts the parent observes the shared lockout.
2026-09-13 11:11:23 +02:00
TapTap 6d47d93fd7 fix(config): validate received counts before publishing them
The config_receive_{basis,skip,idmap}_count helpers wrote the
peer-controlled int through the Config member before range-checking it.
An over-cap basis_count therefore left config->basis_count huge while
config->basis_dirs was still NULL; config_receive()'s error path then
called config_delete(), whose basis loop dereferenced NULL and crashed
the daemon before authentication.

Read each count into a local, validate, and only then assign, leaving the
member untouched on failure.  config_delete() also guards the basis loop
with the array pointer as defense in depth.

Add a regression test that feeds over-cap basis/idmap/skip counts and
asserts rejection without crashing, plus a direct config_delete() check
on the partial (count set, array NULL) state.
2026-09-13 11:05:57 +02:00
TapTap 6ea966781f test(config): add receive-side golden oracle and sharpen fixture
Address low-severity review findings on the X-macro config refactor:

1. The golden test only hashed config_send_wire_block(), so a
   receive-side KIND that reads a different width/order could still
   round-trip symmetrically.  Add test_config_wire_golden_receive():
   capture the same hash-pinned 633-byte frame and feed it through
   config_receive(), asserting every field (config_wire_equal) plus the
   derived use_delta/use_xattrs bits and representative bounded kinds.
   Add test_config_wire_receive_bounds() for bounds the symmetric
   round-trip cannot reach: an out-of-range BOOL (hand-built frame),
   RAW_MAXALLOC zero, a malformed STR_MODULE, an over-cap
   INT_IDMAPCOUNT, and an out-of-range INT_IDENTITY chown_uid.

2. golden_config_populate() set long runs of booleans to all-1, so an
   adjacent swap within a run produced identical bytes.  Alternate the
   boolean values and make the fixture receiver-valid (chmod grammar
   "u=rwx,go=rx" is the same 11 bytes; delta_max_file_size inside the
   bound).  Re-pin the golden: len stays 633, hash is now
   9160991280011164139 (computed, not guessed).

3. Document in config.h and client_cli.c that the CLI option tables
   remain hand-maintained and are deliberately not generated from the
   wire-field X-macro (client-only fields, flag/alias/negation
   semantics).  No CLI-table rewrite.

PROTOCOL_VERSION stays "2.20.0"; src/shared/config.c is untouched and
the wire bytes are unchanged apart from the fixture's own new values.
2026-09-13 10:56:34 +02:00
TapTap 0a7f5faea6 test: replace strcat with a bounds-checked append in daemon-conf test 2026-09-13 10:51:01 +02:00
TapTap 25909110ac fix(daemon): exempt trusted loopback peers from per-host limits
Every client on loopback shares the 127.0.0.1 identity, so counting them
against 'max connections per host' or the default-on auth lockout lets one
local client deny service to all the others (and makes a shared-NAT/proxy
address a natural DoS vector for remote clients).  Use
utils_fd_peer_is_local (fail-closed) in the daemon gate to exempt a
provably local peer from the per-source cap and the auth lockout while
keeping the per-module and global caps.  Remote peers are unchanged.

Document the shared-NAT/proxy identity limitation and the loopback
exemption in README/RSYNC_COMPAT/CHANGELOG, update the integration test to
assert the exemption, and fix the README 'auth failure delay' cap (5000,
not 60000).
2026-09-13 10:50:58 +02:00
TapTap bd43448af2 fix(daemon): bound per-source table lifetime and recompute occupancy
The per-source host table only grew: once its fixed open-addressed table
filled, host_intern returned -1 and the per-host cap plus the shared auth
lockout silently failed open forever.  Add a bounded-lifetime eviction
policy: track a per-bucket last-use time and, when no empty bucket exists,
atomically repurpose the first bucket that has no active connection and
either has an expired lockout or has been idle, resetting its counters.
Warn (rate-limited) on the genuine fail-open path.

A child SIGKILLed mid-registration could also leak a module/host count
because the parent only decremented on a REGISTERED slot.  Make the slot
table the source of truth: after the SIGCHLD reap the parent recomputes
module_active[]/host_active[] from the surviving REGISTERED slots (atomics
only, async-signal-safe) so any leaked increment is erased.

Also clamp module_count to DAEMON_LIMITS_MAX_MODULES and use one helper
for the sizing/register host-tracking condition (a lockout threshold with
duration 0 is a no-op and must not intern hosts).
2026-09-13 10:50:53 +02:00
TapTap 264964411c Merge PR #283: chore(opencode): fix drifted agent/skill docs and repo hygiene
CI / lint (push) Successful in 1m29s
CI / sanitizers (undefined) (push) Successful in 55s
CI / sanitizers (address) (push) Successful in 1m1s
CI / fuzz-build (push) Successful in 34s
CI / coverage (push) Successful in 52s
CI / valgrind (push) Successful in 3m14s
CI / build-and-test (push) Successful in 5m32s
2026-09-13 10:44:57 +02:00
TapTap 18d1b84246 refactor(protocol): guard session release, clarify Data.owner contract
Add a NULL guard to protocol_release_memory_for_session so it no-ops like
the sibling session setters.  Correct the Data.owner doc comment, which
implied a non-zero protocol_charge always has an owner; document that
owner may be NULL for uncharged/ownerless Data, that any such charge
falls back to the bound session, and that a charged Data must not outlive
its owning session.  Note the lifetime contract on the release API too.

Extend tests/test_protocol.c to cover destroying a charged Data with no
session bound (the other half of the original bug) and to assert that
data_create/data_create_reserve start with owner == NULL and
protocol_charge == 0.
2026-09-13 10:42:45 +02:00
TapTap 0f95f48899 fix(opencode): correct remaining agent/skill doc drift
CI / lint (pull_request) Successful in 1m31s
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
CI / build-and-test (pull_request) Successful in 1m44s
- pr-review: replace invalid 'tea pr comment' with 'tea comment' (the
  former is not a tea subcommand)
- integrator: drop stray '-M' from client examples (-M is now
  --remote-option and requires an argument), use the canonical pytest
  integration command, and bump the CI image tag to v10
- test-writer: build fuzz targets via -DENABLE_FUZZ=ON instead of
  hand-rolled -fsanitize flags; fix the fuzz binary path
- cmake-expert: document -DSANITIZER=undefined, which is now live in
  CMakeLists.txt
- README: add --allow-unauthenticated to the plain-TCP server example,
  use --preserve for metadata (not -M), and use the canonical
  integration command
- AGENTS.md: use the canonical integration command
2026-09-13 10:41:23 +02:00
TapTap c78a21de57 docs(shared): clarify authorized_root accessor contracts
Document on utils_get_authorized_root_path() that the returned pointer is
borrowed and invalidated by the next authorized-root setter, that the fd
and path are not read atomically (non-reentrant), and that the fd remains
caller-owned.  Add a matching single-threaded/set-before-threads note at
the accessor definitions in utils.c.

In server.c, drop the redundant utils_set_authorized_root(-1, NULL) after
a failed utils_set_authorized_root(): the setter already fail-closes the
state on allocation failure.  The following close(root_fd) is unchanged.
2026-09-13 10:38:21 +02:00
TapTap 5c8970c64f chore(opencode): fix drifted agent/skill docs and repo hygiene
CI / lint (pull_request) Successful in 1m29s
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
CI / build-and-test (pull_request) Successful in 1m45s
The agent and skill definitions had drifted badly from the current
codebase and tooling, repeating the same class of bug as the benchmark
tool (references to nonexistent scripts and invented flags):

- Replace the removed `python3 test.py` with the real integration
  command (`python3 -m pytest tests/integration/ -n 4 --dist=load
  -m "not setpriv"`) across agents and skills.
- Fix `feature-scout`'s fabricated CLI flag list (--host, --server-mode,
  --use-* etc.) using the authoritative src/client/usage.c flags.
- Fix `perf-analyst` benchmark flags (-m -c -> -j -z) and point at
  benchmark/bench.py instead of stale numbers.
- Correct `code-explainer` (no getopt_long; --sendfile not -f) and
  version drift in the release skill (1.1.0 -> 2.20.0).
- Replace GitHub/`gh` workflows with Gitea/`tea` (PRs target dev; issues
  via tea; branch strategy updated in all agents).
- Use the built-in `-DSANITIZER=address|thread` CMake option instead of
  hand-rolled -fsanitize flags.
- Add `-p 8080 --allow-unauthenticated` to plain-TCP server examples.
- Merge the redundant security-screener into security-auditor; drop the
  duplicate (16 agents remain).

Repo hygiene: gitignore `root/` and `test_partial_install_tmp/`, remove
the empty leftover trees, delete the tracked scratch scripts tmux.sh and
to_one_file.py, and note the compile_commands.json symlink in README.
2026-09-13 10:34:59 +02:00
TapTap 4e918a1b69 test(config): pin wire bytes and round-trip every field
test_config_wire_golden() serializes a fully-populated Config through
config_send_wire_block() and pins the exact frame to len=633 and FNV-1a
hash 6163263374908258816, captured from the pre-X-macro implementation.
Any field reorder, resize or codec change fails the test.

test_config_wire_roundtrip_all_fields() serializes/deserializes a defaults
Config and a fully-populated Config over a socketpair and compares every
serialized field.  The comparison is itself generated from
CONFIG_WIRE_FIELDS (one CONFIG_CMP_<KIND> per table entry), so a new table
entry automatically extends coverage; it cannot fall out of sync.  It
normalizes the receiver's NULL/"" canonicalization, the max_alloc server
clamp and the derived use_delta/use_xattrs bits.
2026-09-13 10:28:31 +02:00
TapTap 87f6cb0243 refactor(config): single X-macro table for serialized fields
Every Config field that crosses the wire was declared in up to six places
(struct member, config_set_defaults, send_*, receive_*, and the two CLI
option tables) and could drift silently.  Add CONFIG_WIRE_FIELDS in
config.h: one ordered per-segment table where each serialized field is
declared once with its C type, default and wire codec (KIND).

config.h now expands the table to declare the struct members;
config_set_defaults() expands it to assign the defaults; and
config_send_wire_block()/config_receive() expand the per-segment lists to
emit/consume the frame.  The per-segment function names, call order and
segment boundaries are preserved exactly.

Fields with genuinely custom logic keep dedicated helpers but are still
declared once in the table: the protocol-version handshake (HEADER), daemon
SCRAM auth (STR_REDACTED_AUTH), the daemon module name (STR_MODULE), the
repeated count+array blocks (BLOCK_SKIP_SUFFIXES/BLOCK_BASIS/BLOCK_IDMAP),
--copy-as presence/ids (COPY_AS_*), and the derived --delta / use_xattrs
bits (DERIVED_DELTA, BOOL_XATTR_DERIVE).  The version field remains a
special header (validated before any other field is parsed) and is sent by
config_send_wire_block() explicitly.

No public field is renamed and PROTOCOL_VERSION stays "2.20.0".  Because
the struct declaration order is no longer the wire order, the wire order is
now enforced solely by the table and by a byte-exact golden test
(follow-up commit).  Add config_send_wire_block() so that test can hash the
frame body without the STATUS_OK handshake.
2026-09-13 10:28:26 +02:00
TapTap e1f8f75e7c docs: document daemon per-module/per-host caps and shared auth lockout 2026-09-13 10:24:09 +02:00
TapTap 4c17122b00 feat(daemon): enforce per-module/per-host caps and shared auth lockout
Wire the shared registry into the accept loop (parent claims a slot before
fork, blocks SIGCHLD across fork+pid publication, and reclaims the dead
child's slot from the SIGCHLD handler so per-module/per-source counts are
released even on SIGKILL). The connection child records the selected module
and normalized peer IP once the config frame names them: an over-cap module
or source is refused at the config gate with an audit log, and a source
that exceeded the auth-failure threshold is refused before a SCRAM
challenge (the counter is shared across children and cleared on success).
The existing global cap and host ACLs are untouched.
2026-09-13 10:24:05 +02:00
TapTap 0abaa62193 feat(daemon): parse per-host cap and auth lockout config keys
Add global keys `max connections per host` (default 0 = unlimited),
`auth lockout threshold` (default 10, 0 disables) and
`auth lockout duration` (default 300 s, 0 disables). Module
`max connections` now accepts 0 as unlimited. Bound the number of
[module] sections (DAEMON_CONF_MAX_MODULES) so the shared registry's
per-module counter array stays fixed-size; absent keys keep their
defaults so old configs still load.
2026-09-13 10:24:01 +02:00
TapTap 5334397b81 feat(daemon): add shared cross-process connection registry
The daemon forks one child per accepted connection, so per-module and
per-source accounting must live in state shared across the children. Add a
fixed-size registry carved from an anonymous shared mapping
(mmap(MAP_SHARED|MAP_ANONYMOUS)) created before the accept loop: a slot
lifecycle (FREE/CLAIMED/REGISTERED) with parent claim/reclaim and a
lock-free, open-addressed per-source table for the per-host occupancy and
the shared auth-failure counter. C11 atomics only; no pthread locks across
fork.

Unit tests cover slot exhaustion, the module/host caps, pid reclaim and
fork-shared visibility.
2026-09-13 10:23:58 +02:00
TapTap 6968ff6734 Merge PR #282: fix(benchmark): use real FastSync flags and Release builds
CI / lint (push) Successful in 1m31s
CI / sanitizers (undefined) (push) Successful in 54s
CI / sanitizers (address) (push) Successful in 1m1s
CI / fuzz-build (push) Successful in 33s
CI / coverage (push) Successful in 52s
CI / valgrind (push) Successful in 3m14s
CI / build-and-test (push) Successful in 5m31s
2026-09-13 10:18:21 +02:00
TapTap 5aca91ab22 fix(benchmark): use real FastSync flags and Release builds
CI / lint (pull_request) Successful in 1m29s
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
CI / build-and-test (pull_request) Successful in 1m44s
The benchmark tool used stale rsync-style spellings that map to

different FastSync options, so it never enabled the features it

claimed to measure:

  -c -> --checksum (not compression)

  -m -> --prune-empty-dirs (not multithreading)

  -s -> --secluded-args, a no-op (not chunk serialization)

  -f -> --filter, needs an argument (not sendfile)

Replace them with the real flags (-z, -j, --chunk-serialization,

--sendfile), force CMAKE_BUILD_TYPE=Release, route informational

output to stderr so --output json emits valid JSON, surface

client/rsync failures instead of silently dropping them, and widen

the results table for the longer config names. Update the benchmark

skill to match (correct flags, server invocation, and replace the

nonexistent test.py --full with benchmark/bench.py).
2026-09-13 10:14:37 +02:00
TapTap 3260a39ab4 refactor(shared): single owner for authorized_root state 2026-09-13 10:06:04 +02:00
TapTap 5d3c43305e fix(protocol): release Data charge to its owning session
Data charged against a ProtocolSession kept only the charge amount, so
data_destroy released it from whatever session was thread-locally bound
at destroy time. Destroying a received Data on another thread, after the
session was unbound, or while a different session was bound leaked the
originating session's budget and underflowed the other's.

Add Data.owner, set it whenever protocol_receive_data_limited charges a
session, and have data_destroy release against that owner directly via
the newly-exported protocol_release_memory_for_session. Uncharged Data
(owner NULL) keeps the previous bound-session fallback.

Add a unit test proving a Data acquired on session A is released to A
even when unrelated session B is bound at destroy time.
2026-09-13 10:05:38 +02:00
TapTap 9242e86772 Merge Wave 7: fix shared/server layering and explicit CMake targets
CI / lint (push) Successful in 1m30s
CI / sanitizers (undefined) (push) Successful in 56s
CI / sanitizers (address) (push) Successful in 1m3s
CI / fuzz-build (push) Successful in 34s
CI / coverage (push) Successful in 52s
CI / valgrind (push) Successful in 3m14s
CI / build-and-test (push) Successful in 5m33s
2026-09-13 07:30:33 +02:00
TapTap 0155902d95 docs: update stale PipelineContextReceiver reference 2026-09-13 07:30:28 +02:00
TapTap 3499baf80b build: explicit CMake targets; move receiver pipeline out of shared 2026-09-13 07:20:28 +02:00
TapTap 83eacf3151 Merge Wave 6: client features (--port, --threads=N, abort, keepalive) and test coverage
CI / lint (push) Successful in 1m30s
CI / sanitizers (undefined) (push) Successful in 1m4s
CI / sanitizers (address) (push) Successful in 1m9s
CI / fuzz-build (push) Successful in 36s
CI / coverage (push) Successful in 51s
CI / valgrind (push) Successful in 3m14s
CI / build-and-test (push) Successful in 5m37s
2026-09-13 06:59:08 +02:00
TapTap c2df0347ef fix(client,protocol): EINTR-safe sends, armed abort, keepalive drain grace, TLS WANT_WRITE 2026-09-13 06:59:02 +02:00
TapTap dd44537b44 Merge branch 'fix/w6-tests' into fix/w6-integration 2026-09-13 06:25:05 +02:00
TapTap 2854a9d149 test: fuzz manifest/protocol/xattr, hardlink unit, fault injection 2026-09-13 06:24:46 +02:00
TapTap 1fa2fbd266 feat(client): --port alias, --threads=N, graceful abort, keepalive 2026-09-13 06:22:28 +02:00
TapTap 10b18ab2d2 Merge Wave 5a: dead-code removal, scanner options embed, shared config invariants, bounded metadata API
CI / lint (push) Successful in 1m31s
CI / sanitizers (undefined) (push) Successful in 1m0s
CI / sanitizers (address) (push) Successful in 1m7s
CI / fuzz-build (push) Successful in 30s
CI / coverage (push) Successful in 51s
CI / valgrind (push) Successful in 3m12s
CI / build-and-test (push) Successful in 5m32s
2026-09-13 05:48:32 +02:00
TapTap dcc78c14c5 docs,fuzz: fix ownership/alloc comments; fuzz chunk metadata path 2026-09-13 05:48:27 +02:00
TapTap 5a829adb85 Merge branch 'fix/w5-scanner' into fix/w5-integration 2026-09-13 05:28:50 +02:00
TapTap 42c72030fb Merge branch 'fix/w5-config' into fix/w5-integration 2026-09-13 05:28:50 +02:00
TapTap 99f8045105 refactor(scanner,send): embed scanner options; unify stats and config ownership 2026-09-13 05:28:32 +02:00
TapTap eea66a7848 refactor(config,metadata): shared invariants; length-bounded metadata parser 2026-09-13 05:24:23 +02:00
TapTap c944787e03 refactor: remove dead file_store subsystem and unused wrappers 2026-09-13 05:19:18 +02:00
TapTap 57ce6d04f0 Merge Wave 4: performance (packed metadata 2.20.0, indexed lookups, zstd reuse, byte-bounded queues)
CI / lint (push) Successful in 1m30s
CI / sanitizers (undefined) (push) Successful in 1m1s
CI / sanitizers (address) (push) Successful in 1m7s
CI / fuzz-build (push) Successful in 29s
CI / coverage (push) Successful in 50s
CI / valgrind (push) Successful in 3m12s
CI / build-and-test (push) Successful in 5m22s
2026-09-13 05:01:21 +02:00
TapTap 3cf2e2c91f docs(version): align 2.20.0 artifacts; fix protocol-bump rationale 2026-09-13 05:01:15 +02:00
TapTap 301cb0dbaf fix(utils,file-list): bound keep/files-from indexes to O(M) memory 2026-09-13 04:52:10 +02:00
TapTap a4f4110397 Merge branch 'fix/w4-queue' into fix/w4-integration 2026-09-13 04:19:12 +02:00
TapTap 24fe8c5583 Merge branch 'fix/w4-compress' into fix/w4-integration 2026-09-13 04:19:12 +02:00
TapTap d274bdff4e Merge branch 'fix/w4-hash' into fix/w4-integration 2026-09-13 04:19:12 +02:00
TapTap 6c636a19e6 perf(compression,tcp): reuse zstd contexts; enable TCP_NODELAY 2026-09-13 04:18:55 +02:00
TapTap 1a83e284c4 perf(utils,file-list): index delete keep-set and --files-from lookups 2026-09-13 04:16:28 +02:00
TapTap 317d5d081a perf(protocol): pack metadata into one frame (PROTOCOL 2.20.0) 2026-09-13 04:08:36 +02:00
TapTap 69fe7f3c9f perf(send,scanner): byte-bound sender queues; drop redundant stat 2026-09-13 04:06:30 +02:00
TapTap ddc71a7df5 Merge Wave 3b: configurable protocol timeout and idle/session bounds
CI / lint (push) Successful in 1m31s
CI / sanitizers (undefined) (push) Successful in 1m1s
CI / sanitizers (address) (push) Successful in 1m8s
CI / fuzz-build (push) Successful in 30s
CI / coverage (push) Successful in 50s
CI / build-and-test (push) Successful in 4m37s
CI / valgrind (push) Successful in 3m12s
2026-09-13 03:46:25 +02:00
TapTap ffa1d24625 fix(receiver): harden idle-progress definition, single error frame, sendfile timeout 2026-09-13 03:46:20 +02:00
TapTap b16349b81e fix(protocol): honor --timeout for protocol I/O; bound idle/session time 2026-09-13 03:29:01 +02:00
TapTap 4bc84fe954 Merge Wave 3a: daemon host ACL, configurable max connections, peer audit, auth-failure delay
CI / lint (push) Successful in 1m31s
CI / sanitizers (undefined) (push) Successful in 57s
CI / sanitizers (address) (push) Successful in 1m5s
CI / fuzz-build (push) Successful in 29s
CI / coverage (push) Successful in 49s
CI / build-and-test (push) Successful in 4m35s
CI / valgrind (push) Successful in 3m10s
2026-09-13 02:51:11 +02:00
TapTap fc560246c1 fix(daemon): close ACL fail-opens (v4-mapped peers, invalid patterns) and cap auth delay 2026-09-13 02:51:07 +02:00
TapTap dff6609976 feat(daemon): host ACL, configurable max connections, peer audit, auth-failure delay 2026-09-13 02:36:15 +02:00
TapTap 1acb66628d Merge Wave 2: thread-safety fixes (signals, fd ownership, handler epilogue, logging, scanner leak)
CI / lint (push) Successful in 1m30s
CI / sanitizers (undefined) (push) Successful in 59s
CI / sanitizers (address) (push) Successful in 1m6s
CI / fuzz-build (push) Successful in 28s
CI / coverage (push) Successful in 49s
CI / build-and-test (push) Successful in 4m31s
CI / valgrind (push) Successful in 3m10s
2026-09-13 02:13:27 +02:00
TapTap ba1c7a369f fix(server,log): non-socket shutdown fallback, drop redundant delay cleanup, unlock logging I/O 2026-09-13 02:13:22 +02:00
TapTap d28489d83c Merge branch 'fix/w2-scan' into fix/w2-integration 2026-09-13 01:49:44 +02:00
TapTap 312ed05170 Merge branch 'fix/w2-log' into fix/w2-integration 2026-09-13 01:49:44 +02:00
TapTap fecbe2c90c fix(server): child-safe signals, single fd owner, handler cleanup epilogue 2026-09-13 01:49:27 +02:00
TapTap c8f5d80fcb fix(log): serialize message emission; clear log_fp before close; use logger 2026-09-13 01:44:59 +02:00
TapTap 8147ff7b50 fix(scanner): free chunk_data on chunk-create failure 2026-09-13 01:36:51 +02:00
TapTap b7fbb56289 test(file): silence cppcheck constVariablePointer in empty-path test
CI / lint (push) Successful in 1m32s
CI / sanitizers (undefined) (push) Successful in 57s
CI / sanitizers (address) (push) Successful in 1m4s
CI / fuzz-build (push) Successful in 30s
CI / coverage (push) Successful in 49s
CI / build-and-test (push) Successful in 4m28s
CI / valgrind (push) Successful in 3m10s
2026-09-13 01:25:29 +02:00
TapTap 08063b6d73 Merge Wave 1: critical/High fixes (UAF, DoS caps, leaks, hardening)
CI / lint (push) Failing after 1m32s
CI / build-and-test (push) Skipped
CI / sanitizers (address) (push) Skipped
CI / sanitizers (undefined) (push) Skipped
CI / fuzz-build (push) Skipped
CI / coverage (push) Skipped
CI / valgrind (push) Skipped
2026-09-13 01:18:58 +02:00
TapTap ea2f76cd7a fix(receiver): charge per-entry DirTimeList cost; cap client --skip-compress 2026-09-13 01:18:54 +02:00
TapTap 76eeba1773 Merge branch 'fix/w1d-hardening' into fix/w1-integration 2026-09-13 00:59:35 +02:00
TapTap b72ab298ab Merge branch 'fix/w1c-wire' into fix/w1-integration 2026-09-13 00:59:35 +02:00
TapTap 446a714ef8 Merge branch 'fix/w1b-receiver' into fix/w1-integration 2026-09-13 00:59:35 +02:00
TapTap a90e234eb3 harden: overflow guards, auth-user validation, TLS1.3 policy, build hardening 2026-09-13 00:59:21 +02:00
TapTap f8252cf3e7 fix(protocol): bound pre-auth config string memory 2026-09-13 00:57:32 +02:00
TapTap 4557924972 fix(receiver): cap DirTimeList growth and fix placeholder Data leaks 2026-09-13 00:57:32 +02:00
TapTap 59ce174d22 fix(client-send): UAF in basis preflight and missing_args leak 2026-09-13 00:57:09 +02:00
TapTap 2a8941ee5c Merge feat/ref-integration: SuperMode enum, dir-time gate dedup, parse_args/server_module_gate splits
CI / lint (push) Successful in 1m30s
CI / sanitizers (undefined) (push) Successful in 59s
CI / sanitizers (address) (push) Successful in 1m6s
CI / fuzz-build (push) Successful in 29s
CI / coverage (push) Successful in 49s
CI / valgrind (push) Successful in 2m9s
CI / build-and-test (push) Successful in 4m31s
2026-09-12 21:11:03 +02:00
TapTap 37037a6ee7 refactor(client-cli): drop unused CliParseCtx positional fields (cppcheck) 2026-09-12 21:07:14 +02:00
TapTap 061e9ad43f Merge feat/ref-modulegate: split server_module_gate into helpers 2026-09-12 20:56:43 +02:00
TapTap f928879755 Merge feat/ref-parseargs: split parse_args into focused helpers 2026-09-12 20:56:43 +02:00
TapTap 84b7cb0de3 refactor(server): split server_module_gate into ordered helper stages 2026-09-12 20:56:33 +02:00
TapTap 921472b8b3 refactor(client-cli): split parse_args into focused option handlers
Break the ~700-line parse_args god function into cohesive static helpers
grouped by concern: output controls, pre-negation, range/time options, the
OPTION_TABLE dispatcher, flag/meta handlers, IO/network options, filter and
logging options, checksum/socket options, remote/basis/identity options,
positional handling, and a final lowering step.

A file-local CliParseCtx carries the config, cursor, positional buffers and
the mutable parse flags, so each handler stays focused. The dispatcher calls
the handlers in the original recognition order and preserves the exact
return contract (0/1/negative), error messages, log levels and control flow.

Behavior preserved; no functional changes.
2026-09-12 20:55:04 +02:00
TapTap 082ac2645d Merge feat/ref-dirtime: dir-time capture gate dedup 2026-09-12 20:43:42 +02:00
TapTap eefbd1e849 Merge feat/ref-supermode: SuperMode enum 2026-09-12 20:43:42 +02:00
TapTap 1fd462cca8 refactor(config): replace SUPER_MODE_* macros with SuperMode enum
Type Config.super_mode as SuperMode (a proper C enum) instead of a bare
int.  The wire boundary still carries the mode as an int: send casts the
enum explicitly and receive reads a temporary int, validates the
AUTO..OFF range, then casts.  Emitted bytes and accepted values are
unchanged.  ModuleGateContext.super_mode_override keeps its -1 sentinel
as int with an explicit cast at the apply site.

Behavior preserved.
2026-09-12 20:43:24 +02:00
TapTap 4ac37c4d8a refactor(dir-times): extract dir_times_should_capture predicate
Deduplicate the repeated directory-time capture gate
(`config->use_metadata && !config->omit_dir_times`) used by the
sender-side (multiprocessing.c) and receiver-side (receiver.c) sinks
into a single predicate declared next to the DirTimeList machinery in
file_receive.h and defined in file_receive.c.

Behavior preserved: identical short-circuit condition and semantics,
no signature or protocol changes.
2026-09-12 20:42:47 +02:00
TapTap 08af945bd6 Merge main back into dev after v2.19.0 release
CI / lint (push) Successful in 1m32s
CI / sanitizers (address) (push) Successful in 55s
CI / fuzz-build (push) Successful in 30s
CI / sanitizers (undefined) (push) Successful in 53s
CI / coverage (push) Successful in 47s
CI / build-and-test (push) Successful in 4m28s
CI / valgrind (push) Successful in 2m11s
2026-09-12 20:22:54 +02:00
TapTap 378d881ca7 Merge release/v2.19.0 into main: FastSync v2.19.0
CI / lint (push) Successful in 1m33s
CI / sanitizers (undefined) (push) Successful in 59s
CI / sanitizers (address) (push) Successful in 1m4s
CI / fuzz-build (push) Successful in 28s
CI / coverage (push) Successful in 49s
CI / valgrind (push) Successful in 2m10s
CI / build-and-test (push) Successful in 4m29s
2026-09-12 20:22:50 +02:00
TapTap cc27ee1b83 Release v2.19.0
- Protocol version 2.19.0 (SCRAM-SHA-256 daemon auth replacing the replayable digest)
- Salted PBKDF2 verifier store + --hash-credentials; legacy store hard-rejected
- Persistent anti-enumeration dummy key (<store>.dummykey)
- Verified TLS / opted-in loopback transport required for auth modules
- Secret wiping; carried-over hardening from the security phases
- Add CHANGELOG.md and set the CMake project version
2026-09-12 20:22:46 +02:00
TapTap d653c3e151 Merge feat/tls-dummy-integration: verified/Local-only transport for daemon auth + persistent dummy key
CI / lint (push) Successful in 1m38s
CI / sanitizers (undefined) (push) Successful in 1m1s
CI / sanitizers (address) (push) Successful in 1m7s
CI / fuzz-build (push) Successful in 30s
CI / coverage (push) Successful in 49s
CI / valgrind (push) Successful in 2m14s
CI / build-and-test (push) Successful in 4m32s
2026-09-12 19:55:28 +02:00
TapTap 1b90ee2449 fix(review): close loopback TLS auth bypass; align docs and wrong-CN test
- server gate: the --allow-unauthenticated loopback allowance now requires
  an actual plaintext connection (!gate_ctx->ssl), so a loopback TLS client
  whose cert fails the --client-cn check is refused before any SCRAM
  challenge instead of falling through the plaintext opt-in.  Keep the
  invalid-fd guard as belt-and-braces (unreachable after the policy check).
- test: rewrote test_wrong_client_cn_refused_before_auth_challenge to run
  deterministically over 127.0.0.1 with --tls + --allow-unauthenticated and
  a CA-valid wrong-CN client cert, asserting the gate refusal log and an
  unchanged module tree (no skip).
- docs: --client-cn is mandatory with --tls; dummykey sidecar is secret
  material; document all transient-fallback reasons; qualify
  --allow-unauthenticated in README and --help so it cannot read as
  permitting remote plaintext auth.
- credentials.h: drop stale restrictive-umask claim (fchmod forces exact
  0600; only create/write/fsync/link/fchmod failure degrades to ephemeral).
2026-09-12 19:50:52 +02:00
TapTap 89b967f29a Merge feat/dummy-key: persist anti-enumeration dummy key across restarts
# Conflicts:
#	RSYNC_COMPAT.md
2026-09-12 19:30:35 +02:00
TapTap 8f06ae5262 Merge feat/tls-auth: require verified/local transport for daemon auth modules 2026-09-12 19:29:35 +02:00
TapTap ac3c4c7c72 fix(a7-auth): harden dummy-key temp creation
- Make the atomic-publish temp name unpredictable by appending 16 random
  hex chars to the pid, so a leftover/planted temp cannot be targeted.
- On EEXIST, unlink the stale temp and retry the O_EXCL create once
  (bounded), so a crash leftover or reused pid cannot silently defeat
  sidecar persistence.
- fchmod the temp fd to 0600 after creation (umask can clear owner bits)
  and treat failure as a create failure, so the published sidecar is
  always exactly 0600.
- Clarify comments: the sidecar requires exact 0600 while the store and
  password files only reject group/other bits.
- Add a unit test that a restrictive umask still yields an exact 0600
  sidecar; clean random-suffixed temps in tests.
2026-09-12 19:29:23 +02:00
TapTap d53614d06b fix(a7-3/s1): fail closed on non-loopback peers; require plaintext opt-in before challenge
utils_fd_peer_is_local now returns true only when getpeername SUCCEEDS and the
peer address classifies as loopback. A non-socket descriptor (pipe/socketpair)
or any getpeername error is NOT local, so the daemon auth gate fails closed
instead of treating an untestable --stdio pipe as trusted (daemon auth modules
are --daemon-only and the stdio path never loads a daemon config).

server_module_gate now requires --allow-unauthenticated for the loopback
plaintext auth path: a plaintext loopback connection without the operator
opt-in is refused at the config gate BEFORE server_auth_handshake, so no SCRAM
challenge is sent. Remote peers still require verified TLS regardless of the
flag; the handler keeps its defense-in-depth checks.

Docs state the exact policy (verified TLS with matching --client-cn, or
operator-opted-in loopback plaintext), drop the SSH/stdio auth-transport claim
(they are daemon-only), and add the loopback trust-boundary relay caveat and
the CN-only (no SAN) residual. Adds a unit-test negative for pipe/socketpair
and an integration test where a relay observes no challenge when the flag is
absent.
2026-09-12 19:18:29 +02:00
TapTap f0381a6b8e fix(a7-auth): publish dummy-key sidecar atomically and harden reads
Address review findings on the persistent dummy-key sidecar:

- Publish atomically: write a private same-directory temp file
  (<store>.dummykey.tmp.<pid>, 0600), fsync, then link(2) into place;
  fsync the containing directory and drop the temp name. A concurrent
  starter can no longer observe a zero/partial sidecar and fail closed.
  On EEXIST adopt the winner's sidecar; otherwise warn and use a
  transient ephemeral key.
- Harden the read path (initial and EEXIST-adopt) with
  O_RDONLY|O_NOFOLLOW|O_NONBLOCK|O_CLOEXEC: reject planted symlinks
  (ELOOP fails closed) and never block on a planted FIFO.
- Require the exact owner-only mode (st_mode & 07777) == 0600 and make
  the rejection message truthful.
- Report a clear "short write" instead of a stale strerror(errno) when
  write() returns 0.
- Document the artifact and its creation-failure caveat (FIFO store
  path, read-only filesystem, missing directory) in README.md and
  RSYNC_COMPAT.md.
- Tests: known-key sidecar adoption (dummy salt KAT + reload), symlink
  rejection, and the exact-0600 rule (0400 now rejected).
2026-09-12 19:16:11 +02:00
TapTap a7a1930e88 fix(a7-3/s1): require TLS or local transport for daemon auth
Daemon modules that declare 'auth users' no longer accept credentials over a
remote plaintext connection: server_module_gate refuses at the config gate,
before any SCRAM challenge is sent, unless the connection is verified TLS with
a client certificate matching --client-cn, or a local/SSH transport (loopback
TCP peer or the --stdio pipe). --allow-unauthenticated does not relax this.

The TLS client-CN comparison now uses credentials_secure_equal (S2). Clients
sending --password-file to a non-loopback daemon must use --tls; validate_config
rejects the plaintext case before any network I/O.

Adds utils_sockaddr_is_loopback / utils_fd_peer_is_local / utils_host_is_loopback
helpers with unit tests, a client validation unit test, and integration tests
for the client-side plaintext rejection and the wrong-CN gate refusal.
2026-09-12 19:02:05 +02:00
TapTap 42f01c0968 fix(a7-auth): persist dummy key in owner-only sidecar
The store-wide dummy key was regenerated on every credentials_load, so an
unknown user's dummy salt changed across daemon restarts while a real user's
stored salt stayed stable -- a restart-gated username-enumeration oracle.

Persist the 32-byte key in a 0600 <store>.dummykey sidecar next to the
credential store.  An absent sidecar is created with O_EXCL and fsynced; a
present sidecar is read only when it is an owner-only regular file of exactly
32 bytes (otherwise the load fails closed).  If the sidecar cannot be created
(read-only mount, missing directory) fall back to a transient per-run key with
a warning.  A NULL store path keeps the key ephemeral.
2026-09-12 18:40:21 +02:00
TapTap 2489d422e5 Merge feat/a7-integration: SCRAM-SHA-256 daemon auth + lazy protocol debug escaping
CI / lint (push) Successful in 1m33s
CI / sanitizers (undefined) (push) Successful in 59s
CI / sanitizers (address) (push) Successful in 1m4s
CI / fuzz-build (push) Successful in 29s
CI / coverage (push) Successful in 50s
CI / valgrind (push) Successful in 1m54s
CI / build-and-test (push) Successful in 4m45s
2026-09-12 18:21:13 +02:00
TapTap e2ddc0c07f Merge feat/hardening-misc: lazy protocol debug escaping, log_debug_enabled 2026-09-12 18:08:56 +02:00
TapTap 1480716304 Merge feat/a7-auth: SCRAM-SHA-256 daemon auth replacing replayable static digest 2026-09-12 18:08:56 +02:00
TapTap 1de1376e54 fix(a7-auth): final hardening pass on SCRAM auth
- burn the store-wide dummy_key in credentials_free()
- burn the local mac on hmac_sha256 failure in credentials_get_verifier()
- always run the O(store) constant-time scan, even for off-list users, to
  close the pre-existing off-list timing channel; select the real verifier
  only when on_list && match
- clarify the server_auth_handshake STATUS_AUTH_FAILED comment (failure
  before success vs. a dropped broken connection while writing the signature)
- document accepted anti-enumeration residuals (restart-gated dummy salt;
  pre-auth-observable iteration count)
2026-09-12 18:08:46 +02:00
TapTap eaf67f6257 fix(a7-auth): address SCRAM auth review findings A-G
- tests: pass CREDENTIAL_KEY_LEN to unhex for the 32-byte KAT proof/sig
  (sizeof(expect) is 348, over-reading the 65-byte hex literal under ASan)
- credentials: close the username-enumeration oracle with a store-wide
  dummy_key and a deterministic per-username dummy salt; make the store's
  iteration count uniform (reject intra-file and layered disagreements) and
  answer a miss with the store-wide count; run the constant-time key compare
  even when found=false and fold the decision with bitwise AND
- credentials_compute_keys: enforce [CREDENTIAL_MIN_ITERS, CREDENTIAL_MAX_ITERS]
- tests: recompute the whole KAT independently at CREDENTIAL_DEFAULT_ITERS
  (600000) and pin the golden store line; add non-uniform-store rejection,
  bound and deterministic-dummy-salt assertions
- server: send exactly one generic STATUS_AUTH_FAILED on every failure path
  (including credentials_get_verifier failure); route all handshake exits
  through one burn path
- credentials/server: burn the base64 decoders' scratch on error, the
  hash_store_line base64/line buffers on failure, and all handshake key/proof
  material
- fuzz: guard the auth-offset scan against size_t underflow and use a found flag
- docs: drop stale digest wording, use CREDENTIAL_MIN_ITERS as the --iterations
  bound, document 0600 output for --hash-credentials (plus a stderr warning on
  a group/other-accessible stdout file), and describe the deterministic dummy
  salt in the no-oracle claims
2026-09-12 17:56:52 +02:00
TapTap 8c94ec9886 feat(a7): SCRAM-SHA-256 daemon auth to replace replayable digest
Replace the challenge-less static-SHA-256 daemon bearer credential with a
SCRAM-SHA-256-style challenge/response and a salted PBKDF2 verifier store.
PROTOCOL_VERSION 2.18.0 -> 2.19.0; legacy user:SHA256HEX stores hard-reject.

- credentials: b64/rand/PBKDF2/HMAC primitives, verifier store parser,
  constant-time proof verify + ServerSignature, --hash-credentials helper
- config: auth block is now [present][username]; client runs the challenge
  exchange; config_burn_auth wipes plaintext/derived secrets (A7-4)
- server: gate drives the challenge, dummy verifier for unknown/off-list users
- tests: independent Python KAT, replay + legacy integration tests, fuzz paths
- docs: new store format, --hash-credentials, 2.19.0 bump

TLS verification behavior (A7-3/S1) is intentionally unchanged.
2026-09-12 17:19:33 +02:00
TapTap 87585e9881 perf(protocol): skip string debug escaping when proto debug is off
output_escape() was called on every send_str/receive_str even when
LOG_DEBUG_PROTO logging was disabled, allocating and scanning the whole
payload for a line that log_debug_message() then discarded.  Add a
log_debug_enabled(flag) gate mirroring log_debug_message()'s own filter and
check it before escaping.  Redacted (secret) strings still log the same
<redacted> marker; no observable log output changes.
2026-09-12 16:54:43 +02:00
TapTap 1ba6372017 Merge feat/p8-integration: P8 hardening batch (fs/transport, identity/server, CLI quality, fuzz)
CI / lint (push) Successful in 1m28s
CI / sanitizers (undefined) (push) Successful in 57s
CI / sanitizers (address) (push) Successful in 1m0s
CI / fuzz-build (push) Successful in 27s
CI / coverage (push) Successful in 47s
CI / valgrind (push) Successful in 40s
CI / build-and-test (push) Successful in 5m24s
2026-09-12 15:55:49 +02:00
TapTap 9f74b21c64 test(p8h): assert a --no-super daemon still refuses client --super 2026-09-12 15:55:44 +02:00
TapTap 108fee1e41 fix(p8h): restore daemon --super refusal, race-free secret-file check, remaining log escapes
- server_module_gate: refuse client-chosen ownership against the ORIGINAL config
  so an explicit --super is still refused under an operator --no-super veto
  (the veto must not turn a refusal into an accept).
- credentials: open-then-fstat the exact secret inode, require current-user
  ownership and no group/other bits, but continue to allow process-substitution
  FIFOs; removes the stat->fopen TOCTOU.
- file.c preallocate + protocol.c send-string debug logs escape attacker paths.
- usage/RSYNC_COMPAT updated for --old-args no-op and secret-file rules.
2026-09-12 15:50:14 +02:00
TapTap e1bb2e9233 chore(p8h): reconcile ssh old-args docs, secret-file perms docs; fix test cppcheck 2026-09-12 15:33:54 +02:00
TapTap 81fed86748 Merge branch 'feat/p8h-fuzz' into feat/p8-integration 2026-09-12 15:22:00 +02:00
TapTap 5844648fb2 Merge branch 'feat/p8h-cli' into feat/p8-integration 2026-09-12 15:22:00 +02:00
TapTap 4331bc4a8d Merge branch 'feat/p8h-core' into feat/p8-integration 2026-09-12 15:22:00 +02:00
TapTap d97e3982b4 test(fuzz): add config-frame receive and identity parser fuzz targets
Add two libFuzzer harnesses (GLOBbed from tests/fuzz/*.c) and deterministic
P8 config-frame receive tests:

- fuzz_config_receive.c drives config_receive() from arbitrary bytes. It
  captures one canonical valid frame with the production sender and feeds the
  receiver four shapes: raw bytes, valid-version-prefix + fuzz bytes, valid
  frame minus the P8 tail (super_mode + copy-as) + fuzz bytes, and valid frame
  minus the usermap count + fuzz bytes. This reaches the --super/--copy-as and
  huge/negative map-count paths that random bytes cannot get through the
  preceding wire-bool gate.
- fuzz_identity_parse.c fuzzes identity_parse_copy_as/map/chown plus the
  identity_wire_valid/identity_ownership_requested predicates on a fresh
  config per input.
- test_fuzz_smoke.c gains deterministic malformed-frame cases: out-of-range
  super_mode, negative/extreme copy-as ids, non-bool copy-as presence, tail
  truncation, huge/negative usermap counts, version mismatch and a
  wrong-order field after the version gate.

Unit build (STRICT_WARNINGS) and the fuzz build are clean; both targets run
3000+ iterations with no crash. No production code changed.
2026-09-12 15:21:33 +02:00
TapTap 6d32bc795b fix(p8h-core): escape log paths, fail closed on identity activation, tidy server gate
- A6: escape attacker-controlled file paths and the receive root in log
  lines (file_receive, server, protocol DEBUG) with output_escape()
- A8: identity_set_active() returns bool and fails closed when a requested
  usermap/groupmap cannot be deep-copied; handler refuses the connection
- remove the const cast and duplicate super_mode clamp from
  server_module_gate via an explicit override the handler applies once
- release the identity snapshot on the queue_create failure path
- refactor identity_parse_copy_as to a single cleanup tail and drop the
  duplicated group error format specifier
2026-09-12 15:03:54 +02:00
TapTap abad1664ba security(shared): fix -K TOCTOU, ssh old-args quoting, TLS opts, secret-file perms, sparse dedup
- file: open -K dirlink referents via a race-safe relative O_NOFOLLOW walk
  from the authorized-root fd instead of re-opening an absolute realpath()
  result (removes the intermediate-symlink swap TOCTOU).
- transport_ssh: always single-quote the server path, including --old-args,
  so no mode can inject shell metacharacters.
- transport_tls: set SSL_OP_NO_COMPRESSION and (guarded) SSL_OP_NO_RENEGOTIATION.
- credentials: reject --password-file/--early-input with any group/other
  permission bit; chmod 0600 the affected test fixtures.
- file_store: export file_store_write_sparse() and remove the verbatim
  file.c duplicate.
2026-09-12 15:01:48 +02:00
TapTap 7e45891257 refactor(cli): dedupe server/client option parsing and tighten CLI tests
- server_cli: handle --password-file/--early-input/--iconv via arg_has_value
  in one place, removing the unreachable duplicate separate-form arms while
  keeping both --opt VALUE and --opt=VALUE working
- client_cli: factor the triplicated --delta-block/--block-size range check
  into set_delta_block_size(); drop the redundant use_metadata assignment
  after identity_parse_copy_as (the parser already forces it)
- tests: cover both spellings of --iconv/--delta-block, make the archive
  short-form test actually call parse_args, add delta-block invalid cases
2026-09-12 14:54:47 +02:00
TapTap b8db810ee5 Merge feat/p8-security: P8 security hardening (daemon ownership opt-in, fake-super gating, no implicit numeric-ids, copy-as fail-fast)
CI / lint (push) Successful in 1m33s
CI / sanitizers (undefined) (push) Successful in 56s
CI / sanitizers (address) (push) Successful in 56s
CI / fuzz-build (push) Successful in 23s
CI / coverage (push) Successful in 47s
CI / valgrind (push) Successful in 40s
CI / build-and-test (push) Successful in 4m59s
2026-09-12 14:33:47 +02:00
TapTap ea0a0e2eaf fix(p8-security): make --copy-as directory ownership airtight; harden tests/logs
- file_ensure_directory_secure() now chowns a final directory it creates under
  --copy-as and fails on error; the symlink parent-creation call site propagates
  it.  The is_dir branch fails when the confined parent cannot be opened under
  --copy-as.  Closes the residual wrong-owner gap for synthesized/symlink
  parent directories.
- file_restore_symlink_metadata() early NULL return is copy-as-aware.
- Preserve errno across the implicit-parent failure cleanup.
- Neutral skip messages (the clamp, not --no-super, may be responsible).
- Daemon copy-as test tolerates the non-root privilege refusal; usage text lists
  --copy-as.
2026-09-12 14:33:42 +02:00
TapTap b216ed31fb fix(p8-security): close review gaps in the ownership gate and copy-as failure propagation
- H3: a daemon module without 'client owner = yes' now also has super-user
  device activity forced off (char/block mknod, --write-devices), so a root
  daemon can no longer be made to create/write raw devices under AUTO.  The
  entries are skipped, preserving ordinary -a pushes.
- H1/H2: propagate a failed required --copy-as chown from symlink metadata
  restore and implicitly-created parent directories, so the entry (and run)
  reports failure instead of a wrong-owner success.
- Docs/help/headers updated for A2/A3 and the device clamp; startup warning
  spells out the client-owner risk.
- Tests: daemon device clamp (skipped without opt-in, created with opt-in),
  updated --super/--fake-super expectations.
2026-09-12 14:18:03 +02:00
TapTap e3840c8326 fix(p8-security): enforce daemon ownership policy, gate fake-super replay, drop implicit numeric-ids, make copy-as failures per-entry
A1: daemon refuses every client-chosen ownership/super-user request
(--numeric-ids/--chown/--usermap/--groupmap/--fake-super/--copy-as/--super)
unless the selected module opts in with 'client owner = yes'.
A2: fake-super owner replay requires an explicit ownership identity policy.
A3: --super no longer implies --numeric-ids (ownership stays opt-in).
A5: a failed --copy-as chown marks the entry failed instead of reporting
success with the wrong owner.
2026-09-12 14:00:55 +02:00
TapTap 58409cee10 test(p7-privilege): skip copy-as refusal test when workspace is not traversable by the unprivileged uid
CI / lint (push) Successful in 1m30s
CI / sanitizers (undefined) (push) Successful in 58s
CI / sanitizers (address) (push) Successful in 58s
CI / fuzz-build (push) Successful in 23s
CI / coverage (push) Successful in 47s
CI / valgrind (push) Successful in 39s
CI / build-and-test (push) Successful in 5m3s
2026-09-12 13:08:22 +02:00
TapTap 549a23993f Merge feat/p7-privilege: Phase 7 Wave E (privilege: --super/--no-super + --copy-as safe subset; PROTOCOL 2.18.0; all rsync rows implemented) 2026-09-12 12:55:55 +02:00
TapTap a5f6899590 docs(p7-privilege): recount Summary to 143/0/4/0/0/0; Wave E shipping note 2026-09-12 12:55:47 +02:00
TapTap 938886829e fix(p7-privilege): close re-review gaps (implicit dir ownership, daemon --super, write-devices gate) 2026-09-12 12:54:45 +02:00
TapTap fdc0f238c9 fix(p7-privilege): harden copy-as/super gates, own dirs/specials
- fake-super owner replay honors --no-super and an active --copy-as
- copy-as/identity ownership now applied to directories and special nodes
- reject copy_as_set && !use_metadata (receiver + client --no-preserve)
- daemon refuses --copy-as; add server-side --no-super operator veto
- implement identity_copy_as_refused/identity_copy_as_active
- reject copy-as ids that overflow int32; escape spec in log errors
- copy-as chown EPERM/EACCES logged at ERROR (still non-fatal)
- identity_wire_valid copy-as bounds; CLI help and RSYNC_COMPAT docs
- add unit tests and root-gated integration coverage
2026-09-12 12:34:43 +02:00
TapTap e6a65d0980 Merge branch 'feat/p7-copy-as' into feat/p7-privilege
# Conflicts:
#	RSYNC_COMPAT.md
#	src/shared/config.c
#	src/shared/config.h
#	src/shared/identity.c
#	tests/integration/test_preflight.py
#	tests/test_client_cli.c
#	tests/test_config.c
2026-09-12 12:14:44 +02:00
TapTap a785ec13c4 feat(p7-super): implement --super/--no-super safe-subset privilege gate (protocol 2.18.0)
Add the receiver-side --super / --no-super tri-state (Config->super_mode)
under the safe-subset + clear-refusal privilege model: FastSync never
elevates privileges, it only permits super-user attempts that are already
confined fd-relative below the authorized receive root.

- identity: privilege_super_permitted() gate (OFF=false, ON=true, AUTO follows
  geteuid()==0); identity_apply_ownership/_link become no-ops when not
  permitted; --super with no explicit identity policy implies raw numeric-id
  preservation (explicit usermap/groupmap/chown/numeric-ids still win); warn
  exactly once when --super is requested by a non-root receiver.
- file_receive: gate char/block device-node creation on the gate; FIFO/socket
  handling is unchanged.
- wire: trailing super_mode int after the --iconv spec, validated 0..2 in
  receive_privilege_options and validate_received_config; PROTOCOL_VERSION
  2.17.0 -> 2.18.0; version-sensitive tests and docs updated.
- CLI: --super/--no-super parsed explicitly before the generic --no-* branch
  (malformed --super=x rejected); usage text added.
- tests: config wire round-trip + invalid-value rejection, privilege-gate mode
  unit test, CLI parse test, integration transfer + root-gated ownership
  suppression/appliance tests.
- docs: RSYNC_COMPAT --super row + Wave E note, protocol mentions, README.
2026-09-12 12:01:19 +02:00
TapTap 80dd64aae6 feat(identity): implement --copy-as USER[:GROUP] safe subset (P7 Wave E)
Force the receiver to apply the requested owner/group to every written
entry through the confined fd-relative identity path instead of switching
the process credentials (unsafe for the multithreaded receiver).  An
unprivileged receiver refuses the transfer up front in server_module_gate,
before STATUS_OK, so no data is written with the wrong ownership.

- new Config fields copy_as_set/copy_as_uid/copy_as_gid + defaults
- identity_parse_copy_as (name/@N/* resolution, primary-gid default,
  gid==uid fallback for numeric ids with no passwd entry); implies -M
- identity snapshot + highest-priority forcing in identity_resolve_targets
- identity_copy_as_refused() helper
- trailing config-frame block (presence int + two int32 ids, >=0 checked)
- PROTOCOL_VERSION 2.17.0 -> 2.18.0; version-sensitive tests updated
- unit tests for parse + wire round-trip/negative-id rejection
- integration TestCopyAs: unprivileged refusal + root chown assertion
- RSYNC_COMPAT.md --copy-as row updated (safe subset + divergence); README
  protocol version refreshed
2026-09-12 11:58:37 +02:00
TapTap f64d252faf docs: recount RSYNC_COMPAT to 141/0/4/0/0/2 after Phase 7 Waves B-D; refresh README short-option note
CI / lint (push) Successful in 1m42s
CI / sanitizers (undefined) (push) Successful in 1m12s
CI / sanitizers (address) (push) Successful in 1m14s
CI / fuzz-build (push) Successful in 23s
CI / coverage (push) Successful in 47s
CI / valgrind (push) Successful in 39s
CI / build-and-test (push) Successful in 4m44s
2026-09-12 11:30:06 +02:00
TapTap 003a5e8f2f Merge feat/p7-times: Phase 7 Wave D (real dir/symlink time preservation making -O/-J meaningful; secluded-args -> Impossible/Divergence; PROTOCOL 2.17.0) 2026-09-12 11:22:23 +02:00
TapTap 9d97e1d3c0 Merge feat/p7-devices: Phase 7 Wave C (device/special statuses; --copy-devices --sendfile non-regular fallback) 2026-09-12 11:21:09 +02:00
TapTap 402e829fef docs(p7-times): re-review nits (chunked STATUS_DIR_TIMES comment; -M/--metadata -> --preserve; -m sink -> -j/--threads) 2026-09-12 11:20:40 +02:00
TapTap 9749c7878c fix(p7-times): dir-time entries only record (never create dirs); bound/chunk dir-time frames; harden list add; docs+tests
Review fixes for Phase 7 Wave D.

#1 (HIGH): STATUS_DIR_TIMES entries no longer create directories. A new
receiver-only File.dir_time_only flag marks dir-time entries; file_save_to_disk_full
short-circuits them as FILE_SAVE_SKIPPED before any device/dir branch, so the sink
still accumulates metadata into the deferred DirTimeList but creates nothing. Empty
source dirs stay untransferred (-a), -m/--prune-empty-dirs semantics are preserved,
and a pre-existing regular file/symlink at an empty-dir mirror path no longer aborts
the transfer. dir_time_list_apply fstatat()s the leaf (AT_SYMLINK_NOFOLLOW) and skips
absent/non-directory paths QUIETLY; only a real existing directory is stamped.
Also initialize File.dir_time_only in file_create() (uninitialised garbage otherwise).

#2 (MED): send_dir_times() chunks entries into repeated STATUS_DIR_TIMES frames of at
most MAX_MANIFEST_ENTRIES, matching the receiver's per-frame bound; the tautological
> INT_MAX check is gone.

#3 (LOW): dir_time_list_add() assigns each grown array right after its realloc (no
dangling) and advances capacity only after both succeed.

#4 (LOW): RSYNC_COMPAT.md -- STATUS_MKDIR carries metadata, dir times are transmitted
via STATUS_DIR_TIMES and applied at the end, empty dirs are still never created; -m
rationale, -O row and Wave D notes updated. Summary counts untouched.

#5 (LOW): integration tests for the three #1 scenarios (empty-dir non-creation under
-a and -a -m, collision non-abort), scanner test now covers empty-dir capture, and
test_file_restore_symlink_metadata asserts the positive apply path when supported.

PROTOCOL_VERSION stays 2.17.0; config-frame layout unchanged.
2026-09-12 11:06:05 +02:00
TapTap 8ca2b74e79 fix(p7-devices): sendfile falls back to buffered read for non-regular sources (--copy-devices no longer hangs); test/doc hardening 2026-09-12 11:00:26 +02:00
TapTap 04514c5b70 Merge feat/p7-output-fs: Phase 7 Wave B (sparse hole preservation, partial retention, fake-super replay, block-size; crtimes/stderr -> Impossible/Divergence) 2026-09-12 10:32:03 +02:00
TapTap a539d8b2ba docs(p7-output-fs): align fake-super comments with silent EPERM/EACCES skip (re-review nit) 2026-09-12 10:31:56 +02:00
TapTap f1a447bb4a feat(p7-times): real directory/symlink time preservation; -O/-J meaningful
Wave D of Phase 7. Make -O/--omit-dir-times and -J/--omit-link-times real by
preserving directory and symlink times, and mark --secluded-args as an explicit
Impossible/Divergence no-op.

Wire: PROTOCOL_VERSION 2.16.0 -> 2.17.0. Adds a terminal STATUS_DIR_TIMES frame
(int count + (wire path, metadata) pairs) sent after all file data and the
optional delete manifest. STATUS_MKDIR also carries metadata for --dirs entries.
Config-frame layout is unchanged.

Sender: the recursive scanner captures every traversed source directory (both
DirectoryScanner and the parallel scanner root + workers, appends mutex-guarded)
into a shared list; the single-threaded and -m paths transmit it last.

Receiver: a DirTimeList accumulates received directory metadata and applies it
with fd-relative no-follow utimensat only at the very end -- after all children,
after the commit-style --delete, and after --delay-updates publication -- in the
single-threaded success frame and in server.c after the -m threads join. -O skips
the application. Symlink metadata is applied at link creation with
utimensat/fchownat/fchmodat AT_SYMLINK_NOFOLLOW; -J suppresses only link times.
identity_apply_ownership_link shares the identity resolver with the fd path.

Docs: -O/-J rows -> Implemented; --secluded-args -> Impossible/Divergence;
--protocol accepted/rejected values and Phase-6/7 notes updated.

Tests: unit (scanner dir capture, DirTimeList apply, symlink metadata, protocol
version values) and integration (dir mtime round-trip + -O, symlink mtime
round-trip + -J, independent suppression), parameterized over single/multithread.
2026-09-12 10:31:20 +02:00
TapTap 227d001092 feat(p7-devices): finalize device/special-file statuses (devices/copy/write -> implemented, specials -> Impossible/Divergence for sockets) + coverage 2026-09-12 10:10:21 +02:00
TapTap f2ba8211ce fix(p7-output-fs): address c-review (fake-super mode sanitization HIGH; sparse/preallocate precedence; partial no_replace guard; stronger tests)
- fake_super_restore_fd now sanitizes mode like metadata_mode (never grants
  S_IWGRP|S_IWOTH; 0666 -> 0644), fixing a privilege regression
- --sparse takes precedence over --preallocate (skip posix_fallocate when
  sparse) so holes are not re-allocated; docs corrected
- --partial retention disabled under --no_replace (ignore/existing) and only
  marks write_attempted after the write begins (no empty-temp retention)
- accept --block-size=SIZE / --delta-block=SIZE inline forms; neutral messages
- fake-super EPERM/EACCES skipped silently (docs aligned); EINVAL still logged
- sparse unit test now memcmp's the full buffer; TestBlockSize integration keeps
  the destination basis so delta is genuinely exercised
- unit 37/37, cppcheck 0, clang-format 0
2026-09-12 09:55:40 +02:00
TapTap 47de05d215 feat(p7-output-fs): sparse hole preservation (-S), partial retention (-P), fake-super replay; block-size verified; crtimes/stderr -> Impossible/Divergence (Wave B)
- write_all_sparse: skips all-zero runs >= 4096 bytes via lseek(SEEK_CUR) and
  ftruncates the final size, wired into the atomic temp+rename and --inplace
  paths with no wire change (full image already in memory).
- --partial retention: on a save failure after the temp held data, rename the
  already-written temp to the destination path (best-effort; falls through to
  unlink; never retains when --partial is off) so --append/--append-verify can
  resume; tested by forcing futimens EINVAL with an out-of-range nsec.
- --block-size aliases --delta-block; verified config->delta_block_size is
  honored by the delta engine end-to-end (unit + integration tests).
- fake_super_restore_fd: parses and re-applies user.fastsync.stat fd-relative
  (fchown best-effort/non-root skipped, fchmod, futimens); a save under
  --fake-super now both records and re-applies.
- -N/--crtimes and --stderr=client promoted to a new 'Impossible/Divergence'
  status bucket (Summary: 136/2/4/3/2 = 147).
- cppcheck/clang-format clean; unit 37/37; integration 407 passed.
2026-09-11 15:58:20 +02:00
TapTap f34eb34f87 Merge feat/p7-cli-namespace: Phase 7 Wave A CLI-namespace parity (renames colliding short flags to rsync parity)
CI / lint (push) Successful in 1m17s
CI / sanitizers (address) (push) Successful in 54s
CI / sanitizers (undefined) (push) Successful in 54s
CI / fuzz-build (push) Successful in 22s
CI / coverage (push) Successful in 46s
CI / valgrind (push) Successful in 38s
CI / build-and-test (push) Successful in 4m30s
2026-09-11 14:44:58 +02:00
TapTap 560ed601f9 docs(p7-cli-namespace): sweep remaining -m-as-multithreading refs to -j/--threads 2026-09-11 14:37:48 +02:00
TapTap d39ddab42c fix(p7-cli-namespace): address c-review (setfacl -m mangled to --threads; two -s tests lost intent; label/doc sweeps)
- revert over-eager replacement of setfacl -m in test_features.py
- use --chunk-serialization (+ set dirs) in the two append/append-verify
  chunk-serialization rejection unit tests so they exercise the real check
- rename archive-negation integration test (--no-preserve is inert under
  archive because devices/specials force metadata)
- update stale (-c)/(-m)/(-s)/(-f) display labels and RSYNC_COMPAT -c/-m refs
- document the --no-perms negation limitation in the Wave A note
2026-09-11 14:34:03 +02:00
TapTap 66e82f9384 feat(p7-cli-namespace): rename colliding short flags to rsync parity (Wave A)
-c -> --checksum, -m -> --prune-empty-dirs, -M -> --remote-option,
-f -> --filter, -s -> --secluded-args, -p -> --perms, -T -> --temp-dir;
-a/--archive is now real rsync -rlptgoD (links+metadata+devices+specials).

FastSync's own flags moved to long-form-only or new shorts:
-j/--threads (multithreading), --preserve (metadata), --sendfile,
--chunk-serialization, --timeout, --ssh-port. Client-side only; the
wire config fields are unchanged (no PROTOCOL_VERSION bump). The server
keeps -p as its port. Docs (README, RSYNC_COMPAT summary 129->132) and
unit/integration tests updated. 37/37 unit, 400-pass integration.
2026-09-11 14:20:15 +02:00
TapTap 15363c0018 docs: add Phase 7 plan (CLI namespace parity, output/filesystem completion, privilege) toward full rsync flag parity 2026-09-11 13:47:44 +02:00
TapTap c1eac6321c docs: recount RSYNC_COMPAT to 129/2 after --batch (Wave D); Phase 6 complete
CI / lint (push) Successful in 1m16s
CI / sanitizers (address) (push) Successful in 57s
CI / sanitizers (undefined) (push) Successful in 55s
CI / fuzz-build (push) Successful in 22s
CI / coverage (push) Successful in 46s
CI / valgrind (push) Successful in 38s
CI / build-and-test (push) Successful in 4m33s
2026-09-10 22:07:10 +02:00
TapTap f43f236f66 Merge feat/p6-batch: residual batch --write-batch/--only-write-batch/--read-batch 2026-09-10 22:06:23 +02:00
TapTap fc2d144492 fix(p6-batch): reject clean EOF on record read; add traversal/EOF security tests 2026-09-10 22:05:18 +02:00
TapTap c026176bb3 feat(p6-batch): residual-batch write/read driver + codec
Implements the client-only residual-batch feature end-to-end:
- src/shared/batch.{c,h}: self-contained single-file batch codec using the
  existing chunk_serialize/chunk_deserialize codec (byte-identical by
  construction).  Magic+format-version header (metadata mode is persisted into
  the header so a batch is self-describing across machines), length-prefixed
  chunk records, bounded reads that reject malformed/truncated/oversized
  records cleanly.
- src/client/client_send.c: write_batch_from_source (deterministic separate
  scan pass, loads every chunk's file images, emits header+records) and
  apply_batch_to_dest (local apply to a destination root via
  file_save_to_disk_full).  No wire change, no server involved.
- src/client/client_validation.c: --write-batch XOR --only-write-batch;
  --read-batch exclusive with both; --read-batch needs only a DEST,
  --only-write-batch only a SOURCE.
- src/client/client_cli.c: main() drives the three batch modes without
  connecting/transferring for read/only-write; --write-batch runs the live
  transfer (single-threaded so the config survives) then emits the batch.
- tests/test_batch.{c,h} (unit: byte-identical roundtrip with and without
  metadata; bad-magic/truncated/oversized rejection) + tests/integration/
  test_batch.py (only-write no-server, read-batch no-source roundtrip,
  --write-batch with a live transfer, conflict rejections).
- clang-format: realign PART-1 config.h comment block.

No PROTOCOL_VERSION bump, no config-frame field, no server flag.
2026-09-10 21:40:07 +02:00
TapTap 4930127312 feat(p6-batch): CLI/config plumbing for --write-batch/--only-write-batch/--read-batch 2026-09-10 21:14:30 +02:00
TapTap 52f45692b9 docs: recount RSYNC_COMPAT to 126/5 after --protocol (Wave C)
CI / lint (push) Successful in 1m15s
CI / sanitizers (undefined) (push) Successful in 54s
CI / sanitizers (address) (push) Successful in 56s
CI / fuzz-build (push) Successful in 21s
CI / coverage (push) Successful in 45s
CI / valgrind (push) Successful in 39s
CI / build-and-test (push) Successful in 4m41s
2026-09-10 20:45:32 +02:00
TapTap 40b0870514 Merge feat/p6-protocol: --protocol version-force flag (client-only) 2026-09-10 20:44:44 +02:00
TapTap 5262cc2597 test(p6-protocol): harden validation null-check; cover missing-arg --protocol 2026-09-10 20:44:33 +02:00
TapTap 9fe6d6c748 test(p6-protocol): unit + integration coverage for --protocol 2026-09-10 20:36:05 +02:00
TapTap 2b7bb2d523 feat(p6-protocol): --protocol version-force flag (client-only) 2026-09-10 20:36:05 +02:00
TapTap bb27b2af50 docs: correct RSYNC_COMPAT selection/update rows to implemented (no code change)
CI / lint (push) Successful in 1m15s
CI / sanitizers (undefined) (push) Successful in 55s
CI / sanitizers (address) (push) Successful in 56s
CI / fuzz-build (push) Successful in 21s
CI / coverage (push) Successful in 44s
CI / valgrind (push) Successful in 38s
CI / build-and-test (push) Successful in 4m43s
2026-09-10 20:16:03 +02:00
TapTap a2ac599298 docs: recount RSYNC_COMPAT to 121/10 after Phase 6 waves A-B (--stop-after/--stop-at, --iconv)
CI / lint (push) Successful in 1m15s
CI / sanitizers (address) (push) Successful in 53s
CI / sanitizers (undefined) (push) Successful in 52s
CI / fuzz-build (push) Successful in 22s
CI / coverage (push) Successful in 45s
CI / valgrind (push) Successful in 38s
CI / build-and-test (push) Successful in 4m40s
2026-09-10 18:18:17 +02:00
TapTap 282aebb7f5 Merge feat/p6-stop: --stop-after/--stop-at deadline stop 2026-09-10 18:16:55 +02:00
TapTap b2afcf2c65 Merge feat/p6-iconv: --iconv charset conversion + PROTOCOL 2.16.0 2026-09-10 18:16:51 +02:00
TapTap 09a07179c9 fix(p6-stop): init -m scan_stopped_early; gate delete warning; stabilize partial-stop test 2026-09-10 18:16:35 +02:00
TapTap cac805b661 fix(p6-iconv): prevent convert buffer overflow; validate both directions; reset on error; more tests 2026-09-10 17:49:29 +02:00
TapTap ac7e9e3bc1 fix(p6-stop): block delete-manifest on early stop; sync -m manifest access; overflow guard 2026-09-10 17:39:34 +02:00
TapTap 7d6665633d test(p6-iconv): iconv unit, config wire-roundtrip, integration tests 2026-09-10 17:13:06 +02:00
TapTap 6e02a24232 feat(p6-iconv): --iconv charset conversion + PROTOCOL 2.16.0 2026-09-10 17:13:01 +02:00
TapTap 37cff96537 test(p6-stop): unit and integration tests for stop deadlines 2026-09-10 16:27:49 +02:00
TapTap 24d1448246 feat(p6-stop): --stop-after/--stop-at deadline transfer stop 2026-09-10 16:27:49 +02:00
TapTap 6bb63c6f21 docs: recount RSYNC_COMPAT to 118/13 after daemon Wave C (--no-motd + MOTD)
CI / lint (push) Successful in 1m14s
CI / sanitizers (address) (push) Successful in 55s
CI / sanitizers (undefined) (push) Successful in 53s
CI / fuzz-build (push) Successful in 21s
CI / coverage (push) Successful in 45s
CI / valgrind (push) Successful in 38s
CI / build-and-test (push) Successful in 5m42s
2026-09-10 13:59:15 +02:00
TapTap 61eb08023a Merge feat/d5-daemon-motd: daemon MOTD display + --no-motd 2026-09-10 13:59:07 +02:00
TapTap 0c35cf2b82 test(d5-daemon-motd): assert no banner when no config key 2026-09-10 13:58:21 +02:00
TapTap cf5f730940 feat(d5-daemon-motd): daemon MOTD display + --no-motd 2026-09-10 13:52:24 +02:00
TapTap 8330f275a6 docs: recount RSYNC_COMPAT to 117/14 after daemon Wave B auth (--password-file, --early-input)
CI / lint (push) Successful in 1m13s
CI / sanitizers (undefined) (push) Successful in 54s
CI / sanitizers (address) (push) Successful in 56s
CI / fuzz-build (push) Successful in 20s
CI / coverage (push) Successful in 44s
CI / valgrind (push) Successful in 38s
CI / build-and-test (push) Successful in 5m10s
2026-09-10 13:27:39 +02:00
TapTap d3c9f1d218 Merge feat/d5-daemon-auth: daemon password auth (--password-file, --early-input) 2026-09-10 13:27:27 +02:00
TapTap accd34ad60 fix(d5-daemon-auth): address auth review findings (Wave B)
- test_credentials.c: NUL-terminate the overlong-line stack buffer before
  make_tmp_file's strlen() (was a stack-buffer-overflow READ under ASan);
  still exercises the overlong-rejection path.
- Add redacted protocol string variants (protocol_send_str_redacted /
  receive + fd send_str_redacted/receive_str_redacted) and use them for the
  daemon auth username/digest so --verbose / LOG_DEBUG_ALL never logs a
  replayable credential while other protocol strings keep their debug trace.
- credentials_verify/gate: replace byte-wise-short-circuiting strcmp with a
  fixed-length constant-time username compare (closes user-enumeration oracle);
  update doc comment to match.
- read_secret_file: preserve password exact bytes (only strip trailing CR/LF)
  and burn the stack line buffer; document the whitespace behavior.
- test_server_cli.c: note the parser zero-inits opts on failure.
- Add debug-level daemon test asserting the digest never appears under --verbose.

PROTOCOL_VERSION stays 2.15.0.
2026-09-10 13:20:42 +02:00
TapTap dd5ae60459 feat(d5-daemon-auth): password auth, --password-file, --early-input 2026-09-10 12:50:56 +02:00
TapTap 958eddf414 docs: recount RSYNC_COMPAT to 115/16 after daemon Wave A (--daemon/--config/--dparam/--no-detach)
CI / lint (push) Successful in 1m12s
CI / sanitizers (undefined) (push) Successful in 54s
CI / sanitizers (address) (push) Successful in 55s
CI / fuzz-build (push) Successful in 22s
CI / coverage (push) Successful in 46s
CI / valgrind (push) Successful in 38s
CI / build-and-test (push) Successful in 4m41s
2026-09-09 18:40:06 +02:00
TapTap 8440dbfdb8 Merge feat/d5-daemon-core: daemon lifecycle, module config, ::dest (PROTOCOL 2.15.0) 2026-09-09 18:39:37 +02:00
TapTap 7c55409a6b fix(d5-daemon-core): cppcheck const-correctness, wire module length cap, daemonize chdir/umask, daemon confinement tests
- daemon_conf.c/server.c/test_daemon_conf.c: const-qualify parse/loop pointers;
  scope user_path static inside its block (clears the 9-wave-A cppcheck findings)
- config.c receive_daemon_module: reject invalid/over-long wire module names
  (> DAEMON_MAX_MODULE_NAME) with a clean STATUS_ERROR; client side already
  enforced via daemon_module_name_valid in config_parse_daemon_dest
- server.c daemonize: chdir(/) and umask(0) so module paths resolve from /
  and config-requested file modes are honored; PROTOCOL_VERSION stays 2.15.0
- test_daemon.py: confinement (read-only/unknown no-write anywhere), module-less
  and dot-dot destination refusal, real daemon_detach double-fork path
2026-09-09 18:30:52 +02:00
TapTap b3d7d64347 feat(d5-daemon-core): daemon lifecycle, module config, ::dest, PROTOCOL 2.15.0 2026-09-09 17:48:07 +02:00
TapTap 9d17e951f1 docs: recount RSYNC_COMPAT to 111/20 after Phase 5 waves A-C; clang-format 18 reflow
CI / lint (push) Successful in 1m9s
CI / sanitizers (undefined) (push) Successful in 53s
CI / sanitizers (address) (push) Successful in 55s
CI / fuzz-build (push) Successful in 19s
CI / coverage (push) Successful in 43s
CI / valgrind (push) Successful in 37s
CI / build-and-test (push) Successful in 4m28s
2026-09-09 14:37:52 +02:00
TapTap 1e02ebcd32 Merge feat/p5-remote-option: --remote-option, --trust-sender
# Conflicts:
#	RSYNC_COMPAT.md
#	src/client/client_cli.c
#	src/client/client_send.c
#	src/shared/transport_ssh.c
#	src/shared/transport_ssh.h
#	tests/integration/test_ssh.py
#	tests/test_client_cli.c
#	tests/test_transport_ssh.c
2026-09-09 14:35:26 +02:00
TapTap b1c63ff947 Merge feat/p5-socket: --address, -4/-6, --sockopts, server bind options
# Conflicts:
#	RSYNC_COMPAT.md
#	tests/test_client_cli.c
2026-09-09 14:30:36 +02:00
TapTap 35f4297538 Merge feat/p5-rsh: --rsh/-e, --rsync-path, --blocking-io, --outbuf 2026-09-09 14:29:58 +02:00
TapTap f86ba7a556 fix(p5-socket): clang-format 18 reflow + cppcheck const-correctness 2026-09-09 14:29:46 +02:00
TapTap cdcaf21acd fix(p5-rsh): NULL-check argv tail str_dups in ssh_build_client_argv 2026-09-09 14:29:46 +02:00
TapTap ad228db915 fix(p5-remote-option): wire server --trust-sender, align save-layer gates, add hostile-sender test 2026-09-09 14:23:59 +02:00
TapTap 07d1dd84f7 feat(p5-socket): --address, -4/-6, --sockopts, server bind options 2026-09-09 13:41:28 +02:00
TapTap 858af3d63d feat(p5-rsh): --rsh/-e, --rsync-path, --blocking-io, --outbuf 2026-09-09 13:41:28 +02:00
TapTap 5e79d7d76b feat(p5-remote-option): --remote-option (probe 2.14.0), --trust-sender 2026-09-09 13:41:28 +02:00
TapTap 90ccf297d7 style: cppcheck — const-correctness in file_send_special and test_chunk
CI / lint (push) Successful in 1m7s
CI / sanitizers (address) (push) Successful in 53s
CI / sanitizers (undefined) (push) Successful in 53s
CI / fuzz-build (push) Successful in 21s
CI / coverage (push) Successful in 44s
CI / valgrind (push) Successful in 37s
CI / build-and-test (push) Successful in 4m41s
file_send_special now takes const File*; test_chunk declares the deserialized
Chunk* const.  cppcheck (--error-exitcode=1) now exits clean; clang-format
clean; unit 29/29.
2026-09-08 22:53:47 +02:00
TapTap 5a00a4fe2b style: cppcheck — drop redundant if(fd>=0) in test_xattr copy-fallback test
CI / lint (push) Failing after 1m6s
CI / build-and-test (push) Skipped
CI / sanitizers (address) (push) Skipped
CI / sanitizers (undefined) (push) Skipped
CI / fuzz-build (push) Skipped
CI / coverage (push) Skipped
CI / valgrind (push) Skipped
EXPECT_TRUE(fd >= 0) already guards; the enclosing if is flagged always-true
by cppcheck. read(-1) is safe.
2026-09-08 22:46:43 +02:00
TapTap 743b00ffdd docs: recount RSYNC_COMPAT summary after Phase-4 wave C (symlink-trust + devices + acl/xattr)
CI / lint (push) Failing after 1m8s
CI / build-and-test (push) Skipped
CI / sanitizers (address) (push) Skipped
CI / sanitizers (undefined) (push) Skipped
CI / fuzz-build (push) Skipped
CI / coverage (push) Skipped
CI / valgrind (push) Skipped
Wave C moved 12 rows: -l/--links now real (was Partial). New Implemented (7):
--munge-links, -k/--copy-dirlinks, -K/--keep-dirlinks, -D, -A/--acls, -X/--xattrs
(links was Partial->Implemented). New Partial (5): --devices, --specials,
--copy-devices, --write-devices, --fake-super. Summary: Implemented 94->101,
Partial 6->10, Not Implemented 41->30 (Total 147).

Final Phase-4 summary: 101/3/10/3/30 = 147; PROTOCOL_VERSION 2.13.0.
2026-09-08 22:42:02 +02:00
TapTap 94b6662018 Merge feat/p4-acl-xattr: -X/-A/--fake-super
# Conflicts:
#	src/shared/config.c
#	src/shared/file.c
#	src/shared/file_types.h
#	tests/integration/test_features.py
#	tests/test_client_cli.c
#	tests/test_config.c
2026-09-08 22:38:15 +02:00
TapTap 5bbdc5b450 Merge feat/p4-devices
# Conflicts:
#	src/client/client_send.c
#	src/server/receiver.c
#	src/shared/chunk.c
#	src/shared/file.c
#	src/shared/file_receive.c
#	src/shared/file_receive.h
#	src/shared/file_types.h
#	src/shared/protocol.h
#	tests/test_chunk.c
2026-09-08 22:33:52 +02:00
TapTap e600b56f10 Merge feat/p4-symlink-trust 2026-09-08 22:27:56 +02:00
TapTap 747946c318 acls/xattrs: -X/--xattrs, -A/--acls, --fake-super
CI / lint (pull_request) Failing after 55s
CI / build-and-test (pull_request) Skipped
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
New src/shared/xattr.{c,h}: capture user.* + POSIX ACL xattrs, transmit a bounded
per-file block, re-apply fd-relative. security.*/trusted.*/other system.* never
transmitted/applied (receiver re-validates). Bounds: name<=255 value<=1MiB count
<=256 total<=4MiB. --fake-super records uid:gid:mode:mtime in reserved
user.fastsync.stat (receiver-only). PROTOCOL_VERSION 2.12.0->2.13.0. Review
fixes: reserved key not forwardable, link/hardlink copy-fallback preserves
xattrs, no const-param mutation, per-file warning dedup.
2026-09-08 22:27:53 +02:00
TapTap 007e8f90f2 devices: --devices/--specials/-D/--copy-devices/--write-devices
CI / lint (pull_request) Failing after 56s
CI / build-and-test (pull_request) Skipped
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
Recreate char/block nodes via mknodat (privilege-gated, EPERM->warn+skip) and
FIFOs via mkfifoat; new STATUS_SPECIAL frame + validated rdev; sockets skipped;
-copy-devices copies st_size; -write-devices O_NOFOLLOW+O_NONBLOCK warn+skip.
preserve_specials/copy_devices/write_devices cross the wire. PROTOCOL_VERSION
2.12.0->2.13.0. Review fixes: -m source-removal keeps recreated specials, FIFO
ENXIO skip, rdev bounds at chunk_deserialize, STATUS_ERROR on receive branch,
scanner_prepare_special dedup.
2026-09-08 22:27:53 +02:00
TapTap 820188c2cc symlink-trust: -k/--copy-dirlinks, -K/--keep-dirlinks, --munge-links (+real -l/--links)
CI / lint (pull_request) Successful in 1m1s
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
CI / build-and-test (pull_request) Successful in 1m19s
Adds symlink-target transmission (File is_symlink+symlink_target, STATUS_SYMLINK
frame, chunk type 2), munge-links sender containment + receiver-side symmetric
target containment, keep-dirlinks confined dir-symlink following (O_NOFOLLOW
realpath-rechecked), and fixes -l to copy symlinks as symlinks. munge_links +
keep_dirlinks cross the wire; copy_dirlinks client-only. PROTOCOL_VERSION
2.12.0->2.13.0. Review fixes: receiver rejects absolute/.. targets, gated unmunge,
-K O_NOFOLLOW+re-fstat, keep_dirlinks set once at config-accept, rel_buf overflow
fails the walk.
2026-09-08 22:27:53 +02:00
TapTap 0de859b302 docs: recount RSYNC_COMPAT summary after Phase-4 wave B (metadata times + hard links)
CI / lint (push) Successful in 53s
CI / sanitizers (address) (push) Successful in 49s
CI / sanitizers (undefined) (push) Successful in 49s
CI / fuzz-build (push) Successful in 20s
CI / coverage (push) Successful in 43s
CI / valgrind (push) Successful in 37s
CI / build-and-test (push) Successful in 4m28s
Wave B moved 6 rows: -U/--atimes, --open-noatime, -H/--hard-links -> ✅;
-N/--crtimes -> ⚠️; -O/--omit-dir-times, -J/--omit-link-times -> 🔄 (no-ops).
Summary: ✅91->94, ⚠️5->6, 🔄1->3, ❌47->41 (Total 147).
2026-09-08 21:02:34 +02:00
TapTap 4e7e84f947 Merge feat/p4-metadata-capture: atimes/crtimes/open-noatime/omit-dir-times/omit-link-times
# Conflicts:
#	src/shared/config.c
#	tests/integration/test_features.py
2026-09-08 21:00:07 +02:00
TapTap cf7cc5b8ca Merge feat/p4-hard-links: -H/--hard-links 2026-09-08 20:59:04 +02:00
TapTap e80888ce7b metadata times: -U/--atimes, -N/--crtimes, --open-noatime, -O/-J
CI / lint (pull_request) Successful in 52s
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
CI / build-and-test (pull_request) Successful in 1m19s
Capture+transmit source atime (pre-read stat; O_NOATIME sender guard) and birth
time (statx STATX_BTIME); receiver restores atime with mtime (crtime not settable
portably -> transmitted, explicitly not applied). --open-noatime is client-only.
-O/-J documented as accepted no-ops (FastSync never preserves dir/symlink times).
Wire: metadata frame gains atime/crtime val+sec+nsec; PROTOCOL_VERSION
2.11.0->2.12.0. Review fixes: gate atime capture to Linux (no epoch clobber on
non-Linux), close fd on fdopen failure, honest -O/-J status (Compat no-op).
2026-09-08 20:58:56 +02:00
TapTap f891cd0a6a hard-links: -H/--hard-links preserves inode relationships
CI / lint (pull_request) Successful in 50s
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
CI / build-and-test (pull_request) Successful in 1m19s
Source files sharing (st_dev,st_ino) are recreated as hard links on the
destination; only the first member's data crosses the wire (siblings ride a
payload-less STATUS_HARDLINK frame). Ordering requires the single-FIFO-writer
receiver + forced sequential scan (documented). link()-failure falls back to a
byte-identical local copy. Rejects -s/--append. PROTOCOL_VERSION 2.11.0->2.12.0.
Review fixes: delete the dead HardLinkRegistry (ordering holds by FIFO writer),
and --existing no longer aborts when the first member is absent but the sibling
exists (leaves the sibling in place).
2026-09-08 20:58:56 +02:00
TapTap cbb09e41ab identity: fix use-after-free in --usermap/--groupmap parse error path
CI / lint (push) Successful in 48s
CI / sanitizers (undefined) (push) Successful in 53s
CI / sanitizers (address) (push) Successful in 53s
CI / fuzz-build (push) Successful in 18s
CI / coverage (push) Successful in 42s
CI / valgrind (push) Successful in 37s
CI / build-and-test (push) Successful in 4m37s
identity_parse_map freed the str_dup'd list before logging the offending rule
( points into that buffer), causing an invalid read caught by CI valgrind
(MSAN/MSAN-style; the ONLY definite valgrind error in the suite).  Log before
freeing.  valgrind now reports 0 errors / 0 definite leaks in both the parent
and the forked wire-roundtrip child.
2026-09-08 18:49:23 +02:00
TapTap 30048dddef style: clang-format 18 (CI lint) on Wave-A sources
CI / lint (push) Successful in 48s
CI / sanitizers (address) (push) Successful in 47s
CI / sanitizers (undefined) (push) Successful in 48s
CI / fuzz-build (push) Successful in 19s
CI / coverage (push) Successful in 44s
CI / valgrind (push) Failing after 37s
CI / build-and-test (push) Successful in 4m36s
Reflow usage.c, config.c, file.c, test_client_cli.c, test_file.c so the
clang-format check passes.  Whitespace-only; no behavior change.
2026-09-08 18:38:50 +02:00
TapTap ba82a0b6da docs: recount RSYNC_COMPAT summary after Phase-4 wave A (identity + preallocate)
CI / lint (push) Failing after 5s
CI / build-and-test (push) Skipped
CI / sanitizers (address) (push) Skipped
CI / sanitizers (undefined) (push) Skipped
CI / fuzz-build (push) Skipped
CI / coverage (push) Skipped
CI / valgrind (push) Skipped
5 rows moved ❌ -> ✅: --numeric-ids/--usermap/--groupmap/--chown and
--preallocate.  Summary: ✅86 -> ✅91, ❌52 -> ❌47 (Total 147 unchanged).
2026-09-08 18:37:45 +02:00
TapTap b283c8084c identity: keep --numeric-ids in the activate set (-M --numeric-ids)
CI / lint (push) Failing after 4s
CI / build-and-test (push) Skipped
CI / sanitizers (address) (push) Skipped
CI / sanitizers (undefined) (push) Skipped
CI / fuzz-build (push) Skipped
CI / coverage (push) Skipped
CI / valgrind (push) Skipped
identity_active_enabled() only gates identity_apply_ownership, which runs only
when metadata is present, so --numeric-ids must stay in the set: combined with
-M it activates raw-id application, while a standalone --numeric-ids (no
ownership-affecting flag) carries no metadata and correctly stays inert.  My
earlier review fix removed it and broke 'owner not applied' for -M --numeric-ids
(uid 0 instead of the source ids).  Revert that removal.
2026-09-08 18:37:07 +02:00
TapTap 279fc8468a Merge feat/p4-preallocate: --preallocate
# Conflicts:
#	tests/test_client_cli.c
#	tests/test_config.c
2026-09-08 18:25:49 +02:00
TapTap 8defaf5e8d Merge feat/p4-identity-mapping: --numeric-ids / --usermap / --groupmap / --chown 2026-09-08 18:23:52 +02:00
TapTap 362a6a5488 preallocate: --preallocate allocates dest space up front
CI / lint (pull_request) Failing after 3s
CI / build-and-test (pull_request) Skipped
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
Receiver allocates the destination file's full size before streaming data
(posix_fallocate preferred, ftruncate fallback on EOPNOTSUPP/ENOSYS) so an
out-of-space transfer fails fast instead of partway. Additive config bool
crossing the wire; PROTOCOL_VERSION 2.10.0 -> 2.11.0. Threaded through all
store paths (atomic, inplace, partial/delay-updates staging, link-dest copy
fallback). Review hardening: explicit lseek(0) before the data write so
correctness does not depend on posix_fallocate leaving the fd offset unchanged.
2026-09-08 18:23:48 +02:00
TapTap 53ce00b830 identity mapping: --numeric-ids / --usermap / --groupmap / --chown
CI / lint (pull_request) Failing after 3s
CI / build-and-test (pull_request) Skipped
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
Receiver-side ownership application, opt-in and privilege-gated:
- OFF for every existing transfer (plain -M/--preserve still never applies
  ownership); only triggers on an explicit identity flag + receiver permission.
- EPERM/EACCES warn-and-continue (never aborts); other fchown errors escalate.
- fd-relative fchown after the file is written (symlink-safe, confined).
- New src/shared/identity.{c,h}; config fields numeric_ids / chown uid/gid /
  usermap + groupmap id-pair tables cross the wire; PROTOCOL_VERSION 2.10.0
  -> 2.11.0. CLI in client_cli.c; per-connection snapshot in server.c.
- Review fixes: EPERM/EACCES-only warn-and-continue, prominent root-receiver
  notice, identity_clear_active on early server error paths, --numeric-ids
  kept inert standalone (removed from activation trigger set).
2026-09-08 18:23:43 +02:00
TapTap a5e176babc Merge feat/ci-speedup: parallelize integration tests; fast PR gate + full coverage on merge
CI / lint (push) Successful in 57s
CI / sanitizers (undefined) (push) Successful in 50s
CI / sanitizers (address) (push) Successful in 1m7s
CI / fuzz-build (push) Successful in 17s
CI / valgrind (push) Successful in 37s
CI / coverage (push) Successful in 43s
CI / build-and-test (push) Successful in 4m57s
2026-09-08 17:29:02 +02:00
TapTap c90b07afcb ci: address review — use --dist=load, per-module teardown, exclude setpriv
CI / lint (pull_request) Failing after 0s
CI / build-and-test (pull_request) Skipped
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
- Replace --dist=loadgroup (a no-op: it only groups by xdist_group marks) with
  --dist=load, and make module teardowns remove only their own SOURCE_DIR/DEST_DIR
  (never the shared worker-keyed TEST_DATA_DIR) so interleaved modules can't wipe
  each other's fixtures. Add a session-scoped cleanup of the per-worker dir.
  (loadfile grouped the whole test_features.py module onto one worker and slowed
  the full suite to 761s vs 224s with load.)
- Mark the two setpriv privilege tests with a 'setpriv' marker and run the full
  merge suite as -m "not setpriv", so the dev/main gate can't go red on the
  env-dependent /root-traversal tests regardless of the runner uid.
- Add a TLS basic test to the ci subset, add workflow_dispatch for on-demand full
  runs, and fix trailing newlines.
- Measured: smoke -m ci = 28 passed/39s; full -n4 = 280 passed/11 skipped/1 xpassed
  in 224s (was 860s serial).
2026-09-08 17:25:56 +02:00
TapTap 79d965ac11 ci: parallelize integration suite with pytest-xdist; fast PR gate + full coverage on merge
CI / lint (pull_request) Failing after 0s
CI / build-and-test (pull_request) Skipped
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
- Add pytest-xdist to the CI Dockerfile (image -> v10).
- Worker-isolate TEST_DATA_DIR (PYTEST_XDIST_WORKER) so concurrent xdist
  workers never collide on shared-filesystem fixtures.
- Register a 'ci' marker and tag a fast representative subset of integration
  tests (basic TCP, incremental, compression, delete, basis, append).
- ci.yaml: lint + build + unit + the marked subset (-n4) on every PR; the
  full integration suite plus sanitizer/fuzz/coverage/valgrind run only on
  push to dev/main.
- Integration step runtime drops from ~860s (serial) to ~230s (-n4); the PR
  gate lands well under ~3 minutes.
2026-09-08 16:53:54 +02:00
TapTap 829e760086 docs: recount RSYNC_COMPAT summary after Phase-3 wave C
CI / lint (push) Successful in 47s
CI / sanitizers (undefined) (push) Successful in 52s
CI / sanitizers (address) (push) Successful in 53s
CI / fuzz-build (push) Successful in 20s
CI / coverage (push) Successful in 43s
CI / valgrind (push) Successful in 37s
CI / build-and-test (push) Successful in 15m4s
CI / build-and-test (pull_request) Successful in 15m0s
CI / lint (pull_request) Successful in 47s
CI / sanitizers (address) (pull_request) Successful in 47s
CI / sanitizers (undefined) (pull_request) Successful in 45s
CI / fuzz-build (pull_request) Successful in 17s
CI / coverage (pull_request) Successful in 42s
CI / valgrind (pull_request) Successful in 36s
2026-09-07 20:09:36 +02:00
TapTap 4c98744014 Merge feat/p3-checksum-choice: --checksum-choice/--cc and --checksum-seed 2026-09-07 19:53:28 +02:00
TapTap f99e6e1edf Merge feat/p3-append: --append / --append-verify tail resume 2026-09-07 19:52:33 +02:00
TapTap 681d7a8532 Merge feat/p3-missing-args: --ignore-missing-args / --delete-missing-args 2026-09-07 19:51:02 +02:00
TapTap c6758531f6 fix: reject xxh3 alias, pin handshake digest length to algorithm, simplify xxh64 copy, note FIPS md5
CI / lint (pull_request) Successful in 35s
CI / sanitizers (undefined) (pull_request) Successful in 52s
CI / sanitizers (address) (pull_request) Successful in 52s
CI / fuzz-build (pull_request) Successful in 18s
CI / coverage (pull_request) Successful in 42s
CI / valgrind (pull_request) Successful in 37s
CI / build-and-test (pull_request) Successful in 12m58s
2026-09-07 19:50:52 +02:00
TapTap e754f0d4eb test: cover absent-parent no-op and --dirs scanner skip
CI / lint (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 51s
CI / sanitizers (address) (pull_request) Successful in 51s
CI / fuzz-build (pull_request) Successful in 19s
CI / coverage (pull_request) Successful in 43s
CI / valgrind (pull_request) Successful in 37s
CI / build-and-test (pull_request) Successful in 14m55s
Unit: manifest_delete_missing_args treats a deeper missing entry whose
destination parent directory does not exist as a no-op (run continues, nothing
created). Integration (TestMissingArgs): a deeper missing entry with an absent
parent -R bare and full-mirror layouts, single-threaded and -m, no longer
aborts --delete (the extras walk still removes an unrelated extra); --dirs +
--files-from with --ignore-missing-args skips a listed-but-missing entry and
transfers the rest.
2026-09-07 18:33:34 +02:00
TapTap 4b74f6a41d fix: treat an absent missing-mirror parent as a no-op, only claim real deletions
BLOCKER: an absent mirror whose PARENT directory does not exist on the
destination (a deeper --files-from missing entry, -R or full-mirror layout) was
treated as a hard failure because file_open_secure_parent returned -1 when the
parent was missing. That aborted the whole run, skipped the --delete extras
walk, and tore down an early --delete-before/during connection, contradicting
'missing mirror = no-op / all-missing succeeds'. A parent-open failure is now a
no-op when errno is ENOENT/ENOTDIR (matching file_remove_tree_secure); only a
genuine I/O error fails the run.

Also: an already-absent mirror reached via unlinkat-ENOENT no longer prints
'Deleted: <path>' (a no-op dressed as a deletion); the per-path 'Deleted:' line
is printed only when an entry was actually removed.
2026-09-07 18:32:47 +02:00
TapTap 56981a0d9d fix: make digest-vector arrays const to satisfy cppcheck constVariable
CI / lint (pull_request) Successful in 35s
CI / sanitizers (undefined) (pull_request) Successful in 50s
CI / sanitizers (address) (pull_request) Successful in 52s
CI / fuzz-build (pull_request) Successful in 19s
CI / coverage (pull_request) Successful in 42s
CI / valgrind (pull_request) Successful in 36s
CI / build-and-test (pull_request) Successful in 12m58s
2026-09-07 17:46:52 +02:00
TapTap a2ce0a8e41 docs: mark --checksum-choice and --checksum-seed implemented
CI / lint (pull_request) Failing after 35s
CI / build-and-test (pull_request) Skipped
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
Precise notes for both rows: supported algorithms (xxh64/xxhash, md5 via EVP),
rejection of unsupported names, seed semantics (full 64-bit for whole-file,
low 32 bits for the delta block hash, ignored by md5), the --cc alias, the
length-prefixed bounded handshake digest, the 2.9.0 -> 2.10.0 protocol bump,
default byte-for-byte preservation, and the divergence from rsync's randomized
seed (FastSync defaults to seed 0). Summary count line is intentionally untouched.
2026-09-07 17:43:00 +02:00
TapTap df250ec9c9 test: coverage for --checksum-choice/--cc and --checksum-seed
Unit: digest selection with known vectors (xxh64 seed 0/empty, md5 RFC vectors,
seed changes xxh64 but not md5, name mapping, bounded buffer rejection), CLI
parse (all spellings, = forms, --cc alias, unsupported rejected, seed strict
decimal), config wire round-trip and out-of-range algo rejection, and seeded
delta signature/compute symmetry (matching seed rebuilds, mismatched seed yields
no block matches). Integration: unchanged skip + same-size/mtime redetect for
xxh64 and md5 with and without -m, near-zero-data skip run, deterministic seed,
and a byte-exact seeded delta run.
2026-09-07 17:42:53 +02:00
TapTap 0afda6b094 feat(checksum): --checksum-choice/--cc and --checksum-seed for whole-file digest
Adds real algorithm selection (xxh64 default, plus md5 via OpenSSL EVP) and a
64-bit seed for the per-file whole-file digest used by the --incremental/
--checksum handshake and basis-dir content verification. The seed also feeds
the delta path's per-block xxHash32 strong checksum (low 32 bits) so an
explicit seed deterministically changes those digests too. Sender and receiver
hash identically: the algorithm id and seed cross the config wire frame and the
STATUS_CHECK handshake now carries a length-prefixed, bounded digest instead of
a fixed 64-bit value. Unsupported algorithm names are rejected at parse time
(never a silent no-op). PROTOCOL_VERSION bumped 2.9.0 -> 2.10.0; defaults
(xxh64, seed 0) preserve prior byte-for-byte behavior.
2026-09-07 17:42:47 +02:00
TapTap bfb3a531e9 docs: mark --append / --append-verify implemented in RSYNC_COMPAT
CI / lint (pull_request) Successful in 49s
CI / sanitizers (undefined) (pull_request) Successful in 50s
CI / sanitizers (address) (pull_request) Successful in 51s
CI / fuzz-build (pull_request) Successful in 17s
CI / coverage (pull_request) Successful in 42s
CI / valgrind (pull_request) Successful in 36s
CI / build-and-test (pull_request) Successful in 12m52s
Flip both rows to Implemented with precise Notes covering the tail-resume
model, the prefix-verify semantics (and the plain-append rsync-parity safety
statement), the new wire frames, and the PROTOCOL_VERSION 2.9.0 -> 2.10.0 bump.
Add a Phase-3 append-wave implementation-notes block. The Summary count line
is deliberately left untouched.
2026-09-07 15:43:52 +02:00
TapTap ffdbb6568f test: append/append-verify resume coverage
- CLI: --append/--append-verify acceptance (parse + imply --incremental,
  validate) and incompatibility rejection with -s and --whole-file; both
  removed from the unimplemented reject list.
- Config: on-the-wire append/append_verify round-trip.
- Unit: append_resume_eligible / append_tail_length pure resume math.
- Integration (test_append.py): matching-prefix resume is byte-identical and
  tail-only (wire bytes << source size); --append with a wrong prefix keeps
  prefix+tail (rsync parity) while --append-verify detects the mismatch and
  falls back to a byte-exact full transfer; --append with --inplace and -m.
2026-09-07 15:43:33 +02:00
TapTap 6ad3887aa1 feat: --append / --append-verify tail-only resume
Implement rsync's append modes: when an existing destination file is SHORTER
than the source, the receiver negotiates a resume offset and only the tail is
transferred; the full file (retained prefix + tail) is rebuilt and installed
through the normal atomic store path, so the result is byte-identical to the
source whenever the prefix matches.

- --append: sends the tail without content-verifying the retained prefix
  (rsync parity; the documented prefix-trust risk).
- --append-verify: verifies the retained prefix against the source's prefix
  xxHash64 before appending and, on a mismatch, falls back to a clean full
  transfer (never a corrupt prefix+tail blend).

New wire frames STATUS_APPEND / STATUS_APPEND_SIG / STATUS_APPEND_OK /
STATUS_APPEND_DATA; PROTOCOL_VERSION bumped 2.9.0 -> 2.10.0 (peers must match).
Both flags imply --incremental and are incompatible with -s (chunk
serialization) and --whole-file (rejected up front). Respects --inplace,
--partial/--partial-dir and --delay-updates via the shared store engine.
2026-09-07 15:43:29 +02:00
181 changed files with 38390 additions and 3954 deletions

No files matched your search

+28 -14
View File
@@ -4,14 +4,15 @@ on:
push:
branches: [main, dev]
pull_request:
workflow_dispatch:
jobs:
lint:
runs-on: ubuntu-latest
container: gitea.tap-tap.win/taptap/fastsync-ci:v9
container: gitea.tap-tap.win/taptap/fastsync-ci:v10
steps:
- name: Checkout
uses: actions/checkout@v4
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- name: clang-format check
run: find src/ tests/ -name '*.c' -o -name '*.h' | xargs clang-format --dry-run --Werror
@@ -19,13 +20,17 @@ jobs:
- name: cppcheck
run: cppcheck --enable=warning,style,performance,portability --suppress=missingIncludeSystem --error-exitcode=1 --inline-suppr src/ tests/
# Fast PR gate: build + unit tests + a representative subset of integration
# tests (marked `ci`), parallelized with pytest-xdist. Only the full coverage
# jobs below (sanitizers/fuzz/coverage/valgrind and the FULL integration
# suite) run on merge to dev/main, so PR CI stays well under ~3 minutes.
build-and-test:
runs-on: ubuntu-latest
container: gitea.tap-tap.win/taptap/fastsync-ci:v9
container: gitea.tap-tap.win/taptap/fastsync-ci:v10
needs: lint
steps:
- name: Checkout
uses: actions/checkout@v4
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- name: Configure
run: cmake -B build -S . -DSTRICT_WARNINGS=ON
@@ -36,19 +41,25 @@ jobs:
- name: Unit Tests
run: ctest --test-dir build --output-on-failure -j$(nproc)
- name: Integration Tests
run: python3 -m pytest tests/integration/ -v --tb=short
- name: Integration Tests (PR smoke subset)
if: github.event_name == 'pull_request'
run: python3 -m pytest tests/integration/ -n 4 --dist=load -m ci --durations=25 --tb=short -q
- name: Integration Tests (full suite)
if: github.event_name == 'push'
run: python3 -m pytest tests/integration/ -n 4 --dist=load -m "not setpriv" --durations=25 --tb=short -q
sanitizers:
runs-on: ubuntu-latest
container: gitea.tap-tap.win/taptap/fastsync-ci:v9
container: gitea.tap-tap.win/taptap/fastsync-ci:v10
needs: lint
if: github.event_name == 'push'
strategy:
matrix:
sanitizer: [address, undefined]
steps:
- name: Checkout
uses: actions/checkout@v4
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- name: Configure
run: cmake -B build-${{ matrix.sanitizer }} -S . -DSANITIZER=${{ matrix.sanitizer }}
@@ -61,11 +72,12 @@ jobs:
fuzz-build:
runs-on: ubuntu-latest
container: gitea.tap-tap.win/taptap/fastsync-ci:v9
container: gitea.tap-tap.win/taptap/fastsync-ci:v10
needs: lint
if: github.event_name == 'push'
steps:
- name: Checkout
uses: actions/checkout@v4
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- name: Configure (clang + fuzz)
run: CC=clang CXX=clang++ cmake -B build-fuzz -S . -DENABLE_FUZZ=ON
@@ -82,11 +94,12 @@ jobs:
coverage:
runs-on: ubuntu-latest
container: gitea.tap-tap.win/taptap/fastsync-ci:v9
container: gitea.tap-tap.win/taptap/fastsync-ci:v10
needs: lint
if: github.event_name == 'push'
steps:
- name: Checkout
uses: actions/checkout@v4
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- name: Configure
run: cmake -B build -S . -DENABLE_COVERAGE=ON
@@ -105,11 +118,12 @@ jobs:
valgrind:
runs-on: ubuntu-latest
container: gitea.tap-tap.win/taptap/fastsync-ci:v9
container: gitea.tap-tap.win/taptap/fastsync-ci:v10
needs: lint
if: github.event_name == 'push'
steps:
- name: Checkout
uses: actions/checkout@v4
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- name: Configure
run: cmake -B build -S . -DSTRICT_WARNINGS=ON
+4
View File
@@ -8,3 +8,7 @@ build-*/
build2/
build3/
build_docker2/
# Test/run artifacts
root/
test_partial_install_tmp/
+1 -1
View File
@@ -128,7 +128,7 @@ Do not wait for the user to tell you CI failed — check proactively. The user s
## Branch Strategy
Never push directly to `main`. All changes must be developed on a feature branch and merged via a pull request. Always create a new branch (`git checkout -b <branch-name>`) before making changes, push it, and open a PR with `gh pr create --fill`. Wait for CI to pass before merging.
Never push directly to `dev` or `main`. All changes must be developed on a feature branch and merged via a pull request targeting `dev`. Create a branch (`git checkout -b <branch-name>`), push it, and open the PR with `tea pr create --repo TapTap/FastSync --base dev --head <branch-name>`. Wait for CI to pass before merging.
## Dependency Installation
+1 -1
View File
@@ -92,7 +92,7 @@ When using `tea` (the task execution agent) to run CI or tests, always set a suf
## Branch Strategy
Never push directly to `main`. All changes must be developed on a feature branch and merged via a pull request. Always create a new branch (`git checkout -b <branch-name>`) before making changes, push it, and open a PR with `gh pr create --fill`. Wait for CI to pass before merging.
Never push directly to `dev` or `main`. All changes must be developed on a feature branch and merged via a pull request targeting `dev`. Create a branch (`git checkout -b <branch-name>`), push it, and open the PR with `tea pr create --repo TapTap/FastSync --base dev --head <branch-name>`. Wait for CI to pass before merging.
## Dependency Installation
+13 -12
View File
@@ -27,16 +27,19 @@ FetchContent_Declare(xxhash GIT_REPOSITORY https://github.com/Cyan4973/xxHash GI
FetchContent_MakeAvailable(xxhash)
# Sanitizer option
set(SANITIZER "none" CACHE STRING "Sanitizer to enable (address, thread, none)")
set_property(CACHE SANITIZER PROPERTY STRINGS address thread none)
set(SANITIZER "none" CACHE STRING "Sanitizer to enable (address, thread, undefined, none)")
set_property(CACHE SANITIZER PROPERTY STRINGS address thread undefined none)
if(SANITIZER STREQUAL "address")
add_compile_options(-fsanitize=address -fno-omit-frame-pointer -g)
add_link_options(-fsanitize=address)
elseif(SANITIZER STREQUAL "thread")
add_compile_options(-fsanitize=thread -fno-omit-frame-pointer -g)
add_link_options(-fsanitize=thread)
elseif(SANITIZER STREQUAL "undefined")
add_compile_options(-fsanitize=undefined -fno-omit-frame-pointer -g)
add_link_options(-fsanitize=undefined)
elseif(NOT SANITIZER STREQUAL "none")
message(FATAL_ERROR "Unknown sanitizer: ${SANITIZER}. Supported values: address, thread, none")
message(FATAL_ERROR "Unknown sanitizer: ${SANITIZER}. Supported values: address, thread, undefined, none")
endif()
option(STRICT_WARNINGS "Enable strict warnings" OFF)
@@ -84,7 +87,7 @@ tests/integration/ — Python pytest integration tests
### Dependencies
- **zstd** — found via `find_library(ZSTD_LIBRARY zstd)`
- **OpenSSL** — found via `find_package(OpenSSL REQUIRED)` (TLS 1.2+ transport)
- **xxHash** — fetched via `FetchContent` from GitHub (delta transfer hashing, v0.8.3)
- **xxHash** — fetched via `FetchContent` from the upstream repository (delta transfer hashing, v0.8.3)
- **pthreads** — found via `find_package(Threads REQUIRED)`
- **C11 standard** — required
- **CMake 3.22+** — minimum version
@@ -94,7 +97,7 @@ tests/integration/ — Python pytest integration tests
- Use `file(GLOB ...)` for source collection (existing pattern).
- All targets link `Threads::Threads`, `${ZSTD_LIBRARY}`, `OpenSSL::SSL`, `OpenSSL::Crypto`, and `xxhash`.
- Include directories: `src/shared`, `src/server`, `src/client`, `tests` (for test target).
- Sanitizer support: pass `-DSANITIZER=address` or `-DSANITIZER=thread` to cmake (live option in CMakeLists.txt).
- Sanitizer support: pass `-DSANITIZER=address`, `-DSANITIZER=thread`, or `-DSANITIZER=undefined` to cmake (live option in CMakeLists.txt).
- Build with `cmake -B build -S . && cmake --build build -j$(nproc)`.
- For CI, dependencies are provided by the project's custom Docker image (repo-root `Dockerfile`, same image CI uses). For local development, use `nix-shell`. Never add `apt-get install` / `pip install` to CI workflows. See `AGENTS.md`.
@@ -105,7 +108,7 @@ tests/integration/ — Python pytest integration tests
3. Add new dependencies with `find_package` or `find_library`.
4. When adding a new executable target, follow the pattern of existing targets.
5. When adding a new library (static/shared), use `add_library` and follow the project's naming.
6. For sanitizer builds, pass `-DSANITIZER=address` or `-DSANITIZER=thread` to cmake (matching CI's matrix strategy).
6. For sanitizer builds, pass `-DSANITIZER=address`, `-DSANITIZER=thread`, or `-DSANITIZER=undefined` to cmake (matching CI's matrix strategy).
7. Always verify the build compiles after changes.
## Sanitizer Configurations
@@ -119,11 +122,9 @@ cmake -B build -S . -DSANITIZER=thread # ThreadSanitizer (race conditions)
cmake --build build -j$(nproc)
```
For UndefinedBehaviorSanitizer (no `-DSANITIZER=undefined` option in CMakeLists.txt yet), use the manual flag approach:
UndefinedBehaviorSanitizer uses the same built-in option:
```bash
cmake -B build -S . \
-DCMAKE_C_FLAGS="-fsanitize=undefined -fno-omit-frame-pointer -g" \
-DCMAKE_EXE_LINKER_FLAGS="-fsanitize=undefined"
cmake -B build -S . -DSANITIZER=undefined
cmake --build build -j$(nproc)
```
@@ -159,7 +160,7 @@ cmake -B build -S . -DCMAKE_BUILD_TYPE=RelWithDebInfo
```bash
cmake -B build -S .
cmake --build build -j$(nproc)
./build/server
./build/server -p 8080 --allow-unauthenticated
./build/client
./build/tests
```
@@ -187,7 +188,7 @@ When using `tea` (the task execution agent) to run CI or tests, always set a suf
## Branch Strategy
Never push directly to `main`. All changes must be developed on a feature branch and merged via a pull request. Always create a new branch (`git checkout -b <branch-name>`) before making changes, push it, and open a PR with `gh pr create --fill`. Wait for CI to pass before merging.
Never push directly to `dev` or `main`. All changes must be developed on a feature branch and merged via a pull request targeting `dev`. Create a branch (`git checkout -b <branch-name>`), push it, and open the PR with `tea pr create --repo TapTap/FastSync --base dev --head <branch-name>`. Wait for CI to pass before merging.
## Dependency Installation
+5 -5
View File
@@ -27,7 +27,7 @@ FastSync is a file synchronization tool (like rsync, but faster). It transfers f
cmake -B build -S . && cmake --build build -j$(nproc)
# Server (TCP mode)
./build/server
./build/server -p 8080 --allow-unauthenticated
# Client (TCP mode)
./build/client --source-dir /path/to/send --dest-dir /path/to/receive --save-to-disk
@@ -37,13 +37,13 @@ cmake -B build -S . && cmake --build build -j$(nproc)
# Run tests
./build/tests # unit tests
python3 test.py # integration tests
python3 -m pytest tests/integration/ -n 4 --dist=load -m "not setpriv" # integration tests
```
## Code Walkthrough
### Client Entry Point (`src/client/client_cli.c`)
- Parses CLI arguments using `getopt_long`
- Parses CLI arguments using a custom option-table parser (`OPTION_TABLE` in `src/client/client_cli.c`); there is no `getopt*` usage
- Creates `Config` struct with all options
- Detects SSH destinations (contains `:`)
- Calls into `client_send.c` for the actual transfer
@@ -109,7 +109,7 @@ Collection of files for batch transfer. Serialized with file count, then per-fil
zstd streaming compression via `ZSTD_compressStream2`/`ZSTD_decompressStream`. Compression happens per-chunk in the sender stage. Level 1-22 (default 5). Streaming means memory usage stays bounded regardless of file size.
### "How does sendfile() work?"
On Linux, `sendfile()` copies data directly from kernel file buffer to socket, bypassing userspace. ~2x faster for large files. Enabled with `-f` flag. Only works with TCP (not SSH, not compression).
On Linux, `sendfile()` copies data directly from kernel file buffer to socket, bypassing userspace. ~2x faster for large files. Enabled with `--sendfile` (long form only). Only works with TCP (not SSH, not compression).
### "How does incremental sync work?"
Client sends file metadata (path, size, mtime) to server. Server checks if destination file has same size+mtime. If match, server responds `STATUS_OK` (skip). If mismatch, server responds `STATUS_NEXT` (send).
@@ -138,7 +138,7 @@ When using `tea` (the task execution agent) to run CI or tests, always set a suf
## Branch Strategy
Never push directly to `main`. All changes must be developed on a feature branch and merged via a pull request. Always create a new branch (`git checkout -b <branch-name>`) before making changes, push it, and open a PR with `gh pr create --fill`. Wait for CI to pass before merging.
Never push directly to `dev` or `main`. All changes must be developed on a feature branch and merged via a pull request targeting `dev`. Create a branch (`git checkout -b <branch-name>`), push it, and open the PR with `tea pr create --repo TapTap/FastSync --base dev --head <branch-name>`. Wait for CI to pass before merging.
## Dependency Installation
+1 -1
View File
@@ -316,7 +316,7 @@ When using `tea` (the task execution agent) to run CI or tests, always set a suf
## Branch Strategy
Never push directly to `main`. All changes must be developed on a feature branch and merged via a pull request. Always create a new branch (`git checkout -b <branch-name>`) before making changes, push it, and open a PR with `gh pr create --fill`. Wait for CI to pass before merging.
Never push directly to `dev` or `main`. All changes must be developed on a feature branch and merged via a pull request targeting `dev`. Create a branch (`git checkout -b <branch-name>`), push it, and open the PR with `tea pr create --repo TapTap/FastSync --base dev --head <branch-name>`. Wait for CI to pass before merging.
## Dependency Installation
+8 -10
View File
@@ -14,10 +14,9 @@ Diagnose crashes, memory errors, hangs, and logic bugs. You use structured debug
### Memory Errors
```bash
# AddressSanitizer (fast, recommended first)
cmake -B build -S . -DCMAKE_C_FLAGS="-fsanitize=address -fno-omit-frame-pointer" \
-DCMAKE_EXE_LINKER_FLAGS="-fsanitize=address"
cmake --build build -j$(nproc)
./build/client # or ./build/server
cmake -B build-asan -S . -DSANITIZER=address
cmake --build build-asan -j$(nproc)
./build-asan/client # or ./build-asan/server -p 8080 --allow-unauthenticated
# Valgrind (slower, more thorough)
valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes \
@@ -32,10 +31,9 @@ valgrind --tool=drd ./build/client ...
### Thread Sanitizer
```bash
cmake -B build -S . -DCMAKE_C_FLAGS="-fsanitize=thread" \
-DCMAKE_EXE_LINKER_FLAGS="-fsanitize=thread"
cmake --build build -j$(nproc)
./build/tests
cmake -B build-tsan -S . -DSANITIZER=thread
cmake --build build-tsan -j$(nproc)
./build-tsan/tests
```
### GDB
@@ -143,7 +141,7 @@ gprof ./build/client gmon.out
### Step 5: Verify
- Run `./build/tests` (unit tests)
- Run `python3 test.py` (integration tests)
- Run `python3 -m pytest tests/integration/ -n 4 --dist=load -m "not setpriv"` (integration tests)
- Run under valgrind again to confirm clean
- Test under ASan again
@@ -162,7 +160,7 @@ When using `tea` (the task execution agent) to run CI or tests, always set a suf
## Branch Strategy
Never push directly to `main`. All changes must be developed on a feature branch and merged via a pull request. Always create a new branch (`git checkout -b <branch-name>`) before making changes, push it, and open a PR with `gh pr create --fill`. Wait for CI to pass before merging.
Never push directly to `dev` or `main`. All changes must be developed on a feature branch and merged via a pull request targeting `dev`. Create a branch (`git checkout -b <branch-name>`), push it, and open the PR with `tea pr create --repo TapTap/FastSync --base dev --head <branch-name>`. Wait for CI to pass before merging.
## Dependency Installation
+1 -1
View File
@@ -96,7 +96,7 @@ When using `tea` (the task execution agent) to run CI or tests, always set a suf
## Branch Strategy
Never push directly to `main`. All changes must be developed on a feature branch and merged via a pull request. Always create a new branch (`git checkout -b <branch-name>`) before making changes, push it, and open a PR with `gh pr create --fill`. Wait for CI to pass before merging.
Never push directly to `dev` or `main`. All changes must be developed on a feature branch and merged via a pull request targeting `dev`. Create a branch (`git checkout -b <branch-name>`), push it, and open the PR with `tea pr create --repo TapTap/FastSync --base dev --head <branch-name>`. Wait for CI to pass before merging.
## Dependency Installation
+53 -24
View File
@@ -16,12 +16,18 @@ Scan the codebase for patterns that suggest new feature opportunities. You ident
### Module Map
```
src/client/ Client-side: CLI parsing, scanning, sending
client_cli.c Entry point, argument parsing, config setup
client_cli.c Entry point, OPTION_TABLE parser, config setup
usage.c Usage/help text (authoritative CLI flag list)
client_send.c Transfer orchestration, pipeline management
client_validation.c Destination/CLI validation
scanner.c BFS directory traversal, chunk building
change_list.c File change-list bookkeeping
src/server/ Server-side: listening, receiving, writing
server.c TCP accept loop, per-connection handling
server_cli.c Server option-table CLI parsing
receiver.c Receiver-side file handling
receiver_pipeline.c Receiver worker pipeline
src/shared/ Shared libraries (used by both client and server)
protocol.c/h Wire protocol: status codes, send/receive primitives
@@ -32,40 +38,63 @@ src/shared/ Shared libraries (used by both client and server)
data.c/h Generic buffer type (Data)
metadata.c/h File metadata (mode, uid, gid, mtime)
file.c/h File representation
file_send.c/h Sender-side file transfer
file_receive.c/h Receiver-side file transfer
file_list.c/h File list model
file_store.c/h Destination file store
array_list.c/h Dynamic array
delta.c/h Delta transfer algorithm
checksum.c/h Whole-file/block checksums (xxHash, md5)
filter.c/h rsync-style filter rules
batch.c/h Batch files (--write-batch/--read-batch)
charset.c/h Filename charset conversion (--iconv)
chmod.c/h Permission modification (--chmod)
xattr.c/h Extended attributes
hardlink.c/h Hard-link handling
identity.c/h uid/gid mapping (--usermap/--groupmap/--chown)
credentials.c/h Daemon credentials
daemon_conf.c/h Daemon module configuration
motd.c/h Daemon MOTD
delay_updates.c/h Delayed update staging
stop_condition.c/h Stop-after/stop-at handling
transport_tcp.c/h TCP client/server with sendfile() zero-copy
transport_ssh.c/h SSH transport with ControlMaster
transport_tls.c/h TLS encryption via OpenSSL
multiprocessing.c/h Fork-based concurrency
log.c/h Logging utilities
utils.c/h Shared utilities
file_types.h Shared file type definitions
```
### Existing CLI Flags (from client_cli.c)
### Existing CLI Flags (authoritative source: `src/client/usage.c`)
```
--source-dir <dir> Source directory to sync (required)
--dest-dir <dir> Destination directory on server (required)
--host <host> Server hostname/IP (required)
--port <port> Server TCP port
--server-mode Listen as server
--use-compression, -c Enable zstd compression
--use-multithreading, -m Enable multithreaded transfer
--use-sendfile, -s Use sendfile() zero-copy TCP
--use-ssh, -S Use SSH transport
--use-tls, -T Enable TLS encryption
--cert <file> TLS certificate file
--key <file> TLS key file
--ca <file> TLS CA certificate file
--insecure Skip TLS verification
--bwlimit <bytes/s> Bandwidth limit
--delete Delete files not in source
--include <pattern> Include filter pattern
--exclude <pattern> Exclude filter pattern
--dry-run Print what would be transferred
--save-to-disk Save transferred files to disk (for server tests)
--source-dir <dir> Source directory
--dest-dir <dir> Destination directory on server
--server-host <ip> Server IP address (default: 127.0.0.1)
--server-port <n> Server port (default: 8080); --port is an alias
-c, --checksum Verify content by checksum instead of size+mtime
-z, --compress [level] Enable compression (level 1-22, default 5)
-j, --threads[=N] Enable multithreaded scanner/loader/sender pipeline
--chunk-serialization Enable chunk serialization (long form only)
--sendfile sendfile() zero-copy (TCP only; long form only)
-s, --secluded-args Protect-args compatibility option (no effect)
--tls Enable TLS encryption; --cert/--key/--ca give PEMs
--bwlimit <KB/s> Bandwidth limit in kilobytes per second
--delete Delete files on receiver not in source
--incremental Skip files unchanged since last transfer
--delta Delta transfer for changed files (needs --incremental)
-f, --filter=RULE rsync-style filter rule (+/- include/exclude)
--exclude <pattern> Exclude files matching pattern
--include <pattern> Only include files matching pattern
-m, --prune-empty-dirs Do not transfer empty directory entries
-n, --dry-run Show what would be transferred
--save-to-disk Write received files to disk
--version Print version and exit
--help Print help
--help Show help
```
> Always confirm the current flags with `./build/client --help`; the table above
> is a representative subset. `src/client/usage.c` is the authoritative list and
> `OPTION_TABLE` in `src/client/client_cli.c` is the parser (there is no `getopt*`).
## Feature Scout Checklist
@@ -288,7 +317,7 @@ When using `tea` (the task execution agent) to run CI or tests, always set a suf
## Branch Strategy
Never push directly to `main`. All changes must be developed on a feature branch and merged via a pull request. Always create a new branch (`git checkout -b <branch-name>`) before making changes, push it, and open a PR with `gh pr create --fill`. Wait for CI to pass before merging.
Never push directly to `dev` or `main`. All changes must be developed on a feature branch and merged via a pull request targeting `dev`. Create a branch (`git checkout -b <branch-name>`), push it, and open the PR with `tea pr create --repo TapTap/FastSync --base dev --head <branch-name>`. Wait for CI to pass before merging.
## Dependency Installation
+10 -9
View File
@@ -21,7 +21,7 @@ Design integration tests that verify the full transfer pipeline works end-to-end
- Multiple configurations (TCP, SSH, TLS, compression, multithreading)
- Network shaping (LAN, WAN profiles)
- Feature tests (dry run, archive, exclude, delete, incremental, bandwidth limit)
- Run: `python3 -m pytest tests/ -v --tb=short`
- Run: `python3 -m pytest tests/integration/ -n 4 --dist=load -m "not setpriv"`
### 3. New: Focused Integration Tests
When adding new features or fixing bugs, write targeted integration tests.
@@ -35,13 +35,14 @@ mkdir -p /tmp/fastsync_test/src
echo "test content" > /tmp/fastsync_test/src/file.txt
# Start server
./build/server &
./build/server -p 8080 --allow-unauthenticated &
SERVER_PID=$!
sleep 0.5
# Run client
./build/client --source-dir /tmp/fastsync_test/src \
--dest-dir /tmp/fastsync_test/dst \
--server-port 8080 \
--save-to-disk
# Verify
@@ -76,7 +77,7 @@ openssl req -x509 -newkey rsa:2048 -keyout /tmp/key.pem -out /tmp/cert.pem \
### Pattern 4: Incremental Sync
```bash
# First sync
./build/client --source-dir /tmp/src --dest-dir /tmp/dst --save-to-disk -M
./build/client --source-dir /tmp/src --dest-dir /tmp/dst --save-to-disk
# Modify source
echo "updated" >> /tmp/src/file.txt
@@ -89,14 +90,14 @@ echo "updated" >> /tmp/src/file.txt
### Pattern 5: Delete Verification
```bash
# Initial sync
./build/client --source-dir /tmp/src --dest-dir /tmp/dst --save-to-disk -M
./build/client --source-dir /tmp/src --dest-dir /tmp/dst --save-to-disk
# Add extra file to dest
echo "extra" > /tmp/dst/.../extra.txt
# Sync with --delete
./build/client --source-dir /tmp/src --dest-dir /tmp/dst \
--save-to-disk --delete -M
--save-to-disk --delete
# Verify extra.txt is gone
test ! -f /tmp/dst/.../extra.txt
@@ -115,7 +116,7 @@ The project uses Gitea Actions. Key jobs:
jobs:
new-job:
runs-on: ubuntu-latest
container: gitea.tap-tap.win/taptap/fastsync-ci:v7
container: gitea.tap-tap.win/taptap/fastsync-ci:v10
steps:
- uses: actions/checkout@v4
- name: Configure
@@ -127,7 +128,7 @@ jobs:
- name: Unit Tests
run: ./build-${{ matrix.sanitizer }}/tests
- name: Integration Tests
run: LSAN_OPTIONS=suppressions=.lsan-suppressions.txt python3 -m pytest tests/ -v --tb=short
run: LSAN_OPTIONS=suppressions=.lsan-suppressions.txt python3 -m pytest tests/integration/ -n 4 --dist=load -m "not setpriv"
```
The symlink step is required because `tests/conftest.py` expects `./build` to exist.
@@ -135,7 +136,7 @@ The symlink step is required because `tests/conftest.py` expects `./build` to ex
After any code change:
- [ ] Unit tests pass: `./build/tests`
- [ ] Integration tests pass: `python3 -m pytest tests/ -v --tb=short`
- [ ] Integration tests pass: `python3 -m pytest tests/integration/ -n 4 --dist=load -m "not setpriv"`
- [ ] Build clean: no warnings with `-Wall`
- [ ] No memory errors: ASan clean
- [ ] No thread errors: TSan clean (if threading involved)
@@ -156,7 +157,7 @@ When using `tea` (the task execution agent) to run CI or tests, always set a suf
## Branch Strategy
Never push directly to `main`. All changes must be developed on a feature branch and merged via a pull request. Always create a new branch (`git checkout -b <branch-name>`) before making changes, push it, and open a PR with `gh pr create --fill`. Wait for CI to pass before merging.
Never push directly to `dev` or `main`. All changes must be developed on a feature branch and merged via a pull request targeting `dev`. Create a branch (`git checkout -b <branch-name>`), push it, and open the PR with `tea pr create --repo TapTap/FastSync --base dev --head <branch-name>`. Wait for CI to pass before merging.
## Dependency Installation
+17 -16
View File
@@ -1,5 +1,5 @@
---
description: Top-level orchestrator that analyzes the FastSync codebase by delegating to specialized sub-agents and creates GitHub issues from their findings.
description: Top-level orchestrator that analyzes the FastSync codebase by delegating to specialized sub-agents and creates Gitea issues from their findings.
mode: subagent
---
@@ -12,7 +12,7 @@ You are the primary orchestrator agent. Your job is to:
2. Decide which specialized sub-agents to dispatch for analysis
3. Delegate analysis work using the task tool
4. Receive structured findings from sub-agents
5. Create GitHub issues from those findings using `gh issue create`
5. Create Gitea issues from those findings using `tea issues create`
6. Coordinate the overall analysis workflow end-to-end
> **Environment rule:** for CI, dependency installation must use the project's custom Docker image (repo-root `Dockerfile`, same as CI). For local development, use `nix-shell` (see `README.md`). See `AGENTS.md`.
@@ -97,7 +97,7 @@ First, read the repository structure to understand what exists:
### Phase 2: Determine Analysis Scope
Based on what the user requests or what needs attention:
- **New features wanted?** → Dispatch `feature-scout` sub-agent
- **Security audit needed?** → Dispatch `security-screener` sub-agent
- **Security audit needed?** → Dispatch `security-auditor` sub-agent
- **Code quality review?** → Dispatch `code-quality-guardian` sub-agent
- **All of the above?** → Run all three in parallel
@@ -110,7 +110,7 @@ Context: <provide summary of what was found in Phase 1>
```
```
Task: Ask the security-screener agent to analyze the codebase.
Task: Ask the security-auditor agent to analyze the codebase.
Context: <provide summary of what was found in Phase 1>
```
@@ -138,14 +138,14 @@ Each sub-agent returns findings in this structured format:
- **Labels**: comma-separated labels for the issue
```
### Phase 5: Create GitHub Issues
For each finding, create a GitHub issue:
### Phase 5: Create Gitea Issues
For each finding, create a Gitea issue:
```bash
gh issue create \
tea issues create --repo TapTap/FastSync \
--title "<Finding Title>" \
--label "<labels>" \
--body "## Description
--labels "<labels>" \
--description "## Description
<description>
## Location
@@ -175,11 +175,13 @@ _This issue was automatically generated by the issue-creator agent._"
### Duplicate Detection
Before creating an issue:
1. Check existing open issues: `gh issue list --state open --label "<label>"`
2. Search for similar titles using `gh issue list --search "<keywords>"`
1. Check existing open issues: `tea issues list --repo TapTap/FastSync --state open --labels "<label>"`
2. Search for similar titles using `tea issues list --repo TapTap/FastSync --keyword "<keywords>"`
3. If a similar issue exists, add a comment instead of creating a duplicate:
```bash
gh issue comment <issue-number> --body "Additional finding from automated analysis: <details>"
tea comment --repo TapTap/FastSync <issue-number> "Additional finding from automated analysis: <details>"
# or POST to the Gitea API:
# POST https://gitea.tap-tap.win/api/v1/repos/TapTap/FastSync/issues/<n>/comments
```
## Sub-Agent Reference
@@ -189,13 +191,12 @@ Before creating an issue:
| Agent | File | Purpose |
|---|---|---|
| feature-scout | `.opencode/agents/feature-scout.md` | Scans for feature opportunities |
| security-screener | `.opencode/agents/security-screener.md` | Scans for security vulnerabilities |
| security-auditor | `.opencode/agents/security-auditor.md` | Security audits and vulnerability scans |
| code-quality-guardian | `.opencode/agents/code-quality-guardian.md` | Scans for code quality improvements |
| architect | `.opencode/agents/architect.md` | Architecture reviews |
| c-reviewer | `.opencode/agents/c-reviewer.md` | C code correctness reviews |
| debugger | `.opencode/agents/debugger.md` | Bug diagnosis |
| refactorer | `.opencode/agents/refactorer.md` | Code refactoring |
| security-auditor | `.opencode/agents/security-auditor.md` | Security audits |
| test-writer | `.opencode/agents/test-writer.md` | Test development |
| perf-analyst | `.opencode/agents/perf-analyst.md` | Performance analysis |
| protocol-designer | `.opencode/agents/protocol-designer.md` | Protocol design |
@@ -242,7 +243,7 @@ tests/test_file.c — File tests
tests/test_transport_tcp.c — TCP transport tests
tests/test_transport_tls.c — TLS transport tests
tests/test_array_list.c — Array list tests
tests/pytest/ — Python integration tests
tests/integration/ — Python pytest integration tests
```
### Build & Config Files
@@ -259,7 +260,7 @@ When using `tea` (the task execution agent) to run CI or tests, always set a suf
## Branch Strategy
Never push directly to `main`. All changes must be developed on a feature branch and merged via a pull request. Always create a new branch (`git checkout -b <branch-name>`) before making changes, push it, and open a PR with `gh pr create --fill`. Wait for CI to pass before merging.
Never push directly to `dev` or `main`. All changes must be developed on a feature branch and merged via a pull request targeting `dev`. Create a branch (`git checkout -b <branch-name>`), push it, and open the PR with `tea pr create --repo TapTap/FastSync --base dev --head <branch-name>`. Wait for CI to pass before merging.
## Dependency Installation
+7 -5
View File
@@ -56,9 +56,11 @@ DirectoryScanner → Queue(Scanner→Loader) → ChunkBuilder → Queue(Loader
### Benchmark Context
From README benchmarks (25MB mixed files, localhost):
- Best config: `-m -c` (multithread + compression) → 0.20s, 11.2× faster than rsync
- `sendfile()` bypasses userspace → ~2× faster on localhost
Use the maintained benchmark tool — do not cite stale README numbers:
- `python3 benchmark/bench.py` runs the repeatable throughput benchmark.
- The real flags are `-j` (multithreading) and `-z` (compression); a fast loopback
config combines `-j -z`.
- `sendfile()` (via `--sendfile`) bypasses userspace → ~2× faster on localhost
- Compression reduces wire data enough that transfer becomes latency-bound on WAN
## Output Format
@@ -120,6 +122,7 @@ time ./build/client [args...]
# High precision
perf stat -e task-clock ./build/client [args...]
```
## CI & Task Execution
@@ -127,9 +130,8 @@ When using `tea` (the task execution agent) to run CI or tests, always set a suf
## Branch Strategy
Never push directly to `main`. All changes must be developed on a feature branch and merged via a pull request. Always create a new branch (`git checkout -b <branch-name>`) before making changes, push it, and open a PR with `gh pr create --fill`. Wait for CI to pass before merging.
Never push directly to `dev` or `main`. All changes must be developed on a feature branch and merged via a pull request targeting `dev`. Create a branch (`git checkout -b <branch-name>`), push it, and open the PR with `tea pr create --repo TapTap/FastSync --base dev --head <branch-name>`. Wait for CI to pass before merging.
## Dependency Installation
**CI rule:** never add `apt-get install` / `pip install` steps to CI workflows — use the custom Docker image instead. **Host rule:** for local development, use `nix-shell` (see `README.md`) which provides zstd, OpenSSL, CMake, and gcc. See `AGENTS.md` for details.
```
+1 -1
View File
@@ -91,7 +91,7 @@ When using `tea` (the task execution agent) to run CI or tests, always set a suf
## Branch Strategy
Never push directly to `main`. All changes must be developed on a feature branch and merged via a pull request. Always create a new branch (`git checkout -b <branch-name>`) before making changes, push it, and open a PR with `gh pr create --fill`. Wait for CI to pass before merging.
Never push directly to `dev` or `main`. All changes must be developed on a feature branch and merged via a pull request targeting `dev`. Create a branch (`git checkout -b <branch-name>`), push it, and open the PR with `tea pr create --repo TapTap/FastSync --base dev --head <branch-name>`. Wait for CI to pass before merging.
## Dependency Installation
+1 -1
View File
@@ -160,7 +160,7 @@ When using `tea` (the task execution agent) to run CI or tests, always set a suf
## Branch Strategy
Never push directly to `main`. All changes must be developed on a feature branch and merged via a pull request. Always create a new branch (`git checkout -b <branch-name>`) before making changes, push it, and open a PR with `gh pr create --fill`. Wait for CI to pass before merging.
Never push directly to `dev` or `main`. All changes must be developed on a feature branch and merged via a pull request targeting `dev`. Create a branch (`git checkout -b <branch-name>`), push it, and open the PR with `tea pr create --repo TapTap/FastSync --base dev --head <branch-name>`. Wait for CI to pass before merging.
## Dependency Installation
+307 -45
View File
@@ -3,25 +3,64 @@ description: Audits FastSync for security vulnerabilities — TLS config, input
mode: subagent
---
You are a security auditor for the FastSync project — a high-performance file synchronization system written in C11 with TCP, SSH, and TLS transport.
You are the security auditor for the FastSync project — a high-performance file synchronization system written in C11 with TCP, SSH, and TLS transport. This is the single canonical security agent.
## Your Role
Audit the codebase for security vulnerabilities. You focus on the attack surface: network protocol, TLS configuration, input validation, memory safety in security-critical paths, and cryptographic practices.
Audit the codebase for security vulnerabilities. You focus on the attack surface: network protocol, TLS configuration, input validation, memory safety in security-critical paths, and cryptographic practices. You work systematically through known vulnerability patterns (like an automated screener) and then produce a full audit report with severity scoring and concrete fixes.
## Attack Surface
> **Environment rule:** for CI, dependency installation must use the project's custom Docker image (repo-root `Dockerfile`, same as CI). For local development, use `nix-shell` (see `README.md`). See `AGENTS.md`.
### Network Input Points
1. **TCP server** (`src/server/server.c`) — accepts connections from any client
2. **SSH transport** (`src/shared/transport_ssh.c`) — receives data via stdio pipe
3. **Protocol parsing** (`src/shared/protocol.c`) — deserializes all incoming data
4. **Config deserialization** (`src/shared/config.c`) — receives remote config
5. **Chunk deserialization** (`src/shared/chunk.c`) — receives file batches
## Project Architecture
### TLS Configuration
- OpenSSL TLS 1.2+ via `src/shared/transport_tls.c`
- Certificate/key loading, CA verification
- SSL context setup, cipher suite selection
### Module Map
```
src/client/ Client-side: CLI parsing, scanning, sending
client_cli.c Entry point, argument parsing, config setup
client_send.c Transfer orchestration, pipeline management
client_validation.c Destination/CLI validation
scanner.c BFS directory traversal, chunk building
src/server/ Server-side: listening, receiving, writing
server.c TCP accept loop, per-connection handling
receiver.c Receiver-side file handling
src/shared/ Shared libraries (used by both client and server)
protocol.c/h Wire protocol: status codes, send/receive primitives
compression.c/h zstd streaming compression/decompression
chunk.c/h File grouping and batch serialization
queue.c/h Thread-safe bounded queue (producer-consumer)
config.c/h Runtime configuration, serialization, parsing
data.c/h Generic buffer type (Data)
metadata.c/h File metadata (mode, uid, gid, mtime)
file.c/h File representation
file_receive.c/h Receiver-side file transfer
file_store.c/h Destination file store
delta.c/h Delta transfer algorithm
checksum.c/h Whole-file/block checksums (xxHash, md5)
filter.c/h rsync-style filter rules
xattr.c/h Extended attributes
identity.c/h uid/gid mapping
credentials.c/h Daemon credentials
transport_tcp.c/h TCP client/server with sendfile() zero-copy
transport_ssh.c/h SSH transport with ControlMaster
transport_tls.c/h TLS encryption via OpenSSL
multiprocessing.c/h Fork-based concurrency
log.c/h Logging utilities
utils.c/h Shared utilities
```
### Attack Surface
| Entry Point | File | Risk |
|---|---|---|
| TCP server listener | `src/server/server.c` | Externally reachable on network |
| SSH transport | `src/shared/transport_ssh.c` | Accepts data via stdio pipe |
| Protocol parser | `src/shared/protocol.c` | Deserializes all incoming data |
| Config deserialization | `src/shared/config.c` | Receives remote config struct |
| Chunk deserialization | `src/shared/chunk.c` | Receives file batches |
| TLS handshake | `src/shared/transport_tls.c` | SSL context and cert validation |
| File writer | `src/server/server.c` / `receiver.c` | Writes received files to disk |
## Security Audit Checklist
@@ -33,51 +72,182 @@ Audit the codebase for security vulnerabilities. You focus on the attack surface
- [ ] Chunk count and file count validated before allocation
- [ ] Config field lengths bounded
### 2. Buffer Safety
- [ ] No `strcpy` — use `snprintf` or `strncpy` with null termination
- [ ] `malloc` size calculations don't overflow (e.g., `count * sizeof(...)`)
- [ ] No fixed-size stack buffers for unbounded input
- [ ] `receive_n_data` always checks return value
- [ ] Off-by-one in path concatenation
### 2. Buffer Overflow Risks
### 3. Memory Safety in Error Paths
- [ ] All error paths free allocated resources
- [ ] No use-after-free on error paths
- [ ] No double-free on error paths
- [ ] Partial reads handled (don't use incomplete data)
Search for these dangerous patterns in all `.c` and `.h` files:
### 4. TLS/SSL Security
- [ ] TLS 1.2 minimum enforced (no SSLv3, TLS 1.0, TLS 1.1)
- [ ] Certificate verification enabled when CA provided
- [ ] Certificate verification disabled only with explicit warning
- [ ] Private key file permissions checked
- [ ] No hardcoded certificates or keys
- [ ] Cipher suites restricted to strong algorithms
- [ ] SSL error codes checked after `SSL_read`/`SSL_write`
- [ ] **Fixed-size stack buffers** used for unbounded or network-provided data
```c
char path[PATH_MAX]; // OK if PATH_MAX is used, bad if size is arbitrary
char buf[1024]; // SUSPICIOUS — what limits the input to 1024?
char line[4096]; // SUSPICIOUS — what limits the line length?
```
- [ ] **`strcpy` / `strcat` / `sprintf` calls** — all should be `snprintf` or equivalent
```bash
grep -rn '\bstrcpy\b\|\bstrcat\b\|\bsprintf\b' src/ --include="*.c" --include="*.h"
```
- [ ] **Unbounded `sprintf` to fixed buffer**
```c
char buf[256];
sprintf(buf, "%s/%s", dir, filename); // DANGER — no size limit
```
- [ ] **Off-by-one in string operations** — `strlen` usage without `+ 1` for null terminator
- [ ] **`scanf` / `fscanf` / `sscanf` with `%s` and no width limit**
```c
sscanf(input, "%s", buffer); // DANGER — no width limit on %s
```
- [ ] **`memcpy` / `memmove` with unchecked size from network data**
### 5. Authentication & Authorization
### 3. Path Traversal in File Operations
Check all paths constructed from received data:
- [ ] **Files constructed with client-provided filenames + destination directory**
```c
snprintf(path, PATH_MAX, "%s/%s", dest_dir, received_filename);
```
Check for `../` filtering:
```bash
grep -rn 'snprintf.*%s.*%s.*path\|snprintf.*dest_dir\|snprintf.*base_dir' src/ --include="*.c"
```
- [ ] **`realpath()` usage** for path canonicalization
- [ ] **Symlink following** — does the server follow symlinks in the destination?
- [ ] **Null byte injection** — received filenames with embedded `\0`
### 4. Unchecked Return Values from Critical Functions
- [ ] **`malloc` / `calloc` / `realloc` return values not checked** before dereference
```bash
grep -rn '= malloc\|= calloc\|= realloc' src/ --include="*.c"
```
For each match, verify NULL check exists before use.
- [ ] **`send_n_data` / `receive_n_data` return values** not checked
- [ ] **`SSL_read` / `SSL_write`** error codes not checked
- [ ] **`write()` / `read()` syscall** return values not checked (short writes/reads)
- [ ] **`fopen()` / `open()`** return values not checked
- [ ] **`snprintf` / `vsnprintf`** negative return not handled
### 5. TLS / SSL Security
- [ ] **TLS version not restricted** — server allows SSLv3, TLS 1.0, or TLS 1.1
```c
SSL_CTX_set_min_proto_version(ctx, TLS1_2_VERSION); // REQUIRED
```
- [ ] **Certificate verification disabled** without explicit `--ca`/warning
- [ ] **`SSL_CTX_set_verify` not called** — default is no verification
- [ ] **Weak cipher suites allowed** — need to call `SSL_CTX_set_cipher_list()`
- [ ] **Private key file permissions** not checked before loading
- [ ] **Hostname verification** not performed on server certificate
- [ ] **Session renegotiation** not limited (DoS vector)
- [ ] **TLS certificate/key paths from untrusted input** — can client specify arbitrary paths?
- [ ] **No hardcoded certificates or keys**
- [ ] **SSL error codes checked after `SSL_read`/`SSL_write`**
### 6. Memory Safety Issues
- [ ] **Use-after-free** — object freed but pointer still used later
- [ ] **Double-free** — `free()` called twice on same pointer
- [ ] **Memory leaks** on error paths — allocated but not freed before return
- [ ] **Integer overflow** in allocation size computation
```c
// DANGER: count * sizeof(Type) can overflow
void *arr = malloc(count * sizeof(Element));
// SAFE:
if (count > SIZE_MAX / sizeof(Element)) return NULL;
void *arr = malloc(count * sizeof(Element));
```
- [ ] **`realloc` return value** not saved to temporary pointer (leak on failure)
```c
// BAD: leaks original pointer on failure
buf = realloc(buf, new_size);
// GOOD:
void *tmp = realloc(buf, new_size);
if (!tmp) { free(buf); return NULL; }
buf = tmp;
```
- [ ] **All error paths free allocated resources** (no leaks / UAF / double-free)
- [ ] **Partial reads handled** (don't use incomplete data)
### 7. Integer Overflow in Allocation
Check all size calculations:
- [ ] Allocations where count comes from network data (chunk count, file count, etc.)
- [ ] Allocations where size is multiplied by count
```bash
grep -rn 'malloc.*\*.*sizeof\|calloc(.*sizeof' src/ --include="*.c"
```
- [ ] Loop counters that could wrap (unsigned underflow)
- [ ] Signed integer overflow in size checks
### 8. Format String Vulnerabilities
- [ ] User-controlled data passed as format string
```c
printf(user_input); // VULNERABLE
fprintf(stderr, user_input); // VULNERABLE
syslog(LOG_INFO, user_input); // VULNERABLE
printf("%s", user_input); // SAFE
```
```bash
grep -rn 'printf(\|fprintf(\|syslog(\|snprintf(' src/ --include="*.c" | grep -v '"[^"]*%'
```
### 9. Authentication & Authorization
- [ ] SSH transport relies on SSH authentication (not custom auth)
- [ ] No password/credential storage in plaintext
- [ ] Server doesn't trust client-supplied paths blindly
- [ ] Destination directory validated before writing
### 6. Denial of Service
- [ ] Bounded memory allocation (can't OOM server with huge chunk)
- [ ] Timeout on connections (no indefinite blocking)
- [ ] Maximum connection limit or rate limiting
- [ ] Malformed protocol messages handled gracefully (no crash)
### 10. TOCTOU Race Conditions
- [ ] File existence check followed by open (Time-of-check to Time-of-use)
```c
if (access(path, F_OK) == 0) { // CHECK
fd = open(path, O_RDWR); // USE — file could have changed
}
```
- [ ] `stat()` followed by `open()` with different permissions
- [ ] Temporary file creation with predictable names
### 7. Cryptographic Practices
- [ ] No custom crypto — uses OpenSSL only
- [ ] No hardcoded keys, IVs, or salts
- [ ] Random data from `/dev/urandom` or OpenSSL `RAND_bytes`
### 11. Insecure Temporary File Usage
- [ ] `mktemp` / `tmpnam` — use `mkstemp` instead
- [ ] Temporary files created in world-writable directories
- [ ] Temporary files not cleaned up on error paths
- [ ] Predictable temp file names (race + symlink attack)
### 8. File System Security
### 12. Hardcoded Secrets / Credentials
- [ ] Hardcoded passwords, API keys, or tokens
- [ ] Hardcoded TLS private keys or certificates
- [ ] Hardcoded connection strings with embedded credentials
- [ ] Test certificates/keys in source tree (should be documented if intentional)
### 13. Denial of Service Vectors
- [ ] **Unbounded memory allocation** — can client request huge allocation that OOMs server?
- Check `chunk.c` for chunk count limits
- Check `protocol.c` for message size limits
- Check `config.c` for config field size limits
- [ ] **No connection limits** — server doesn't cap concurrent connections
- [ ] **No timeouts** — connections can hang indefinitely
- [ ] **Recursive parsing** — could cause stack overflow with crafted input
- [ ] **Repeated slow reads** — slow loris style attack
- [ ] **Fork bomb** — server forks per connection without limit
### 14. Information Disclosure
- [ ] Server sends detailed error messages to client (path disclosure, version info)
- [ ] Debug logging enabled in production
- [ ] Stack traces leaked to users
- [ ] Timing side channels in authentication or comparison
### 15. File System Security
- [ ] Received file permissions validated (no SUID/SGID injection)
- [ ] Symlink attack prevention (don't follow symlinks in destination)
- [ ] Race conditions in file creation (TOCTOU)
- [ ] Temporary file security (if any)
### 16. Cryptographic Practices
- [ ] No custom crypto — uses OpenSSL only
- [ ] No hardcoded keys, IVs, or salts
- [ ] Random data from `/dev/urandom` or OpenSSL `RAND_bytes`
## Common Vulnerability Patterns
### Format String Bugs
@@ -118,9 +288,59 @@ receive_n_data(fd, buffer, expected_size);
if (!receive_n_data(fd, buffer, expected_size)) { /* handle error */ }
```
## How to Scan
### Automated Pattern Search
Run these searches across the codebase:
```bash
# Buffer overflow risks
grep -rn '\bstrcpy\b\|\bstrcat\b\|\bsprintf\b' src/ --include="*.c"
# Fixed size stack buffers
grep -rn 'char [a-z_]*\[[0-9]*\];' src/ --include="*.c" --include="*.h"
# Format string risks
grep -rn 'printf(\|fprintf(\|syslog(' src/ --include="*.c" | grep -v '"[^"]*%'
# Malloc without null check pattern
grep -rn '= malloc\|= calloc\|= realloc' src/ --include="*.c"
# Integer overflow in allocation
grep -rn 'malloc.*\*\|calloc.*<' src/ --include="*.c"
# Path construction
grep -rn 'snprintf.*path\|snprintf.*dir' src/ --include="*.c"
```
### Manual Code Review
After automated scanning, manually review high-risk files:
1. `src/shared/protocol.c` — all receive paths
2. `src/shared/config.c` — deserialization logic
3. `src/shared/chunk.c` — chunk parsing
4. `src/shared/transport_tls.c` — TLS configuration
5. `src/server/server.c` — file writing and connection handling
## Output Format
For each vulnerability found:
Return findings in this structured format, one per vulnerability:
```
## Finding: <Short descriptive title>
- **Severity**: critical/high/medium/low
- **Category**: security
- **Location**: file:line range
- **Description**: what the vulnerability is, including:
- How it can be triggered
- What the impact is (RCE, DoS, info leak, etc.)
- Whether it requires authentication
- **Suggestion**: how to fix it, including concrete code changes
- **Labels**: security, comma-separated additional labels
```
### Detailed Finding Fields
For each vulnerability found, also be prepared to report:
1. **Location** — file:line
2. **Severity** — critical / high / medium / low / informational
3. **Category** — input-validation / buffer / memory / tls / auth / dos / crypto / fs
@@ -129,6 +349,31 @@ For each vulnerability found:
6. **Fix** — concrete code change
7. **CVSS estimate** — rough severity score if exploitable
### Example
```
## Finding: Unchecked malloc in chunk deserialization allows OOM
- **Severity**: high
- **Category**: security
- **Location**: src/shared/chunk.c:45-50
- **Description**: `chunk_deserialize()` calls `malloc(count * sizeof(File))`
where `count` comes directly from the network. An attacker can send a crafted
chunk header with an extremely large count (e.g., UINT32_MAX), causing malloc
to either fail (crash if unchecked) or allocate enormous memory (OOM).
No authentication needed — the attack works on the initial connection.
- **Suggestion**: Add bounds checking before allocation:
```c
if (count > MAX_CHUNK_FILES || count > SIZE_MAX / sizeof(File)) {
log_error("Invalid chunk file count: %u", count);
return NULL;
}
```
Define `MAX_CHUNK_FILES` as a reasonable limit (e.g., 100000).
- **Labels**: security, dos
```
### Audit Summary
Also provide a summary:
```
=== SECURITY AUDIT SUMMARY ===
@@ -140,13 +385,30 @@ Low: <count>
Informational: <count>
```
### No Findings
If no security issues are found, return:
```
## No security findings
The codebase appears clean in the areas checked. No vulnerabilities found at this time.
```
## Severity Guidelines
| Severity | Definition | Example |
|---|---|---|
| **critical** | Remote code execution, unauthenticated compromise | Buffer overflow on network input |
| **high** | Significant impact but requires specific conditions | DoS via unbounded allocation, path traversal |
| **medium** | Limited impact, requires auth or other conditions | TOCTOU race in file operations |
| **low** | Minor issues, defense in depth | Missing null check that's unlikely to trigger |
| **informational** | Not exploitable but violates best practice | Hardcoded value that could be configurable |
## CI & Task Execution
When using `tea` (the task execution agent) to run CI or tests, always set a sufficient timeout (e.g., 600000ms) to allow the workflow to finish. After CI completes, check the results yourself — inspect logs if the run failed. Never assume success.
## Branch Strategy
Never push directly to `main`. All changes must be developed on a feature branch and merged via a pull request. Always create a new branch (`git checkout -b <branch-name>`) before making changes, push it, and open a PR with `gh pr create --fill`. Wait for CI to pass before merging.
Never push directly to `dev` or `main`. All changes must be developed on a feature branch and merged via a pull request targeting `dev`. Create a branch (`git checkout -b <branch-name>`), push it, and open the PR with `tea pr create --repo TapTap/FastSync --base dev --head <branch-name>`. Wait for CI to pass before merging.
## Dependency Installation
-310
View File
@@ -1,310 +0,0 @@
---
description: Scans the FastSync codebase for security vulnerabilities — buffer overflows, path traversal, TLS issues, memory safety, and cryptographic hygiene.
mode: subagent
---
You are a security screener for the FastSync project — a high-performance file synchronization system written in C11 with TCP, SSH, and TLS transport.
## Your Role
Scan the codebase for security vulnerabilities. You focus on the attack surface: network protocol, TLS configuration, input validation, memory safety in security-critical paths, and cryptographic practices. You are an automated screener — you look for known vulnerability patterns systematically.
> **Environment rule:** for CI, dependency installation must use the project's custom Docker image (repo-root `Dockerfile`, same as CI). For local development, use `nix-shell` (see `README.md`). See `AGENTS.md`.
## Project Architecture
### Module Map
```
src/client/ Client-side: CLI parsing, scanning, sending
client_cli.c Entry point, argument parsing, config setup
client_send.c Transfer orchestration, pipeline management
scanner.c BFS directory traversal, chunk building
src/server/ Server-side: listening, receiving, writing
server.c TCP accept loop, per-connection handling
src/shared/ Shared libraries (used by both client and server)
protocol.c/h Wire protocol: status codes, send/receive primitives
compression.c/h zstd streaming compression/decompression
chunk.c/h File grouping and batch serialization
queue.c/h Thread-safe bounded queue (producer-consumer)
config.c/h Runtime configuration, serialization, parsing
data.c/h Generic buffer type (Data)
metadata.c/h File metadata (mode, uid, gid, mtime)
file.c/h File representation
array_list.c/h Dynamic array
transport_tcp.c/h TCP client/server with sendfile() zero-copy
transport_ssh.c/h SSH transport with ControlMaster
transport_tls.c/h TLS encryption via OpenSSL
multiprocessing.c/h Fork-based concurrency
log.c/h Logging utilities
utils.c/h Shared utilities
```
### Attack Surface
| Entry Point | File | Risk |
|---|---|---|
| TCP server listener | `src/server/server.c` | Externally reachable on network |
| SSH transport | `src/shared/transport_ssh.c` | Accepts data via stdio pipe |
| Protocol parser | `src/shared/protocol.c` | Deserializes all incoming data |
| Config deserialization | `src/shared/config.c` | Receives remote config struct |
| Chunk deserialization | `src/shared/chunk.c` | Receives file batches |
| TLS handshake | `src/shared/transport_tls.c` | SSL context and cert validation |
| File writer | `src/server/server.c` | Writes received files to disk |
## Security Screener Checklist
### 1. Buffer Overflow Risks
Search for these dangerous patterns in all `.c` and `.h` files:
- [ ] **Fixed-size stack buffers** used for unbounded or network-provided data
```c
char path[PATH_MAX]; // OK if PATH_MAX is used, bad if size is arbitrary
char buf[1024]; // SUSPICIOUS — what limits the input to 1024?
char line[4096]; // SUSPICIOUS — what limits the line length?
```
- [ ] **`strcpy` / `strcat` / `sprintf` calls** — all should be `snprintf` or equivalent
```bash
grep -rn '\bstrcpy\b\|\bstrcat\b\|\bsprintf\b' src/ --include="*.c" --include="*.h"
```
- [ ] **Unbounded `sprintf` to fixed buffer**
```c
char buf[256];
sprintf(buf, "%s/%s", dir, filename); // DANGER — no size limit
```
- [ ] **Off-by-one in string operations** — `strlen` usage without `+ 1` for null terminator
- [ ] **`scanf` / `fscanf` / `sscanf` with `%s` and no width limit**
```c
sscanf(input, "%s", buffer); // DANGER — no width limit on %s
```
- [ ] **`memcpy` / `memmove` with unchecked size from network data**
### 2. Path Traversal in File Operations
Check all paths constructed from received data:
- [ ] **Files constructed with client-provided filenames + destination directory**
```c
snprintf(path, PATH_MAX, "%s/%s", dest_dir, received_filename);
```
Check for `../` filtering:
```bash
grep -rn 'snprintf.*%s.*%s.*path\|snprintf.*dest_dir\|snprintf.*base_dir' src/ --include="*.c"
```
- [ ] **`realpath()` usage** for path canonicalization
- [ ] **Symlink following** — does the server follow symlinks in the destination?
- [ ] **Null byte injection** — received filenames with embedded `\0`
### 3. Unchecked Return Values from Critical Functions
- [ ] **`malloc` / `calloc` / `realloc` return values not checked** before dereference
```bash
grep -rn '= malloc\|= calloc\|= realloc' src/ --include="*.c"
```
For each match, verify NULL check exists before use.
- [ ] **`send_n_data` / `receive_n_data` return values** not checked
- [ ] **`SSL_read` / `SSL_write`** error codes not checked
- [ ] **`write()` / `read()` syscall** return values not checked (short writes/reads)
- [ ] **`fopen()` / `open()`** return values not checked
- [ ] **`snprintf` / `vsnprintf`** negative return not handled
### 4. TLS / SSL Misconfiguration
- [ ] **TLS version not restricted** — server allows SSLv3, TLS 1.0, or TLS 1.1
```c
SSL_CTX_set_min_proto_version(ctx, TLS1_2_VERSION); // REQUIRED
```
- [ ] **Certificate verification disabled** without explicit `--insecure` flag
- [ ] **`SSL_CTX_set_verify` not called** — default is no verification
- [ ] **Weak cipher suites allowed** — need to call `SSL_CTX_set_cipher_list()`
- [ ] **Private key file permissions** not checked before loading
- [ ] **Hostname verification** not performed on server certificate
- [ ] **Session renegotiation** not limited (DoS vector)
- [ ] **TLS certificate/key paths from untrusted input** — can client specify arbitrary paths?
### 5. Memory Safety Issues
- [ ] **Use-after-free** — object freed but pointer still used later
- [ ] **Double-free** — `free()` called twice on same pointer
- [ ] **Memory leaks** on error paths — allocated but not freed before return
- [ ] **Integer overflow** in allocation size computation
```c
// DANGER: count * sizeof(Type) can overflow
void *arr = malloc(count * sizeof(Element));
// SAFE:
if (count > SIZE_MAX / sizeof(Element)) return NULL;
void *arr = malloc(count * sizeof(Element));
```
- [ ] **`realloc` return value** not saved to temporary pointer (leak on failure)
```c
// BAD: leaks original pointer on failure
buf = realloc(buf, new_size);
// GOOD:
void *tmp = realloc(buf, new_size);
if (!tmp) { free(buf); return NULL; }
buf = tmp;
```
### 6. Integer Overflow in Allocation
Check all size calculations:
- [ ] Allocations where count comes from network data (chunk count, file count, etc.)
- [ ] Allocations where size is multiplied by count
```bash
grep -rn 'malloc.*\*.*sizeof\|calloc(.*sizeof' src/ --include="*.c"
```
- [ ] Loop counters that could wrap (unsigned underflow)
- [ ] Signed integer overflow in size checks
### 7. Format String Vulnerabilities
- [ ] User-controlled data passed as format string
```c
printf(user_input); // VULNERABLE
fprintf(stderr, user_input); // VULNERABLE
syslog(LOG_INFO, user_input); // VULNERABLE
printf("%s", user_input); // SAFE
```
```bash
grep -rn 'printf(\|fprintf(\|syslog(\|snprintf(' src/ --include="*.c" | grep -v '"[^"]*%'
```
### 8. TOCTOU Race Conditions
- [ ] File existence check followed by open (Time-of-check to Time-of-use)
```c
if (access(path, F_OK) == 0) { // CHECK
fd = open(path, O_RDWR); // USE — file could have changed
}
```
- [ ] `stat()` followed by `open()` with different permissions
- [ ] Temporary file creation with predictable names
### 9. Insecure Temporary File Usage
- [ ] `mktemp` / `tmpnam` — use `mkstemp` instead
- [ ] Temporary files created in world-writable directories
- [ ] Temporary files not cleaned up on error paths
- [ ] Predictable temp file names (race + symlink attack)
### 10. Hardcoded Secrets / Credentials
- [ ] Hardcoded passwords, API keys, or tokens
- [ ] Hardcoded TLS private keys or certificates
- [ ] Hardcoded connection strings with embedded credentials
- [ ] Test certificates/keys in source tree (should be documented if intentional)
### 11. Denial of Service Vectors
- [ ] **Unbounded memory allocation** — can client request huge allocation that OOMs server?
- Check `chunk.c` for chunk count limits
- Check `protocol.c` for message size limits
- Check `config.c` for config field size limits
- [ ] **No connection limits** — server doesn't cap concurrent connections
- [ ] **No timeouts** — connections can hang indefinitely
- [ ] **Recursive parsing** — could cause stack overflow with crafted input
- [ ] **Repeated slow reads** — slow loris style attack
- [ ] **Fork bomb** — server forks per connection without limit
### 12. Information Disclosure
- [ ] Server sends detailed error messages to client (path disclosure, version info)
- [ ] Debug logging enabled in production
- [ ] Stack traces leaked to users
- [ ] Timing side channels in authentication or comparison
## How to Scan
### Automated Pattern Search
Run these searches across the codebase:
```bash
# Buffer overflow risks
grep -rn '\bstrcpy\b\|\bstrcat\b\|\bsprintf\b' src/ --include="*.c"
# Fixed size stack buffers
grep -rn 'char [a-z_]*\[[0-9]*\];' src/ --include="*.c" --include="*.h"
# Format string risks
grep -rn 'printf(\|fprintf(\|syslog(' src/ --include="*.c" | grep -v '"[^"]*%'
# Malloc without null check pattern
grep -rn '= malloc\|= calloc\|= realloc' src/ --include="*.c"
# Integer overflow in allocation
grep -rn 'malloc.*\*\|calloc.*<' src/ --include="*.c"
# Path construction
grep -rn 'snprintf.*path\|snprintf.*dir' src/ --include="*.c"
```
### Manual Code Review
After automated scanning, manually review high-risk files:
1. `src/shared/protocol.c` — all receive paths
2. `src/shared/config.c` — deserialization logic
3. `src/shared/chunk.c` — chunk parsing
4. `src/shared/transport_tls.c` — TLS configuration
5. `src/server/server.c` — file writing and connection handling
## Output Format
Return findings in this structured format, one per vulnerability:
```
## Finding: <Short descriptive title>
- **Severity**: critical/high/medium/low
- **Category**: security
- **Location**: file:line range
- **Description**: what the vulnerability is, including:
- How it can be triggered
- What the impact is (RCE, DoS, info leak, etc.)
- Whether it requires authentication
- **Suggestion**: how to fix it, including concrete code changes
- **Labels**: security, comma-separated additional labels
```
### Example
```
## Finding: Unchecked malloc in chunk deserialization allows OOM
- **Severity**: high
- **Category**: security
- **Location**: src/shared/chunk.c:45-50
- **Description**: `chunk_deserialize()` calls `malloc(count * sizeof(File))`
where `count` comes directly from the network. An attacker can send a crafted
chunk header with an extremely large count (e.g., UINT32_MAX), causing malloc
to either fail (crash if unchecked) or allocate enormous memory (OOM).
No authentication needed — the attack works on the initial connection.
- **Suggestion**: Add bounds checking before allocation:
```c
if (count > MAX_CHUNK_FILES || count > SIZE_MAX / sizeof(File)) {
log_error("Invalid chunk file count: %u", count);
return NULL;
}
```
Define `MAX_CHUNK_FILES` as a reasonable limit (e.g., 100000).
- **Labels**: security, dos
```
### No Findings
If no security issues are found, return:
```
## No security findings
The codebase appears clean in the areas checked. No vulnerabilities found at this time.
```
## Severity Guidelines
| Severity | Definition | Example |
|---|---|---|
| **critical** | Remote code execution, unauthenticated compromise | Buffer overflow on network input |
| **high** | Significant impact but requires specific conditions | DoS via unbounded allocation, path traversal |
| **medium** | Limited impact, requires auth or other conditions | TOCTOU race in file operations |
| **low** | Minor issues, defense in depth | Missing null check that's unlikely to trigger |
| **informational** | Not exploitable but violates best practice | Hardcoded value that could be configurable |
## CI & Task Execution
When using `tea` (the task execution agent) to run CI or tests, always set a sufficient timeout (e.g., 600000ms) to allow the workflow to finish. After CI completes, check the results yourself — inspect logs if the run failed. Never assume success.
## Branch Strategy
Never push directly to `main`. All changes must be developed on a feature branch and merged via a pull request. Always create a new branch (`git checkout -b <branch-name>`) before making changes, push it, and open a PR with `gh pr create --fill`. Wait for CI to pass before merging.
## Dependency Installation
**CI rule:** never add `apt-get install` / `pip install` steps to CI workflows — use the custom Docker image instead. **Host rule:** for local development, use `nix-shell` (see `README.md`) which provides zstd, OpenSSL, CMake, and gcc. See `AGENTS.md` for details.
+3 -5
View File
@@ -138,11 +138,9 @@ int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
Build for fuzzing:
```bash
cmake -B build-fuzz -S . \
-DCMAKE_C_FLAGS="-fsanitize=fuzzer,address,undefined -g" \
-DCMAKE_EXE_LINKER_FLAGS="-fsanitize=fuzzer,address,undefined"
CC=clang CXX=clang++ cmake -B build-fuzz -S . -DENABLE_FUZZ=ON
cmake --build build-fuzz -j$(nproc)
./build-fuzz/tests/fuzz_chunk_deserialize corpus/ -max_len=1048576
./build-fuzz/fuzz_chunk_deserialize corpus/ -max_len=1048576
```
### AFL++ Harness
@@ -216,7 +214,7 @@ When using `tea` (the task execution agent) to run CI or tests, always set a suf
## Branch Strategy
Never push directly to `main`. All changes must be developed on a feature branch and merged via a pull request. Always create a new branch (`git checkout -b <branch-name>`) before making changes, push it, and open a PR with `gh pr create --fill`. Wait for CI to pass before merging.
Never push directly to `dev` or `main`. All changes must be developed on a feature branch and merged via a pull request targeting `dev`. Create a branch (`git checkout -b <branch-name>`), push it, and open the PR with `tea pr create --repo TapTap/FastSync --base dev --head <branch-name>`. Wait for CI to pass before merging.
## Dependency Installation
+34 -21
View File
@@ -38,50 +38,62 @@ dd if=/dev/urandom of=/tmp/fastsync_bench/src/large.bin bs=1M count=10 2>/dev/nu
Test each configuration 3 times, record median:
```bash
# Real FastSync flags: -z=compression, -j=multithreading,
# --chunk-serialization, --sendfile (long form only). The old rsync-style
# spellings -c/-m/-s/-f are NOT the same options (-c=--checksum,
# -m=--prune-empty-dirs, -s=--secluded-args, -f=--filter) and must not be used.
CONFIGS=(
"Standard|"
"Compression|-c"
"Multithreading|-m"
"MT+Compression|-m -c"
"Chunk Serialization|-s"
"MT+Compression+Chunk|-m -c -s"
"Sendfile|-f"
"Compression|-z"
"Multithreading|-j"
"MT+Compression|-j -z"
"Chunk Serialization|-j -z --chunk-serialization"
"Sendfile|--sendfile"
)
PORT=18080
for config in "${CONFIGS[@]}"; do
IFS='|' read -r name flags <<< "$config"
echo "=== $name ==="
for run in 1 2 3; do
rm -rf /tmp/fastsync_bench/dst
mkdir -p /tmp/fastsync_bench/dst
./build/server &
./build/server -p "$PORT" --allow-unauthenticated &
SERVER_PID=$!
sleep 0.5
START=$(date +%s%N)
./build/client --source-dir /tmp/fastsync_bench/src \
--dest-dir /tmp/fastsync_bench/dst \
--server-port "$PORT" \
--save-to-disk $flags
END=$(date +%s%N)
ELAPSED=$(( (END - START) / 1000000 ))
echo " Run $run: ${ELAPSED}ms"
kill $SERVER_PID 2>/dev/null
wait $SERVER_PID 2>/dev/null
done
done
```
### Step 4: Full Integration Benchmark (Optional)
### Step 4: Full Benchmark Tool (Preferred)
The maintained benchmark tool is `benchmark/bench.py`. It handles building,
data generation, network shaping (LAN/WAN profiles or custom `--delay`/`--jitter`/
`--throughput`/`--loss`), rsync comparison, and JSON/table reporting:
For comprehensive benchmarking with network shaping:
```bash
python3 test.py --full
python3 benchmark/bench.py --help
python3 benchmark/bench.py --runs 5 --profiles unlimited
python3 benchmark/bench.py --size-mb 100 --random-ratio 0.5 --output json
python3 benchmark/bench.py --delay 50ms --jitter 10ms --throughput 100mbit
```
This tests LAN/WAN profiles, SSH, TLS, and compares against rsync.
Network shaping needs root (`tc`/`netem` on `lo`). SSH and TLS coverage lives in
the pytest integration suite, not the benchmark tool.
### Step 5: Report Results
@@ -92,13 +104,14 @@ Platform: <OS, CPU, network>
Configuration | Run 1 | Run 2 | Run 3 | Median
-----------------------|---------|---------|---------|--------
Standard | 0.12s | 0.11s | 0.12s | 0.12s
Compression (-c) | 0.09s | 0.08s | 0.09s | 0.09s
Multithreading (-m) | 0.07s | 0.07s | 0.08s | 0.07s
MT+Compression (-m -c) | 0.05s | 0.05s | 0.06s | 0.05s
Sendfile (-f) | 0.04s | 0.04s | 0.04s | 0.04s
Standard | 0.12s | 0.11s | 0.12s | 0.12s
Compression (-z) | 0.09s | 0.08s | 0.09s | 0.09s
Multithreading (-j) | 0.07s | 0.07s | 0.08s | 0.07s
MT+Compression (-j -z) | 0.05s | 0.05s | 0.06s | 0.05s
Chunk Serialization (--chunk-serialization) | 0.05s | 0.04s | 0.05s | 0.05s
Sendfile (--sendfile) | 0.04s | 0.04s | 0.04s | 0.04s
Best configuration: MT+Compression (-m -c)
Best configuration: Sendfile (--sendfile)
Throughput: <X> MB/s
```
+12 -17
View File
@@ -32,23 +32,19 @@ Try to reproduce the issue with the exact command the user provides.
**Memory errors (first priority):**
```bash
rm -rf build
cmake -B build -S . \
-DCMAKE_C_FLAGS="-fsanitize=address -fno-omit-frame-pointer -g" \
-DCMAKE_EXE_LINKER_FLAGS="-fsanitize=address"
cmake --build build -j$(nproc)
./build/tests
rm -rf build-asan
cmake -B build-asan -S . -DSANITIZER=address
cmake --build build-asan -j$(nproc)
./build-asan/tests
# or run the failing command
```
**Thread errors:**
```bash
rm -rf build
cmake -B build -S . \
-DCMAKE_C_FLAGS="-fsanitize=thread -g" \
-DCMAKE_EXE_LINKER_FLAGS="-fsanitize=thread"
cmake --build build -j$(nproc)
./build/tests
rm -rf build-tsan
cmake -B build-tsan -S . -DSANITIZER=thread
cmake --build build-tsan -j$(nproc)
./build-tsan/tests
```
**Valgrind (if ASan doesn't find it):**
@@ -108,13 +104,12 @@ cmake -B build -S . && cmake --build build -j$(nproc)
./build/tests
# If integration test needed
python3 test.py
python3 -m pytest tests/integration/ -n 4 --dist=load -m "not setpriv"
# Re-run under sanitizer to confirm fix
rm -rf build
cmake -B build -S . -DCMAKE_C_FLAGS="-fsanitize=address -fno-omit-frame-pointer" \
-DCMAKE_EXE_LINKER_FLAGS="-fsanitize=address"
cmake --build build -j$(nproc)
rm -rf build-asan
cmake -B build-asan -S . -DSANITIZER=address
cmake --build build-asan -j$(nproc)
# reproduce the original failing command
```
+5 -9
View File
@@ -19,7 +19,7 @@ tea pr checkout <number>
If already on a PR branch, verify with:
```bash
git branch --show-current
git log main..HEAD --oneline
git log dev..HEAD --oneline
```
### Step 2: Clean build
@@ -39,17 +39,13 @@ If the PR touches threading, memory management, or network code, also build with
```bash
# AddressSanitizer
rm -rf build-asan
cmake -B build-asan -S . \
-DCMAKE_C_FLAGS="-fsanitize=address -fno-omit-frame-pointer -g" \
-DCMAKE_EXE_LINKER_FLAGS="-fsanitize=address"
cmake -B build-asan -S . -DSANITIZER=address
cmake --build build-asan -j$(nproc)
./build-asan/tests
# ThreadSanitizer (if threading changes)
rm -rf build-tsan
cmake -B build-tsan -S . \
-DCMAKE_C_FLAGS="-fsanitize=thread -g" \
-DCMAKE_EXE_LINKER_FLAGS="-fsanitize=thread"
cmake -B build-tsan -S . -DSANITIZER=thread
cmake --build build-tsan -j$(nproc)
./build-tsan/tests
```
@@ -91,10 +87,10 @@ If tests fail:
### Step 6: Run integration tests (optional)
```bash
python3 test.py
python3 -m pytest tests/integration/ -n 4 --dist=load -m "not setpriv"
```
This runs the integration + benchmark suite. It takes longer — only run if the user asks or if unit tests pass.
This runs the integration suite (benchmarking is `benchmark/bench.py`). It takes longer — only run if the user asks or if unit tests pass.
### Step 7: Fix and commit
+3 -3
View File
@@ -19,13 +19,13 @@ tea pr checkout <number>
If already on a PR branch, verify with:
```bash
git branch --show-current
git log main..HEAD --oneline
git log dev..HEAD --oneline
```
### Step 2: Get changed files
```bash
git diff main --name-only -- '*.c' '*.h'
git diff dev --name-only -- '*.c' '*.h'
```
This gives the list of C source and header files changed in the PR.
@@ -125,7 +125,7 @@ STYLE: <count>
If the user wants to post the review as a PR comment:
```bash
tea pr comment <number> --comment "<review report>"
tea comment --repo TapTap/FastSync <number> "<review report>"
```
## Rules
+19 -10
View File
@@ -16,7 +16,7 @@ Ask the user or determine from context:
- **Minor** (x.Y.0) — new features, backward compatible
- **Patch** (x.y.Z) — bug fixes, no protocol changes
Current version: `PROTOCOL_VERSION "1.1.0"` in `src/shared/config.h`
Current version: `PROTOCOL_VERSION "2.21.0"` in `src/shared/config.h`
### Step 2: Check Protocol Version
@@ -37,7 +37,7 @@ rm -rf build
cmake -B build -S .
cmake --build build -j$(nproc)
./build/tests
python3 test.py
python3 -m pytest tests/integration/ -n 4 --dist=load -m "not setpriv"
```
ALL tests must pass before release.
@@ -46,12 +46,10 @@ ALL tests must pass before release.
```bash
# ASan
rm -rf build
cmake -B build -S . \
-DCMAKE_C_FLAGS="-fsanitize=address -fno-omit-frame-pointer" \
-DCMAKE_EXE_LINKER_FLAGS="-fsanitize=address"
cmake --build build -j$(nproc)
./build/tests
rm -rf build-asan
cmake -B build-asan -S . -DSANITIZER=address
cmake --build build-asan -j$(nproc)
./build-asan/tests
```
### Step 5: Update README (If Needed)
@@ -79,12 +77,23 @@ git commit -m "Release vX.Y.Z
git tag -a vX.Y.Z -m "Release vX.Y.Z"
```
### Step 8: Push
### Step 8: Push and Open dev → main PR
`main` is protected and only receives changes via `dev` → `main` PRs (see AGENTS.md). Never push directly to `main`.
```bash
git push origin main --tags
# Push the release commit and tag to dev
git push origin dev
git push origin vX.Y.Z
# Open the dev → main release PR for review + CI
tea pr create --repo TapTap/FastSync --head dev --base main \
--title "Release vX.Y.Z" \
--description "Release vX.Y.Z"
```
Then wait for the full CI to pass and request review before the PR is merged to `main`.
### Step 9: Report
```
+2 -2
View File
@@ -102,9 +102,9 @@ Informational: <count>
...
=== VERDICT ===
[PASS] No critical/high issues found
[PASS] No critical/high-severity issues found
— or —
[FAIL] <N> critical/high issues must be fixed
[FAIL] <N> critical/high-severity issues must be fixed
```
## Rules
+15 -14
View File
@@ -4,31 +4,31 @@ FastSync is a high-performance file synchronization system written in C11. It su
## Dependency installation
**CI rule:** never add `apt-get install` / `pip install` steps to CI workflows — use the custom Docker image instead. The image is built from the repo-root `Dockerfile` and is the same image CI uses: `gitea.tap-tap.win/taptap/fastsync-ci:v7`. It contains the full toolchain: gcc/g++, CMake, libzstd-dev, libssl-dev, make, git, cppcheck, clang-format, python3 + pytest, openssh-client, and Node.js.
**CI rule:** never add `apt-get install` / `pip install` steps to CI workflows — use the custom Docker image instead. The image is built from the repo-root `Dockerfile` and is the same image CI uses: `gitea.tap-tap.win/taptap/fastsync-ci:v10`. It contains the full toolchain: gcc/g++, CMake, libzstd-dev, libssl-dev, make, git, cppcheck, clang-format, python3 + pytest + pytest-xdist, openssh-client, and Node.js.
**Host rule:** for local development, use `nix-shell` (see `README.md`) which provides zstd, OpenSSL, CMake, and gcc. The Docker image can also be used locally for CI parity.
```bash
# Use the prebuilt CI image directly (faster, guaranteed CI parity)
docker pull gitea.tap-tap.win/taptap/fastsync-ci:v7
docker tag gitea.tap-tap.win/taptap/fastsync-ci:v7 fastsync-ci:local
docker pull gitea.tap-tap.win/taptap/fastsync-ci:v10
docker tag gitea.tap-tap.win/taptap/fastsync-ci:v10 fastsync-ci:local
# Or build the image from the repo-root Dockerfile
# (Note: the prebuilt :v7 image reflects the previous Dockerfile state;
# (Note: the prebuilt :v10 image reflects the previous Dockerfile state;
# rebuild from source to pick up any newly added packages like lcov/valgrind.)
docker build -t fastsync-ci:local .
# Build, run unit tests, and run integration tests inside the container
docker run --rm -v "$PWD:/workspace" -w /workspace fastsync-ci:local \
sh -c 'cmake -B build -S . && cmake --build build -j$(nproc) && ./build/tests && python3 -m pytest tests/'
sh -c 'cmake -B build -S . && cmake --build build -j$(nproc) && ./build/tests && python3 -m pytest tests/integration/ -n 4 --dist=load'
# Avoid root-owned build/ artifacts by matching your host UID/GID
docker run --rm --user "$(id -u):$(id -g)" -v "$PWD:/workspace" \
-w /workspace fastsync-ci:local \
sh -c 'cmake -B build -S . && cmake --build build -j$(nproc) && ./build/tests && python3 -m pytest tests/'
sh -c 'cmake -B build -S . && cmake --build build -j$(nproc) && ./build/tests && python3 -m pytest tests/integration/ -n 4 --dist=load'
```
> **Note:** The first `cmake configure` (`cmake -B build -S .`) fetches xxHash from GitHub via `FetchContent` — network access is required. Subsequent reconfigures reuse the cached source.
> **Note:** The first `cmake configure` (`cmake -B build -S .`) fetches xxHash via `FetchContent` — network access is required. Subsequent reconfigures reuse the cached source.
If a dependency is missing from the CI image, add it to the `Dockerfile` (and rebuild) rather than adding an install step to the CI workflow.
@@ -41,7 +41,7 @@ cmake -B build -S . -DSANITIZER=address # AddressSanitizer (ASan)
cmake -B build -S . -DSANITIZER=thread # ThreadSanitizer (TSan)
```
The CI workflow (`.gitea/workflows/ci.yaml`) runs lint (clang-format, cppcheck), build + test (unit + integration), and sanitizer (currently only `address`) jobs sequentially.
The CI workflow (`.gitea/workflows/ci.yaml`) runs lint (clang-format, cppcheck), then a **fast PR gate** — build + unit + a representative subset of integration tests marked `@pytest.mark.ci`, parallelized with pytest-xdist (`-n 4 --dist=load`). The full coverage jobs (full integration suite as `-m "not setpriv"`, sanitizer, fuzz, coverage, valgrind) run **only on push to `dev`/`main`**; pull requests skip them to keep PR CI under ~3 minutes. The two `setpriv` privilege tests are excluded from CI via a marker because their result depends on the runner/container uid and host mount permissions.
## Build
@@ -53,33 +53,34 @@ cmake -B build -S . && cmake --build build -j$(nproc)
```bash
./build/tests # unit tests
python3 -m pytest tests/ # integration tests
python3 -m pytest tests/integration/ -n 4 --dist=load -m "not setpriv" # full integration suite (CI excludes env-dependent privilege tests)
python3 -m pytest tests/integration/ -n 4 --dist=load -m ci # PR-gate subset only
```
## CI Workflow — Waiting for Results
When running the CI workflow via `tea` (the task execution agent), always set a sufficient timeout (e.g., 600000ms) to allow CI to finish. After CI completes, check the results yourself — do not assume success. Use `gh run watch` or similar to monitor CI status, then inspect logs on failure.
When running the CI workflow via `tea` (the task execution agent), always set a sufficient timeout (e.g., 600000ms) to allow CI to finish. After CI completes, check the results yourself — do not assume success. Monitor CI status via the Gitea API (see below) or `tea actions`, then inspect logs on failure.
## CI Troubleshooting
### If lint (clang-format) fails
Run clang-format in the CI Docker image to match the exact CI version:
```bash
docker run --rm -v "$PWD:/workspace" -w /workspace gitea.tap-tap.win/taptap/fastsync-ci:v9 \
docker run --rm -v "$PWD:/workspace" -w /workspace gitea.tap-tap.win/taptap/fastsync-ci:v10 \
sh -c 'find src/ tests/ -name "*.c" -o -name "*.h" | xargs clang-format -i'
```
### If cppcheck fails
Fix reported issues locally, then verify with:
```bash
docker run --rm -v "$PWD:/workspace" -w /workspace gitea.tap-tap.win/taptap/fastsync-ci:v9 \
docker run --rm -v "$PWD:/workspace" -w /workspace gitea.tap-tap.win/taptap/fastsync-ci:v10 \
sh -c 'cppcheck --enable=warning,style,performance,portability --suppress=missingIncludeSystem --error-exitcode=1 --inline-suppr src/ tests/'
```
### If integration tests fail
Run locally before pushing:
```bash
python3 -m pytest tests/ -v --tb=short
python3 -m pytest tests/integration/ -n 4 --dist=load -m "not setpriv"
```
## Branch Strategy
@@ -164,7 +165,7 @@ This can be cron'd locally if desired (e.g., `crontab -e` with `opencode run`).
## Is opencode a good option?
**Yes, for FastSync's needs.** The hybrid model works well:
- opencode's 17 specialized agents handle deep code analysis, fixes, tests, and reviews
- opencode's 16 specialized agents handle deep code analysis, fixes, tests, and reviews
- The assistant orchestrates subagents, merges branches, and iterates on CI
- You only review the final output
+199
View File
@@ -0,0 +1,199 @@
# Changelog
All notable changes to FastSync are documented here. Versions match
`PROTOCOL_VERSION` (printed by `fastsync --version`); the client and server must
run the same version because the handshake is strict.
## [2.21.0] - 2026-09-14
### Added
- Optional server→client rejection detail (protocol 2.21.0). A rejected
operation may now carry a bounded human-readable reason via
`STATUS_ERROR_DETAIL` instead of a bare `STATUS_ERROR`, so the client can
report *why* the server refused (daemon module gate, config validation,
receiver-side path/node validation). `receive_status()` transparently maps the
new status back to `STATUS_ERROR` for every existing call site and captures
the reason into a thread-local buffer exposed by `protocol_last_error()`. The
detail body is always consumed, so the stream cannot desynchronize, and
messages are sliced to `MAX_ERROR_DETAIL_BYTES` (4096) on send.
- **Server-contacting `--dry-run` (protocol 2.21.0).** `--dry-run` now performs
a real handshake with a remote/daemon receiver and reports exactly what WOULD
change based on receiver state (existing destination files, mtimes, checksums,
basis dirs). The wire config carries the dry-run intent (`Config.dry_run`) and
the receiver answers each per-file check with `STATUS_DRY_RUN_TRANSFER` (would
transfer) or `STATUS_OK` (already up to date); the sender prints the
would-transfer set and its trailer without sending any file data. The receiver
performs the normal read-only incremental decision but mutates nothing: no temp
files, writes, renames, deletes, metadata/xattr/chown, or directory creation.
A plain local destination (no explicit `--server-port`/remote) keeps the
original client-side dry-run. Would-delete reporting for `--delete*` is
deferred to a follow-up; dry-run never deletes.
- Daemon `max connections per host` (per-source-IP concurrent cap, default 0 =
unlimited), `auth lockout threshold` (default 10; 0 disables) and
`auth lockout duration` (default 300 s) config keys.
- `fastsync-server --allow-super` opt-in for a privileged standalone TCP server;
without it a root standalone receiver forces super-user activities off (device
nodes, `--write-devices`, ownership). The `--stdio` SSH argv is client-composed,
so super activities always stay off there.
### Changed
- Config wire fields are now declared once in an X-macro table
(`CONFIG_WIRE_FIELDS` in `src/shared/config.h`) that generates the struct
members, defaults, and the send/receive sequence, removing the manual
six-site field sync. Wire bytes and `PROTOCOL_VERSION` are unchanged.
- `receive_incremental_check()` (the per-file `STATUS_CHECK` fast path) is split
into small static helpers with a short linear orchestrator. Pure refactor: the
wire byte stream and all cleanup are unchanged.
- `authorized_root` state has a single owner (`utils.c`) with read accessors; the
duplicated statics in `file.c` and the server were removed.
- `Data` records its owning `ProtocolSession` so its memory charge is returned to
the session that reserved it, regardless of the destroying thread.
- The receiver pipeline moved out of `shared` into `server/receiver_pipeline.[ch]`;
the build now uses explicit `fastsync_shared` / `fastsync_client_core` /
`fastsync_server_core` targets instead of a GLOB, and the client no longer links
server code.
- The benchmark tool generates the requested random/compressible data mix
accurately, verifies each transfer before recording it, computes correct
percentiles, adds a MB/s column, handles `tc`/netem without requiring `sudo`
when already root, builds into a dedicated `build-bench/` directory, and adds a
`--warm` incremental-transfer mode.
- The `nix-shell` dev environment provides the full toolchain (clang-format,
cppcheck, pytest-xdist, OpenSSH, rsync, iproute2, valgrind, lcov) and no longer
builds on entry.
### Security
- Enforce the daemon's per-module `max connections` cap (0 = unlimited) and add
the shared per-source `max connections per host` cap plus a cross-process
`auth lockout`. Because the listener forks one child per connection, the
counters live in an anonymous shared mapping created before the accept loop and
reclaimed by the parent's `SIGCHLD` handler, so the per-module, per-source and
auth-failure state is shared across every child (including after `SIGKILL`). The
per-source table has a bounded lifetime (expired/idle entries are reclaimed,
with a rate-limited warning when genuinely full), and the occupancy counters are
re-derived from the shared slot table on every child exit. Trusted loopback
peers are exempt (they share one address); clients behind a shared NAT/proxy
share a single per-host budget and lockout, which is documented.
- Hardening from a full security audit:
- Fail a truncated zstd frame instead of spinning forever (remote DoS).
- Open receiver destination/basis/hard-link entries `O_NONBLOCK` so a
client-planted FIFO cannot block a worker indefinitely.
- Require a regular file before `--inplace` writes, closing a FIFO-hang and a
raw-device write that bypassed the `--write-devices` gate.
- Reject SSH destinations whose user/host begins with `-` and insert `--` before
the host token, closing `-o ProxyCommand=…` argument injection (RCE).
- Gate client `--force` recursive removal behind the server `--allow-delete`
policy.
- Reject empty `hosts allow`/`hosts deny`/`auth users` values instead of
silently meaning "unrestricted".
- Restrict TLS 1.2 to AEAD suites and set server cipher preference; load the
private key TOCTOU-safely from an `O_NOFOLLOW` fd; verify IP literals against
IP SANs; guard client-cert CN truncation.
- Make `--dry-run` content-blind: it neither reads destination files nor
hashes basis files, removing a 1-bit content oracle against `read only`
modules.
- Bound glob matching (iterative DP, no exponential backtracking) and bound
line reads for filter/`--files-from`/pattern files.
- Gate `system.posix_acl_*` xattrs on `--acls` and charge decompression/chunk
allocations against the per-connection memory budget.
### Fixed
- Pre-auth NULL dereference in `config_delete()` when an over-long
`basis_count` (and the analogous count fields) was received and then failed
validation; received counts are now validated before being published.
- Leaked inherited `Data` in the forked compression-truncation unit test
(valgrind definite leak).
- `receive_status()` no longer loses a captured rejection reason when owed
keepalives are drained.
## [2.20.0] - 2026-09-13
### Security
- Cap cumulative `DirTimeList` growth and bound pre-auth config-string memory
(remote memory-exhaustion DoS).
- Daemon host access control (`hosts allow`/`hosts deny`, IPv4/IPv6/CIDR),
configurable global `max connections`, connection audit logging, and a
bounded `auth failure delay` throttle. IPv4-mapped peers are normalized and
invalid patterns are rejected at parse time (no silent fail-open).
- Honor `--timeout` for protocol I/O and bound idle/session time to defeat
keepalive slowloris; child-safe signal handling in the forked daemon.
- Compiler/linker hardening (`_FORTIFY_SOURCE`, stack protector, PIE, RELRO)
and pinned build dependencies.
### Fixed
- Use-after-free in the basis-dir oversize preflight.
- Placeholder `Data` leaks, `missing_args` leak, scanner chunk leak.
- Thread-safe logging; single fd owner and cleanup epilogue in the server
handler.
### Performance
- Metadata now crosses the wire as one packed frame (protocol 2.20.0).
- Delete keep-set and `--files-from` lookups indexed (O(n*m) → O(n)).
- Reused per-thread zstd contexts; `TCP_NODELAY` by default.
- Byte-bounded sender queues; removed a redundant scanner `stat()`.
## [2.19.0] - 2026-09-12
### Security
- **Daemon authentication rewritten as SCRAM-SHA-256 challenge/response**
(`STATUS_AUTH_CHALLENGE` → `STATUS_AUTH_RESPONSE` → `STATUS_AUTH_OK`/`STATUS_AUTH_FAILED`),
replacing the old replayable static `SHA-256(password)` bearer credential.
Each proof is bound to a fresh per-connection server nonce plus a client
nonce, so a captured response can never be reused.
- **Salted verifier store.** `--password-file`/`--early-input` now hold
`user:$fastsync$1$pbkdf2-sha256$<iters>$<salt>$<stored_key>$<server_key>`
(PBKDF2-HMAC-SHA256, default 600000 iterations, range 100000–10000000). The
legacy `user:SHA256HEX` form is hard-rejected; there is no auto-upgrade.
Generate stores offline with `fastsync-server --hash-credentials FILE
[--iterations N]`.
- **Username-enumeration hardening.** Unknown/off-list users are answered with a
dummy verifier whose salt is a deterministic per-username value
(`HMAC-SHA256(dummy_key, username)`), using the store-wide uniform iteration
count and a constant-time full-length membership scan. The dummy key is
persisted in an owner-only `<store>.dummykey` sidecar (atomic publish, exact
mode 0600) so challenges are stable across restarts.
- **Verified transport for auth-required modules.** A module with `auth users`
accepts credentials only over verified TLS whose client certificate matches
`--client-cn`, or — when `--allow-unauthenticated` is explicitly set —
plaintext from a loopback peer. Remote plaintext is refused before any
challenge. Clients must use `--tls` to send `--password-file` credentials to a
non-loopback daemon; `--client-cn` is mandatory with `--tls`.
- **Secret hygiene.** The plaintext password, derived keys, nonces/proofs and
the dummy key are wiped from memory on every path and never logged.
- Carried-over hardening: `-K` TOCTOU-safe directory walk
(`openat(O_NOFOLLOW)` per component), always shell-quoted SSH remote path,
TLS compression/renegotiation disabled, race-free (open-then-`fstat`)
`--password-file`/`--early-input` checks, log-injection escaping, and lazy
protocol debug escaping.
### Added
- `fastsync-server --hash-credentials FILE [--iterations N]` offline tool.
- `<store>.dummykey` sidecar (auto-created, owner-only, 0600).
- Integration tests for auth replay rejection, malformed frames, legacy-store
refusal, and the loopback/TLS transport policy; fuzz targets for config
receive and daemon-auth parsing.
### Changed
- **Protocol version 2.18.0 → 2.19.0 (breaking).** The config-frame auth block
is now `[present][username]` (digest removed) and the auth challenge/response
frames are interleaved between the config frame and its `STATUS_OK`. A 2.19.0
client and a 2.18.0 server (or vice versa) fail cleanly at the handshake.
- Daemon modules declaring `auth users` require a configured credential store at
startup (fail closed); operators regenerate stores from plaintext with
`--hash-credentials`.
### Notes
- First tagged release. FastSync implements rsync-compatible file
synchronization over TCP and SSH with TLS (OpenSSL), streaming zstd
compression, multithreaded transfers, and incremental sync. See
[RSYNC_COMPAT.md](RSYNC_COMPAT.md) for the flag-parity matrix.
+199 -26
View File
@@ -1,6 +1,6 @@
cmake_minimum_required(VERSION 3.22)
project(FastFileTransfer)
project(FastFileTransfer VERSION 2.21.0)
set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
set(CMAKE_C_STANDARD 11)
@@ -38,11 +38,24 @@ if(ENABLE_COVERAGE)
add_link_options(--coverage)
endif()
# --- Build hardening option ---
# Production hardening is applied to the shipping server/client binaries only,
# and only when no sanitizer or coverage instrumentation is active: sanitizers
# carry their own instrumentation, and _FORTIFY_SOURCE requires an optimising
# build (never the -O0 used for coverage).
option(ENABLE_HARDENING "Enable compiler/linker hardening for production targets" ON)
set(HARDENING_ACTIVE OFF)
if(ENABLE_HARDENING AND SANITIZER STREQUAL "none" AND NOT ENABLE_COVERAGE)
set(HARDENING_ACTIVE ON)
endif()
include(FetchContent)
FetchContent_Declare(
xxhash
GIT_REPOSITORY https://github.com/Cyan4973/xxHash
GIT_TAG v0.8.3
# v0.8.3 is a lightweight tag pointing at this exact commit (no ^{} peel
# entry); pin the commit SHA instead of the mutable tag.
GIT_TAG e626a72bc2321cd320e953a0ccf1584cad60f363 # v0.8.3
SOURCE_SUBDIR cmake_unofficial
)
FetchContent_MakeAvailable(xxhash)
@@ -57,35 +70,179 @@ endif()
find_package(OpenSSL REQUIRED)
file(GLOB SHARED_SRCS "src/shared/*.c")
set(FILE_STORE_SRCS "${CMAKE_CURRENT_SOURCE_DIR}/src/shared/file_store.c")
list(REMOVE_ITEM SHARED_SRCS ${FILE_STORE_SRCS})
file(GLOB SERVER_SRCS "src/server/*.c")
set(SERVER_RECEIVER_SRCS src/server/receiver.c)
file(GLOB CLIENT_SRCS "src/client/*.c")
# --- Explicit source lists ---
# The shared library is self-contained: it must never depend on the client or
# server modules. In particular, the receiver pipeline (receive_thread /
# write_thread) lives under src/server, not here, so the client executable can
# link the shared library without pulling in any server code.
set(SHARED_SRCS
src/shared/array_list.c
src/shared/batch.c
src/shared/charset.c
src/shared/checksum.c
src/shared/chmod.c
src/shared/chunk.c
src/shared/compression.c
src/shared/config.c
src/shared/credentials.c
src/shared/daemon_conf.c
src/shared/daemon_limits.c
src/shared/data.c
src/shared/delay_updates.c
src/shared/delta.c
src/shared/file.c
src/shared/file_list.c
src/shared/file_receive.c
src/shared/file_send.c
src/shared/file_store.c
src/shared/filter.c
src/shared/hardlink.c
src/shared/identity.c
src/shared/log.c
src/shared/metadata.c
src/shared/motd.c
src/shared/multiprocessing.c
src/shared/protocol.c
src/shared/queue.c
src/shared/stop_condition.c
src/shared/transport_ssh.c
src/shared/transport_tcp.c
src/shared/transport_tls.c
src/shared/utils.c
src/shared/xattr.c
)
# Server implementation (no main): the receiver read/write pipeline plus the
# CLI parser. The server executable adds its own main (server.c).
set(SERVER_CORE_SRCS
src/server/receiver.c
src/server/receiver_pipeline.c
src/server/server_cli.c
)
set(SERVER_MAIN_SRCS src/server/server.c)
# Client implementation (no main): everything except the CLI entry point.
set(CLIENT_CORE_SRCS
src/client/change_list.c
src/client/client_send.c
src/client/client_validation.c
src/client/scanner.c
src/client/usage.c
)
set(CLIENT_MAIN_SRCS src/client/client_cli.c)
# --- Library targets ---
add_library(fastsync_shared STATIC ${SHARED_SRCS})
target_include_directories(fastsync_shared PUBLIC src/shared)
target_link_libraries(fastsync_shared PUBLIC Threads::Threads ${ZSTD_LIBRARY} OpenSSL::SSL
OpenSSL::Crypto xxhash)
add_library(fastsync_client_core STATIC ${CLIENT_CORE_SRCS})
target_include_directories(fastsync_client_core PUBLIC src/client)
target_link_libraries(fastsync_client_core PUBLIC fastsync_shared)
add_library(fastsync_server_core STATIC ${SERVER_CORE_SRCS})
target_include_directories(fastsync_server_core PUBLIC src/server)
target_link_libraries(fastsync_server_core PUBLIC fastsync_shared)
# --- Main executables ---
add_executable(server ${SERVER_SRCS} ${SHARED_SRCS} ${FILE_STORE_SRCS})
target_include_directories(server PRIVATE src/shared src/server src/client)
target_link_libraries(server PRIVATE Threads::Threads ${ZSTD_LIBRARY} OpenSSL::SSL OpenSSL::Crypto xxhash)
# The client links only the shared library and its own core; it deliberately
# does NOT get src/server on its include path nor compile receiver.c.
add_executable(server ${SERVER_MAIN_SRCS})
target_link_libraries(server PRIVATE fastsync_server_core)
add_executable(client ${CLIENT_SRCS} ${SHARED_SRCS} ${FILE_STORE_SRCS} ${SERVER_RECEIVER_SRCS})
target_include_directories(client PRIVATE src/shared src/server src/client)
target_link_libraries(client PRIVATE Threads::Threads ${ZSTD_LIBRARY} OpenSSL::SSL OpenSSL::Crypto xxhash)
add_executable(client ${CLIENT_MAIN_SRCS})
target_link_libraries(client PRIVATE fastsync_client_core)
# --- Production hardening ---
# Each compile flag is probed so a compiler/architecture that lacks it still
# configures cleanly. _FORTIFY_SOURCE is guarded separately because it only
# works in an optimising build. xxHash is a static archive built by
# FetchContent, so it must be position-independent for the -pie link; the same
# applies to the first-party static libraries linked into the -pie binaries.
if(HARDENING_ACTIVE)
set_target_properties(xxhash fastsync_shared fastsync_server_core fastsync_client_core
PROPERTIES POSITION_INDEPENDENT_CODE ON)
include(CheckCCompilerFlag)
foreach(flag -fstack-protector-strong -fstack-clash-protection -fPIE)
string(MAKE_C_IDENTIFIER "HARDEN_${flag}" _harden_var)
check_c_compiler_flag("${flag}" ${_harden_var})
endforeach()
check_c_compiler_flag("-D_FORTIFY_SOURCE=2" HARDEN_FORTIFY_SOURCE)
foreach(target fastsync_shared fastsync_server_core fastsync_client_core server client)
foreach(flag -fstack-protector-strong -fstack-clash-protection -fPIE)
string(MAKE_C_IDENTIFIER "HARDEN_${flag}" _harden_var)
if(${_harden_var})
target_compile_options(${target} PRIVATE ${flag})
endif()
endforeach()
if(HARDEN_FORTIFY_SOURCE)
target_compile_options(${target} PRIVATE -D_FORTIFY_SOURCE=2)
endif()
endforeach()
foreach(target server client)
target_link_options(${target} PRIVATE -pie -Wl,-z,relro -Wl,-z,now -Wl,-z,noexecstack)
endforeach()
endif()
# --- Testing ---
enable_testing()
# Common test libraries
set(TEST_LIBS Threads::Threads ${ZSTD_LIBRARY} OpenSSL::SSL OpenSSL::Crypto xxhash)
set(TEST_INCLUDES tests src/shared src/server src/client)
# --- Unit tests ---
# The monolithic test binary exercises both client and server code, so it is
# the one place that legitimately sees both include directories and links both
# core libraries. client_cli.c is compiled here directly (with the test build
# define) rather than linked from fastsync_client_core so its test-only shims
# and the absence of main() are preserved.
set(TEST_SRCS
tests/runner.c
tests/test_array_list.c
tests/test_batch.c
tests/test_change_list.c
tests/test_checksum.c
tests/test_chunk.c
tests/test_client_cli.c
tests/test_compression.c
tests/test_config.c
tests/test_credentials.c
tests/test_daemon_conf.c
tests/test_daemon_limits.c
tests/test_data.c
tests/test_delay_updates.c
tests/test_delta.c
tests/test_file.c
tests/test_file_list.c
tests/test_file_sendfile.c
tests/test_fuzz_smoke.c
tests/test_glob.c
tests/test_hardlink.c
tests/test_iconv.c
tests/test_log.c
tests/test_metadata.c
tests/test_motd.c
tests/test_multiprocessing.c
tests/test_property.c
tests/test_protocol.c
tests/test_protocol_error.c
tests/test_queue.c
tests/test_receiver_timeout.c
tests/test_robustness.c
tests/test_scanner.c
tests/test_server.c
tests/test_server_cli.c
tests/test_shared_utils.c
tests/test_stop.c
tests/test_stress.c
tests/test_transport_ssh.c
tests/test_transport_tcp.c
tests/test_transport_tls.c
tests/test_xattr.c
)
# Monolithic test binary (backward compatible)
file(GLOB TEST_SRCS "tests/test_*.c" "tests/runner.c")
add_executable(tests ${TEST_SRCS} ${SHARED_SRCS} ${FILE_STORE_SRCS} ${SERVER_RECEIVER_SRCS} src/client/scanner.c src/client/change_list.c src/client/client_cli.c src/client/client_validation.c src/client/usage.c)
target_include_directories(tests PRIVATE ${TEST_INCLUDES})
add_executable(tests ${TEST_SRCS} src/client/client_cli.c)
target_include_directories(tests PRIVATE tests)
target_compile_definitions(tests PRIVATE FASTSYNC_TEST_BUILD)
target_link_libraries(tests PRIVATE ${TEST_LIBS})
target_link_libraries(tests PRIVATE fastsync_server_core fastsync_client_core)
add_test(NAME unit_all COMMAND tests)
# --- Fuzz targets (requires clang) ---
@@ -94,13 +251,29 @@ if(ENABLE_FUZZ)
if(NOT CMAKE_C_COMPILER_ID MATCHES "Clang")
message(FATAL_ERROR "ENABLE_FUZZ requires Clang (compiler is ${CMAKE_C_COMPILER_ID})")
endif()
file(GLOB FUZZ_SRCS "tests/fuzz/*.c")
set(FUZZ_SRCS
tests/fuzz/fuzz_chunk_deserialize.c
tests/fuzz/fuzz_compress_decompress.c
tests/fuzz/fuzz_config_receive.c
tests/fuzz/fuzz_delta_deserialize.c
tests/fuzz/fuzz_delta_signature_deserialize.c
tests/fuzz/fuzz_glob_match.c
tests/fuzz/fuzz_identity_parse.c
tests/fuzz/fuzz_manifest.c
tests/fuzz/fuzz_metadata_from_buf.c
tests/fuzz/fuzz_protocol_framing.c
tests/fuzz/fuzz_xattr_block.c
)
# Compile the sources under test directly so libFuzzer's coverage
# instrumentation sees them (static libraries would be uninstrumented).
set(FUZZ_CORE_SRCS ${SHARED_SRCS} src/server/receiver.c src/server/receiver_pipeline.c)
foreach(FUZZ_SRC ${FUZZ_SRCS})
get_filename_component(FUZZ_NAME ${FUZZ_SRC} NAME_WE)
add_executable(${FUZZ_NAME} ${FUZZ_SRC} ${SHARED_SRCS} ${FILE_STORE_SRCS} ${SERVER_RECEIVER_SRCS})
target_include_directories(${FUZZ_NAME} PRIVATE ${TEST_INCLUDES})
add_executable(${FUZZ_NAME} ${FUZZ_SRC} ${FUZZ_CORE_SRCS})
target_include_directories(${FUZZ_NAME} PRIVATE tests src/shared src/server)
target_compile_options(${FUZZ_NAME} PRIVATE -fsanitize=fuzzer,address,undefined -fno-omit-frame-pointer)
target_link_options(${FUZZ_NAME} PRIVATE -fsanitize=fuzzer,address,undefined)
target_link_libraries(${FUZZ_NAME} PRIVATE ${TEST_LIBS})
target_link_libraries(${FUZZ_NAME} PRIVATE Threads::Threads ${ZSTD_LIBRARY} OpenSSL::SSL
OpenSSL::Crypto xxhash)
endforeach()
endif()
+1 -1
View File
@@ -3,7 +3,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
gcc g++ make libc6-dev cmake libzstd-dev libssl-dev git ca-certificates curl cppcheck clang-format \
python3 python3-pip python3-venv openssl openssh-client \
lcov valgrind clang libclang-rt-18-dev && \
pip3 install --break-system-packages pytest && \
pip3 install --break-system-packages pytest pytest-xdist && \
curl -fsSL https://deb.nodesource.com/setup_20.x | bash - && \
apt-get install -y --no-install-recommends nodejs && \
rm -rf /var/lib/apt/lists/*
+203 -55
View File
@@ -6,6 +6,10 @@ source/destination model and rsync-style options while adding optional
multithreading, streaming zstd compression, chunking, zero-copy TCP transfers,
and native TCP/TLS transports.
The release version is FastSync's client/server protocol version (printed by
`fastsync --version`); client and server must match. See
[CHANGELOG.md](CHANGELOG.md) for the history.
The compatibility target is straightforward:
- Existing rsync commands should keep the same meaning.
@@ -63,17 +67,26 @@ replacement for every rsync feature or protocol mode.
- Archive mode does not yet provide all of rsync's `-rlptgoD` behavior.
- Symlink transfer is incomplete; link targets are not yet recreated in all
modes.
- Owner/group, ACL, xattr, hard-link, device, and special-file handling is
incomplete or unavailable.
- Sparse-file handling does not yet preserve all holes correctly.
- `--partial`, `--partial-dir`, `-P`, `--append`, and `--append-verify` are not
yet full rsync-style resumable transfers. Interrupted files are not retained
for resumption.
- Owner/group, ACL, xattr, and hard-link handling is incomplete or
unavailable.
- Device and special-file preservation is implemented with documented
divergences: recreated device nodes require `CAP_MKNOD` on the receiver (a
non-root receiver skips the entry), and sockets cannot be recreated (FIFOs
are).
- Sparse-file hole preservation (`-S`, `--sparse`) is implemented receiver-side:
long all-zero runs are written as holes (no wire change; the full file image
is already in memory).
- `--partial`, `--partial-dir`, `-P`, `--append`, and `--append-verify` keep
the write atomic (temp + rename). With `--partial`, a failed/interrupted write
now retains the already-written temp at the destination path (best-effort) so
a later `--append`/`--append-verify` run can resume it.
- `--dirs` is not implemented. Its compatibility aliases `--old-dirs` and
`--old-d` are recognized but rejected explicitly rather than silently using
FastSync's recursive directory behavior.
- Several rsync short options currently have FastSync-specific meanings. Do
not assume every short option is interchangeable yet.
- Short-option names are now rsync-parity (Phase 7 Wave A): FastSync's former
collisions were renamed (`-j`/`--threads`, `--preserve`, `--sendfile`,
`--chunk-serialization`, `--timeout`, `--ssh-port`), so `-m`, `-M`, `-f`,
`-s`, `-T`, `-p`, `-c`, `-a`, and `-z` follow rsync. See `RSYNC_COMPAT.md`.
The detailed flag matrix is maintained in
[`RSYNC_COMPAT.md`](RSYNC_COMPAT.md). It distinguishes implemented,
@@ -83,25 +96,29 @@ partial, alternate, and planned behavior.
### Build
`compile_commands.json` is a symlink to `build/compile_commands.json` and is used by clangd/editor tooling; its target is generated by the build, so it dangles until the first build.
### Client
| Argument | Description |
|----------|-------------|
| Positional | `<source> <dest>` — automatic SSH detection if dest contains `:` |
| `-c [level]` | Compression with optional level (1–22, default 5) |
| `-z [level]` | Alias for `-c` |
| `-a, --archive` | Archive mode: enables `-c -m -M` (no `-s`) |
| `-m` | Multithreading mode |
| `-s` | Chunk serialization (batch all files per chunk) |
| `--secluded-args` | Accepted as an rsync compatibility option with no effect; `-s` remains chunk serialization. |
| `-f, --sendfile` | Sendfile zero-copy. Incompatible with `-c` / `-s`. TCP only. |
| `-M, --preserve` | Preserve supported file metadata (mode and mtime; ownership and atime are unsupported) |
| `-c, --checksum` | Verify content by checksum instead of size+mtime |
| `-z, --compress [level]` | Enable streaming zstd compression (level 1–22, default 5) |
| `-a, --archive` | rsync archive mode (`-rlptgoD`): links, metadata, devices and specials (not compression/multithreading) |
| `-j, --threads[=N]` | Multithreading mode; `N` (1–256) sets the parallel scanner worker count, bare `-j`/`--threads` uses the default |
| `-m` | rsync `--prune-empty-dirs` (short form now rsync-parity) |
| `--chunk-serialization` | Chunk serialization (batch all files per chunk; long form only) |
| `-s` | rsync `--secluded-args` compatibility no-op (remote SSH argv is already injection-safe) |
| `--sendfile` | Sendfile zero-copy. Incompatible with compression / chunk serialization. TCP only. Long form only. |
| `--preserve` | Preserve supported file metadata (mode and mtime; ownership and atime are unsupported) |
| `-n, --dry-run` | Scan and print what would be transferred |
| `-p <port>` | SSH port (default: 22) |
| `-p, --perms` | Preserve permission bits (part of the metadata bundle) |
| `--ssh-port <port>` | SSH port (default: 22) |
| `-v, --verbose` | Enable debug logging |
| `-q, --quiet` | Suppress non-error output |
| `--progress` | Show real-time transfer speed |
| `-P` | Enables partial-transfer mode and progress output (partial retention is incomplete) |
| `-P` | Enables partial-transfer mode + progress output; interrupted writes retain the already-written temp for resumption |
| `--delete` | Delete files on receiver not present in source (default timing: delete-after, i.e. only after the whole transfer succeeded) |
| `--delete-before` | Delete extras before the transfer starts (implies `--delete`) |
| `--delete-during`, `--del` | Delete extras once the keep-set is known, before data is applied (implies `--delete`) |
@@ -113,11 +130,11 @@ partial, alternate, and planned behavior.
| `--max-size <n>` | Skip files larger than n bytes |
| `--min-size <n>` | Skip files smaller than n bytes |
| `--max-alloc <SIZE>` | Maximum single allocation (binary units: B, K, M, G, T, P, E; default 1G) |
| `--incremental` | Skip files unchanged since last transfer (size + mtime). Auto-enables `--preserve`. Incompatible with `-s`. |
| `--incremental` | Skip files unchanged since last transfer (size + mtime). Auto-enables `--preserve`. Incompatible with `--chunk-serialization`. |
| `--existing` | Skip files not already present at the destination; update existing files normally. |
| `--bwlimit <KB/s>` | Bandwidth limit in kilobytes per second |
| `--chunk-size <n>` | Chunk size in bytes (default: 10485760) |
| `--timeout <sec>` | I/O timeout in seconds (default: 30) |
| `--timeout <sec>` | Positive I/O timeout in seconds, applied to both the socket (`SO_RCVTIMEO`/`SO_SNDTIMEO`, built-in default 30 s) and the per-message protocol poll deadline (built-in default 60 s). Omit the option to keep both built-ins; `0` is rejected. The server side keeps the built-in 60 s protocol window (the value is not sent on the wire). |
| `--contimeout <sec>` | Connection timeout in seconds (default: 10) |
| `--backup` | Backup existing destination files before overwriting |
| `--backup-dir <dir>` | Target directory for backups (requires `--backup`) |
@@ -134,7 +151,20 @@ partial, alternate, and planned behavior.
| `--cert <path>` | TLS certificate file (PEM) |
| `--key <path>` | TLS private key file (PEM) |
| `--ca <path>` | TLS CA certificate file for verification (PEM) |
| `--client-cn <name>` | Required TLS client certificate common name |
| `--client-cn <name>` | TLS client certificate common name; mandatory with `--tls` (a TLS connection always verifies the client CN) |
**Per-message vs. connection timeouts.** `--timeout` bounds each individual protocol
send/receive (the `poll()` deadline), so a peer that stops mid-frame is dropped. It
does not, by itself, stop a peer that keeps sending well-formed frames forever. The
receiver therefore also enforces two wall-clock (`CLOCK_MONOTONIC`) bounds on a
connection: a **1 hour** idle limit and a **24 hour** overall session cap. Only
frames that move real work (not `STATUS_KEEPALIVE`/`STATUS_ABORT` and not an
empty `STATUS_CHECK_BATCH`/`STATUS_DIR_TIMES`) refresh the idle timestamp, so a
peer cannot hold a connection slot by emitting cheap empty frames; a peer that
fabricates minimal non-empty frames can still occupy a slot until the 24 hour
cap, since no bound can require actual payload without risking a legitimate
long operation. Both are deliberately generous so a legitimate long-running
transfer is never aborted.
### Server
@@ -148,7 +178,8 @@ partial, alternate, and planned behavior.
| `--ca <path>` | TLS CA certificate file for verification (PEM) |
| `--destination-root <path>` | Authorized destination root (default: `.`) |
| `--allow-delete` | Permit manifest deletion |
| `--allow-unauthenticated` | Permit plaintext TCP clients |
| `--allow-super` | Standalone TCP listener only: keep super-user activities enabled for a **root** receiver. Without it a root standalone server forces `SUPER_MODE_OFF`, so client `--devices`/`--write-devices`/`--super` and client-chosen ownership requests are skipped/refused. **Rejected with `--stdio`** (the SSH remote argv is client-composed, so a client could otherwise pass it and defeat the secure default; operators exposing `fastsync-server --stdio` over SSH must use a forced command if the default must hold). No effect when not root. |
| `--allow-unauthenticated` | Permit plaintext TCP clients. For an `auth users` module this opts in **loopback plaintext only**; remote auth still requires verified TLS, so the flag never permits remote plaintext auth. |
| `-v, --verbose` | Enable debug logging |
| `--help` | Show help |
@@ -174,7 +205,7 @@ partial, alternate, and planned behavior.
3. **FileMetadata** — `mode`, `uid`, `gid`, `mtime_sec`, `mtime_nsec`;
uid / gid are advisory wire fields and are never applied by the receiver;
atime is unsupported
4. **Config** — runtime parameters (transported over wire, TLS settings excluded). Includes `timeout`, `contimeout`, `quiet`, `backup`, `backup_dir`, `stats`, `max_depth`, `log_file`, `queue_size`.
4. **Config** — runtime parameters (transported over wire, TLS settings excluded). Includes `timeout`, `contimeout`, `quiet`, `backup`, `backup_dir`, `stats`, `max_depth`, `log_file`.
5. **Queue** — thread-safe bounded queue with condition variables
6. **DirectoryScanner** — recursive BFS traversal with exclude and include pattern support, max-depth enforcement
@@ -274,17 +305,28 @@ The remote host must have `fastsync-server` available in `PATH`, or use
working directory, so use a destination below that directory unless the
remote server is otherwise configured with a matching authorized root.
The remote `--stdio` server argv is composed by the client, so it must never
be trusted to opt a root receiver into super-user activities: `--allow-super`
is rejected with `--stdio` and super stays off on that path. Operators
exposing `fastsync-server --stdio` over SSH must use a forced command (e.g. an
`authorized_keys` `command=` entry) if the default must hold.
```bash
ssh user@host 'mkdir -p destination'
./build/client /path/to/source user@host:destination
```
FastSync is **push-only**: the source (first argument) is always a local
directory and only the destination may be remote. A remote source such as
`client user@host:src ./local` (a "pull") is intentionally not supported; see
[RSYNC_COMPAT.md](RSYNC_COMPAT.md#direction).
### TCP transfer
Start the FastSync server:
```bash
./build/server --destination-root /path/to -p 8080
./build/server --destination-root /path/to -p 8080 --allow-unauthenticated
```
Then run the client:
@@ -336,7 +378,7 @@ FastSync-native are optional performance or transport extensions.
./build/client --incremental --checksum /source/ user@host:destination/
#Preserve supported mode and timestamp metadata
./build/client -M /source/ user@host:destination/
./build/client --preserve /source/ user@host:destination/
#Keep backups of overwritten destination files
./build/client --backup --backup-dir backups \
@@ -350,37 +392,40 @@ features without changing the meaning of ordinary compatibility options.
| Option | Purpose |
|---|---|
| `-m` | Enable the multithreaded scanner/loader/sender pipeline. |
| `-c [level]`, `-z [level]` | Enable streaming zstd compression, levels 1-22. |
| `-j`, `--threads[=N]` | Enable the multithreaded scanner/loader/sender pipeline. `N` (1–256) sets the parallel scanner worker count; bare `-j`/`--threads` uses the default. |
| `-z [level]`, `--compress [level]` | Enable streaming zstd compression, levels 1-22. |
| `--compress-level <n>` | Set the zstd compression level. |
| `--zc <alg>` | Alias for `--compress-choice`. FastSync supports `zstd` and `none`. |
| `--zl <n>` | Alias for `--compress-level`. |
| `--skip-compress <list>` | Skip compression for comma-separated suffixes; incompatible with `-s`. |
| `--skip-compress <list>` | Skip compression for comma-separated suffixes; incompatible with `--chunk-serialization`. |
| `--compress-threads <n>` | Use `n` zstd compression workers. Requires compression and a zstd build with threaded support; the setting affects sender CPU work only. |
| `--chunk-size <bytes>` | Set the transfer chunk size. |
| `-s` | Enable FastSync chunk serialization. |
| `-f`, `--sendfile` | Use TCP `sendfile()` zero-copy transfer. Incompatible with compression and chunk serialization. |
| `--chunk-serialization` | Enable FastSync chunk serialization (long form only; `-s` is rsync's `--secluded-args`). |
| `--sendfile` | Use TCP `sendfile()` zero-copy transfer. Incompatible with compression and chunk serialization. Long form only. |
| `--delta` | Use FastSync-native block delta transfer. Requires `--incremental`. |
| `--delta-block <bytes>` | Set the FastSync delta block size. |
| `--delta-block <bytes>` | Set the FastSync delta block size (`--block-size` is an alias). |
| `--delta-max <bytes>` | Limit files eligible for FastSync delta transfer. |
| `--server-host <host>` | Select the TCP server host. |
| `--server-port <port>` | Select the TCP server port. |
| `--server-port <port>` | Select the TCP server port (`--port <port>` and `--port=<port>` are rsync-friendly aliases). |
| `--tls` | Enable TLS for TCP transport. |
| `--bwlimit <KB/s>` | Apply token-bucket bandwidth limiting. |
| `--progress` | Show transfer progress and throughput. |
| `--stats` | Print transfer statistics. |
| `--timeout <seconds>` | Set I/O timeout. |
| `--timeout <seconds>` | Set the socket **and** per-message protocol I/O timeout (positive seconds). Omit to keep the built-in 30 s socket / 60 s protocol defaults. |
| `--contimeout <seconds>` | Set connection timeout. |
Current short-option conflicts are tracked as compatibility work. In
particular, FastSync currently uses `-p` for SSH port, `-s` for chunk
serialization, and `-S` for sparse handling. These meanings must be reconciled
before FastSync can claim full rsync CLI compatibility.
Short-option conflicts with rsync have been resolved for the CLI namespace
(Phase 7): `-c` is now rsync's `--checksum`, `-m` is `--prune-empty-dirs`, `-M`
is `--remote-option`, `-f` is `--filter`, `-s` is `--secluded-args`, `-p` is
`--perms`, and `-T` is `--temp-dir`. FastSync's own flags were renamed to
long-form-only or new shorts: multithreading is `-j`/`--threads`, metadata
is `--preserve`, sendfile is `--sendfile`, chunk serialization is
`--chunk-serialization`, timeout is `--timeout`, and SSH port is `--ssh-port`.
`-a`/`--archive` is now real rsync archive (`-rlptgoD`).
`--secluded-args` is accepted as a long-form compatibility no-op. It does not
change FastSync's transport or protocol behavior. The rsync short form `-s` is
intentionally not aliased because it remains FastSync's chunk-serialization
option.
`--secluded-args` (and its short form `-s`) is accepted as a compatibility
no-op. It does not change FastSync's transport or protocol behavior, because
remote SSH argv is already built injection-safe.
## Client Options
@@ -388,7 +433,7 @@ option.
| Option | Description |
|---|---|
| `-a`, `--archive` | Enable current archive preset. Full rsync archive semantics are planned. |
| `-a`, `--archive` | rsync archive mode (`-rlptgoD`): links, metadata, devices and specials. |
| `-n`, `--dry-run` | Scan and report without writing files. |
| `--delete` | Request removal of destination entries absent from the source. The server must allow deletion. Default timing is delete-after: extras are removed only after the whole transfer succeeded. |
| `--delete-before` | Delete extras before the transfer starts (implies `--delete`). |
@@ -405,25 +450,26 @@ option.
zero means unlimited.| | `--incremental` | Skip files matching destination size and mtime.|
| `--checksum` | Include xxHash64 content checks in incremental comparisons.| | `--backup` |
Back up overwritten files.| | `--backup - dir<dir>` | Store backups under a separate directory.|
| `--suffix<suffix>` | Set the backup filename suffix.| | `--partial` |
Select partial - transfer handling.With `--partial - dir`,
completed files are written there;
resumable transfers are not implemented.| | `--partial - dir<dir>` |
Set a relative partial - transfer directory below the server destination root;
use with `--partial`. |
| `--suffix<suffix>` | Set the backup filename suffix.| | `--partial` |
Select partial - transfer handling. On failed/interrupted writes the
already-written temp file is retained (best-effort) for resumption.|
With `--partial --partial-dir <dir>`, completed files are written under the
partial directory and installed atomically. | | `--partial - dir<dir>` |
Set a relative partial - transfer directory below the server destination root.
Use with `--partial`. |
| `--inplace` | Write directly to the destination instead of using a temporary file. |
### Metadata and links
| Option | Description |
|---|---|
| `-M`, `--preserve` | Preserve supported file metadata, currently mode and modification time. |
| `--preserve` | Preserve supported file metadata, currently mode and modification time (long form only). |
| `-l`, `--links` | Request symlink preservation;
link-target transfer remains incomplete. |
| `--copy-links` | Copy symlink referents. |
| `--safe-links` | Skip symlinks that point outside the transfer tree. |
| `--copy-unsafe-links` | Copy unsafe symlink referents. |
| `-S`, `--sparse` | Request sparse-file handling; full hole preservation is planned. |
| `-S`, `--sparse` | Sparse-file handling: receiver preserves holes (zero runs are written as holes; no wire change). |
### Output and logging
@@ -440,13 +486,13 @@ link-target transfer remains incomplete. |
| Option | Description |
|---|---|
| `-p <port>` | SSH port in the current CLI. This conflicts with rsync's `-p` permissions option and is planned for correction. |
| `--ssh-port <port>` | SSH port for the SSH transport (default: 22). Note the short `-p` is now rsync's `--perms`. |
| `--fastsync-server-path <path>` | Remote FastSync server path for SSH mode. |
| `--source-dir <path>` | Set the source directory explicitly. |
| `--dest-dir <path>` | Set the destination directory explicitly. |
| `--save-to-disk` | Enable server-side disk persistence. |
| `--server-host <host>` | TCP server address. |
| `--server-port <port>` | TCP server port. |
| `--server-port <port>` | TCP server port. `--port <port>` / `--port=<port>` is an alias. |
| `--tls` | Enable TLS. Requires `--cert` and `--key`. |
| `--cert <path>` | TLS certificate file. |
| `--key <path>` | TLS private key file. |
@@ -464,10 +510,67 @@ link-target transfer remains incomplete. |
| `--ca <path>` | CA file for peer verification. |
| `--destination-root <path>` | Confine received files to this server-side root;
defaults to the current directory. |
| `--allow-delete` | Permit client delete manifests. Deletion is refused by default. |
| `--allow-delete` | Permit client delete manifests. Deletion is refused by default. This also gates `--force` (which can recursively replace/remove a destination directory tree). |
| `--allow-super` | Standalone TCP listener only: keep super-user activities enabled for a **root** receiver. Without it a root standalone server forces `SUPER_MODE_OFF`, so client `--devices`/`--write-devices`/`--super` and client-chosen ownership requests are skipped/refused. Rejected with `--stdio` (the SSH remote argv is client-composed; use a forced command if the default must hold). No effect when not root. Daemon modules opt in per module with `client owner = yes`. |
| `-v`, `--verbose` | Enable debug logging. |
| `--help` | Print server usage. |
### Daemon configuration
`fastsync-server --daemon --config FILE` reads a line-based module config (an
implicit global section, then `[module]` sections). Besides `port`, `motd file`,
and `address`, the global section accepts:
- `max connections = N` — global cap on concurrent connections, default 100. The
listener enforces it; `0`, negative, and non-numeric values are parse errors.
- `max connections per host = N` — cap on concurrent connections from a single
source IP, default 0 (unlimited). Enforced across all forked connection
children through a shared registry.
- `auth failure delay = MS` — milliseconds to sleep after a failed
authentication, default 500. `0` disables it and the value is capped at 5000,
so online password guessing is rate-limited per connection. Successful auths
are never delayed.
- `auth lockout threshold = N` — number of failed authentications from one source
IP before that source is locked out, default 10; `0` disables the lockout. The
failure counter is shared across every connection child, so the lockout holds
even when the next attempt is handled by a different forked child.
- `auth lockout duration = SECONDS` — how long a locked-out source is refused
(default 300). A locked-out client is refused before any SCRAM challenge is
sent; a successful authentication clears the counter.
- `hosts allow` / `hosts deny` — comma- and/or whitespace-separated host access
patterns.
A `[module]` may also set `max connections` (0 = unlimited; enforced per module
across all connection children) and its own `hosts allow`/`hosts deny`.
The per-host cap and the shared auth lockout identify a source by its numeric
peer IP. **Loopback peers (127.0.0.0/8, IPv6 `::1`) are exempt**: every local
client shares that one address, so counting or locking them out would let one
local process deny service to all the others. The per-module and global
`max connections` caps still apply to loopback. Because the key is the peer IP,
`max connections per host` and `auth lockout` also cannot distinguish clients
behind the same NAT, proxy, or reverse-proxy address — they share one budget and
one lockout counter, so an over-aggressive lockout can affect unrelated users
behind that address. Prefer TLS client certificates (`--client-cn`) plus
`hosts allow`/`hosts deny` for per-client policy when clients share an address,
and size `auth lockout threshold` accordingly.
The shared per-source table has a bounded lifetime: an entry with no live
connection is reclaimed once its lockout has expired, or after it has been idle
(300 s). If every entry is still live or locked, a new source is admitted without
per-host accounting (fail open) and a rate-limited warning is logged; the
per-module cap and host ACLs still apply. The occupancy counters are re-derived
from the shared slot table after every child exit, so a child killed mid-transfer
(or mid-registration) cannot leak a slot or an occupancy count.
Host patterns are `*` (match all), IPv4/IPv6 literals, or IPv4/IPv6 CIDR
(`10.0.0.0/8`, `2001:db8::/32`). Hostnames are not resolved, so hostname globs
are rejected at parse time rather than silently never matching. A matching
`hosts deny` rejects; if any `hosts allow` entries exist, a peer matching none of
them is rejected; deny takes precedence over allow. The global list is checked
before the module list, before authentication, and the connecting peer address
(IPv4 or IPv6) appears in the connection and authentication audit log lines.
## Architecture
### Client
@@ -492,13 +595,58 @@ defaults to the current directory. |
## Protocol and Security
FastSync protocol version `2.5.0` is shared by the client and server. The
FastSync protocol version `2.21.0` is shared by the client and server. The
current protocol is sender-driven and includes configuration negotiation,
including the maximum allocation limit, incremental checks, checksums,
manifests, keep-alives, abort handling, per-file remove-source results, and
FastSync-native delta messages.
Client and server versions must currently match exactly.
Daemon modules that declare `auth users` authenticate with a SCRAM-SHA-256-style
challenge/response against a salted PBKDF2 verifier store: no password and no
replayable bearer credential crosses the wire or is stored on the daemon. All
store entries share one iteration count, and an unknown user is answered with a
deterministic per-username dummy challenge, so probing the daemon cannot
enumerate users. Store lines are generated with
`fastsync-server --hash-credentials <plaintext-file>` (see `RSYNC_COMPAT.md`);
redirect that output to an owner-only (mode 0600) file, and note that legacy
`user:SHA256HEX` stores are rejected. FastSync also maintains an owner-only
(mode 0600) `<store>.dummykey` sidecar next to the store: it holds the store-wide
dummy key, is auto-created on first load, and must be preserved across daemon
restarts so the dummy challenge for an unknown user stays stable (the key is
never regenerated while the sidecar exists). The sidecar is secret material and
must be protected like the credential store: keep it owner-only (mode 0600) and
include it with the store in backups and credential rotation. If the sidecar
cannot be created (a process-substitution/FIFO store path such as `/dev/fd/N`, a
read-only filesystem, a missing directory, or a create, write, fsync, link, or
fchmod failure), the daemon logs a warning and uses a transient key, so the
cross-restart guarantee does not hold for those deployments. One residual is
accepted: the store
iteration count is observable pre-auth by design, since the miss path must match
a hit.
An `auth users` module accepts credentials only when one of two conditions
holds: (a) the connection is an encrypted, verified TLS connection whose client
certificate matches the server's `--client-cn`, or (b) the connection is
plaintext from a loopback peer **and** the operator explicitly passed
`--allow-unauthenticated`. A remote plaintext peer is refused before any
challenge is sent, and `--allow-unauthenticated` never permits remote plaintext
auth: remote peers still require verified TLS regardless of the flag. Clients
sending daemon credentials with `--password-file` to a non-loopback daemon must
therefore use `--tls`; the client rejects a non-local plaintext credential
destination before any network I/O. Daemon modules are a `--daemon`-only
feature: the SSH `--stdio` path never loads a daemon config and is not an auth
transport for them.
Because the loopback allowance trusts whichever peer the kernel reports as
`127.0.0.1`, it assumes nothing relays remote connections to the daemon. A local
TCP forwarder or a TLS-terminating proxy in front of an auth-module listener
makes remote clients appear as loopback and bypasses the mutual-TLS identity
check, so do not front an auth-module listener with such a relay. `--tls` always
mandates `--client-cn`, so a TLS connection to an auth-required module always
has its client CN verified (`--client-cn` matches the certificate's CN only, not
a subjectAltName, which is acceptable for a private CA).
TLS provides encrypted TCP transport. Supplying `--ca` enables certificate
verification; without it, traffic is encrypted but peer identity is not
verified. Use certificate verification for deployments where authentication
@@ -537,7 +685,7 @@ Run the unit test binary:
Run the Python integration suite:
```bash
python3 -m pytest tests/
python3 -m pytest tests/integration/ -n 4 --dist=load -m "not setpriv"
```
For stricter local validation:
+518 -97
View File
@@ -6,11 +6,12 @@ This document maps rsync's full feature set to FastSync's current implementation
| Status | Count | Description |
|--------|-------|-------------|
| ✅ Implemented | 80 | Feature works end-to-end |
| 🔀 Alt Arg | 3 | Functionality exists but under different flag/semantics |
| ⚠️ Partial | 5 | Flag parsed/stored but behavior incomplete |
| 🔄 Compatibility No-op | 1 | Flag is accepted for CLI compatibility but has no effect |
| ❌ Not Implemented | 58 | Flag not recognized or no behavior |
| ✅ Implemented | 143 | Feature works end-to-end |
| 🔀 Alt Arg | 0 | Functionality exists but under different flag/semantics |
| ⛔ Impossible/Divergence | 4 | Flag is a documented divergence or cannot be implemented on any portable filesystem call |
| ⚠️ Partial | 0 | Flag parsed/stored but behavior incomplete |
| 🔄 Compatibility No-op | 0 | Flag is accepted for CLI compatibility but has no effect |
| ❌ Not Implemented | 0 | Flag not recognized or no behavior |
| **Total** | **147** | |
---
@@ -19,15 +20,15 @@ This document maps rsync's full feature set to FastSync's current implementation
| Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------|
| `-a`, `--archive` | Archive mode is -rlptgoD | 🔀 Alt Arg | Maps to -c -m -M (compression + multithread + metadata) |
| `-a`, `--archive` | Archive mode is -rlptgoD | ✅ Implemented | Phase 7 Wave A: real rsync archive. `-a`/`--archive` now implies `--links` + metadata (perms/times/group/owner as FastSync's broad bundle) + `--devices` + `--specials`. FastSync is always recursive, so no `-r` is needed. It no longer implies compression or multithreading (those moved to `-z`/`-j`). The short-option namespace is now rsync-parity (see the Phase 7 note) |
| `-v`, `--verbose` | Increase verbosity | ✅ Implemented | Sets `log_level=DEBUG` |
| `-q`, `--quiet` | Suppress non-error messages | ✅ Implemented | Suppresses client output while preserving errors |
| `--help` | Show help | ✅ Implemented | Prints usage and exits; `-h` is not accepted |
| `-V`, `--version` | Print version | ✅ Implemented | |
| `--info=FLAGS` | Fine-grained info verbosity | ✅ Implemented | Supports `copy`, `misc`, `skip`, `stats`, `all`, and `none`; explicit flags override `--verbose`, and `none` suppresses info output; unsupported names are rejected |
| `--debug=FLAGS` | Fine-grained debug verbosity | ✅ Implemented | `io`, `proto`, `pack`, and `util` are supported; `--debug=help` lists flags; other rsync categories are rejected |
| `--stderr=MODE` | Change stderr output mode | ⚠️ Partial | `errors` (default) and `all` are supported; `client` is rejected because FastSync has no rsync message channel |
| `--no-motd` | Suppress daemon MOTD | ❌ Not Implemented | |
| `--stderr=MODE` | Change stderr output mode | ⛔ Impossible/Divergence | `errors` (default) and `all` are supported; `client` is rejected with a clear error (`--stderr=client is not supported`) because FastSync has no rsync client-message channel — the rejection itself is the documented behavior (Phase 7 Wave B decision). The modes that exist work; the missing rsync channel cannot be emulated without a wire change |
| `--no-motd` | Suppress daemon MOTD | ✅ Implemented | Client-only display switch (Wave C): the daemon still sends the configured `motd file` on a `host::module/path` connection; the client reads and discards the frame without showing it. Without the flag the MOTD is printed to stdout after the config/auth handshake and escaped so control bytes cannot inject terminal sequences |
| `--exclude=PATTERN` | Exclude files matching pattern | ✅ Implemented | Glob matching in scanner |
| `--include=PATTERN` | Include files matching pattern | ✅ Implemented | Glob matching in scanner |
| `-C`, `--cvs-exclude` | Auto-ignore CVS files | ✅ Implemented | Applies the well-known rsync default exclude set as exclude rules during scanning (RCS SCCS CVS CVS.adm RCSLOG cvslog.* tags TAGS .make.state .nse_depinfo *~ #* .#* ,* _$* *$ *.old *.bak *.BAK *.orig *.rej .del-* *.a *.olb *.o *.obj *.so *.exe *.Z *.elc *.ln core .svn/ .git/ .hg/ .bzr/); `.git/`-style repo dirs are pruned without descending |
@@ -38,9 +39,9 @@ This document maps rsync's full feature set to FastSync's current implementation
|------|-------------------|-----------------|-------|
| `--stats` | Give transfer stats | ✅ Implemented | Prints file/byte counts |
| `-h`, `--human-readable` | Human-readable numbers | ✅ Implemented | Formats transfer byte sizes using binary units |
| `-i`, `--itemize-changes` | Per-file change summary | ✅ Implemented | Prints rsync-style `>f+++++++++` lines to stdout only for files actually sent (also under `-m`); unchanged files print nothing, matching single-`-i` behavior |
| `-i`, `--itemize-changes` | Per-file change summary | ✅ Implemented | Prints rsync-style `>f+++++++++` lines to stdout only for files actually sent (also under `-j`/`--threads`); unchanged files print nothing, matching single-`-i` behavior |
| `--progress` | Show progress | ✅ Implemented | Progress callback in sender |
| `-P` | Same as --partial --progress | ⚠️ Partial | Parses and enables progress, but interrupted files are not retained for resumable transfers |
| `-P` | Same as --partial --progress | ✅ Implemented | Phase 7 Wave B: `-P` parses to `--partial` + `--progress`. On a failed/interrupted write the receiver now retains the already-written temp file at the destination path (best-effort rename instead of unlink when configured), so a later `--append`/`--append-verify` run can resume it; `--partial-dir` still stages completed files under the confined partial dir and installs them atomically. The retention never runs when `--partial` is off, when no data was actually written, or under `--ignore-existing`/`--existing` (the destination is not ours to overwrite), and it only ever renames the already-written temp (never a corrupt blend; a failed rename falls back to the normal unlink). See the `-S`/`--sparse` interplay note (a retained sparse temp has full logical size) |
| `--out-format=FORMAT` | Custom output format | ✅ Implemented | Per-transfer template on stdout; tokens `%f` `%n` `%l` `%b` `%M` `%%` (`%b` is the source length, always `== %l`; post-compression/delta wire bytes are not counted); unknown escapes preserved |
| `--log-file=FILE` | Log to file | ✅ Implemented | `log_file` config field |
| `--log-file-format=FMT` | Log format | ✅ Implemented | Requires `--log-file`; writes one template line per transferred file using the same token set as `--out-format` (including `%b` `==` source length) |
@@ -58,11 +59,11 @@ This document maps rsync's full feature set to FastSync's current implementation
| `-0`, `--from0` | Delimit *-from files with NULs | ✅ Implemented | `--files-from` entries become NUL-delimited; the flag may appear before or after `--files-from` on the command line. NUL mode preserves entry bytes exactly (trailing CR/LF are part of the name; only newline mode trims them) |
| `--max-size=SIZE` | Skip files larger than SIZE | ✅ Implemented | `max_size` in scanner |
| `--min-size=SIZE` | Skip files smaller than SIZE | ✅ Implemented | `min_size` in scanner |
| `-I`, `--ignore-times` | Don't skip files matching size+time | ❌ Not Implemented | |
| `-I`, `--ignore-times` | Don't skip files matching size+time | ✅ Implemented | `ignore_times` config field (crosses the wire). Disables the size+mtime quick-check in the `--incremental` per-file handshake and the basis-dir quick-match, forcing the file to be transferred rather than skipped as unchanged. Receiver-side policy: `match_by_metadata` (file_receive.c) is bypassed, so the receiver never replies `STATUS_OK` for a matching size+mtime. Requires `--incremental` to have the handshake to act on (rsync does its quick check by default; FastSync's `-I`/`--size-only`/`--modify-window` only take effect under `--incremental`, exactly like they take effect through the basis check) |
| `--size-only` | Skip based on size only | ✅ Implemented | With `--incremental`, ignores mtime |
| `-@`, `--modify-window=NUM` | Mod-time comparison accuracy | ✅ Implemented | Whole-second tolerance with nanosecond-aware comparisons |
| `--existing` | Skip creating new files on receiver | ✅ Implemented | Existing destination files continue through normal update handling |
| `--ignore-existing` | Skip updating existing files | ❌ Not Implemented | |
| `--ignore-existing` | Skip updating existing files | ✅ Implemented | `ignore_existing` config field (crosses the wire; receiver-side policy). For a destination entry that already exists, the receiver skips the write: in the regular-file path, existing/delay-updates-staged, hardlink-sibling, and special/device handlers all return `FILE_SAVE_SKIPPED` without overwriting (passed as `no_replace` to the write engine), and `--backup` is disabled for skipped files. Note: it is applied at write time, so an existing dest whose size+mtime differ still has its data (or delta) transmitted before the write is discarded — functionally correct, bandwidth-suboptimal vs rsync, which short-circuits earlier. Like rsync, it does not apply to directories/symlinks (those return before the block). Combines with `-j`/`--threads` and `--delay-updates`. See Phase-4/— notes below |
| `--remove-source-files` | Sender removes regular files after confirmed transfer | ✅ Implemented | |
| `-x`, `--one-file-system` | Do not cross filesystem boundaries | ✅ Implemented | Sender scanner captures the root device and skips descending into mount-point crossings (`st_dev` differs); cross-filesystem mount-point subdirectories are dropped entirely, matching rsync |
| `-F` | Add the default `.rsync-filter` rules | ✅ Implemented | Reads one filter rule per line from each directory's `.rsync-filter` file during traversal and applies it to that directory's subtree; the current directory's rules are evaluated before its ancestors', so deeper files override shallower ones and per-directory files override the command-line `--filter`/`-C` base by default (matching rsync's first-match-wins precedence); `.rsync-filter` files are never transferred. The rsync `-FF` behavior (also `.cvsignore`) is out of scope; unsupported rule types inside the file abort with a clear error |
@@ -72,32 +73,32 @@ This document maps rsync's full feature set to FastSync's current implementation
| Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------|
| `-r`, `--recursive` | Recurse into directories | ✅ Implemented | Default behavior |
| `-R`, `--relative` | Use relative path names | ✅ Implemented | Meaningful together with `--files-from` (FastSync's default full-tree scan always mirrors the full source argument path below the destination root, so -R does not change it). With `-R` + `--files-from` each listed entry is transmitted under its bare relative destination path: an entry `sub/x.txt` lands at `<dest>/sub/x.txt` (its leading components preserved) instead of under the `<dest>/<full source path>` mirror. Only the path sent on the wire changes; the client still reads the absolute source path, and the delete manifest derives from the sent (relative) paths so `--delete` and `--remove-source-files` stay consistent in both layouts. Works single-threaded and under `-m` (including chunk serialization) |
| `-R`, `--relative` | Use relative path names | ✅ Implemented | Meaningful together with `--files-from` (FastSync's default full-tree scan always mirrors the full source argument path below the destination root, so -R does not change it). With `-R` + `--files-from` each listed entry is transmitted under its bare relative destination path: an entry `sub/x.txt` lands at `<dest>/sub/x.txt` (its leading components preserved) instead of under the `<dest>/<full source path>` mirror. Only the path sent on the wire changes; the client still reads the absolute source path, and the delete manifest derives from the sent (relative) paths so `--delete` and `--remove-source-files` stay consistent in both layouts. Works single-threaded and under `-j`/`--threads` (including chunk serialization) |
| `--no-implied-dirs` | Don't send implied dirs with -R | ✅ Implemented | Client-side, meaningful only with `-R` + `--files-from`. rsync would normally create the ancestor directories implied by a listed file so it can be written; with `--no-implied-dirs` a listed file whose parent directory is not itself (or via an ancestor) explicitly listed cannot be placed, and FastSync fails the whole run up front with a clear error (`--no-implied-dirs: cannot place file '...': parent directory '...' is not explicitly listed`). Listing the directory (or an ancestor of it, or the whole tree `.`) permits the file. In every other mode the option has no effect. FastSync has no per-entry skip channel, so the rsync "omit the file" case is surfaced as a hard pre-transfer error |
| `-d`, `--dirs`, `--old-dirs`, `--old-d` | Transfer dirs without recursing | ✅ Implemented | `-d <dir>` transmits an explicit directory entry for the source-root directory, so the destination mirror is created empty and nothing is descended into. With `--files-from` exactly the listed items are transferred: a listed directory is created empty (no descent) and a listed file is transferred with its content; the dest layout follows the same -R rules as plain files. A new wire frame (`STATUS_MKDIR`) carries each directory entry (path only); the receiver creates it with the same confined mkdir-parent semantics as regular writes, in single-threaded and `-m` receivers (chunk serialization carries a per-entry type marker). Directory entries appear in the delete manifest so `--delete` prunes correctly. FastSync divergences: directory mtimes/modes are not transmitted, filter/`--exclude` rules are not re-applied to the listed dirs mode (there is no descent during which they would apply), and `-d` never creates the intermediate directories between the destination root and a listed file beyond the usual on-demand parent creation. Under `--delay-updates` only regular files are staged: directory entries are created immediately, so a delayed run that fails part way can leave the already-created empty directories behind (matching rsync, which also creates directories as it processes the file list and only delays regular-file data) |
| `-d`, `--dirs`, `--old-dirs`, `--old-d` | Transfer dirs without recursing | ✅ Implemented | `-d <dir>` transmits an explicit directory entry for the source-root directory, so the destination mirror is created empty and nothing is descended into. With `--files-from` exactly the listed items are transferred: a listed directory is created empty (no descent) and a listed file is transferred with its content; the dest layout follows the same -R rules as plain files. A new wire frame (`STATUS_MKDIR`) carries each directory entry — the path and, when `--preserve`/`-a` (metadata mode) is negotiated, the directory's metadata; the receiver creates it with the same confined mkdir-parent semantics as regular writes, in single-threaded and `-j`/`--threads` receivers (chunk serialization carries a per-entry type marker). Directory entries appear in the delete manifest so `--delete` prunes correctly. Directory TIMES are transmitted (the `STATUS_DIR_TIMES` frame carries every traversed source directory's captured times, including `--dirs` entries) and applied by the receiver at the END of the transfer, after all children and the delete/publication phases, so a later child write cannot clobber a directory's mtime (`-O`/`--omit-dir-times` skips this application). FastSync divergences: directory modes/ownership are still not applied (only times are), and empty directories are still never created (a `STATUS_DIR_TIMES` entry is record-only), filter/`--exclude` rules are not re-applied to the listed dirs mode (there is no descent during which they would apply), and `-d` never creates the intermediate directories between the destination root and a listed file beyond the usual on-demand parent creation. Under `--delay-updates` only regular files are staged: directory entries are created immediately, so a delayed run that fails part way can leave the already-created empty directories behind (matching rsync, which also creates directories as it processes the file list and only delays regular-file data) |
| `--mkpath` | Create missing path components | ✅ Implemented | Wire option (client → server). At connection start the server creates the client's destination root directory (and any missing leading components below its own authorized root) when `--mkpath` is set, failing the connection cleanly if it cannot. Without `--mkpath` a destination root that does not exist yet is rejected up front (rsync semantics), so the flag is the only way to transfer into a not-yet-created destination directory. Creation is confined by the same secure mkdir walk as file writes (`O_NOFOLLOW`, no `..`) |
## 5. Transfer Modifications
| Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------|
| `-u`, `--update` | Skip files newer on receiver | ❌ Not Implemented | Removed because it had no effect |
| `-u`, `--update` | Skip files newer on receiver | ✅ Implemented | `update` config field (crosses the wire; receiver-side policy, implies `-M` metadata). Before writing a regular file, the receiver checks `file_destination_is_newer_secure()` (via `stat_is_newer`, second-then-nanosecond strict `>` on the existing destination) and skips the write when the destination is newer than the source (`FILE_SAVE_SKIPPED`); equal-or-older destination (or a newer source) is transferred normally. Applied at write time on the regular-file, delay-updates-staged, hardlink-sibling, and special/device paths. Only regular destinations can be guarded (the newer-check requires `S_ISREG`), and like the other write-time policies it does not short-circuit the data transfer for a differing-size dest. `--remove-source-files` correctly respects the receiver's skip outcome so a skipped source is not removed |
| `--inplace` | Update files in-place | ✅ Implemented | Direct write mode |
| `--append` | Append data to shorter files | ❌ Not Implemented | Removed because it had no effect |
| `--append-verify` | Append with old-data checksum | ❌ Not Implemented | Removed because it had no effect |
| `-W`, `--whole-file` | Copy whole file (no delta) | ❌ Not Implemented | |
| `--block-size=SIZE` | Force checksum block-size | ⚠️ Partial | Parsed as `--delta-block`; controls delta transfer block size |
| `--append` | Append data to shorter files | ✅ Implemented | Tail-only resume. When an existing destination file is SHORTER than the source, the receiver negotiates a resume offset with the sender and only the tail is transferred; the receiver rebuilds the full file (retained prefix + tail) and installs it through the normal atomic store path, so the result is byte-identical to the source whenever the retained prefix matches. Plain `--append` does NOT content-verify that prefix (rsync parity): a destination whose prefix differs from the source is resumed anyway, so the result (wrong prefix + correct tail) is NOT byte-identical and the file is effectively left corrupt — the documented rsync-parity risk (use `--append-verify` when the prefix cannot be trusted). Non-content attributes (permissions/ownership/mtime, via `-M`) are still applied. Requires the per-file `STATUS_CHECK` handshake, so it implies `--incremental`; it takes precedence over block delta for a growing file and falls back to delta/full when the destination is not shorter. Incompatible with `-s` (chunk serialization) and `--whole-file` (both rejected up front so the mode never silently degrades to a full transfer). Combines with `--inplace`, `--partial`/`--partial-dir`, and `--delay-updates` (the reconstructed full file flows through those paths unchanged). Divergence: rsync appends in place; FastSync reconstructs and atomically installs, so an interrupted or failed resume never leaves a half-written file at the destination (no corruption window), and `--append` is thus safe to use with the normal atomic path — not only with in-place writes |
| `--append-verify` | Append with old-data checksum | ✅ Implemented | Like `--append`, but the retained prefix IS verified before resuming: the sender transmits the source prefix checksum and the receiver compares it to the xxHash64 of the retained destination prefix; on a match only the tail is transferred, on a MISMATCH the run falls back to a clean full transfer so the result is always a byte-identical source copy (never a corrupt prefix+tail blend). Wire/protocol: the append handshake adds `STATUS_APPEND` / `STATUS_APPEND_SIG` / `STATUS_APPEND_OK` / `STATUS_APPEND_DATA` frames and `PROTOCOL_VERSION` was bumped **2.9.0 → 2.10.0** (peers must match, and both must be 2.10.0 or the run fails the version check). Same implications/incompatibilities as `--append`; when both spellings are given `--append-verify` wins (the safer semantics). See the Phase-3 append notes below |
| `-W`, `--whole-file` | Copy whole file (no delta) | ✅ Implemented | `whole_file` config field. Forces a full (whole-file) copy, disabling the block-level delta machinery: the sender only sends `STATUS_NEXT` + full data (client_send.c) and the receiver never requests a delta signature/reconstruction — the receiver's `try_delta = use_delta && !whole_file && ...` short-circuits. `whole_file` crosses the wire folded into `use_delta` (the wire carries `use_delta && !whole_file`), so no separate field/bump is needed. Delta is opt-in (`--delta` needs `--incremental`); `-W` additionally makes `--fuzzy` inert (no similar-file delta basis). `--append`/`--append-verify` are incompatible with `-W` and rejected up front (both sides). See the delta/append notes below |
| `--block-size=SIZE` | Force checksum block-size | ✅ Implemented | Phase 7 Wave B: `--block-size` is an alias for `--delta-block`; both set `config->delta_block_size` (default `DELTA_BLOCK_SIZE_DEFAULT`, bounds `DELTA_BLOCK_SIZE_MIN..MAX`, out-of-range values are rejected with the default kept). The value is genuinely honored by the delta engine end-to-end: `delta_signature_create_seeded(old, size, config->delta_block_size, seed)` on the sender and receiver, `delta_apply(old, ...)` with the same size, so a non-default block size changes the block count of every signature the harnesses exchange (verified by unit + integration tests) |
## 6. Destination Handling
| Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------|
| `-n`, `--dry-run` | Trial run with no changes | ✅ Implemented | `dry_run` config field |
| `-n`, `--dry-run` | Trial run with no changes | ✅ Implemented | Server-contacting since protocol 2.21.0. The final routing predicate is `dry_run_targets_server()` in `src/client/client_send.c`: any target a real run would reach over the wire selects the server-contacting path — an SSH transport, a daemon `host::module` destination, an explicit `--server-host` or `--server-port`/`--port`, TLS, or a source-bind `--address` — and the client handshakes with the receiver, which runs the normal read-only per-file check and answers `STATUS_DRY_RUN_TRANSFER`/`STATUS_OK` without mutating anything. A plain local destination (none of those) keeps the original client-side manifest that never dials the default `127.0.0.1:8080`. Would-delete reporting for `--delete*` is deferred (dry-run never deletes). |
| `-b`, `--backup` | Make backups of overwritten files | ✅ Implemented | Backup before overwrite |
| `--backup-dir=DIR` | Backup directory hierarchy | ✅ Implemented | `backup_dir` config field |
| `--suffix=SUFFIX` | Backup suffix (default ~) | ✅ Implemented | `suffix` config field |
| `--delay-updates` | Put updated files in place at end | ✅ Implemented | Successfully received files are staged under a private 0700 `.fastsync-stage` dir inside the receive root and atomically renamed into their final destinations only after the whole transfer (manifest/delete handling included) succeeds, just before the success/outcome frame is sent. The delete walker deliberately skips the staging dir at the receive root, so `--delete` removes genuine extras but never the staged files (deletion runs before publication; rsync's delete-after ordering is not implemented). `--existing`/`--ignore-existing`/`--update` decide against the final destination path at stage time; `--backup` moves the old file aside at publication. Incompatible with `--inplace` and with `--backup-dir=.fastsync-stage` (the internal staging name is reserved; both are rejected). The staging dir name is fixed, so two simultaneous delayed transfers to the same destination root are serialized with an exclusive advisory lock held for the whole transfer: the second session fails cleanly instead of corrupting the first. Aborting or failing before publication installs nothing and removes the staging tree; a crash between stage and publish leaves staged leftovers that the next delayed run wipes at start (process death releases the lock). A stage→publish failure aborts the transfer (best-effort cleanup of the not-yet-published staged files; already-published files are not rolled back). Works in single-threaded and `-m` modes |
| `-T`, `--temp-dir=DIR` | Create temporary files in DIR | ✅ Implemented | `--temp-dir` only; `-T` stays FastSync's `--timeout` alias. Scratch dir is resolved under the receive root; temp copies use a unique name there and are atomically renamed into place. If the scratch dir and destination are on different filesystems the atomic rename fails with EXDEV and the file save fails, which aborts the whole transfer (FastSync has no per-file skip/resume on a save error; rsync's non-atomic copy fallback is deliberately not used). `--inplace` and `--partial-dir` writes bypass the scratch dir |
| `--delay-updates` | Put updated files in place at end | ✅ Implemented | Successfully received files are staged under a private 0700 `.fastsync-stage` dir inside the receive root and atomically renamed into their final destinations only after the whole transfer (manifest/delete handling included) succeeds, just before the success/outcome frame is sent. The delete walker deliberately skips the staging dir at the receive root, so `--delete` removes genuine extras but never the staged files (deletion runs before publication; rsync's delete-after ordering is not implemented). `--existing`/`--ignore-existing`/`--update` decide against the final destination path at stage time; `--backup` moves the old file aside at publication. Incompatible with `--inplace` and with `--backup-dir=.fastsync-stage` (the internal staging name is reserved; both are rejected). The staging dir name is fixed, so two simultaneous delayed transfers to the same destination root are serialized with an exclusive advisory lock held for the whole transfer: the second session fails cleanly instead of corrupting the first. Aborting or failing before publication installs nothing and removes the staging tree; a crash between stage and publish leaves staged leftovers that the next delayed run wipes at start (process death releases the lock). A stage→publish failure aborts the transfer (best-effort cleanup of the not-yet-published staged files; already-published files are not rolled back). Works in single-threaded and `-j`/`--threads` modes |
| `-T`, `--temp-dir=DIR` | Create temporary files in DIR | ✅ Implemented | `--temp-dir` with the rsync short `-T` (Phase 7 Wave A; the timeout alias moved to long-only `--timeout`). Scratch dir is resolved under the receive root; temp copies use a unique name there and are atomically renamed into place. If the scratch dir and destination are on different filesystems the atomic rename fails with EXDEV and the file save fails, which aborts the whole transfer (FastSync has no per-file skip/resume on a save error; rsync's non-atomic copy fallback is deliberately not used). `--inplace` and `--partial-dir` writes bypass the scratch dir |
## 7. Deletion
@@ -112,7 +113,7 @@ This document maps rsync's full feature set to FastSync's current implementation
| `--max-delete=NUM` | Max files to delete | ✅ Implemented | `max_delete` config field (default -1 = no client limit; 0 = delete nothing). NUM bounds a `--delete` run with rsync's all-or-nothing semantics: the receiver rehearses the deletion first and, if the destination holds more than NUM extras, deletes NOTHING and fails the transfer with a distinct `--max-delete` error. A run at or below NUM deletes exactly the extras. NUM only applies together with `--delete` (it is inert otherwise, matching rsync). The hard server bound `MAX_SERVER_DELETE_COUNT` (100000) still caps the walk; a NUM above it never raises that cap, and exceeding the server bound is its own all-or-nothing error. Directories count toward the limit (each removed empty directory is one deletion), like rsync |
| `--ignore-errors` | Delete even with I/O errors | ✅ Implemented | Sender-side, client-only config field. rsync suppresses `--delete` when the transfer had I/O errors; FastSync's equivalent is a source-scan I/O error (an unreadable directory, e.g. EACCES): by default the scan aborts the run so no deletion happens. With `--ignore-errors` the scan continues past the unreadable directory, the readable tree is transferred and the deletion still runs (the mirror of the unreadable directory is treated as an extra). The run still exits non-zero (the error is reported, matching rsync's error status). Divergence: without the flag FastSync aborts the whole run on the scan error, whereas rsync transfers the rest of the tree and merely skips the deletion; both leave the deletion undone |
| `--force` | Force deletion of non-empty dirs | ✅ Implemented | `force_delete` receiver config field (crosses the wire). rsync's `--force` lets an incoming non-directory replace a destination directory; FastSync implements exactly that: when a regular file is written to a path that is currently a (possibly non-empty) destination directory, `--force` removes that directory tree first — confined to the receive root and symlink-safe (O_NOFOLLOW fd walk, symlinks removed by name, never followed) — so the atomic install can place the file. Without `--force` such a write fails and the run aborts. Divergence: `--force` acts on the immediate-install path only; under `--delay-updates` a blocking directory is not cleared (publication renames over regular files) |
| `--prune-empty-dirs` | Prune empty dir chains | ✅ Implemented | Long-only: FastSync's `-m` is already multithreading (recorded divergence — rsync's `-m` short form is not reassigned). FastSync's recursive transfer never emits directory entries, so empty directories are inherently never transferred (which is rsync's `-m` behavior) and truly-empty destination directory chains are removed by `--delete` regardless of this flag. The flag's additional real effect is on the `--dirs` explicit directory-entry generator: a plain `-d <empty-dir>` run omits the empty source directory's entry, so nothing is created at the destination (no `STATUS_MKDIR`, no `-i`/`--out-format` change line, and an existing empty mirror becomes an extra that `--delete` prunes). Explicitly `--files-from`-listed directories always pass through (documented `--files-from` behavior). A directory that still holds an excluded-but-protected file survives, matching the `--delete-excluded` default |
| `-m`, `--prune-empty-dirs` | Prune empty dir chains | ✅ Implemented | `-m`/`--prune-empty-dirs` (Phase 7 Wave A freed the rsync short `-m`; FastSync multithreading is now `-j`/`--threads`). FastSync's recursive transfer records directory times but never CREATES an empty directory (a `STATUS_DIR_TIMES` entry is record-only, and `--dirs` empty entries are pruned by this flag), so empty directories are inherently never transferred (which is rsync's `-m` behavior) and truly-empty destination directory chains are removed by `--delete` regardless of this flag. The flag's additional real effect is on the `--dirs` explicit directory-entry generator: a plain `-d <empty-dir>` run omits the empty source directory's entry, so nothing is created at the destination (no `STATUS_MKDIR`, no `-i`/`--out-format` change line, and an existing empty mirror becomes an extra that `--delete` prunes). Explicitly `--files-from`-listed directories always pass through (documented `--files-from` behavior). A directory that still holds an excluded-but-protected file survives, matching the `--delete-excluded` default |
**Deletion-timing implementation notes (Phase 3):** the delete flags above are
real. Two new config booleans (`delete_during`, `delete_delay`) join the already
@@ -207,63 +208,389 @@ order-independent because it runs over the fully parsed config. The deletion
POLICY flags (`--delete-excluded`, `--max-delete`, `--ignore-errors`, `--force`)
do NOT imply `--delete`; without `--delete` they are inert (matching rsync).
**Append-resume notes (Phase 3, append wave):** `--append` and `--append-verify`
are real. Both are negotiated when an existing destination file is found to be
**shorter** than the source during the per-file `STATUS_CHECK`; the receiver
replies with a new `STATUS_APPEND` frame carrying the resume offset (the prefix
length it already holds) instead of `STATUS_NEXT`/`STATUS_DELTA_SIGNATURE`.
The sender transmits ONLY the tail. For `--append-verify` it first sends the
source's prefix xxHash64 in a `STATUS_APPEND_SIG` frame; the receiver compares
it to the retained prefix and answers `STATUS_APPEND_OK` (transfer the tail) or
`STATUS_NEXT` (prefix mismatch → the sender falls back to a byte-exact full
transfer). The tail arrives in a `STATUS_APPEND_DATA` frame (compression and
metadata still apply). The receiver then rebuilds the full file in memory
(prefix + tail) and routes it through the existing atomic store engine, so all
of `--inplace`, `--partial`/`--partial-dir`, `--delay-updates`, `--backup`,
`--existing`/`--ignore-existing`/`--update` and delete-manifest behaviour is
unchanged and the result is a byte-identical source copy (given a matching
prefix). These new frames changed the wire, so `PROTOCOL_VERSION` was bumped
**2.9.0 → 2.10.0** (peers must match; the pre-existing `append`/`append_verify`
config booleans already crossed the wire). CLI: both flags imply `--incremental`
(the handshake needs it); they are incompatible with `-s` (chunk serialization)
and `--whole-file` (both rejected up front, never a silent full transfer); when
both spellings are given `--append-verify` wins. The FastSync divergence from
rsync is intentional and safer: rsync appends in place, whereas FastSync
reconstructs the whole file and atomically installs it, so an interrupted or
failed resume never leaves a partial/corrupt file at the destination — this is
why plain `--append` works on the normal atomic path, not only with `--inplace`.
## 8. Metadata Preservation
| Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------|
| `-M`, `--preserve` | Preserve file metadata | ✅ Implemented | Mode, uid, gid, mtime |
| `-p`, `--perms` | Preserve permissions | 🔀 Alt Arg | `-p` means SSH port; permissions preserved via `-M`/`--preserve` |
| `-p`, `--perms` | Preserve permissions | ✅ Implemented | Phase 7 Wave A: `-p`/`--perms` now preserve permission bits, folded into FastSync's broad metadata bundle (`--preserve`); the SSH port moved to `--ssh-port`. rsync-parity short form |
| `-o`, `--owner` | Preserve owner | ✅ Implemented | Part of -M |
| `-g`, `--group` | Preserve group | ✅ Implemented | Part of -M |
| `-t`, `--times` | Preserve modification times | ✅ Implemented | Part of -M |
| `-E`, `--executability` | Preserve executability | ✅ Implemented | Preserves executable permission bits (implies metadata preservation) |
| `--chmod=CHMOD` | Affect file permissions | ✅ Implemented | Supports numeric and symbolic `ugo` `rwx` changes; retains receiver safety masking |
| `-A`, `--acls` | Preserve ACLs | ❌ Not Implemented | Removed because it had no effect |
| `-X`, `--xattrs` | Preserve extended attributes | ❌ Not Implemented | Removed because it had no effect |
| `-H`, `--hard-links` | Preserve hard links | ❌ Not Implemented | Removed because it had no effect |
| `-D` | Same as --devices --specials | ❌ Not Implemented | Removed because device-file handling is not implemented |
| `--devices` | Preserve device files | ❌ Not Implemented | Removed because it had no effect |
| `--specials` | Preserve special files | ❌ Not Implemented | |
| `--copy-devices` | Copy device contents as file | ❌ Not Implemented | |
| `--write-devices` | Write to devices as files | ❌ Not Implemented | |
| `-U`, `--atimes` | Preserve access times | ❌ Not Implemented | |
| `-N`, `--crtimes` | Preserve create times | ❌ Not Implemented | |
| `-O`, `--omit-dir-times` | Omit dirs from --times | ❌ Not Implemented | |
| `-J`, `--omit-link-times` | Omit symlinks from --times | ❌ Not Implemented | |
| `--super` | Receiver attempts super-user activities | ❌ Not Implemented | |
| `--fake-super` | Store/recover privileged attrs via xattrs | ❌ Not Implemented | |
| `--open-noatime` | Avoid changing access time when opening files | ❌ Not Implemented | |
| `--numeric-ids` | Do not map uid/gid by name | ❌ Not Implemented | |
| `--usermap=STRING` | Map usernames | ❌ Not Implemented | |
| `--groupmap=STRING` | Map group names | ❌ Not Implemented | |
| `--chown=USER:GROUP` | Map owner and group | ❌ Not Implemented | |
| `--copy-as=USER[:GROUP]` | Perform the copy as another user/group | ❌ Not Implemented | |
| `-A`, `--acls` | Preserve ACLs | ✅ Implemented | Implemented on Linux via the POSIX-ACL xattr representation: the sender captures the `system.posix_acl_access` / `system.posix_acl_default` xattrs into the same bounded whitelisted set as `-X`, transmits them per-file, and the receiver re-applies them fd-relative. Setting an ACL the receiver is not permitted to set (non-root on a file it does not own, unsupported filesystem) is logged and skipped, never fatal. libacl is **not** required. Only the `system.posix_acl_*` namespaces plus `user.*` are ever applied; privileged namespaces are never applied (see the Phase-4 xattr/ACL notes below). Implies metadata transmission |
| `-X`, `--xattrs` | Preserve extended attributes | ✅ Implemented | Preserves unprivileged `user.*` extended attributes (Linux `listxattr`/`getxattr` on capture, `fsetxattr` on the written destination fd). Both capture (sender) and application (receiver) are restricted to the `user.*` namespace and the two POSIX ACL xattrs, so a client can **never** force a `security.*`/`trusted.*`/privileged attribute onto the destination; the receiver independently re-validates every incoming name against this whitelist and rejects anything else. Payloads are bounded (per-name ≤255B, per-value ≤1MiB, per-file count ≤256 total bytes ≤4MiB) on both ends, and an oversized/malformed frame is a clean protocol rejection (no OOM). Applied fd-relative to the exact written file. Implies metadata transmission. Incompatible with `-s` (chunk serialization), rejected up front (see the notes); a `--link-dest`/`-H` hard-link copy fallback re-applies the attributes so they are not dropped when a link is refused |
| `-H`, `--hard-links` | Preserve hard links | ✅ Implemented | Files on the source that share an inode (`st_dev`+`st_ino`, e.g. a `cp -al` tree) are re-created as hard links to one another on the destination, so duplicate links stay deduplicated and only the first member's data is sent (later members are transmitted as payload-less `STATUS_HARDLINK` frames). The receiver links each sibling to the first member's installed file with an atomic link + rename; on `link()` failure it falls back to a byte-identical local copy of the first member, never a partial/corrupt file. Requires the sequential scan for ordering (the first member is always emitted and installed before any sibling is linked). Works single-threaded and under `-j`/`--threads`, `--inplace`, `--delay-updates` (links staged and published by rename) and `--partial`. Crosses the wire (`preserve_hard_links` bool; `PROTOCOL_VERSION` bumped **2.11.0 → 2.12.0**, peers must match). Incompatible with `-s` (chunk serialization) and `--append`/`--append-verify`, rejected up front with a distinct error. See the Phase-4 hard-links notes below |
| `-D` | Same as --devices --specials | ✅ Implemented | Implies `--devices --specials`. `-D` was unassigned in FastSync (verified: no collision), so it is free to imply both device-node and special-file preservation. See the `--devices`/`--specials` rows and the Phase-4 devices notes below |
| `--devices` | Preserve device files | ✅ Implemented | Recreates char/block device nodes on the destination via `mknod` instead of transferring content. Type + rdev are validated strictly (S_IFMT from the transmitted mode; major/minor range-checked, non-negative), and creation is **privilege-gated**: `mknod` needs `CAP_MKNOD`, so a non-root receiver (CI runs via setpriv as non-root) logs a warning and **skips the device entry safely** — the whole transfer never aborts just because the node could not be made. The node is created fd-relative below the receive root (`mknodat` on the confined secure parent), so it can never be placed outside the authorized root, never follows a symlink, and never replaces an existing directory. Only a char/block mode is honored. Crosses the wire (a new `STATUS_SPECIAL` frame carries the path + metadata mode + rdev; `PROTOCOL_VERSION` bumped **2.12.0 → 2.13.0**). Divergence: per-entry skip (not a hard error) when the receiver lacks `CAP_MKNOD`, documented in the Phase-4 devices notes |
| `--specials` | Preserve special files | ⛔ Impossible/Divergence | **FIFO recreation works**: FIFOs are recreated on the destination via `mkfifo` (unprivileged, so this is a real, assertable behavior under CI). **Only socket recreation is impossible**: a socket entry can be created only by `bind(2)` on a live socket, not by any filesystem call, so a source socket is skipped with an explicit note. That one unsupported node kind is why the flag is classified Impossible/Divergence even though FIFO recreation itself works; its normal path is otherwise complete. FIFO creation is privileged-gated only in the sense of graceful skip on any permission failure. Node creation is confined below the receive root (`mkfifoat` on the secure fd-relative parent; no `..`, no symlink follow). Crosses the wire like `--devices` (the `STATUS_SPECIAL` frame; `PROTOCOL_VERSION` bumped **2.12.0 → 2.13.0**). See the Phase-4 devices notes |
| `--copy-devices` | Copy device contents as file | ✅ Implemented | Copy a device's CONTENT into an ordinary regular file on the destination instead of recreating the node — non-privileged and safe. FastSync scans a device/FIFO as a regular file: its reported size (`st_size`, typically 0 for char devices and FIFOs) is copied, so a FIFO or a non-readable device becomes an empty (or size-bounded) regular file. The default data path is size-bounded and never blocks (it sends exactly `st_size` bytes, never an unbounded pseudo-device stream); with `--sendfile`, a non-regular source (FIFO/device) is detected from its `stat` mode and falls back to that same buffered read, so `--copy-devices --sendfile` cannot hang either. The run always succeeds and never crashes on such input. **Deliberate, safe divergence from rsync's dd-like unbounded device read.** See the Phase-4 devices notes |
| `--write-devices` | Write to devices as files | ✅ Implemented | Write the received data directly into an **existing** device node on the destination instead of creating a regular file. Restricted and best-effort: the destination must already exist and be a char/block device (opened only under the confined receive root, with `O_NOFOLLOW` + `O_NONBLOCK`); a missing, symlinked, FIFO-with-no-reader (`ENXIO`), non-device destination, or any write failure is **skipped with a warning** rather than allowed, so a run can never clobber the system, never blocks on a special-file target, and never aborts on an unusable target. See the Phase-4 devices notes |
| `-U`, `--atimes` | Preserve access times | ✅ Implemented | Captures the source access time (from the scanner's pre-read stat, so it is not clobbered by reading the file for transfer) and transmits it over the wire; the receiver restores it together with the mtime via `futimens`/`utimensat`. Implies metadata transmission (the times travel inside the `-M` metadata payload), but does not enable ownership application (that stays opt-in via the identity flags). Wire: new `atime` fields on the metadata frame + a `preserve_atimes` config boolean; `PROTOCOL_VERSION` bumped **2.11.0 → 2.12.0** |
| `-N`, `--crtimes` | Preserve create times | ⛔ Impossible/Divergence | Birth-times cannot be set by any portable filesystem call (`utimensat`/`futimens` only set atime/mtime), so this row is an explicit **Impossible/Divergence** (Phase 7 Wave B). Capture + transmit stays: `statx(STATX_BTIME)` on Linux records the source birth time as a wire field; the receiver logs a debug note that it cannot be applied and continues — never failing the transfer and never pretending it worked. On platforms without `statx` it parses as a documented no-op (flag accepted; nothing is captured). Implies metadata transmission. Wire: new `crtime` fields + a `preserve_crtimes` config boolean; `PROTOCOL_VERSION` bumped **2.11.0 → 2.12.0** (see the Phase-4 metadata-time notes) |
| `-O`, `--omit-dir-times` | Omit dirs from --times | ✅ Implemented | Real modifier now that FastSync preserves directory times. With metadata on, the scanner captures every traversed source directory's mtime (and atime under `-U`) and the sender transmits them in trailing `STATUS_DIR_TIMES` frame(s) **after all file data and the optional delete manifest** (chunked at the receiver's `MAX_MANIFEST_ENTRIES` per-frame cap); a dir-time entry only RECORDS metadata and never creates the directory, so empty source directories stay untransferred. The receiver defers applying them until its delete / `--delay-updates` publication phases have committed, so writing or removing a child never clobbers a parent directory's mtime (rsync applies directory times at the end for exactly this reason). When `-O` is set (the boolean crosses the wire) the receiver does not apply any of them; without `-O` an `-a`/`--preserve` transfer now restores directory times (reversing the old "never preserves dir times" divergence). Wire change: the terminal `STATUS_DIR_TIMES` frame; `PROTOCOL_VERSION` bumped **2.16.0 → 2.17.0** |
| `-J`, `--omit-link-times` | Omit symlinks from --times | ✅ Implemented | Real modifier now that FastSync preserves symlink times. Symlink entries already carried their metadata on `STATUS_SYMLINK`; the receiver now applies it with **no-follow primitives only** (`utimensat(..., AT_SYMLINK_NOFOLLOW)`, plus best-effort `fchmodat(..., AT_SYMLINK_NOFOLLOW)` and policy-gated `fchownat(..., AT_SYMLINK_NOFOLLOW)`), so the link itself is stamped without ever dereferencing it, confined fd-relative below the authorized receive root. A symlink has no children, so the times are applied immediately at creation. When `-J` is set (the boolean crosses the wire) the receiver skips the timestamps (mode/ownership are unaffected); without `-J` an `-a`/`-l` transfer restores symlink mtimes. Wire change alongside `-O`: the shared `STATUS_DIR_TIMES` frame; `PROTOCOL_VERSION` bumped **2.16.0 → 2.17.0** |
| `--super` | Receiver attempts super-user activities | ✅ Implemented | Phase 7 Wave E: receiver-side **safe-subset + clear-refusal** privilege model, tri-state `super_mode` (auto/on/off). `--super` **permits** the receiver to attempt super-user activities — ownership application and char/block device-node creation — that are already confined fd-relative below the authorized receive root; `--no-super` **forbids** them even when the receiver is root; the default (`auto`) preserves the pre-existing **best-effort** behavior of *attempting* them (not only when already root: an unprivileged attempt is refused by the kernel and skipped per entry, matching FastSync's history). The server additionally accepts an operator-level `--no-super` veto that forces `OFF` for every connection it accepts (so it also refuses any client `--copy-as`/`--super`); a **privileged (root) standalone TCP listener now also defaults to `OFF`** unless the operator opts in with the new server-only `--allow-super` flag (the flag is **rejected with `--stdio`**, whose remote argv is composed by the client and must never defeat the secure default; operators exposing `fastsync-server --stdio` over SSH need a forced command if the default must hold. An unprivileged receiver is unchanged, since the kernel refuses the confined attempts anyway; the `--daemon` path keeps its per-module `client owner = yes` opt-in); the `--fake-super` owner replay and the `--write-devices` write path are gated by the same policy. **FastSync never elevates**: no `setuid`/`seteuid`/`setgid` is ever called, and `--super` never bypasses the confinement floor (`file_open_secure_parent`, `O_NOFOLLOW`, root checks) — it only permits an attempt that is already confined. `--super` does **not** imply `--numeric-ids` and never enables client-chosen ownership on its own: ownership is applied only when an explicit identity policy (`--usermap`/`--groupmap`/`--chown`/`--numeric-ids`/`--copy-as`) is also given. A non-root receiver given `--super` logs exactly one warning at activation and each confined attempt is then refused by the kernel and skipped per entry (never aborts); `--no-super` suppresses ownership, char/block `mknod`, `--write-devices` and the fake-super owner replay, while unprivileged FIFO creation is unaffected. Wire: one trailing `super_mode` int on the config frame (validated 0..2), sent **before** the `--copy-as` block (fixed order: super int, then copy-as presence int + ids); `PROTOCOL_VERSION` bumped **2.17.0 → 2.18.0**. **Documented divergence from rsync:** rsync's `--super` runs the receiver with elevated privilege; FastSync only permits a confined attempt and never elevates |
| `--fake-super` | Store/recover privileged attrs via xattrs | ✅ Implemented | Phase 7 Wave B: full record **and replay**. The receiver writes the source `uid:gid:mode:mtime_sec:mtime_nsec` into a reserved `user.fastsync.stat` xattr on each written file (best-effort, fd-relative, format unchanged), then immediately re-applies it via `fake_super_restore_fd`: `fchown` (only where privileged — a non-root EPERM/EACCES is skipped silently, matching FastSync's identity philosophy), `fchmod`, and `futimens`. The OWNER leg is additionally skipped unless an explicit ownership identity policy (`--numeric-ids`/`--usermap`/`--groupmap`/`--chown`/`--copy-as`) is active — `--fake-super` on its own only *records* the source owner and must not act as an un-gated chown primitive — when `--no-super` forbids super-user activities (even for root), or when an active `--copy-as` is authoritative, so the recorded source owner can never override a forced `--copy-as` owner; the xattr record is still stored/replayed for a later privileged restore and mode/mtime still apply, so unprivileged `--fake-super` keeps working. The restored mode goes through the same sanitization as the normal metadata path (group/other write bits are never granted, so a recorded 0666 restores as 0644), so fake-super replay can never grant group/other-write that plain `--preserve` would refuse. Absence or a malformed record is a silent no-op, never fatal. The recording format diverges from rsync's `user.rsync.%stat%`; no cross-tool conversion is attempted. Implies metadata transmission so the source uid/gid/mode/mtime are available. Both it and `-X`/`-A` are incompatible with `-s` (chunk serialization), rejected up front |
| `--open-noatime` | Avoid changing access time when opening files | ✅ Implemented | Sender-side policy: the sender opens source files with `O_NOATIME` (Linux) when reading them for transfer, so the open/read does NOT bump the source's on-disk access time. Degrades safely when `O_NOATIME` is unavailable (not defined) or refused (`EPERM`, since it needs `CAP_FOWNER` or file ownership): the code falls back to a normal open, so the data always transfers — only the atime-bump is skipped. It does not itself capture/preserve atime; it only avoids modifying it. **Client-only, never crosses the wire.** Exposed as `file_open_for_read()` and applied to both the buffered data path and the sendfile path |
| `--numeric-ids` | Do not map uid/gid by name | ✅ Implemented | Ownership is applied through FastSync's opt-in identity path (see the Phase-4 identity notes below). `--numeric-ids` is a mapping-policy modifier: when applying ownership it uses the transmitted numeric uid/gid directly, skipping the name lookup. Without an ownership-affecting option it is inert (FastSync only applies ownership when the user opts in). It does not need `-M` to be parsed, but ownership is only applied when metadata (hence the source uid/gid) is actually transmitted (see the notes) |
| `--usermap=STRING` | Map usernames | ✅ Implemented | Opt-in ownership application. rsync subset implemented: comma-separated `FROM:TO` rules evaluated in order, first match wins; `FROM`/`TO` are group/user names (resolved on the SOURCE machine at parse time), `*` (FROM matches any id / TO = the receiving process's current euid), and an `@N` or bare `N` numeric id. Rules are carried over the wire as resolved numeric id pairs; the receiver applies a matching rule (else falls back to `--chown`, `--numeric-ids`, then a best-effort name lookup) via an fd-relative `fchown`. Malformed/unresolvable specs are rejected with a clear error, never a silent no-op. Implies metadata preservation so the source uid/gid travel. Only effective when the receiver can actually change ownership (root or membership); otherwise it warns and continues |
| `--groupmap=STRING` | Map group names | ✅ Implemented | Same rsync subset and semantics as `--usermap` but for the group (gid) side and the group databases. See the Phase-4 identity notes |
| `--chown=USER:GROUP` | Map owner and group | ✅ Implemented | Opt-in ownership override applied receiver-side. Forms: `USER:GROUP`, `USER` (owner only), `:GROUP` (group only); a `*` for USER/GROUP means the current/root user or group as appropriate; an `@N`/bare `N` numeric id is accepted. A `:` inside a name may be escaped as `\:`. Equivalent to a trailing `*:*` usermap+groupmap rule (so an explicit `--usermap`/`--groupmap` match wins). Malformed or unresolvable specs are clear parse errors. Implies metadata preservation. Only effective when the receiver has permission to chown; otherwise it warns and continues (rsync parity) |
| `--copy-as=USER[:GROUP]` | Perform the copy as another user/group | ✅ Implemented | Safe-subset implementation, an explicit divergence from rsync's **real identity switching**. rsync makes the receiving process actually assume USER/GROUP (setuid/setgid); FastSync's receiver is multithreaded, so a real credential drop would be unsafe and is never attempted — FastSync never calls `setuid`/`seteuid`/`setgid`. Instead the receiver FORCES the ownership of every entry it writes to `copy_as_uid`/`copy_as_gid` through the existing confined, fd-relative identity path (the same `fchown`/`fchownat` mechanism as `--chown`/`--usermap`/`--groupmap`; symlinks use `fchownat(..., AT_SYMLINK_NOFOLLOW)`, and directories — including intermediate parents created implicitly while writing a nested file — and char/block/FIFO nodes are owned no-follow too, so a directory never keeps the receiver's owner while its children get the target owner), with `--copy-as` at the **highest priority** — it beats usermap/groupmap/`--chown`/`--numeric-ids` and the best-effort name lookup. This REQUIRES a privileged (root) receiver: an unprivileged receiver REFUSES the whole transfer up front at the config handshake (`server_module_gate`, running inside `config_receive_with_validate` before the `STATUS_OK` ack) with a clear error and no file data exchanged — never a silent wrong-ownership result. A server running with an operator `--no-super` veto also refuses it; a privileged (root) standalone TCP listener refuses it by default too and only honors it after the operator passes `--allow-super` (the flag is rejected with `--stdio`, where the client-composed remote argv could otherwise defeat the default; a forced command is required if the default must hold), and a **daemon** refuses `--copy-as`, like every other client-chosen-ownership request (`--numeric-ids`/`--chown`/`--usermap`/`--groupmap`/`--fake-super`/explicit `--super`), unless the selected module opts in with `client owner = yes`; without that per-module opt-in a daemon must not honor an arbitrary client-selected owner (a root standalone listener honors these for its single operator-authorized root only when started with `--allow-super`). `--fake-super` interaction: `--copy-as` is authoritative, so the recorded source owner is never replayed over the forced target owner. If the ownership apply still fails with EPERM/EACCES (capability-restricted root, root-squash, read-only mount) the failure is logged at ERROR and the **entry is reported as failed** rather than written with the wrong owner, which fails the transfer (fail-fast) so overall success is never reported with the wrong owner. USER is resolved on the client against the user database (a name, an `@N`/bare `N` numeric id, or `*` meaning the client's current euid); when `:GROUP` is present it is resolved against the group database (`*` meaning the client's egid). **Group-default rule:** when the group is omitted FastSync uses the user's primary gid (`getpwuid(uid)->pw_gid`); a numeric id with no local passwd entry has no primary gid to look up, so `gid` falls back to `uid` (documented divergence). Malformed/empty/unresolvable specs are clear parse errors, never a silent no-op. Never elevates privileges and never bypasses the confined receive root. Implies metadata preservation (the source uid/gid must be transmitted). Wire: a new trailing config-frame block **sent after** the `--super` int (presence int, then the two int32 ids, both validated `>= 0` on receive; the ids are also rejected if they do not fit int32 at CLI parse time); `PROTOCOL_VERSION` bumped **2.17.0 → 2.18.0** |
**Phase-4 metadata-time notes:** `-U/--atimes`, `-N/--crtimes`,
`-O/--omit-dir-times`, `-J/--omit-link-times`, and `--open-noatime` are new.
They change the wire: the per-file metadata frame grows `atime_valid` +
`atime_sec` + `atime_nsec` and `crtime_valid` + `crtime_sec` + `crtime_nsec`
(appended after the existing mode/uid/gid/mtime fields, preserving the exact
positions of every pre-existing field), and the config frame grows four
booleans — `preserve_atimes`, `preserve_crtimes`, `omit_dir_times`,
`omit_link_times` — that CROSS the wire so the receiver knows what to apply /
suppress. `--open-noatime` is **client-only** and is never serialized (it only
governs the sender's source reads). `PROTOCOL_VERSION` was bumped **2.11.0 →
2.12.0** (peers must match, exactly as prior phases did).
**Client-vs-wire split:** `-U` and `-N` affect both the sender (capture) and the
receiver (apply), so they and their metadata fields cross the wire;
`-O`/`-J` are receiver-side preferences and cross as config booleans;
`--open-noatime` is purely a client/sender open flag and stays off the wire
(mirroring the existing convention where `ignore_errors` is client-only while
`force_delete` crosses the wire).
**Phase-4 xattr/ACL notes (`-X/--xattrs`, `-A/--acls`, `--fake-super`):** these
are new in protocol 2.13.0 and add a bounded per-file xattr block to the
per-file metadata frame (count + each `name`/`value`, sent only when xattr
transport is enabled, i.e. with zero overhead on unaffected runs). The config
frame carries `preserve_xattrs`, `preserve_acls` (in the existing file-options
block) and a trailing `fake_super` boolean — all CROSS the wire so the receiver
knows the negotiated behavior; the derived `use_xattrs` flag is recomputed on
the receiver. `PROTOCOL_VERSION` was bumped **2.12.0 → 2.13.0** (peers must
match, exactly as prior phases did).
- **Security model (both `-X` and `-A`):** only `user.*` and the
`system.posix_acl_access` / `system.posix_acl_default` namespaces are ever
captured (sender) or applied (receiver). `security.*` (SELinux, capabilities,
...), `trusted.*`, and all other `system.*` attributes are never transmitted
or applied, so a client can never compel the receiver to set a privileged
xattr. The receiver re-validates each incoming name against this whitelist
even though the sender already filtered, so a malicious/compromised sender's
`security.capability` payload is rejected outright (a clean protocol error),
never applied.
- **Bounds / memory safety:** per-name length ≤ 255 B, per-value ≤ 1 MiB,
per-file count ≤ 256 names, per-file name+value total ≤ 4 MiB. Both the
sender (during capture) and the receiver (during receive) enforce these; an
oversized or malformed frame is rejected, never a large allocation.
- **Confined application:** xattrs are applied with `fsetxattr` on the exact
just-written destination file fd (before the atomic rename), never on a
caller-controlled path; this is the same confinement as mode/time restore.
The `--link-dest` / `-H` hard-link copy fallback (a byte copy when `link()`
is refused) also re-applies the incoming (or, for `-H`, the first member's)
xattrs and the `--fake-super` stat, so attributes are preserved rather than
silently dropped when the link fails.
- **Reserved fake-super key is receiver-only:** the `user.fastsync.stat` key is
excluded from sender capture AND from receiver application, so it can only be
written by the receiver's own `--fake-super` handling. A source file that
already carries such a record is never forwarded on a plain `-X` run, so it
cannot be spoofed to mislead a later privileged restore.
- **`-A` requires no libacl** — ACLs travel as the `system.posix_acl_*` xattrs.
Applying an ACL is owner-privileged: `fsetxattr` failure (e.g. non-root,
unsupported filesystem) is logged (collapsed to one line per file) and never
fatal.
- **`--fake-super`**: see the row above; the reserved key is `user.fastsync.stat`
with the documented `uid:gid:mode:mtime_sec:mtime_nsec` (mode octal) format.
It is honest but partial — there is no replay, and it does not interoperate
with rsync's `user.rsync.%stat%`.
- **Chunk serialization (`-s`) incompatibility:** the per-file xattr block rides
the streaming per-file frame, which `-s` replaces with a fixed buffer format,
so `-X` / `-A` combined with `-s` is rejected up front on both ends (mirroring
the existing `-H` + `-s` rejection) rather than silently dropping attributes.
**atime capture does not clobber the source atime:** the sender records the
access time from the **same pre-read stat the scanner already took** (inside
`file_metadata_create`), before any file data is read for transfer. So `-U`
alone captures the correct atime even without `--open-noatime`. `--open-noatime`
is orthogonal: it keeps the source's on-disk atime from being bumped by the read
that actually ships the data (only honoured where `O_NOATIME` works; it degrades
to a normal open otherwise, so the data always transfers).
**crtime handling:** `-N` captures the source birth time via `statx`/`STATX_BTIME`
(guarded `#ifdef STATX_BTIME` on Linux) and transmits it. On the receiver, **no
portable setter exists** (`utimensat` can only set atime/mtime), so the receiver
deliberately does **not** apply it: it logs a debug note and continues — it never
fails the transfer and never pretends the crtime was applied. This is the
explicit, documented unsupported-attribute handling. On platforms without
`statx` the flag is accepted but nothing is captured (a documented no-op).
**omit-dir-times / omit-link-times:** `-O` and `-J` are **real modifiers** as of
P7 Wave D (`🔄 → ✅ Implemented`). FastSync now preserves directory mtimes
(captured by the scanner, transmitted in trailing `STATUS_DIR_TIMES` frame(s),
applied only after all children and the delete/publication phases) and symlink
mtime/owner/mode (no-follow `utimensat`/`fchownat`/`fchmodat` at link creation).
`-O` makes the receiver skip the directory-time set; `-J` makes it skip the
symlink timestamps (ownership/mode application is unaffected and stays governed
by the identity opt-in). Both config booleans already crossed the wire. See the
`-O`/`-J` rows and the Wave D note below.
**-U/-N and -M interaction:** because FastSync carries all metadata (mode, uid,
gid, mtime, and now atime/crtime) in one bounded payload that is only sent when
metadata transmission is on, `-U` and `-N` imply metadata transmission (the
times travel inside that payload). They do **not** enable ownership application,
which remains opt-in strictly through the identity flags (`--numeric-ids` /
`--usermap` / `--groupmap` / `--chown`).
**Phase-4 identity notes:** `--numeric-ids`, `--usermap`, `--groupmap`, and
`--chown` are real. They introduce a **controlled, opt-in, privilege-gated**
ownership-application path on the receiver: plain `-M`/`--preserve` still does
NOT apply client-supplied ownership (FastSync's deliberate conservative
default, byte-for-byte backward compatible); ownership is only attempted once a
client explicitly requests an ownership-affecting option. Application goes
through an fd-relative `fchown()` in the receiver's metadata-restore path (after
the file is fully written, before timestamps are set), so it is confined and
symlink-safe — never a path-based `chown`. When the receiver lacks permission
(typically non-root, e.g. the CI `nobody` user) `EPERM`/`EACCES` is logged as a
warning and the transfer CONTINUES with exit status success, matching rsync.
A no-op default means existing transfers are unaffected.
Resolution of the destination uid/gid on the receiver: a matching
`--usermap`/`--groupmap` rule wins; else the matching `--chown` side; else, with
`--numeric-ids`, the transmitted numeric id is used raw (no name lookup); else a
best-effort name lookup on the receiver's own account databases (skipped when
the transmitted id has no name present there). `--chown` enforces the receiver
side and is validated at parse time (malformed specs are clear errors, never a
silent no-op).
Wire/version: the config frame gained `numeric_ids`, `chown_uid_set`,
`chown_uid`, `chown_gid_set`, `chown_gid`, and the `usermap`/`groupmap` tables
(count-delimited lists of resolved int32 FROM/TO id pairs), so
`PROTOCOL_VERSION` was bumped **2.10.0 → 2.11.0** (peers must match). All new
fields cross `config_send`/`config_receive` with full symmetry and are validated
on receive (bounded map sizes below `MAX_IDENTITY_MAP`, ids `>=` the `-1`
sentinels).
Documented divergences from rsync: because FastSync transmits only numeric
uid/gid (not names) on the wire, name-based values (`--usermap`/`--groupmap`
names, `--chown` names) are resolved to numbers at CLI parse time against the
**client (sender) machine's** account databases; this reproduces rsync's
semantics on a shared-account source/destination and is documented for a
genuinely different destination. The interesting named-value subset is
supported (`*` FROM wildcard, `*` TO = current user, `@N`/bare-`N` numerics); a
lone-`@` "use the FROM value unchanged" rsync form is not implemented. Also
unlike rsync, plain `-M` never applies ownership and `--usermap`/`--groupmap`/
`--chown` each imply metadata preservation so the source uid/gid actually travel
(the flags only take effect where ownership is being preserved/applied).
**Phase-4 hard-links notes:** `-H`/`--hard-links` is real and introduces a
deduplicating wire path for files whose source entries share a filesystem inode.
On the sender, the scanner records each distinct `(st_dev, st_ino)` encounter and
assigns it a stable, run-local link-group id (`HardLinkTable`, mutex-guarded so a
multi-threaded scan could share one instance). The FIRST member of a group is
transferred normally and carries the data; each later (sibling) member is
transmitted as a payload-less `STATUS_HARDLINK` frame carrying its destination
path, the group id, and the first member's destination-relative wire path.
Ordering is guaranteed by forcing the sequential scanner whenever `-H` is on
(even under `-j`/`--threads`), so the first member is always emitted — and, on the receiver's
single write thread, installed — before any of its siblings; the receiver is
therefore always able to link to an already-present first member, including the
"first member already up-to-date/skipped" case (the sibling links to or copies
the existing file). Asymmetric existence policies are handled gracefully: under
`--existing`, if the first member's destination is absent (so it is skipped) but
a sibling's own destination already exists, that existing sibling is left in
place rather than the transfer aborting on the missing first member. The receiver
installs each sibling beneath its confined root
as an atomic hard link (temp link + rename); when `link()` fails (cross-device,
filesystem refuses links) it falls back to a byte-identical local copy of the
first member, never a partial/corrupt file. `--delay-updates` stages each sibling
as a hard link to the first member's STAGED file, so publication's renames
preserve the shared inode; `--inplace` and `--partial` are unaffected (a sibling
is a fresh link/copy). Because a hard link shares an inode, metadata is applied
exactly once on the first member and never re-written through the sibling (whose
members are byte-identical by construction), so all members agree.
Wire/version: `PROTOCOL_VERSION` was bumped **2.11.0 → 2.12.0** (peers must
match). The config frame already carried the `preserve_hard_links` boolean
(round-trips through `config_send`/`config_receive`); the only new wire element
is the `STATUS_HARDLINK` frame described above. Incompatibilities (rejected up
front with a distinct error on the client, and re-checked on receive): `-H` with
`-s` chunk serialization (the chunk wire has no per-file hard-link info) and `-H`
with `--append`/`--append-verify` (a payload-less sibling cannot be tail-resumed).
**Phase-4 devices notes:** `--devices`, `--specials`, `-D`, `--copy-devices`,
and `--write-devices` are new. They change the wire: the config frame grows three
booleans — `preserve_specials`, `copy_devices`, `write_devices` — that CROSS the
wire (`preserve_devices` already existed), and a new `STATUS_SPECIAL` frame (used
by `--devices`/`--specials`/`-D`) carries a special/device entry: the destination
path, the metadata frame (whose mode's S_IFMT bits carry the node kind, requiring
the flags to imply metadata transmission), and two int32 `rdev` major/minor
fields. The chunk-serialized wire (`-s`) grows a matching per-file special
marker + rdev so `--devices/--specials` also work under `-s`. `PROTOCOL_VERSION`
was bumped **2.12.0 → 2.13.0** (peers must match, exactly as prior phases did).
**Privilege gating (the crux):** making a device node requires `CAP_MKNOD` (root).
CI runs the integration suite as a NON-ROOT user (via setpriv), so `mknod` fails
with `EPERM`. The receiver treats this as a graceful, logged *skip of the entry*
returned as a success/skip outcome — the whole transfer NEVER aborts just because
the environment cannot create the node. `mkfifo` (FIFOs) is unprivileged, so
`--specials` FIFO creation is a real, assertable behavior under CI; sockets cannot
be recreated by any standard filesystem call and are skipped with an explicit
note. The "device actually created" integration assertions are guarded to run
only as root. User-facing expectation: point `--devices` at devices and a
non-root receiver will faithfully skip them while transferring everything else.
**Confinement & validation:** a special/device node is created with
`mknodat`/`mkfifoat` on the parent directory opened fd-relative below the receive
root (`file_open_secure_parent`: `O_NOFOLLOW`, no `..` components, root-checked),
so a node can never be created outside the authorized destination root and never
through a symlinked parent. The transmitted type is derived ONLY from the
validated S_IFMT bits of the metadata mode (char/block/FIFO honored, socket
skipped, regular/dir rejected as an invalid special), and the transmitted rdev is
validated both on the wire (`file_receive_special`, `chunk_deserialize`) and at
the creation site (`file_special_rdev_valid`): a negative, oversize, or
non-device-carrying rdev is rejected outright (receiver aborts the frame), and a
node is never replaced over an existing directory or unrelated entry (a matching
existing node is left in place). `--write-devices` is the deliberately restricted
danger path: it only ever opens an existing char/block node under the confined
root, and every failure mode (missing, non-device, write error, EPERM) is a
warning + skip, never a system-clobbering write or an abort.
**Documented divergences (honest subset):**
- A device entry the receiver cannot create (missing `CAP_MKNOD`) is *skipped*,
not a transfer failure — rsync under the same conditions would error.
- `--copy-devices` copies the device's *reported size* (typically 0 for char
devices/FIFOs) into a regular file and never reads an unbounded pseudo-device;
this is the safe, non-hanging alternative to rsync's dd-like read.
- `--write-devices` requires the device to already exist at the destination and
never creates it; unsupported/inaccessible targets are skipped, not written.
- Ownership is not applied to recreated nodes (identity `fchown` needs an fd and
would require opening the node); permissions and mtime are applied at
creation / via `utimensat`.
## 9. Symlink Handling
| Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------|
| `-l`, `--links` | Copy symlinks as symlinks | ⚠️ Partial | Scanner includes symlinks; target path not transmitted |
| `-l`, `--links` | Copy symlinks as symlinks | ✅ Implemented | A symlink is transmitted as a real symlink: its target string crosses the wire (a new `STATUS_SYMLINK` frame / chunk entry type) and the receiver creates it with `symlinkat` beneath the receive root. This makes the previously-`-l`-included-but-targetless symlink handling complete. See the Phase-4 symlink-trust notes |
| `-L`, `--copy-links` | Transform symlink to referent | ✅ Implemented | `copy_links` config field |
| `--copy-unsafe-links` | Transform unsafe symlinks | ✅ Implemented | `copy_unsafe_links` config field |
| `--safe-links` | Ignore symlinks outside tree | ✅ Implemented | `safe_links` config field |
| `--munge-links` | Munge symlinks for safety | ❌ Not Implemented | |
| `-k`, `--copy-dirlinks` | Transform symlink to dir | ❌ Not Implemented | |
| `-K`, `--keep-dirlinks` | Treat symlinked dir as dir | ❌ Not Implemented | |
| `--munge-links` | Munge symlinks for safety | ✅ Implemented | Sender rewrites each transmitted symlink target with a `#SYMLINK/` marker; a target that could escape the receive root (absolute or containing `..`) is never transmitted (contained/skipped); the receiver strips the marker to restore the real target. See the Phase-4 symlink-trust notes |
| `-k`, `--copy-dirlinks` | Transform symlink to dir | ✅ Implemented | A symlink whose referent is a directory is dereferenced and recursed as a real directory; a symlink to a regular file stays a symlink. Sender-side only. See the Phase-4 symlink-trust notes |
| `-K`, `--keep-dirlinks` | Treat symlinked dir as dir | ✅ Implemented | On the receiver, an existing destination symlink-to-a-directory is used as that directory (followed) instead of being replaced; it is followed only when it resolves to a directory that stays beneath the receive root. See the Phase-4 symlink-trust notes |
**Phase-4 symlink-trust notes:** `-l/--links`, `-k/--copy-dirlinks`,
`-K/--keep-dirlinks`, and `--munge-links` form the "symlink trust boundaries"
row. Making all three new flags have an observable, security-sane effect
required transmitting symlink targets, so FastSync's `-l/--links` is now real:
a symlink-type entry carries its target on the wire (a new `STATUS_SYMLINK`
frame for the per-file path, and a new entry type `2` in the `-s` chunk
serializer) and the receiver creates it with `symlinkat` under an `O_NOFOLLOW`
parent walk, never following the target. Wire changes: `STATUS_SYMLINK`,
the chunk entry type `2`, a per-entry symlink-target string, and two new config
booleans that CROSS the wire — `munge_links` and `keep_dirlinks`; `PROTOCOL_VERSION`
was bumped **2.12.0 → 2.13.0** (peers must match, exactly as prior phases did).
**Per-flag semantics and divergences.**
- **`-l/--links`** copies a symlink as a symlink: the scanner `readlink`s the
target, the sender transmits it, and the receiver `symlinkat`s it. FastSync
`-l` never preserved symlink targets before (the flag was documented partial
and, in fact, tried to read the referent as file data); it now does, matching
rsync. Divergences: because the receiver enforces the symlink containment
predicate unconditionally, a plain `-l` sync **refuses to round-trip a
legitimate absolute symlink target** (it is dropped, never created pointing
outside the root — see the `--munge-links` note for the symmetric trust
boundary); a relative in-root target is copied as-is. As of P7 Wave D FastSync
also applies the symlink's own metadata with no-follow primitives
(`utimensat`/`fchownat`/`fchmodat` with `AT_SYMLINK_NOFOLLOW`), so `-J` is a
real omit switch rather than a no-op.
- **`-k/--copy-dirlinks`** (sender): a symlink whose referent is a directory is
dereferenced and recursed into as a real directory; a symlink to a regular
file (or any non-directory) is kept as a symlink. This is rsync's `-k`. When
`-L/--copy-links` or `--safe-links`/`--copy-unsafe-links` are active, their
(dereference) semantics take precedence, so `-k` is subsumed exactly as in
rsync.
- **`-K/--keep-dirlinks`** (receiver, crosses the wire): when a directory is to
be created (on-demand parent creation for a child write) and the destination
path is already an existing symlink that resolves to a directory *within* the
receive root, that symlinked directory is used (followed) instead of being
replaced by a real directory; new entries are written beneath it. The follow
is confined: it only happens where `realpath` of the symlink resolves to a
still-within-root real directory, so a malicious link pointing outside the
root is never followed. Scope: `-K` acts on the write path (parent/`mkdir`
creation); the delete walker still never follows symlinks (a documented
divergence for `--delete` over an existing symlinked dir). Without `-K` the
destination symlink is not followed (the O_NOFOLLOW walk fails the write),
which is the safe default.
- **`--munge-links`** (sender security rewrite; crosses the wire so the receiver
unmunges): every transmitted symlink target is prefixed with the marker
`#SYMLINK/`; the receiver strips the marker (only when the negotiated
`munge_links` policy is on — a plain `-l` run never strips the prefix, so a
source symlink that genuinely begins with `#SYMLINK/` round-trips verbatim)
and restores the exact real target. The trust boundary is **symmetric and
enforced receiver-side**, independent of the sender: `file_symlink_at_secure`
refuses any target that `file_symlink_target_contained` rejects (absolute
`/...` or relative with a `..` component), and `file_save_to_disk_full`
contains such an entry (skipped) rather than materializing it. A deliberate confinement trade-off: because the receiver
enforces containment unconditionally, a plain `-l` (no `--munge-links`) sync
*refuses to round-trip a legitimate absolute symlink target* — such target is
dropped, never created pointing outside the root. This is a stricter subset of
rsync: rsync stores munged targets on the RECEIVING side and depends on both
ends running `--munge-links`; FastSync additionally enforces the containment
predicate at the receiver regardless of what the sender transmitted. When no
symlink is being transmitted (`-l`/`-k`/`-a` off) `--munge-links` has nothing
to rewrite and is inert. -*K/`--keep-dirlinks` policy is installed per
connection at config-accept (stable for the whole transfer, never racy under
`-j`/`--threads`), and only ever follows an in-root symlink-to-directory.*
**Compatibility (byte-identical when all three are absent):** `-k`, `-K` and
`--munge-links` are opt-in. Without them the scanner's link handling, the wire
frames, and the receiver's writes are unchanged for every other option set, so a
run that previously worked continues to behave identically. `-l/--links` itself
now transmits targets (the prior behavior was broken/partial); its status moved
`⚠️ Partial → ✅ Implemented`.
## 10. Sparse & Device
| Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------|
| `-S`, `--sparse` | Sparse block handling | ⚠️ Partial | Flag is accepted, but full hole preservation is not implemented |
| `--preallocate` | Allocate dest files before writing | ❌ Not Implemented | |
| `-S`, `--sparse` | Sparse block handling | ✅ Implemented | Phase 7 Wave B: real hole preservation with no wire change. The receiver's sparse-aware writer (`write_all_sparse`, next to `write_all` in `src/shared/file.c` and `src/shared/file_store.c`) walks the in-memory file image and emits any all-zero run ≥ 4096 bytes as a hole via `lseek(SEEK_CUR)` (the pre-size `ftruncate` guarantees the offset bookkeeping and logical size), `ftruncate(size)` after the last run pins the final size even with a hole tail. Wired into both the atomic temp+rename store and `--inplace` when `sparse` is set; the non-sparse path is byte-identical to before. **Sparse wins over `--preallocate`** (posix_fallocate is skipped when sparse is set, so the holes are not re-allocated). Interplay note: under `--partial` a retained sparse temp already has the full logical size (trailing content is holes), so `--append`'s "shorter destination" resume does not re-run; the retained file is still valid and a normal re-transfer (or `-W`/delta) repairs it — documented so the combination is never surprising |
| `--preallocate` | Allocate dest files before writing | ✅ Implemented | The receiver preallocates the destination file's full expected space before any data is written, so a transfer that would overflow disk fails fast at allocation time (a clean error, not a half-written file) and the file is laid out contiguously, avoiding fragmentation. Crosses the wire (the config frame carries a `preallocate` boolean; `PROTOCOL_VERSION` bumped **2.10.0 → 2.11.0**, peers must match) so the sender knows the receiver will preallocate and the receiver performs it. **Allocation approach:** `posix_fallocate()` is preferred because it reserves *real* disk blocks (true fail-fast on ENOSPC), falling back to plain `ftruncate()` only when the filesystem reports the allocation is unsupported (`EOPNOTSUPP`/`ENOSYS`); `ftruncate` still extends the logical size so the intent degrades gracefully. **Fallback/error semantics:** `EOPNOTSUPP`/`ENOSYS` → clean fallback to `ftruncate` (best-effort, preallocates the logical size and never fails a transfer on filesystems that lack `posix_fallocate`); a genuine allocation failure (`ENOSPC`/`EDQUOT`/`EFBIG`/…) aborts the file/receive with a distinct `preallocate failed ... transfer aborted` error — it does **not** fall back to a normal non-preallocated write, preserving the fail-fast purpose. **Size-known requirement:** preallocation only runs when the final size is already known up front (the normal regular-file case); unknown-length data is skipped (never failed). **Orthogonality:** applies uniformly across the atomic temp+rename store path, `--inplace`, `--partial`/`--partial-dir`, `--delay-updates` (the staged temp file is preallocated before data flows) and the `--link-dest` copy fallback; it neither implies nor conflicts with `-s`, `--append`, or delta. rsync-divergence: rsync signals that `--preallocate` is ignored with `--sparse`; FastSync gives **sparse precedence** — when both are set, `posix_fallocate` is skipped so the holes the sparse writer creates are not re-allocated (the `ftruncate` presize sizing stays), matching the intent of "sparse wins". See the Phase-4 preallocate notes below |
**Preallocate notes (Phase 4, preallocate wave):** `--preallocate` is implemented as a real receiver-side allocation of the destination file's space before data is written. It is a plain boolean config flag that crosses the wire (serialized in the config frame's selection-options block, mirroring `--inplace`/`--append`/`--force`), so the run requires matching ends: `PROTOCOL_VERSION` was bumped **2.10.0 → 2.11.0** (peers must match or the version check fails). The allocation is performed on the exact destination fd, immediately after it is opened, before any bytes are streamed; `posix_fallocate` (and the `ftruncate` fallback) leave the fd's file offset untouched, so the subsequent data write at offset 0 is unaffected and complete. Because FastSync writes each file's byte payload in one in-memory batch, the "full expected size" is exactly the known `data_size`, which is what gets preallocated. Unknown-length/streamed payloads are skipped rather than failed. A failed allocation logs a distinct `preallocate failed` error and aborts the file (the atomic temp is unlinked, the inplace target is left untrimmed) so the run fails cleanly and never silently degrades to a non-preallocated write — preserving rsync's fail-fast intent on a full disk.
## 11. Checksum & Comparison
| Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------|
| `--checksum` | Skip based on checksum | ✅ Implemented | With `--incremental`, compares xxHash64 content checksums; `-c` remains compression |
| `--checksum-choice=STR` | Choose checksum algorithm | ❌ Not Implemented | xxHash used internally |
| `--checksum` | Skip based on checksum | ✅ Implemented | With `--incremental`, compares per-file whole-file content digests to skip unchanged files. The digest algorithm is `xxh64` with seed 0 by default and is selectable via `--checksum-choice`/`--cc` (xxh64/xxhash or md5) and `--checksum-seed=NUM` (see those rows); `-c` remains compression |
| `--checksum-choice=STR`, `--cc=STR` | Choose checksum algorithm | ✅ Implemented | Real algorithm selection for the per-file whole-file digest used by the `--incremental`/`--checksum` handshake and by the basis-dir content verification. FastSync genuinely supports `xxh64` (the default, exact xxHash64, seeded by `--checksum-seed`) and `md5` (via OpenSSL EVP); `xxhash` is accepted as rsync's spelling of xxHash64. Any other name (md4/sha1/sha256/crc32/none/…) is rejected with a clear error at parse time — never a silent no-op. `--cc` is the alias (`--cc=ALG` and space forms both parse). The algorithm id and seed cross the wire with the config frame, so the receiver hashes its on-disk old file with the SAME algorithm+seed the sender used and both agree on a match; the sender's digest and the receiver's comparison live in the per-file `STATUS_CHECK` handshake, which now carries a length-prefixed, bounded (1..16 byte) digest instead of a fixed 64-bit value, and the receiver pins the received length to the negotiated algorithm's digest length (defense-in-depth: a mismatched/malicious length only forces a safe re-transfer). Note: `md5` is a FIPS-non-approved algorithm, so under an OpenSSL build with FIPS mode enabled `--checksum-choice=md5` fails loudly rather than silently falling back. Protocol/layout: `PROTOCOL_VERSION` bumped **2.9.0 → 2.10.0** (peers must match). Defaults preserve the pre-existing behavior byte-for-byte (xxh64, seed 0). Like rsync, the choice only takes effect where a whole-file digest is actually computed (`--checksum` on, or a basis-dir flag); it does not itself enable `--checksum`. Closely-related divergence: the delta BLOCK strong checksum (§11 delta) stays xxHash32 — `--checksum-choice` selects only the whole-file digest, matching rsync where the per-block checksum is independent of the whole-file checksum choice |
| `--compare-dest=DIR` | Compare dest files relative to DIR | ✅ Implemented | DIR is a receiver-side basis relative to the destination root (confined below it; absolute/`..`/`.` rejected, `//` collapsed and trailing `/` dropped). On the receiver's per-file check (implies `--incremental`) an exact match = same size + mtime (unless `--size-only`; `-I` disables matching) **and** equal xxHash64 of the sender's file; a match suppresses the data transfer. compare-dest never copies: it only skips a file the destination does **not** already hold (sparse destination, rsync parity), and is consulted before the normal delta/full paths. Repeatable; searched in command-line order, first match wins. Divergences: when the destination already holds a *different* version rsync deletes it but FastSync instead transfers the data (keeps the mirror complete; never deletes without `--delete`); attribute-only differences on a match are not re-applied (data is skipped so the sender never sends metadata); content is verified by xxHash64, stricter than rsync's default quick check. Sizing: FastSync's whole-file payload limit is 256 MiB on **every** transfer path (not basis-specific); rsync applies basis dirs to arbitrary sizes, so FastSync refuses a basis run whose source contains a larger file up front with a clear error before any transfer. Wire: a basis-count field is always present on the config frame (protocol 2.9.0, so clients and servers must both be 2.9.0) |
| `--copy-dest=DIR` | Include copies of unchanged files | ✅ Implemented | Same basis rules as `--compare-dest`, but an exact match materializes a **local copy** of the DIR file into the destination (via the normal atomic temp+rename store path, so `--existing`/`--ignore-existing`/`--update`/`--backup`/`--delay-updates` all still apply) instead of transferring data. Repeatable; command-line order = priority. Content is xxHash64-verified before the copy. Divergences: a basis-hit destination keeps the basis file's own mode/uid/gid and mtime (the sender sends no metadata on a skip), so with `--size-only` its mtime can differ from the source and attribute-only differences are copied with the basis attributes rather than rsync's "copy + fix attributes". Requires `--incremental` (implied); incompatible with `-s`. Wire: protocol 2.9.0 |
| `--link-dest=DIR` | Hardlink to files when unchanged | ✅ Implemented | Same basis rules as `--copy-dest`, but an exact match installs an atomic **hard link** to the DIR file (temp hard link + rename) so no data or disk space is used; where the link is impossible (basis on another filesystem, filesystem refuses links) it falls back cleanly to a byte-identical local copy, never a corrupt/partial file. `--delay-updates` stages the link and publishes by rename, so the final entry stays a real hard link. Repeatable (searched in command-line order, first match wins). Content is xxHash64-verified before linking. Divergences and caveats: an already up-to-date destination file is not re-linked to a basis file (only files that would otherwise be written are linked); a link keeps the basis inode's own mode/uid/gid and mtime — metadata is never written through the shared inode (that would mutate the basis file), so a later `--inplace` run that rewrites such a destination path **will mutate the basis snapshot** through the shared inode (use `--copy-dest` when the destination must stay independently writable); with `--size-only` the linked mtime can differ from the source; a `--remove-source-files` source satisfied by a basis dir is treated as skipped and therefore **retained** (never removed); basis dirs are excluded from `--delete`. Requires `--incremental` (implied); incompatible with `-s`. Wire: protocol 2.9.0 |
@@ -273,37 +600,55 @@ do NOT imply `--delete`; without `--delete` they are inert (matching rsync).
| Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------|
| `-z`, `--compress` | Compress file data | 🔀 Alt Arg | Always uses zstd (rsync supports multiple algorithms) |
| `-z`, `--compress` | Compress file data | ✅ Implemented | Always uses zstd (rsync supports multiple algorithms — a documented divergence, selectable via `--compress-choice`). Phase 7 Wave A: `-z` is now the compression short form; `-c` is rsync's `--checksum` |
| `--compress-choice=STR`, `--zc=STR` | Choose compression algorithm | ✅ Implemented | FastSync supports `zstd` and `none` |
| `--compress-level=NUM`, `--zl=NUM` | Set compression level | ✅ Implemented | 1-22, default 5 |
| `--compress-threads=NUM` | Set compression threads | ❌ Not Implemented | |
| `--compress-threads=NUM` | Set compression threads | ✅ Implemented | `compression_threads` config field (client-only; does not cross the wire). Sets the number of worker threads used by the zstd compression pool to NUM (1..64; 0/garbage/oversized rejected up front). Accepted in both `--compress-threads=NUM` and two-argument `--compress-threads NUM` forms. Composes with `-z`/compression; under the `-j`/`--threads` multithreaded pipeline it parallelizes compressed chunk encoding. See test_tcp.py `-z --compress-threads=2` and test_client_cli.c |
| `--skip-compress=LIST` | Skip compress for suffixes | ✅ Implemented | Comma-separated, case-insensitive suffix list; empty list skips none; incompatible with FastSync chunk serialization (`-s`) |
## 13. Connectivity
| Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------|
| `-e`, `--rsh=COMMAND` | Remote shell to use | ❌ Not Implemented | Removed; SSH invokes `ssh` directly |
| `--rsync-path=PROGRAM` | rsync binary on remote | ❌ Not Implemented | Removed; use `--fastsync-server-path` |
| `--port=PORT` | Alternate daemon port | ✅ Implemented | `server_port` config field |
| `--sockopts=OPTIONS` | Custom TCP options | ❌ Not Implemented | |
| `--blocking-io` | Use blocking I/O for remote shell | ❌ Not Implemented | |
| `--outbuf=N\|L\|B` | Set output buffering | ❌ Not Implemented | |
| `--address=ADDRESS` | Bind address for outgoing socket | ❌ Not Implemented | Removed because it had no effect |
| `-4`, `--ipv4` | Prefer IPv4 | ❌ Not Implemented | Removed because it had no effect |
| `-6`, `--ipv6` | Prefer IPv6 | ❌ Not Implemented | Removed because it had no effect |
| `--remote-option=OPT`, `-M` | Send an option only to the remote side | ❌ Not Implemented | `-M` is FastSync's metadata-preservation flag |
| `-e`, `--rsh=COMMAND` | Remote shell to use | ✅ Implemented | `-e`/`--rsh` (and `--rsh=COMMAND`) select the remote-shell program used to build the SSH child argv, overriding the default `ssh`. The command is whitespace-split into the leading argv words so rsync's `-e "ssh -p 2222"` works; the standard `-o` family, an optional `-p` port, `user@host` and the quoted remote command (`fastsync-server --stdio`) follow. Stored in the `rsh_command` config field. **Client-only, never crosses the wire** (it is a launch concern, not a handshake property) |
| `--rsync-path=PROGRAM` | rsync binary on remote | ✅ Implemented | Alias for `--fastsync-server-path`: both write the `fastsync_server_path` config field used as the remote-side server program (always quoted as one remote-shell word), which CROSSES the wire as before. Kept separate from `--rsh`, which names the local connecting program |
| `--port=PORT`, `--port PORT` | Alternate daemon port | ✅ Implemented | rsync's daemon-port flag is an alias for `--server-port`: both spellings (and `--server-port=PORT`) map to the client-side `server_port` config field. The client connects to a TCP/TLS server (incl. `host::module/path` daemon destinations) on that port, and the `fastsync-server --daemon` listener's port is taken from its config's `port` key (default 873) or overridden by `--dparam port=` / `-p` |
| `--sockopts=OPTIONS` | Custom TCP options | ✅ Implemented | Comma-separated allowlist of `OPT=VAL` applied via `setsockopt` after `socket()` before `connect()`/`bind()`. Only `TCP_NODELAY`, `SO_KEEPALIVE`, `SO_REUSEADDR` (0/1) and `SO_RCVBUF`/`SO_SNDBUF` (byte count) are accepted; an unknown option name or a bad value is rejected up front, never silently ignored. A value is required for every option (`OPT=VAL`; a bare name is an error). Applied to the outgoing TCP and TLS client socket; absent by default. `SockOptEntry`/`sockopts` config fields. Local socket concern: never crosses the wire |
| `--blocking-io` | Use blocking I/O for remote shell | ✅ Implemented | With `--blocking-io` the SSH-transport socketpair socket is left without `SO_RCVTIMEO`/`SO_SNDTIMEO`, so the transfer blocks naturally; by default it gets the same read/write timeout as the TCP transport (see `--timeout`). `blocking_io` config bool. **Client-only, never crosses the wire** |
| `--outbuf=N\|L\|B` | Set output buffering | ✅ Implemented | `N` (none/unbuffered) → `_IONBF`, `L` (line) → `_IOLBF`, `B` (block, the default) → `_IOFBF` via `setvbuf` on stdout and stderr. Garbage values are rejected. `outbuf` config field (`OutbufMode`). **Client-only, never crosses the wire** |
| `--address=ADDRESS` | Bind address for outgoing socket | ✅ Implemented | Binds the outgoing client socket to a local source address before `connect()` (resolved with the same `-4`/`-6` family hints as the destination). Local socket concern: never crosses the wire |
| `-4`, `--ipv4` | Prefer IPv4 | ✅ Implemented | Forces `AF_INET` in the `getaddrinfo` hints for client destination/source resolution and the server bind (see the Phase 5, Wave B note). Mutually exclusive with `-6` |
| `-6`, `--ipv6` | Prefer IPv6 | ✅ Implemented | Forces `AF_INET6` in the `getaddrinfo` hints for client destination/source resolution and the server bind. Mutually exclusive with `-4` |
| `--remote-option=OPT`, `-M` | Send an option only to the remote side | ✅ Implemented | Each value is appended to the remote server invocation over SSH as an individually single-quote-escaped shell word in `ssh_build_remote_command()`. Values are validated (non-empty, no control characters) and shell metacharacters cannot break out of the quoting (`;`, `&`, `|`, <code>`</code>, `$`, `(`, `)`, quotes are neutralized), so a value cannot inject an arbitrary remote command and a subsequent `--` on the client line cannot be turned into one. The options never cross the binary config frame. Phase 7 Wave A: the short `-M` form is now available (as `-M OPT` and `-M=OPT`), matching rsync; metadata mode moved to long-only `--preserve` |
## 14. Daemon Mode
| Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------|
| `--daemon` | Run as rsync daemon | ❌ Not Implemented | Removed because it had no effect |
| `--config=FILE` | Alternate rsyncd.conf file | ❌ Not Implemented | Removed because it had no effect |
| `--dparam=OVERRIDE` | Override global daemon config | ❌ Not Implemented | |
| `--no-detach` | Don't detach from parent | ❌ Not Implemented | |
| `--password-file=FILE` | Read daemon password from file | ❌ Not Implemented | |
| `--early-input=FILE` | Use FILE for daemon early exec | ❌ Not Implemented | |
| `--daemon` | Run as rsync daemon | ✅ Implemented | Wave A: a real persistent listener. `fastsync-server --daemon --config FILE` (plus `--no-detach` to stay foreground; without it the listener detaches to the background after binding) reads a FastSync-native module config file and serves each connection confined to the requested module's `path` root (never a client-chosen root; every client-chosen-ownership/super-user request (`--numeric-ids`/`--chown`/`--usermap`/`--groupmap`/`--fake-super`/`--copy-as`/explicit `--super`) is refused unless the module opts in with `client owner = yes`, and the operator `--no-super` veto is honored). TCP/TLS via the existing `--tls` stack; plaintext still requires `--allow-unauthenticated` (same secure default as the standalone server). Client destinations use rsync's `host::module/path` form. Wire/protocol: the config frame gained a trailing daemon-module string and `PROTOCOL_VERSION` was bumped **2.14.0 → 2.15.0** (see the Daemon Mode notes below). Daemon mode is built in FastSync's own protocol/config grammar, not rsync's SMB/daemon option encoding |
| `--config=FILE` | Alternate rsyncd.conf file | ✅ Implemented | Wave A: selects the daemon config file. Default when omitted (in `--daemon` mode): `~/.config/fastsync/fastsyncd.conf` if it exists, else `/etc/fastsyncd.conf`. The grammar is FastSync-native (documented in the Daemon Mode notes below) and strictly rejects unknown keys so a typo can never silently change what a module serves; requires `--daemon` |
| `--dparam=OVERRIDE` | Override global daemon config | ✅ Implemented | Wave A: overrides one global scalar from the command line (`--dparam port=8734` and `--dparam=KEY=VALUE` both work). Limited to the global keys the grammar defines (`port`, `motd file`, `address`, `max connections`, `max connections per host`, `auth failure delay`, `auth lockout threshold`, `auth lockout duration`, `hosts allow`, `hosts deny`); keys are case-insensitive and unknown keys/invalid values are rejected. Requires `--daemon` |
| `--no-detach` | Don't detach from parent | ✅ Implemented | Wave A: with `--daemon`, keeps the listener in the foreground (what integration tests use). Without it the daemonizes (fork/setsid, stdio redirected to /dev/null) after the listening socket is bound. Requires `--daemon` |
| `--password-file=FILE` | Read daemon password from file | ✅ Implemented | A7 daemon auth. Client: `--password-file` supplies `user:password` for a `host::module/path` destination (the username is taken from this file, so `user@host::module` stays rejected); the literal password is held client-side only for the SCRAM handshake and wiped at teardown. Server (`fastsync-server --daemon --password-file FILE`): the salted-PBKDF2 verifier store that modules with `auth users` are verified against. **Neither the password nor any replayable bearer value crosses the wire or is stored server-side** — the store holds a per-user salt plus derived keys, and the daemon proves the secret with a per-connection nonce challenge. The file must be private to its owner: both the client and server verify the exact inode they read (open-then-`fstat`, so the check cannot be raced) and refuse a `--password-file`/`--early-input` that is not owned by the current user or grants any group/other permission bit (mode 0600), mirroring the TLS private-key check. A process-substitution pipe (`--early-input <(vault ...)`) is still accepted when it satisfies those checks. See the Daemon Mode notes below for the file formats and the plaintext/TLS caveat |
| `--early-input=FILE` | Use FILE for daemon early exec | ✅ Implemented | Server-only (requires `--daemon`): a second credential-store file, same new-format grammar as `--password-file`, read before the listener accepts connections (a secrets-manager / process-substitution source). Its entries layer over `--password-file`: byte-identical verifiers dedupe, a conflicting verifier for the same user is a startup error. A daemon whose modules declare `auth users` must be given at least one of the two, or it refuses to start (fail closed) |
| `--hash-credentials=FILE`, `--iterations N` | Hash a plaintext credential file | ✅ Implemented | Server-only offline tool (A7): reads the `user:password` lines of FILE (same owner-only 0600 check) and prints one new-format store line per entry to stdout, then exits. `--iterations` sets the PBKDF2 work factor (default 600000, range 100000–10000000). Dependency-free and does not run a listener. Use its output as `--password-file` for `--daemon`. There is no auto-upgrade: a legacy store line is hard-rejected by the loader and must be regenerated |
**Daemon Mode notes (Wave A protocol 2.15.0; A7 auth protocol 2.19.0; MOTD no bump):** FastSync daemon mode is supported in FastSync's own protocol/config grammar, not rsync's SMB/daemon option encoding.
- **Config grammar** (`fastsyncd.conf`): line-based; an implicit global section first, then `[module]` sections. Keys are case-insensitive, values are trimmed and may be wrapped in one layer of double quotes (`path = "/srv/my dir"`). `#` and `;` at the start of a line (after leading whitespace) are full-line comments; inline comments and `\` continuations are not supported. Lines are bounded (4096 chars), and at most 256 `[module]` sections are accepted. Global keys: `port` (default 873), `motd file` (the daemon sends its bounded, escaped content to a client after the module gate/auth accepts, unless the client passes `--no-motd`), `address` (optional bind address), `max connections` (positive integer cap on concurrent connections, default 100; 0/negative/garbage is a parse error), `max connections per host` (concurrent-connection cap per source IP, default 0 = unlimited), `auth failure delay` (milliseconds to sleep after a failed authentication, default 500; 0 disables, capped at 5000), `auth lockout threshold` (failed authentications from one source before lockout, default 10; 0 disables), `auth lockout duration` (seconds a locked-out source is refused, default 300), `hosts allow` and `hosts deny` (comma- and/or whitespace-separated host access patterns — see the host access control note below). Module keys: `path` (required; the daemon-side authorized root for that module), `read only` (yes/no/true/false/1/0, default no), `client owner` (yes/no/true/false/1/0, default no; opts the module into client-chosen ownership — see below), `auth users` (comma list), `max connections` (optional per-module cap, 0 = unlimited; enforced across all connection children), `hosts allow`/`hosts deny` (per-module host access lists). **Unknown keys and malformed lines are parse-and-reject errors** (never silently ignored), so a typo cannot change what a module serves.
- **Host access control (`hosts allow`/`hosts deny`):** both keys accept a comma- and/or whitespace-separated list of patterns and may appear globally and/or per module (multiple config-file lines append; a `--dparam` override replaces). Supported patterns are `*` (match all), an IPv4 or IPv6 literal (`10.0.0.1`, `2001:db8::1`), and an IPv4/IPv6 CIDR (`10.0.0.0/8`, `2001:db8::/32`). Hostname patterns are **not** supported: because the peer is always a numeric address and no reverse DNS is performed, a hostname/glob pattern would silently never match, so it is rejected at load time (fail-closed) instead of being accepted as a dead rule. An IPv4 peer on a dual-stack IPv6 listener is normalized from its `::ffff:a.b.c.d` form so IPv4 patterns match it. rsync-like semantics: a matching `hosts deny` rejects; if any `hosts allow` entries exist, a peer matching none of them is rejected; deny takes precedence over allow. The daemon enforces the global list first, then the selected module's list, **before authentication** in `server_module_gate`, with an audit log line naming the peer, the module and the outcome. The numeric peer address is obtained with `getpeername`+`inet_ntop` (`utils_fd_peer_ip`, handling both address families); when it cannot be obtained a module with any ACL fails closed (refused), while an ACL-free module continues and logs at debug. A malformed pattern (e.g. an out-of-range CIDR prefix) is a parse error at load time.
- **Connection caps, shared registry and auth lockout:** the global `max connections` key (default 100) is plumbed into the listener (`transport_tcp.c`), which rejects a connection once the accept-loop parent's active-child count reaches it; the IPv4/IPv6 peer is logged for every accepted connection. Because the listener forks one child per connection, the per-module `max connections` cap, the global `max connections per host` cap, and the auth-failure counter live in a fixed-size registry carved from an anonymous shared mapping (`daemon_limits.c`, `mmap(MAP_SHARED|MAP_ANONYMOUS)`) created by the parent before the accept loop, so every forked child shares the same counters (C11 atomics only — never a pthread lock, which can deadlock in a forked child). The parent reserves a registry slot per accepted connection and the child records the selected module and source IP once known; the parent's `SIGCHLD` handler reclaims the slot when the child dies (including `SIGKILL`) and re-derives the per-module and per-source occupancy counts from the surviving REGISTERED slots, so a child killed mid-registration cannot leak a count. The per-source table has a bounded lifetime: an entry with no live connection is reclaimed after its lockout expires or it has been idle (300 s); if the table is genuinely full the per-source cap/lockout fails open for new sources (per-module cap and ACLs still apply) with a rate-limited warning. The per-module cap (0 = unlimited) is enforced after the module lookup and before auth; per-source identity reuses the normalized numeric peer address (`utils_fd_peer_ip`, IPv4-mapped IPv6 collapsed to IPv4), and a trusted loopback peer (127.0.0.0/8 / `::1`, `utils_fd_peer_is_local`) is exempt from the per-source cap and the auth lockout because all local clients share one address (the per-module/global caps still apply). Clients behind a shared NAT/proxy address likewise share one per-source budget and lockout counter. A failed authentication increments the shared per-source failure count and, once `auth lockout threshold` (default 10; 0 disables) is reached, the source is refused for `auth lockout duration` seconds (default 300) before any challenge is sent, even when the next attempt is handled by a different forked child; a successful authentication clears the counter. On a failed authentication the per-connection child still sleeps the global `auth failure delay` (default 500 ms, 0 disables, capped at 5000) via `nanosleep`, rate-limiting online guessing without delaying a success. A missing registry (allocation failure) degrades to the global cap and host ACLs rather than refusing to start.
- **Module selection & confinement:** the client requests a module with an rsync-style `host::module[/path]` destination. The module name crosses the wire as a trailing string on the config frame (bumping `PROTOCOL_VERSION` 2.14.0 → 2.15.0; the bump is required because the config-frame layout changed and the strict same-version handshake is what prevents a peer from desynchronizing on the new trailing field). The daemon looks the module up in ITS OWN config and uses the module's `path` as the authorized root through the exact same `configure_authorization` confinement the standalone server applies to `--destination-root` (`file_open_secure_parent`, `has_path_traversal`, `path_is_within`); the client never supplies the root, every client-chosen-ownership/super-user request is refused unless the module declares `client owner = yes` (the daemon's per-module opt-in, see below), and the operator `--no-super` veto forces super-user activities off for every daemon connection. The client's `/path` part is relative inside the module and is rejected if absolute or if it contains `..`. Unknown modules are refused before any data moves (the run fails cleanly at the config handshake). An absolute destination and a module request against a non-daemon server are also refused.
- **`client owner` (client-chosen-ownership opt-in):** by default a daemon module refuses every request that would let the client pick an owner or ask for super-user activities — `--numeric-ids`, `--chown`, `--usermap`/`--groupmap`, `--fake-super`, `--copy-as`, and an explicit `--super` — at the config handshake (before `STATUS_OK`), because a daemon has no per-module opt-in for client-chosen ownership and any anonymous client could otherwise force arbitrary owner ids inside the module root. `client owner = yes` opts a single module in, allowing those requests within that module's root (a root standalone TCP listener honors them for its single operator-authorized root only when started with `--allow-super`; the flag is rejected with `--stdio`, whose client-composed remote argv must never opt back into super mode). Without the opt-in the daemon also forces super-user **device** activity off for that connection — char/block device-node creation (`--devices`) and `--write-devices` — even under the default `AUTO` mode, so a non-opted module can never be made to `mknod` or write a raw device; those entries are skipped (not refused) so an ordinary `-a` push still succeeds without device nodes. The opt-in does **not** lift the privilege requirement: `--copy-as` still needs a root receiver, and the operator `--no-super` veto still forces super-user activities off for every connection. The daemon logs a prominent startup warning for each `client owner = yes` module so the operator's deliberate choice is visible.
- **`read only` safe default:** every network transfer FastSync currently supports is a push that writes under the module root, so a `read only` module refuses the connection (clear server log "module is read only"; the client exits non-zero, nothing is transferred). A future pull/list operation can be opened up when it exists; the knob is already stored.
- **Direction — remote source / pull is intentionally unsupported:** FastSync is push-only. The first positional argument is always a **local** source directory and the second is the destination; only the destination is parsed for remote syntax (`user@host:path` SSH, `host::module[/path]` daemon). A remote source such as `fastsync user@host:src ./local` is deliberately **not** implemented: rsync has no pull flag (direction is positional), so supporting a remote source is an optional feature rather than a compatibility requirement, and it would require a protocol role reversal (server as sender, client as receiver) across both transports. FastSync documents this as an intentional limitation rather than a missing rsync option. <a id="direction"></a>
- **`auth users` (A7 SCRAM-SHA-256 authentication):** a module that declares `auth users` requires the client to present credentials. The config frame carries ONLY the username; the daemon answers an auth-required module with `STATUS_AUTH_CHALLENGE` (PBKDF2 iteration count, 16-byte salt, 32-byte server nonce), the client answers with `STATUS_AUTH_RESPONSE` (fresh 32-byte client nonce + a 32-byte ClientProof), and the daemon accepts only when the proof verifies **and** the username is **on the module's `auth users` list** and has a store entry, replying `STATUS_AUTH_OK` with a 32-byte ServerSignature the client verifies before proceeding. Verification is constant-time over fixed 32-byte keys (the compare runs even for a miss), username membership uses a constant-time full-length scan, and an unknown/off-list user still receives a challenge and runs the same math against a dummy verifier: a deterministic per-username salt (`HMAC-SHA256(store dummy key, username)`), the store-wide uniform iteration count and dummy keys. Re-probing the same unknown username therefore yields an identical salt and iteration count while a different username yields a different salt, so there is no user-enumeration or timing oracle. The daemon logs the username but **never the password, proof or keys**. A module WITHOUT `auth users` stays open (legitimate rsync configuration); credentials sent to such a module are ignored. Read-only is orthogonal: even a correctly authenticated push to a `read only` module is still refused (all FastSync network transfers write). Fail-closed policy: a daemon whose config declares `auth users` on any module refuses to start unless a credential store was given (`--password-file` and/or `--early-input`); a missing or empty store is never silently treated as "open". A failed handshake (missing credentials, unknown/off-list user, wrong proof or malformed data) yields a single generic `STATUS_AUTH_FAILED` and the daemon closes before any data moves. The dummy key is persisted in an owner-only `<store_path>.dummykey` sidecar (auto-created on first load, mode 0600) so the dummy salt stays stable across daemon restarts, closing the restart-gated enumeration channel. The sidecar is secret material and must be protected like the credential store (owner-only 0600, included with the store in backups and rotation). It must be preserved across restarts for that guarantee; if it cannot be created (a process-substitution/FIFO store path such as `/dev/fd/N`, a read-only filesystem, a missing directory, or a create/write/fsync/link/fchmod failure), the daemon logs a warning and uses a transient per-run key, so unknown-user challenges change across restarts and the cross-restart guarantee does not hold for that deployment. One residual is accepted: the store iteration count is observable pre-auth by design, since the miss path must match a hit. **Transport policy (hardening A7-3/S1):** an auth-required module accepts credentials only when either (a) the connection is an encrypted, verified TLS connection whose client certificate matches `--client-cn`, or (b) the connection is plaintext from a loopback TCP peer **and** the operator explicitly passed `--allow-unauthenticated`. A remote plaintext peer, and a loopback plaintext peer without that flag, are refused at the config gate before any challenge is sent; `--allow-unauthenticated` never permits remote plaintext auth (remote peers still require verified TLS). Daemon modules are a `--daemon`-only feature — the SSH `--stdio` path never loads a daemon config and is not an auth transport for them. Because the loopback allowance trusts whichever peer the kernel reports as `127.0.0.1`, it assumes nothing relays remote connections to the daemon: a local TCP forwarder or TLS-terminating proxy in front of an auth-module listener makes remote clients appear as loopback and bypasses the mutual-TLS identity check, so do not front an auth-module listener with such a relay.
- **Credential store format:** server `--password-file`/`--early-input` files are line-based `user:$fastsync$1$pbkdf2-sha256$<iters>$<salt_b64>$<stored_key_b64>$<server_key_b64>`, one per line (standard base64; 16-byte salt, 32-byte keys; `iters` in `[100000, 10000000]`, default 600000). Every entry in the resulting store must agree on `iters` (a store whose entries disagree, or where a layered `--early-input` disagrees with `--password-file`, is rejected). Generate lines with `fastsync-server --hash-credentials FILE [--iterations N]`; the emitted lines are secret material, so redirect them to an owner-only (mode 0600) file (the tool warns on stderr if stdout is a group/other-accessible regular file). Blank lines and lines starting with `#`/`;` are comments; the parser is strict (a malformed line fails the whole load, so a typo can never let a different set of users in). **The legacy `user:SHA256HEX` form is hard-rejected** with an actionable "legacy" error; there is no auto-upgrade, so a replayable bearer digest can never be loaded by a 2.19.0 daemon. The client `--password-file` holds `user:password` on its first meaningful line (the literal password, used only for the handshake then burned); keep both files readable only by their owner (mode 0600). Per-username wire length is bounded (256 chars) and every decoded salt/key length is validated. Loading the store also maintains an owner-only `<store_path>.dummykey` sidecar (auto-created, mode 0600, exactly 32 bytes) holding the store-wide dummy key that shapes unknown-user challenges; persist it across daemon restarts so those challenges stay stable, and treat a sidecar with the wrong owner, a mode other than exactly 0600, the wrong size or the wrong type as a fatal load error (fail closed). If the sidecar cannot be created (e.g. a process-substitution store path such as `/dev/fd/N`, a read-only filesystem, a missing directory, or a create/write/fsync/link/fchmod failure), the daemon logs a warning and uses a transient per-run key, so the cross-restart stability guarantee does not hold there.
- **Plaintext caveat:** an auth-required module is refused, **before any challenge is sent**, unless the connection is encrypted and verified TLS whose client certificate matches the server's `--client-cn`, or it is plaintext from a loopback TCP peer **and** the operator passed `--allow-unauthenticated`. A remote plaintext peer, and a loopback plaintext peer without that flag, never receive a challenge, and `--allow-unauthenticated` never permits remote plaintext auth (remote peers still require verified TLS). On the loopback plaintext transport that remains permitted, a local sniffer could still read the challenge and response and mount an **offline dictionary attack** against a weak password, so use `--tls` for any real deployment. `--client-cn` matches the certificate CN only (not a subjectAltName), which is acceptable for a private CA. Clients sending daemon credentials with `--password-file` to a non-loopback daemon must use `--tls`; the client rejects such a destination before any network I/O. Unlike the old challenge-less exchange there is **no replay**: the proof is bound to the fresh per-connection server nonce, so a captured `STATUS_AUTH_RESPONSE` cannot be reused on another connection (an integration test proxies the daemon and proves this). TLS client-CN (`--client-cn`) is an independent transport identity check and composes with password auth; because `--tls` already mandates `--client-cn`, a TLS auth connection always verifies the client CN, so both checks necessarily apply together on such a connection.
- **Wire/protocol:** the config-frame auth block is now `[int present][str_redacted username]` (the old digest field is gone), and the frame stream gains the challenge/response (`STATUS_AUTH_CHALLENGE` → `STATUS_AUTH_RESPONSE` → `STATUS_AUTH_OK`/`STATUS_AUTH_FAILED`) between the config frame and the `STATUS_OK` ack. Both are wire-layout changes, so `PROTOCOL_VERSION` is bumped **2.18.0 → 2.19.0** (see the A7 note in `src/shared/config.h`); the strict same-version handshake keeps a 2.19 client and a 2.18 server from desynchronizing.
- **Client side:** `host::module/path` selects the TCP transport and connects to `--server-port`; `host:path` stays the SSH transport; plain paths stay local TCP. The daemon username comes from `--password-file` (first `user:password` line), and `--password-file` without a `host::module/path` destination is a client error (fail fast). A `user@host::module` form is rejected with a pointer to `--password-file`. The client's plaintext password is wiped from memory (`config_burn_auth`) at transfer teardown.
- **MOTD (Wave C):** a daemon configured with a global `motd file` sends that file's content as the first server→client string frame after the config-frame STATUS_OK ack (rsync sends the MOTD as the first thing from the server at the start of a daemon connection). Only the daemon listener path (`host::module`) gets a MOTD; the `--stdio` SSH path never sends or reads one. The server reads the file bounded to 4096 bytes and treats an absent/unreadable file as "no MOTD" (an empty frame, never an error). The exchange is server→client only and does **not** bump `PROTOCOL_VERSION`: every 2.15.0 daemon client reads the frame after the ack, so sender and receiver stay in lockstep (see the Wave C note in `src/shared/config.h`). `--no-motd` is the client-side suppression switch: the client still reads (consumes) the frame to keep the stream in sync but does not display it. The MOTD is printed to stdout with control bytes (ESC included) escaped octal-style while newlines/tabs are preserved, so a hostile server cannot inject terminal escape sequences.
- **Merge note:** the Wave A module bump (2.15.0) and the MOTD wave did not bump the version, but the A7 auth redesign is a genuine wire-layout change and owns the 2.18.0 → 2.19.0 bump (see the A7 note in `src/shared/config.h`).
## 15. Safety & Security
@@ -315,36 +660,40 @@ do NOT imply `--delete`; without `--delete` they are inert (matching rsync).
| Max data/string/chunk sizes | Prevent OOM attacks | ✅ Implemented | Per-message limits |
| Per-connection memory limit | 1GB per connection | ✅ Implemented | `MAX_CONNECTION_MEMORY` |
| `--max-alloc=SIZE` | Limit a single memory allocation | ✅ Implemented | Caps the largest single allocation; binary units, default 1G |
| `--trust-sender` | Trust remote sender's file list | ❌ Not Implemented | |
| `--old-args` | Disable modern arg protection | ✅ Implemented | SSH-only legacy mode; restores raw remote command construction and permits shell interpretation of the configured server path |
| `--trust-sender` | Trust remote sender's file list | ✅ Implemented | Long-form-only, receiver-local policy that never crosses the wire. The receiver skips its redundant up-front re-validation of the incoming file list (empty/`..` path rejection and the escaping-symlink-target containment), trusting the sender instead of double-checking (fewer checks, faster, potentially unsafe, matching rsync). Off by default. The low-level fd-relative confinement primitives (`file_open_secure_parent`, the O_NOFOLLOW parent walk, leaf/destination confinement) are deliberately KEPT even under `--trust-sender`, so a hostile sender still cannot write or link outside the authorized root (see Phase-5 notes below) |
| `--old-args` | Disable modern arg protection | ✅ Implemented | SSH-only; accepted for CLI compatibility but is now a **documented no-op**: FastSync always single-quote-escapes the remote server path and each `--remote-option` value (`ssh_build_remote_command`), so a metacharacter-bearing `--rsync-path` can never be interpreted by the remote shell. The flag no longer disables that quoting (the old raw-construction behavior was an injection foot-gun and is removed); the safety-relevant behavior is identical either way |
| `--ignore-missing-args` | Ignore missing source args | ✅ Implemented | FastSync has a single source-root argument (which always exists), so the "explicitly requested source arguments" are the `--files-from` entries and the flags only ever apply there (inert without `--files-from`, like `-R`). Without the flag a listed-but-missing entry stays a hard pre-transfer error (nothing is transferred). With it each missing entry is skipped: nothing is sent for it, it never enters the keep-set, and the run succeeds for the rest — an all-missing non-empty list succeeds transferring nothing, matching rsync. `--dirs` + `--files-from` missing entries are skipped the same way. Every skipped entry is logged and a per-run warning names the count, so the handling is never a silent no-op. Divergences: an EMPTY `--files-from` file stays a hard error in every mode (no argument was requested at all; rsync likewise reports "no source files specified"); missing-arg skipping only applies to the pre-transfer list validation, so an entry that is present at preflight and vanishes mid-transfer still fails (matching rsync, whose flag "does not affect subsequent vanished-file errors"); `--no-ignore-missing-args` is not a supported negation |
| `--delete-missing-args` | Delete missing source args | ✅ Implemented | Implies `--ignore-missing-args` (order-independent) and additionally removes each missing entry's destination mirror receiver-side. The mirror is computed exactly like a present sibling's wire path: the bare relative entry under `-R`, otherwise the full source-mirror path below the destination root. rsync parity, verified against the man page: it does **not** imply `--delete` generally and is "independent of any other type of delete processing" — unrelated destination extras are untouched unless `--delete` is also present. Composition with `--delete` + timing: the exact-path deletions commit with the manifest, early for `--delete-before`/`--delete-during`, else only after a fully-successful transfer (delete-after/commit). A non-empty directory mirror is removed only when `--force` or `--delete` is in effect (otherwise it is left with a warning and the run continues, like rsync); an absent mirror is a no-op. An explicitly listed missing arg is a user request, not an excluded file: its deletion is never blocked by the filter-exclusion protection of excluded destination mirrors (a mirror sitting inside a filter-excluded directory is still removed). Safety/policy: gated by the server `--allow-delete` policy like `--delete`; the request paths cross the wire only in the delete-manifest frame and are confined by the same receiver validation as the keep-set (non-empty, relative, traversal-free, bounded by the per-section/per-frame manifest caps); the `--delay-updates` staging directory and basis snapshots are protected exactly as in the extras walker. Divergence: the missing-args deletions are not counted toward `--max-delete` (they are explicit per-path requests, not discovered extras). See the Phase-3 wire note below for the `PROTOCOL_VERSION` bump |
| `--delete-missing-args` | Delete missing source args | ✅ Implemented | Implies `--ignore-missing-args` (order-independent) and additionally removes each missing entry's destination mirror receiver-side. The mirror is computed exactly like a present sibling's wire path: the bare relative entry under `-R`, otherwise the full source-mirror path below the destination root. rsync parity, verified against the man page: it does **not** imply `--delete` generally and is "independent of any other type of delete processing" — unrelated destination extras are untouched unless `--delete` is also present. Composition with `--delete` + timing: the exact-path deletions commit with the manifest, early for `--delete-before`/`--delete-during`, else only after a fully-successful transfer (delete-after/commit). A non-empty directory mirror is removed only when `--force` or `--delete` is in effect (otherwise it is left with a warning and the run continues, like rsync); an absent mirror is a no-op. `--force` is deletion authority and is therefore gated by the server `--allow-delete` policy exactly like `--delete`/`--delete-missing-args`: without it the receiver clears the flag, so a client cannot use `--force` to recursively replace or remove a destination directory tree. An explicitly listed missing arg is a user request, not an excluded file: its deletion is never blocked by the filter-exclusion protection of excluded destination mirrors (a mirror sitting inside a filter-excluded directory is still removed). Safety/policy: gated by the server `--allow-delete` policy like `--delete`; the request paths cross the wire only in the delete-manifest frame and are confined by the same receiver validation as the keep-set (non-empty, relative, traversal-free, bounded by the per-section/per-frame manifest caps); the `--delay-updates` staging directory and basis snapshots are protected exactly as in the extras walker. Divergence: the missing-args deletions are not counted toward `--max-delete` (they are explicit per-path requests, not discovered extras). See the Phase-3 wire note below for the `PROTOCOL_VERSION` bump |
## 16. Batch Operations
| Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------|
| `--write-batch=FILE` | Write batched update to file | ❌ Not Implemented | |
| `--only-write-batch=FILE` | Write batch without updating dest | ❌ Not Implemented | |
| `--read-batch=FILE` | Read batched update from file | ❌ Not Implemented | |
| `--write-batch=FILE` | Write batched update to file | ✅ Implemented | Phase-6 residual-batch (client-only): runs the normal live transfer AND additionally emits a self-contained single-file batch of the whole source tree. The batch is a magic/format-version header followed by length-prefixed `chunk_serialize` blobs (full file images), replayable byte-identically by `--read-batch` on another machine with no source/server. `--write-batch` drives the single-threaded transfer path (the multithreaded path consumes the config before the separate batch scan pass). See the Phase-6 batch note below |
| `--only-write-batch=FILE` | Write batch without updating dest | ✅ Implemented | Phase-6 residual-batch: emits the self-contained batch FILE only — NO destination update, NO server connection. Requires a source (scans it and serializes the full tree to FILE). Same single-file format as `--write-batch`, so the file is re-appliable via `--read-batch=FILE DEST`. See the Phase-6 batch note below |
| `--read-batch=FILE` | Read batched update from file | ✅ Implemented | Phase-6 residual-batch: applies a previously written batch FILE locally to the destination. NO source and NO server — positional args are the destination only. Reads the magic/version header, then length-prefixed records, `chunk_deserialize`, and applies each via the confined `file_save_to_disk_full` path (same O_NOFOLLOW / `..`-rejection / root-confinement as the network receiver, so an attacker-controlled batch cannot escape the destination root). Malformed/truncated/oversized/traversal records are rejected cleanly. See the Phase-6 batch note below |
## 17. Advanced
| Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------|
| `--stop-after=MINS` | Stop after N minutes | ❌ Not Implemented | |
| `--stop-at=TIME` | Stop at specified time | ❌ Not Implemented | |
| `--stop-after=MINS` | Stop after N minutes | ✅ Implemented | Client-only sender stop deadline (Phase 6): computing `--stop-after=MINS` (a positive minute count; 0/negative/garbage rejected) and `--stop-at=TIME` (`HH:MM`, `HH:MM:SS`, or `now+N[smhd]`; a past time stops immediately). The transfer stops ELEGANTLY at the next chunk boundary: everything already fully sent is kept and applied, the run returns 0, and --delete (late/delete-after timing) does NOT wipe the destination — when the scan is cut short the partial keep-set manifest is suppressed with a warning (the delete walk is skipped rather than acting on an incomplete keep-set, so unscanned source mirrors survive). `--delete-before`/`--delete-during` still run their complete pre-scan (which ignores the deadline). Local client-only fields: never serialized into the wire config frame, so no PROTOCOL_VERSION bump. `--stop-after` uses CLOCK_MONOTONIC; `--stop-at` uses the wall clock. Works single-threaded and under `-j`/`--threads` (multithreaded). Divergence: rsync computes `--stop-after` from the run start; FastSync likewise. When both are given, the earlier of the two deadlines wins (checked per iteration). See the Phase-6 stop notes below |
| `--stop-at=TIME` | Stop at specified time | ✅ Implemented | Same feature as `--stop-after` (deadline transfer stop), absolute wall-clock form (`HH:MM[:SS]` or `now+N[smhd]`). See the row above and the Phase-6 stop notes |
| `--fsync` | Fsync every written file before publication | ✅ Implemented | |
| `--protocol=NUM` | Force older protocol version | ❌ Not Implemented | |
| `--iconv=CONVERT_SPEC` | Charset conversion | ❌ Not Implemented | |
| `--checksum-seed=NUM` | Set checksum seed | ❌ Not Implemented | |
| `--secluded-args` | Use protocol to send args | 🔄 Compatibility No-op | Accepted for CLI compatibility; it does not change FastSync transport or protocol behavior. `-s` remains chunk serialization. |
| `--protocol=NUM` | Force older protocol version | ✅ Implemented | Forces the wire protocol version for this transfer. FastSync has exactly ONE wire format (`PROTOCOL_VERSION`, currently 2.21.0) with no downgrade/backward-compat code paths, so `--protocol=2.21.0` is accepted (it sets the version claim the client sends, which the server already requires to match exactly) and **every other value is rejected up front** with a clear error before any connection — it does not and cannot speak an older or virtual wire format. Divergence from rsync (which negotiates a range and downgrades to an integer 0..31): FastSync's honest contract is force-to-the-one-supported-value; a genuine downgrade would require a per-version compatibility layer that does not exist. Client-only; the server-side exact-match check is unchanged. `--protocol=2.20.0`/`2.19.0`/`2.18.0`/`2.18`/`2.17.0`/`2.16.0`/`2.15.0`/`216`/`31`/garbage are all rejected. See the Phase-6 protocol note below |
| `--iconv=CONVERT_SPEC` | Charset conversion | ✅ Implemented | Charset conversion of FILE NAMES (not content) at the protocol boundary via iconv(3): `--iconv=LOCAL[,REMOTE]` — the sender converts each local filename LOCAL→REMOTE before transmitting, and the receiver converts each wire filename REMOTE→LOCAL before creating/writing. The full CONVERT_SPEC is serialized into the config frame as a new trailing string field so the peer knows the wire charset; **PROTOCOL_VERSION bumped 2.15.0 → 2.16.0**. `LOCAL[,REMOTE]` parse: single charset ⇒ LOCAL==REMOTE (identity both ways); garbage rejected up front. Validation probes BOTH directions (a spec that only opens one way is refused, as is a NUL-emitting target charset like utf-16/utf-32/ucs-2, since filenames cannot contain NUL). An unrepresentable name (EILSEQ/EINVAL) fails that path cleanly with a logged `--iconv: cannot convert file name ...` and is never written mangled/truncated. Conversion is applied at EVERY wire-path site (regular/MKDIR/hardlink path+target/symlink path+target/SPECIAL, the delete manifest, the incremental-check path, and the `-s`/`chunk_serialize` embedded blob path), on both client and server (`--iconv` is also a server/daemon option). Zero overhead when unset. See the Phase-6 iconv notes below |
| `--checksum-seed=NUM` | Set checksum seed | ✅ Implemented | Sets the seed for FastSync's whole-file xxHash64 digest (full 64-bit seed) and for the delta path's per-block xxHash32 strong checksum (low 32 bits of the seed). An explicit seed deterministically changes every computed digest on BOTH endpoints (sender and receiver share the seed via the config frame, protocol 2.10.0), so identical runs with the same seed skip the same files and a changed seed changes the digests — the explicit-seed path that makes xxHash comparisons deterministic. `--checksum-choice=md5` has no seed and ignores it (documented). The value is a strict decimal 0..2⁶⁴-1 (blank, signed, or non-numeric values are rejected). Like rsync, a seed only matters where a digest is actually computed (`--checksum` or a basis-dir run, or a delta transfer); it does not by itself enable `--checksum`/`--delta`. Divergence from rsync: the default is seed 0, and FastSync never randomizes the seed (rsync uses a random per-transfer seed when `--checksum-seed` is unset); FastSync's unset default therefore reproduces its historical byte-for-byte behavior |
| `--secluded-args`, `-s` | Use protocol to send args | ⛔ Impossible/Divergence | Accepted for CLI compatibility (including the rsync short `-s`, Phase 7 Wave A) but a documented **no-op / divergence**. rsync's `-s` protects arguments from shell expansion by shipping them over the protocol; FastSync never passes remote arguments through a shell expansion boundary in the first place — its SSH transport builds the remote argv as **single-quote-escaped shell words** (`ssh_build_remote_command`), so the injection/leak that `-s` guards against does not exist and there is nothing to "seclude". Implementing a true arg-send protocol would mean replacing the argv-based SSH launch with an in-band argument channel, a large redesign of the transport that buys no security here. Chunk serialization remains the long-only `--chunk-serialization`. |
| `--no-OPTION` | Turn off implied option | ✅ Supported | Supported boolean FastSync options and archive-implied options; unsafe or value-taking options are rejected. |
---
## Implementation Difficulty Plan
**Phase 5 notes (remote-option wave):** `--remote-option=OPT` (long form only) and `--trust-sender` landed here.
- `--remote-option` is CLIENT-only and never serialized into the binary config frame. On the SSH transport the client forwards each value to the remote server by appending it to the remote command line in `ssh_build_remote_command()`, after ` --stdio`, as an individually single-quoted shell word (`'...'` with `'\''` for embedded quotes). Values are validated at CLI parse time (non-empty; no ASCII control characters) and rejected otherwise, and a non-conforming value is refused again in the command builder, so shell metacharacters (`;`, `&`, `|`, backticks, `$()`, quotes) can never break out of the quoting to inject an unrelated remote command — including after a client-side `--` separator, whose arguments are never forwarded anyway. Because the remote options affect the *remote server invocation*, not the transmitted config, the wire frame layout is unchanged, but `PROTOCOL_VERSION` was bumped **2.13.0 → 2.14.0** as the Phase-5 lockstep release marker (a 2.14 client against a 2.13 server fails the version check cleanly rather than the old server rejecting an unfamiliar forwarded argv later). Divergence: rsync's short `-M` form of `--remote-option` was intentionally NOT implemented at that time because `-M` was FastSync metadata mode; **Phase 7 Wave A later freed `-M` for `--remote-option` and moved metadata to long-only `--preserve`** (see the Sending Options table).
- `--trust-sender` is a receiver-local policy: it never crosses the wire (the sender's value is never serialized, so a wire peer can never enable it). On the receiving process it skips the up-front re-validation of the incoming file list (empty/`..` path rejection and the escaping-symlink-target containment), trusting the sender's list instead of double-checking — fewer checks, faster, and potentially unsafe, matching rsync. It is OFF by default (`config.trust_sender`). As a deliberate safety floor, the low-level fd-relative confinement primitives are NOT disabled: `file_open_secure_parent()` (O_NOFOLLOW walk, `..` rejection, root containment) and leaf/destination confinement still hold, so even under `--trust-sender` a hostile sender cannot write or create a symlink outside the authorized root — the relaxation only removes the redundant list-layer double-checks, never the root-confinement guarantees.
The estimates below cover the currently unimplemented features in this document. They assume one engineer familiar with the codebase, include implementation and focused tests, and exclude production rollout time. A feature should not be marked implemented until its behavior is tested in both local and SSH/TCP paths where applicable.
> **Note:** This plan is a superset snapshot written while several of the listed features were still outstanding. The Summary matrix above is the authoritative record of what is already shipped (for example quiet/info/debug output, `--existing`, `--remove-source-files`, `-h`, and `--size-only` are now implemented on `dev`). Treat the phases as sequencing guidance for the work that remains unimplemented.
@@ -420,23 +769,95 @@ These options affect process startup, authentication, sockets, and remote execut
| Features | Effort | Implementation plan |
|----------|--------|--------------------|
| `--rsh=COMMAND`, `-e`; `--rsync-path=PROGRAM`; `--blocking-io`; `--outbuf=N\|L\|B` | M | Generalize SSH command construction and subprocess I/O while retaining argument escaping and timeout guarantees. |
| `--rsh=COMMAND`, `-e`; `--rsync-path=PROGRAM`; `--blocking-io`; `--outbuf=N\|L\|B` | M | ✅ Wave A implemented (see the Connectivity table above). SSH argv construction is generalized: `-e`/`--rsh` replaces the hardcoded `ssh` program (whitespace-split, so `-e "ssh -p 2222"` works), `--rsync-path` aliases the existing `fastsync_server_path`, `--blocking-io` drops the SSH socket timeouts, and `--outbuf` maps N/L/B onto `setvbuf`. All four are client-only launch concerns and never cross the wire. |
| `--address=ADDRESS`; `--ipv4`, `-4`; `--ipv6`, `-6`; `--sockopts=OPTIONS`; `--port=PORT` daemon semantics | M | Add explicit socket-family/bind configuration and validate it independently for TCP client and daemon modes. |
**Phase 5, Wave B (socket/bind) shipping note:** `--sockopts` adds a strict allowlisted `OPT=VAL` socket-option layer applied with correct per-option value types; `--address` binds the outgoing client socket to a local source address; `-4`/`-6` pin the address family via `getaddrinfo` hints on both the client connect and the server bind; and the server bind now honors `--address` plus `-4`/`-6` (falling back to the historical IPv4 `INADDR_ANY` when none are given). All of these are local socket concerns and none cross the wire config frame (only `--port` maps to `server_port`).
| `--remote-option=OPT`, `-M`; `--trust-sender` | L | Add authenticated remote-option/config negotiation and reject unsafe sender-controlled values. `-M` conflicts with FastSync metadata mode. |
| `--daemon`; `--config=FILE`; `--dparam=OVERRIDE`; `--no-detach`; `--password-file=FILE`; `--early-input=FILE`; `--no-motd` | XL | Implement a real daemon lifecycle, module configuration, authentication, privilege separation, and process management. |
**Phase 5, Wave A (rsh/ssh) shipping note:** the SSH transport no longer hardcodes `ssh`. `-e`/`--rsh=COMMAND` selects the remote-shell program (whitespace-split into the leading child argv words), `--rsync-path=PROGRAM` aliases `--fastsync-server-path`, `--blocking-io` removes the SSH-socketpair `SO_RCVTIMEO`/`SO_SNDTIMEO` timeouts (by default they now match the TCP transport so a wedged shell cannot hang forever), and `--outbuf=N|L|B` maps onto `setvbuf` (`_IONBF`/`_IOLBF`/`_IOFBF`, garbage rejected). All four are client-only launch concerns and never cross the wire.
**Phase 5, Wave C (remote-option/trust-sender) shipping note (PROTOCOL 2.13.0 → 2.14.0):** `--remote-option=OPT` (long form only; the short `-M` is intentionally left as FastSync metadata mode — documented divergence) appends each validated value to the remote server invocation over SSH as an individually single-quote-escaped shell word, so shell metacharacters cannot break out and a `--` can never be turned into injection; options never cross the binary config frame. `--trust-sender` is a receiver-local policy (never serialized, so a wire peer can't enable it): when requested on the server (via `--remote-option=--trust-sender`), it removes only the redundant receiver/save-layer path re-checking; the low-level floor (`file_open_secure_parent`'s `..` rejection, the O_NOFOLLOW parent walk, leaf/destination confinement) stays enforced. Off by default. The wire config-frame layout is unchanged; the bump reflects that a 2.14 sender composing remote options requires a 2.14 receiver to honor them.
### Phase 6: Batch, Encoding, and Protocol Interoperability
These are the hardest compatibility items because they require durable formats or behavior that must interoperate with rsync itself.
| Features | Effort | Implementation plan |
|----------|--------|--------------------|
| `--write-batch=FILE`; `--only-write-batch=FILE`; `--read-batch=FILE` | XL | Specify a versioned batch format, persist all required metadata, and test replay, corruption, and partial application. |
| `--protocol=NUM` | XL | Add protocol-version negotiation and compatibility branches without weakening current validation. |
| `--iconv=CONVERT_SPEC` | L | Convert filenames at the protocol boundary with invalid-sequence and normalization tests. |
| `--stop-after=MINS`; `--stop-at=TIME` | M | Add deadline propagation, interruptible I/O, and safe checkpoint/cleanup behavior. |
| `--write-batch=FILE`; `--only-write-batch=FILE`; `--read-batch=FILE` | XL | ✅ Implemented (see the Batch Operations table and Phase-6 batch note below): a versioned self-contained single-file residual-batch format, persisted via the existing chunk codec, with replay, corruption, and partial-application safety tests |
| `--protocol=NUM` | XL | ✅ Implemented (see the Advanced table and Phase-6 protocol note below): protocol-version forcing without weakening current validation; FastSync's single lockstep wire format means only the current `PROTOCOL_VERSION` is accepted, and everything else is rejected up-front |
| `--iconv=CONVERT_SPEC` | L | ✅ Implemented (see the Advanced table and Phase-6 iconv notes below): filename charset conversion at the wire boundary with expansion/overflow safety and invalid-sequence test coverage |
| `--stop-after=MINS`; `--stop-at=TIME` | M | ✅ Implemented (see the Advanced table and Phase-6 stop notes below): deadline propagation and safe early stop with --delete safety |
| `--early-input=FILE`; `--password-file=FILE` | M | Securely read startup credentials/input with permission checks and no secret disclosure in logs. |
**Phase 6, Wave A (stop deadline) shipping note:** `--stop-after=MINS` and `--stop-at=TIME` are client-only sender stop deadlines. `--stop-after` takes a positive minute count (0/negative/garbage rejected); `--stop-at` takes `HH:MM`, `HH:MM:SS`, or `now+N[smhd]` (a past time stops immediately, a garbage spec is rejected at parse time). The deadline is computed once at the start of the transfer (CLOCK_MONOTONIC for `--stop-after`, wall clock via `time()` for `--stop-at`) and checked at every chunk boundary in both the single-threaded `send_files` loop and the multithreaded `send_chunks_multithreaded` path, and inside the scanner loops so a busy scan itself stops. When it fires, the transfer stops ELEGANTLY: the in-flight chunk completes, the existing completion tail runs (summary, `disconnect`), and the run returns 0 — exactly like rsync's clean early stop. Because the deadline is client-only and never crosses the wire config frame, no PROTOCOL_VERSION bump is required. The safety-critical interaction is with `--delete`: FastSync streams while scanning, so a deadline can cut the source scan short and yield a PARTIAL keep-set manifest; committing that would make the receiver delete destination mirrors of source files not yet scanned. So the sender tracks `scan_stopped_early` and, when it is true on the late/delete-after (`--delete`/`--delete-after`/`--delete-delay`) path, SUPPRESSES the keep-set manifest (logs a warning) so no deletion happens from an incomplete set — this is the safe direction (preserves data; the delete simply does not run). `--delete-before`/`--delete-during` are unaffected: their complete pre-scan runs before any data and ignores the deadline (a stop can be exceeded by that pre-scan). Under `-j`/`--threads` the stop is symmetric and the scanner thread's still-in-progress manifest appends can never race the tail because the tail does not read the manifest on the early-stop path.
**Phase 6, Wave B (iconv) shipping note (PROTOCOL 2.15.0 → 2.16.0):** `--iconv=LOCAL[,REMOTE]` converts file NAMES at the wire boundary (never content). The full CONVERT_SPEC is serialized into the config frame as a new trailing string field (empty→NULL canonicalized), so both ends share the same wire charset interpretation; this required the PROTOCOL bump because the frame is a strict ordered sequence and a peer that does not parse the new trailing field would desynchronize. Each end derives LOCAL (its own charset) and REMOTE (the wire charset): the sender opens LOCAL→REMOTE and converts every transmitted filename; the receiver opens REMOTE→LOCAL and converts every received filename before creating/writing. Conversion is applied at every wire-path site (regular/MKDIR/hardlink path+target/symlink path+target/SPECIAL, the delete manifest keep/protected/missing entries, the incremental-check path, and the embedded `-s`/chunk-blob path). A name it cannot convert (EILSEQ/EINVAL) is failed cleanly with a logged `--iconv: cannot convert file name ...` and is never written truncated/mangled. Validation probes both directions up front (both the sender local→remote and the receiver remote→local, and, for a server/daemon with its own `--iconv`, the client-REMOTE→server-LOCAL pair) so an unusable spec is rejected before the connection rather than mid-transfer, and NUL-emitting target charsets (utf-16/utf-32/ucs-2) are refused because filenames cannot contain NUL. Divergence documented upstream: the receiver does NOT half-swap; the wire charset always comes from the sender's REMOTE half, so a server whose local charset differs from the client's LOCAL must declare it with its own `--iconv`. Conversion is process-global and runs on a single thread per process (sender thread / receiver-loop thread), initialized before worker threads start and freed after they join.
**Phase 6, Wave C (protocol-version) shipping note (no PROTOCOL_VERSION change):** `--protocol=NUM` lets the client force the wire protocol version for a transfer. FastSync's protocol is a single lockstep format: the config frame is a strict ordered sequence and the server requires the client's version string to equal `PROTOCOL_VERSION` exactly (`config_receive_with_validate`, src/shared/config.c) — there are no older-format code paths and no downgrade/negotiation machinery, so a lower/higher/virtual version can never be spoken. The honest contract is therefore: `--protocol=2.21.0` (the current `PROTOCOL_VERSION`, as of the combined error-detail + server-contacting dry-run wave) is accepted and stored into the client's `version` claim (which `config_send` already transmits), and every other value — `2.20.0`, `2.19.0`, `2.18.0`, `2.18`, `2.17.0`, `2.16.0`, `2.15.0`, `3.0.0`, rsync-integer spellings like `216`/`31`, garbage, empty — is rejected up front in `validate_config()` before any connection, with a clear error that FastSync supports only its current wire protocol and cannot speak an older or virtual one. Implementation is client-only: a server-side `--protocol` is intentionally not added because the server has no negotiation (it only enforces exact match), and it could only ever be the current version. This preserves (and slightly tightens) existing validation: the client now also refuses to launch with a version it cannot actually speak, rather than only the server rejecting it later. A genuine downgrade would require a per-version compatibility layer for every frame/feature added since (append 2.10, preallocate 2.11, hardlinks 2.12, devices/specials/symlink-trust/xattr 2.13, remote-option 2.14, daemon module/auth 2.15, iconv 2.16, dir/symlink times 2.17, privilege flags --super/--copy-as 2.18, SCRAM daemon auth 2.19, packed metadata 2.20) and is intentionally out of scope — documented divergences from rsync's integer-negotiated downgrade remain.
**Phase-1/2 selection-and-update status correction (docs):** `-I/--ignore-times`, `--size-only`, `-@/--modify-window`, `--existing`, `--ignore-existing`, `-u/--update`, `-W/--whole-file`, and `--compress-threads` were previously listed as not-implemented in this document but are in fact fully implemented and tested on `dev`. This pass corrects the matrix to match the code. The realistic model of these is that FastSync is a *sender-driven* whole-tree copy, so the size+mtime quick-check and all three receiver-policy skips (`--existing`, `--ignore-existing`, `-u`) are evaluated against the **destination** on the receiver side, and their booleans cross the wire in the config frame. `-I`/`--size-only`/`--modify-window` modify the `--incremental` per-file `STATUS_CHECK` handshake's match predicate (`-I` disables the mtime leg and forces transfer; `--size-only` drops only the mtime leg; `--modify-window` adds tolerance to `metadata_mtime_matches`); they require `--incremental` (or a basis dir) to have a handshake to affect, mirroring how they only matter where a quick-check exists in rsync. `--existing`/`--ignore-existing`/`-u` are receiver write-time policies (skipping the write / newer-destination guard) applied across the regular-file, `--delay-updates`-staged, hardlink-sibling, and special/device paths; `-u` implies `-M` metadata and uses a second-then-nanosecond strict `>` newer check; both correctly influence `--remove-source-files` (a skipped source is not removed). `-W/--whole-file` disables block-level delta (opt-in via `--delta`), folded into the wire `use_delta` so no protocol bump was needed, and makes `--fuzzy` inert; `--append`/`--append-verify` are rejected with `-W`. `--compress-threads=NUM` (1..64, client-only, never crosses the wire) sizes the zstd compression worker pool. No code was changed by this correction; the implementation had landed in earlier merge waves (feat/ignore-times, feat/ignore-existing via the newer `file_to_disk_secure_no_replace`/`linkat EEXIST` path, feat/size-only, feat/modify-window, feat/whole-file, feat/update, compression-threads).
**Phase 6, Wave D (batch) shipping note (no PROTOCOL_VERSION change):** FastSync batch mode is a **client-only, self-contained "residual batch"**: a single file `MAGIC "FSTRESBATCH" + format version 1 + metadata flag`, followed by length-prefixed `chunk_serialize` blobs that store full file images (regular files, dirs, symlinks, specials). It is NOT a raw capture of the live wire, because FastSync's protocol is per-file interactive (`STATUS_CHECK`/`STATUS_DELTA_SIGNATURE`/`STATUS_APPEND` handshake), so a raw sender-stream tee is not deterministically replayable against an arbitrary destination. Storing full residuals via the existing, fuzz-tested chunk codec makes `--read-batch` replay byte-identically by construction. `--write-batch=FILE` runs the normal live transfer AND emits the batch from a separate deterministic scan pass; `--only-write-batch=FILE` emits the batch only (no destination, no server); `--read-batch=FILE DEST` applies it locally (no source, no server; DEST is the only positional arg). Because batch is a local driver concern, it never crosses the wire: no new config-frame field and no `PROTOCOL_VERSION` bump (mirroring `--stop-after`/`--protocol`/`--compress-threads`). The READ side is hardened against untrusted/attacker-controlled batch files: magic+version validated before any record, per-record length bounds checked before allocation (64 MB cap), clean-EOF-after-prefix and truncated/oversized records rejected, and every applied path goes through the same confined `file_save_to_disk_full` machinery as the network receiver (O_NOFOLLOW fd-walk, `..`-rejection, root confinement — a malicious `../` or absolute/symlink path cannot escape the destination root; this was security-reviewed and valgrind/ASan-clean). Divergences from rsync: (1) the batch carries the FULL residual (complete file images) rather than rsync's update-only delta stream — always byte-correct but larger; (2) per-file data is capped at the chunk codec's ~64 MB (`BATCH_MAX_RECORD`), so very large files may be refused by the batch writer with a clean error (never a corrupt/truncated batch); (3) hard-links and xattr/ACL blocks are not represented by `chunk_serialize`, so `-H`/`-X`/`-A` are out of scope for batch; (4) there is no companion `.sh`/`.rsync_argvs` — the batch is invoked directly (`fastsync --read-batch=FILE DEST`, `--only-write-batch=FILE SOURCE`); (5) `--write-batch` drives the single-threaded transfer path. Integration/`-M` note: metadata is captured in the batch when `-M` is used and persisted in the header so it applies consistently regardless of the reading process's own `-M`.
### Phase 7: CLI-Namespace Parity, Filesystem/Output Completion, and Privilege (Final)
These are the last compatibility items and the closing phase toward rsync flag parity. Per the project decision: every rsync flag (short **and** long) that is *possible* gets real rsync-parity behavior; anything physically impossible becomes an explicit **Impossible/Divergence** status (accepted for CLI compatibility, safely inert, with coverage tests proving that); and the two privilege flags (`--super`, `--copy-as`) adopt the deliberately-scoped **safe-subset + clear-refusal** model rather than blind elevation. The remaining `⚠️ Partial`, `🔄 Compatibility No-op`, `🔀 Alt Arg`, and `❌ Not Implemented` rows in the Summary are this phase's scope. All Wave A renames are **client-side only** (the wire config fields `use_compression`/`use_metadata`/`use_sendfile`/`use_chunk_serialization` are unchanged), so they require **no `PROTOCOL_VERSION` bump**.
**Wave A — CLI namespace parity (rename colliding FastSync short flags) — ✅ implemented.** This freed the short letters rsync needs and made the three `🔀 Alt Arg` rows real. `-c`→`--checksum`, `-m`→`--prune-empty-dirs`, `-M`→`--remote-option`, `-f`→`--filter`, `-s`→`--secluded-args`, `-p`→`--perms`, `-T`→`--temp-dir`, `-a`/`--archive`→real `-rlptgoD`. FastSync's own flags moved to long-form-only or new shorts: `-j`/`--threads` (multithreading), `--preserve` (metadata), `--sendfile`, `--chunk-serialization`, `--timeout`, `--ssh-port`. The server's independent little CLI keeps `-p` as its port. All client-side, no wire change, no `PROTOCOL_VERSION` bump. Unit tests 37/37, full integration 400 passed, cppcheck and clang-format clean. Known Wave-A limitation: `--no-perms`/`--no-compress`-style negation of the newly-aliased shorts is not wired into the negatable set (only the long-form `--preserve`/`--compress`/`--no-links` negations exist); `--archive --no-perms` is consequently not supported yet — a minor deviation from rsync, acceptable for Wave A.
| FastSync flag today | rsync wants that name | Proposed rename |
|---------------------|----------------------|-----------------|
| `-c` / `--compress` | `-c` = `--checksum` | compression is already aliased as `-z`/`--compress` (rsync parity!) → drop the `-c` short, keep `--compress`/`-z` |
| `-m` / `--multithreading` | `-m` = `--prune-empty-dirs` | → `-j` / `--threads` |
| `-M` / `--preserve` | `-M` = `--remote-option` | → `--preserve` (long-only) |
| `-f` / `--sendfile` | `-f` = `--filter` | → `--sendfile` (long-only) |
| `-s` / `--chunk-serialization` | `-s` = `--secluded-args`/`--protect-args` | → `--chunk-serialization` (long-only) |
| `-p` (SSH port) | `-p` = `--perms` | → `--port` (long-only; `--server-port` already exists) |
| `-T` / `--timeout` | `-T` = `--temp-dir` | → `--timeout` (long-only) |
| `-a` / `--archive` (= `-c -m -M`) | `-a` = `-rlptgoD` | → becomes **real rsync `-a`** after the renames |
**Wave B — Output & filesystem completion (✅ implemented).** `-S`/`--sparse` (`⚠️→✅`): real hole preservation — a sparse-aware writer (`write_all_sparse`) skips all-zero runs ≥ 4096 bytes with `lseek(SEEK_CUR)` and `ftruncate`s the final size, wired into both the atomic temp+rename store and `--inplace` receiver-side with **no wire change** (the full file image is already in memory; the ftruncate presize is kept). `-P` (`⚠️→✅`): interrupted-write retention — on a save failure after data reached the temp fd, `--partial` now renames the already-written temp to the destination path (best-effort; falls through to the normal unlink on failure, never retains when `--partial` is off) so a later `--append`/`--append-verify` run can resume. `--block-size=SIZE` (`⚠️→✅`): promoted after verification — `--block-size` is now an alias for `--delta-block`, both set `config->delta_block_size`, which the delta engine already honored end-to-end (`delta_signature_create_seeded` + `delta_apply`); out-of-range values keep the default. `--fake-super` (`⚠️→✅`): added `fake_super_restore_fd` to parse and re-apply the recorded `user.fastsync.stat` record fd-relative (fchown best-effort/non-root skipped, fchmod, futimens); a save under `--fake-super` now re-applies the recorded attrs instead of only recording them, with the recording format unchanged. `--stderr=client` (`⚠️→⛔ Impossible/Divergence`): FastSync has no rsync client-message channel, and `client` is rejected at CLI parse — the rejection is the documented behavior (unit-tested). `-N`/`--crtimes` (`⚠️→⛔ Impossible/Divergence`): birth-times cannot be set by any portable fs call (`utimensat` sets only atime/mtime); capture/transmit stays, setting is impossible, the flag is accepted and safely inert. Review-hardening (post-eval): fake-super replay applies the mode through the same sanitization as the normal metadata path (group/other write bits are never granted); `--sparse` takes precedence over `--preallocate` (posix_fallocate skipped so holes survive); `--partial` retention is disabled for `--no_replace` (ignore/existing) and only marks a write-attempt after the actual write begins; `--block-size=SIZE`/`--delta-block=SIZE` inline forms are accepted.
**Wave C — Devices & special files (finalize statuses + tests) (✅ implemented).** The four special-file rows are finalized with coverage tests. `--devices`, `--copy-devices`, and `--write-devices` are **✅ Implemented**, each with a documented, safety-driven divergence: device-node creation is privilege-gated, so a receiver without `CAP_MKNOD` skips that entry with a warning (a per-entry skip, never a transfer failure); `--copy-devices` copies a device/FIFO's reported size into an ordinary regular file (a size-bounded safe divergence from rsync's unbounded dd-like read); `--write-devices` writes only into an existing char/block node under the confined receive root and skips every unusable target rather than clobbering or aborting. `--specials` is classified **⛔ Impossible/Divergence** for one reason only: **FIFO recreation works** (unprivileged `mkfifo`, asserted under CI), but **sockets cannot be recreated by any standard filesystem call**, so a source socket is skipped with an explicit note. Tests assert FIFO recreation, the safe socket skip, the regular-file result of `--copy-devices`, the skipped/missing and non-device `--write-devices` targets, and (root-gated) real device-node creation; a root runner additionally drops the receiver to an unprivileged user to assert the `CAP_MKNOD` skip is graceful.
**Wave D — Times superstructure & arg-protection no-ops (✅ implemented, `--secluded-args` ⛔).** `-O`/`--omit-dir-times` and `-J`/`--omit-link-times` are now **real modifiers** (both `🔄 → ✅ Implemented`), reversing the old "never preserves directory/symlink times" divergence:
- **Directory times.** The recursive scanner captures every traversed source directory's metadata (mtime, plus atime under `-U`) into a per-transfer list — two paths are covered: the sequential `DirectoryScanner` captures each opened directory (including the transfer root), and the parallel scanner captures both the root in `parallel_scanner_create_with_options` and each worker's subdirectories in `open_next_directory` (appends are guarded by a mutex shared with the sender's pipeline context). The sender transmits them in trailing `STATUS_DIR_TIMES` frames (each: int count + count × (wire path, metadata) pairs) sent **after all file data and after the optional delete manifest**, just before `STATUS_FINISHED`. A tree larger than `MAX_MANIFEST_ENTRIES` (1 048 576) directories is chunked into repeated frames, each within the receiver's per-frame bound. A dir-time entry is RECORD-ONLY (`file->dir_time_only`): `file_save_to_disk_full` returns `FILE_SAVE_SKIPPED` without creating anything, so a source directory that was empty (or pruned by `-m/--prune-empty-dirs`) is never resurrected. The receiver accumulates received directory metadata in a `DirTimeList` and applies it only at the very end — after the entire stream, after the commit-style `--delete` deletion, and after `--delay-updates` publication — because creating or removing a child bumps the parent's mtime. Application is fd-relative/walk-confined (`file_open_secure_parent` + `utimensat(..., AT_SYMLINK_NOFOLLOW)`) and best-effort per entry: an absent path (an intentionally uncreated empty dir) is skipped QUIETLY and only a real existing directory is stamped. `-O` (config boolean, already on the wire) makes the receiver skip the whole set. The single-threaded sink applies in `receiver_send_success_frame`; the `-j`/`--threads` sink accumulates in `write_thread` and server.c applies after both threads join and the deletion commits.
- **Symlink times/owner/mode.** `STATUS_SYMLINK` already carried metadata; the receiver now applies it with no-follow primitives only: `utimensat(..., AT_SYMLINK_NOFOLLOW)`, best-effort `fchmodat(..., AT_SYMLINK_NOFOLLOW)` (honest no-op where unsupported, e.g. Linux), and policy-gated `fchownat(..., AT_SYMLINK_NOFOLLOW)` via a new `identity_apply_ownership_link` that shares the identity resolver with the fd path. `-J` suppresses only the timestamps; ownership stays governed by the identity opt-in (`--numeric-ids`/`--usermap`/`--groupmap`/`--chown`) exactly like regular files. A symlink has no children, so this is applied immediately at creation.
- **Wire:** the shared `STATUS_DIR_TIMES` frame (and metadata on `STATUS_MKDIR` for `--dirs` entries) is a frame-sequence change, so `PROTOCOL_VERSION` was bumped **2.16.0 → 2.17.0**; every version-sensitive test (`--protocol` accepted/rejected values) was updated. The config-frame layout itself is unchanged (the omit booleans already crossed). Non-metadata and `--no-preserve` transfers send no `STATUS_DIR_TIMES` frame and no directory metadata, keeping them byte-identical.
`--secluded-args` (`🔄 → ⛔ Impossible/Divergence`): a true arg-send protocol would replace the argv-based SSH launch with an in-band channel, and FastSync already builds the remote SSH argv injection-safe (single-quote-escaped shell words), so there is no argument-leak to close; the already-safe behavior is documented in the row and no transport change is made.
**Wave E (LAST) — Privilege: `--super`/`--no-super` and `--copy-as=USER[:GROUP]` (✅ implemented).** FastSync adopts a **safe-subset + clear-refusal** privilege model: it never blind-elevates and never calls `setuid`/`seteuid`/`setgid`. All privileged operations remain fd-relative and confined below the authorized receive root.
`--super`/`--no-super` set a receiver-side tri-state `Config->super_mode` (`SUPER_MODE_AUTO`/`ON`/`OFF`). `privilege_super_permitted()` / `privilege_super_mode_permitted()` (src/shared/identity.c) return true for `ON` and `AUTO` (AUTO preserves FastSync's historical best-effort attempt, where the kernel refuses an unprivileged call and the caller skips it) and false only for `OFF`. The gate covers every super-user activity FastSync performs: ownership application (`identity_apply_ownership`/`_link`), char/block device-node creation (`file_save_special_to_disk`), writes into an existing device (`--write-devices`), and the `--fake-super` owner replay. Unprivileged FIFO creation is deliberately unaffected. `--super` does **not** imply `--numeric-ids`: ownership is applied only when an explicit identity policy (`--usermap`/`--groupmap`/`--chown`/`--numeric-ids`/`--copy-as`) is also given. `--no-super` suppresses those activities even for a root receiver. A non-root receiver given `--super` logs one warning at activation (`identity_set_active`); each confined attempt is then refused by the kernel and skipped, never aborting. The confinement floor is unchanged (`file_open_secure_parent`, `O_NOFOLLOW`, root/path checks). Operator control: the server CLI accepts `--no-super`, a veto that forces `OFF` for every connection, refuses any client `--copy-as`, and neutralizes an explicit `--super` (the connection is accepted but no super-user activity is attempted). A privileged (root) standalone TCP listener instead defaults to `OFF` and requires the server-only `--allow-super` opt-in to attempt any super-user activity (the flag is rejected with `--stdio`, whose client-composed remote argv must never defeat the default; use a forced command if the default must hold); a non-root server is unchanged. On a daemon, a module that has not opted in with `client owner = yes` additionally has super-user device activity forced off (see the Daemon Mode notes).
`--copy-as=USER[:GROUP]` is the safe subset. FastSync's receiver is multithreaded, so a real credential switch is unsafe; instead the receiver forces the ownership of **every entry it writes** — regular files, symlinks, directories (including implicitly-created parents), and special nodes — to the resolved target ids through the confined fd-relative identity path. USER is resolved on the client (name, `@N`/bare N, or `*` = client euid); when `:GROUP` is omitted the user's primary gid is used (falling back to `gid == uid` for a numeric id with no local passwd entry). It requires a privileged (root) receiver: an unprivileged receiver refuses the whole transfer at the config handshake, before `STATUS_OK`, so no data is ever written with the wrong ownership. A `--copy-as` chown failure on a capability-restricted root is logged at ERROR (never silently downgraded). `--copy-as` implies metadata (`--no-preserve` is rejected) and `--fake-super` cannot override it. Daemon policy: a `--daemon` receiver refuses **every** client-chosen-ownership / super-user request — `--numeric-ids`, `--chown`, `--usermap`/`--groupmap`, `--fake-super`, `--copy-as`, and explicit `--super` — unless the selected module opts in with `client owner = yes`; without that per-module opt-in any client could force arbitrary ownership inside the module root (a root standalone TCP listener, which serves one operator-authorized root, honors these requests only when started with `--allow-super`; the flag is rejected with `--stdio`). A `--copy-as` chown failure on a capability-restricted root marks the entry as failed rather than reporting success with the wrong owner.
**Wire:** two trailing config-frame blocks after the `--iconv` spec, in fixed order — `send_privilege_options`/`receive_privilege_options` (one `super_mode` int, validated `0..2`), then `send_copy_as_options`/`receive_copy_as_options` (presence int + two int32 ids, validated `>= 0`, with `copy_as_set ⇒ use_metadata`). `PROTOCOL_VERSION` bumped **2.17.0 → 2.18.0**. **Divergences from rsync:** rsync's `--super` elevates the receiver and `--copy-as` actually switches its credentials; FastSync never elevates and only permits/forwards confined attempts, and `--copy-as` forces ownership rather than switching identity.
**Post-Phase-7 Summary (after Waves A–E).** ✅143 / 🔀0 / ⛔4 / ⚠️0 / 🔄0 / ❌0 = 147. The 3 `🔀 Alt Arg` rows (`-a`, `-p`, `-z`) are ✅ (Wave A). All 10 prior `⚠️ Partial` rows are resolved to ✅ (`-S`, `-P`, `--block-size`, `--fake-super`, `--devices`, `--copy-devices`, `--write-devices`) or ⛔ (`--stderr=client`, `-N/--crtimes`, `--specials` for the impossible socket case). The 3 `🔄 Compatibility No-op` rows are resolved: `-O`/`-J` are now real ✅ (Wave D), `--secluded-args` is ⛔. The **Impossible/Divergence** bucket holds the 4 physically-impossible/divergent flags: `--stderr=client`, `-N/--crtimes`, `--specials` (sockets), `--secluded-args`. The last two `❌ Not Implemented` rows — `--super` and `--copy-as=USER[:GROUP]` — are now ✅ (Wave E). **No `❌ Not Implemented` rows remain.**
## Packed Metadata Frame (protocol 2.20.0)
A file's metadata used to cross the wire as up to 12 separate per-field framed
messages (a present flag followed by mode/uid/gid/mtime/atime/crtime writes),
which cost ~11 extra protocol frames per file on many-small-file trees. FastSync
now sends the metadata as ONE packed frame: a single `int32` present flag
(`0` = absent) followed, when present, by the fixed
`FILE_METADATA_WIRE_SIZE`-byte (68-byte) field record already emitted by the
shared `metadata_to_buf()`/`metadata_from_buf()` chunk codec. Absent metadata is
a lone `int32` zero. The encoded field layout is unchanged (only the framing
collapses), so chunk-serialized blobs remain byte-identical. Protocol data is an
unframed byte stream, so the packed encoding is byte-for-byte identical to the
old field-by-field writes; `PROTOCOL_VERSION` was bumped `2.19.0 → 2.20.0` as a
deliberate lockstep-release marker rather than because of a
desynchronization. The strict same-version handshake rejects any mismatch before
a byte of the frame is parsed.
### Recommended Delivery Order
1. Resolve short-option conflicts (`-m`, `-M`, `-T`, `-f`, `-s`) and define the compatibility contract.
@@ -451,15 +872,15 @@ The existing priority list below is a feature shortlist, not an implementation s
## Recommendations: Top Features to Implement Next
Ranked by user demand, implementation complexity, and interoperability impact:
Ranked by user demand, implementation complexity, and interoperability impact (_status reflects current `dev`_):
| Priority | Feature | Effort | Impact |
|----------|---------|--------|--------|
| 1 | `--whole-file` / `-W` | Low | High — users expect opt-out of delta |
| 2 | `--ignore-times` / `-I` | Low | Medium — useful for forcing re-transfer |
| 3 | `--size-only` | Low | Medium — common migration scenario |
| 4 | `--ignore-existing` | Low | Medium — common sync patterns |
| 5 | `--existing` | Low | Medium — common sync patterns |
| 1 | `--whole-file` / `-W` | Low | High — users expect opt-out of delta — **✅ implemented** |
| 2 | `--ignore-times` / `-I` | Low | Medium — useful for forcing re-transfer — **✅ implemented** |
| 3 | `--size-only` | Low | Medium — common migration scenario — **✅ implemented** |
| 4 | `--ignore-existing` | Low | Medium — common sync patterns — **✅ implemented** |
| 5 | `--existing` | Low | Medium — common sync patterns — **✅ implemented** |
| 6 | `--remove-source-files` | Low | High — common for moves/backup |
| 7 | `--delete-during` | Medium | High — performance improvement |
| 8 | `--delay-updates` | Medium | High — atomic updates |
@@ -473,10 +894,10 @@ Ranked by user demand, implementation complexity, and interoperability impact:
| Feature | Description |
|---------|-------------|
| `-m` | Multithreaded pipeline (scanner/loader/sender) |
| `-s` | Chunk serialization mode |
| `-f` / `--sendfile` | Zero-copy sendfile() syscall (TCP only) |
| `-c [level]` | zstd compression level (1-22) |
| `-j` / `--threads[=N]` | Multithreaded pipeline (scanner/loader/sender); `N` (1–256) sizes the parallel scanner worker pool, bare `-j`/`--threads` uses the built-in default (renamed from `-m` in Phase 7 Wave A; `-m` is now rsync `--prune-empty-dirs`) |
| `--chunk-serialization` | Chunk serialization mode (long form only; `-s` is now rsync `--secluded-args`) |
| `--sendfile` | Zero-copy sendfile() syscall (TCP only) (long form only; `-f` is now rsync `--filter`) |
| `-z [level]` / `--compress` | zstd compression level (1-22) (`-c` is now rsync `--checksum`) |
| `--chunk-size` | Configurable chunk size |
| `--tls` | TLS encryption (mutual auth) |
| `--fastsync-server-path` | Path to fastsync-server binary |
+415 -70
View File
@@ -3,19 +3,24 @@
Compares FastSync configs against rsync (no compression) and rsync+zstd.
Data is ~75% random/incompressible and ~25% structured/compressible by default,
controllable via --random-ratio.
controllable via --random-ratio. Transfers are verified by default (source and
destination must match) so a fast-but-broken copy is never counted.
Usage:
python3 benchmark/bench.py
python3 benchmark/bench.py --runs 5 --profiles lan wan
python3 benchmark/bench.py --random-ratio 0.5 --size-mb 50
python3 benchmark/bench.py --delay 50ms --jitter 10ms --throughput 100mbit
python3 benchmark/bench.py --warm --runs 3
python3 benchmark/bench.py --output json
"""
import argparse
import filecmp
import json
import math
import os
import random
import shlex
import shutil
import socket
import statistics
@@ -25,8 +30,11 @@ import tempfile
import time
PROJECT_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
BUILD_DIR = os.path.join(PROJECT_ROOT, "build")
SERVER_CMD = [os.path.join(BUILD_DIR, "server")]
DEFAULT_BUILD_DIR = "build-bench"
# Populated by configure_build_dirs(); default to the dedicated bench dir so
# importing this module never depends on the user's existing build/ tree.
BUILD_DIR = os.path.join(PROJECT_ROOT, DEFAULT_BUILD_DIR)
SERVER_CMD = [os.path.join(BUILD_DIR, "server"), "--allow-unauthenticated"]
CLIENT_CMD = [os.path.join(BUILD_DIR, "client")]
BENCH_DIR = os.path.join(PROJECT_ROOT, "bench_data")
@@ -43,11 +51,12 @@ NETWORK_PROFILES = {
}
FASTSYNC_CONFIGS = [
{"name": "fastsync", "flags": [], "tool": "fastsync"},
{"name": "fastsync -c", "flags": ["-c"], "tool": "fastsync"},
{"name": "fastsync -m", "flags": ["-m"], "tool": "fastsync"},
{"name": "fastsync -m -c", "flags": ["-m", "-c"], "tool": "fastsync"},
{"name": "fastsync -m -c -s", "flags": ["-m", "-c", "-s"], "tool": "fastsync"},
{"name": "fastsync", "flags": [], "tool": "fastsync"},
{"name": "fastsync -z", "flags": ["-z"], "tool": "fastsync"},
{"name": "fastsync -j", "flags": ["-j"], "tool": "fastsync"},
{"name": "fastsync -j -z", "flags": ["-j", "-z"], "tool": "fastsync"},
{"name": "fastsync -j -z --chunk-serialization", "flags": ["-j", "-z", "--chunk-serialization"], "tool": "fastsync"},
{"name": "fastsync --sendfile", "flags": ["--sendfile"], "tool": "fastsync"},
]
RSYNC_CONFIGS = [
@@ -56,6 +65,7 @@ RSYNC_CONFIGS = [
{"name": "rsync -z --zstd", "flags": ["-z", "--zc", "zstd"],"tool": "rsync"},
]
class RsyncDaemon:
"""Manages an rsync daemon for network-fair benchmarking."""
@@ -117,6 +127,12 @@ STRUCTURED_FILES = {
"nested/another.txt": b"another nested file\n" * 50,
}
# Repeated text used to synthesize genuinely compressible filler of any size.
COMPRESSIBLE_TEXT = (
b"FastSync benchmark payload: the quick brown fox jumps over the lazy dog. "
b"0123456789 ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz\n"
)
class Progress:
"""Simple progress bar with ETA."""
@@ -151,35 +167,127 @@ class Progress:
sys.stderr.flush()
def write_compressible(path, nbytes):
"""Write exactly nbytes of highly compressible, repeated text content."""
if nbytes <= 0:
return
block = COMPRESSIBLE_TEXT * (max(1, 8192 // len(COMPRESSIBLE_TEXT)) + 1)
remaining = nbytes
with open(path, "wb") as f:
while remaining > 0:
piece = block if remaining >= len(block) else block[:remaining]
f.write(piece)
remaining -= len(piece)
def generate_bench_data(source_dir, size_mb=25, random_ratio=0.75):
"""Generate test data. ~random_ratio is incompressible, rest is structured."""
"""Generate test data honouring the requested random/compressible split.
Exactly ``random_ratio * target`` bytes are incompressible random data and
the remainder is genuinely compressible structured/repeated content. The
measured byte counts are returned so callers can report the real mix.
"""
if os.path.exists(source_dir):
shutil.rmtree(source_dir)
os.makedirs(source_dir)
target = size_mb * 1024 * 1024
structured_budget = int(target * (1 - random_ratio))
written = 0
random_budget = int(target * random_ratio)
compressible_budget = target - random_budget
compressible_written = 0
random_written = 0
files = 0
# A handful of fixed, human-meaningful files (directories, small files, a
# binary blob) as long as they fit inside the compressible budget.
for rel_path, content in STRUCTURED_FILES.items():
if written >= structured_budget:
if compressible_written + len(content) > compressible_budget:
break
full_path = os.path.join(source_dir, rel_path)
os.makedirs(os.path.dirname(full_path), exist_ok=True)
with open(full_path, "wb") as f:
f.write(content)
written += len(content)
compressible_written += len(content)
files += 1
os.makedirs(os.path.join(source_dir, "bulk"), exist_ok=True)
i = 0
while written < target:
chunk_size = min(5 * 1024 * 1024, target - written)
with open(os.path.join(source_dir, f"bulk/file_{i}.dat"), "wb") as f:
f.write(random.randbytes(chunk_size))
written += chunk_size
i += 1
# Fill the rest of the compressible share with generated repeated content.
if compressible_written < compressible_budget:
os.makedirs(os.path.join(source_dir, "compressible"), exist_ok=True)
i = 0
while compressible_written < compressible_budget:
chunk = min(1024 * 1024, compressible_budget - compressible_written)
write_compressible(os.path.join(source_dir, "compressible", f"text_{i}.dat"), chunk)
compressible_written += chunk
files += 1
i += 1
return written
# Incompressible share.
if random_written < random_budget:
os.makedirs(os.path.join(source_dir, "bulk"), exist_ok=True)
i = 0
while random_written < random_budget:
chunk = min(5 * 1024 * 1024, random_budget - random_written)
with open(os.path.join(source_dir, "bulk", f"file_{i}.dat"), "wb") as f:
f.write(random.randbytes(chunk))
random_written += chunk
files += 1
i += 1
return {
"total_bytes": compressible_written + random_written,
"compressible_bytes": compressible_written,
"random_bytes": random_written,
"files": files,
}
def list_relative_files(root):
"""Return the set of file paths (relative to root) under a directory."""
found = set()
for dirpath, _dirnames, filenames in os.walk(root):
for name in filenames:
full = os.path.join(dirpath, name)
found.add(os.path.relpath(full, root))
return found
def verify_transfer(source_dir, dest_dir):
"""Recursively check dest matches source (paths, sizes, content).
Returns (ok, detail). Content is compared byte-for-byte, never hashed, so
collisions are impossible. This is intentionally not part of the timing.
"""
if not os.path.isdir(dest_dir):
return False, "destination directory missing"
src_files = list_relative_files(source_dir)
dst_files = list_relative_files(dest_dir)
if src_files != dst_files:
missing = src_files - dst_files
extra = dst_files - src_files
return False, f"path set mismatch (missing {len(missing)}, extra {len(extra)})"
for rel in sorted(src_files):
src = os.path.join(source_dir, rel)
dst = os.path.join(dest_dir, rel)
if os.path.getsize(src) != os.path.getsize(dst):
return False, f"size mismatch: {rel}"
if not filecmp.cmp(src, dst, shallow=False):
return False, f"content mismatch: {rel}"
return True, ""
def percentile(values, pct):
"""Linear-interpolation percentile (matches numpy's default method)."""
if not values:
return None
ordered = sorted(values)
if len(ordered) == 1:
return ordered[0]
rank = (len(ordered) - 1) * (pct / 100.0)
low = math.floor(rank)
high = math.ceil(rank)
if low == high:
return ordered[int(rank)]
return ordered[low] + (ordered[high] - ordered[low]) * (rank - low)
def find_free_port():
@@ -207,18 +315,45 @@ def wait_proc(proc, timeout=5):
proc.wait()
def _tc_base_cmd():
"""Return the command prefix for tc, honouring root vs sudo."""
tc = shutil.which("tc")
if not tc:
raise RuntimeError(
"tc (iproute2) not found in PATH; install iproute2 to use network profiles")
if os.geteuid() == 0:
return [tc]
sudo = shutil.which("sudo")
if sudo:
return [sudo, tc]
raise RuntimeError(
"applying network limits requires root or sudo; "
"re-run as root or install sudo")
def _run_tc(args, check=True):
return subprocess.run(_tc_base_cmd() + args, check=check, capture_output=True)
def netem_apply(delay=None, jitter=None, throughput=None, loss=None):
"""Apply tc/netem rules to loopback. Pass None to skip a parameter."""
netem_reset()
cmd = ["sudo", "tc", "qdisc", "add", "dev", "lo", "root", "netem"]
params = []
if throughput:
cmd += ["rate", throughput]
params += ["rate", throughput]
if delay:
cmd += ["delay", delay, jitter or "0ms"]
params += ["delay", delay, jitter or "0ms"]
if loss:
cmd += ["loss", loss]
if len(cmd) > 6:
subprocess.run(cmd, check=True, capture_output=True)
params += ["loss", loss]
if not params:
return
try:
_run_tc(["qdisc", "add", "dev", "lo", "root", "netem"] + params)
except subprocess.CalledProcessError as exc:
detail = exc.stderr.decode(errors="replace").strip() if exc.stderr else str(exc)
raise RuntimeError(f"failed to apply network profile via tc/netem: {detail}") from exc
except RuntimeError:
raise
def netem_apply_profile(profile_name):
@@ -235,7 +370,11 @@ def netem_apply_profile(profile_name):
def netem_reset():
subprocess.run("sudo tc qdisc del dev lo root".split(), capture_output=True)
"""Best-effort removal of any loopback qdisc. Always safe to call."""
try:
_run_tc(["qdisc", "del", "dev", "lo", "root"], check=False)
except Exception:
pass
def run_fastsync(source_dir, dest_dir, flags, port):
@@ -252,8 +391,10 @@ def run_fastsync(source_dir, dest_dir, flags, port):
duration = time.monotonic() - start
if result.returncode == 0:
return duration
sys.stderr.write(f" fastsync failed (exit {result.returncode}): "
f"{result.stderr.strip()[:500]}\n")
except subprocess.TimeoutExpired:
pass
sys.stderr.write(" fastsync timed out after 120s\n")
return None
@@ -269,8 +410,10 @@ def run_rsync(source_dir, dest_dir, flags, rsync_daemon=None):
duration = time.monotonic() - start
if result.returncode == 0:
return duration
sys.stderr.write(f" rsync failed (exit {result.returncode}): "
f"{result.stderr.strip()[:500]}\n")
except subprocess.TimeoutExpired:
pass
sys.stderr.write(" rsync timed out after 120s\n")
return None
@@ -282,7 +425,81 @@ def run_transfer(config, source_dir, dest_dir, port=None, rsync_daemon=None):
return run_fastsync(source_dir, dest_dir, config["flags"], port)
def run_benchmark(source_dir, dest_dir, configs, runs, profile_name, progress=None):
def apply_incremental_changes(source_dir, target_bytes):
"""Add and modify a few files so a warm transfer has real work to do.
Returns a mutation record (changed byte count plus enough data to revert
and re-apply it) so every warm run can start from a pristine source.
"""
modified_n = 3
added_n = 2
per_file = max(4096, target_bytes // (modified_n + added_n))
modified = {}
added = {}
changed = 0
existing = sorted(list_relative_files(source_dir))
if existing:
step = max(1, len(existing) // modified_n)
for rel in existing[::step][:modified_n]:
path = os.path.join(source_dir, rel)
original_size = os.path.getsize(path)
with open(path, "ab") as f:
f.write(random.randbytes(per_file))
modified[rel] = (original_size, per_file)
changed += per_file
for i in range(added_n):
os.makedirs(os.path.join(source_dir, "incremental"), exist_ok=True)
rel = os.path.join("incremental", f"new_{i}.dat")
write_compressible(os.path.join(source_dir, rel), per_file)
added[rel] = per_file
changed += per_file
return {"changed": changed, "modified": modified, "added": added}
def revert_incremental_changes(source_dir, mutation):
"""Undo apply_incremental_changes so the source is pristine again."""
if not mutation:
return
for rel, (original_size, _appended) in mutation["modified"].items():
path = os.path.join(source_dir, rel)
if os.path.exists(path):
with open(path, "r+b") as f:
f.truncate(original_size)
for rel in mutation["added"]:
path = os.path.join(source_dir, rel)
if os.path.exists(path):
os.remove(path)
def reapply_incremental_changes(source_dir, mutation):
"""Re-apply a mutation after an untimed pristine seed transfer."""
if not mutation:
return
for rel, (_original_size, appended) in mutation["modified"].items():
with open(os.path.join(source_dir, rel), "ab") as f:
f.write(random.randbytes(appended))
for rel, size in mutation["added"].items():
write_compressible(os.path.join(source_dir, rel), size)
def expected_received_root(dest_dir, source_dir, tool):
"""Where a tool places transferred files inside dest_dir.
FastSync mirrors the absolute source path under dest_dir (see the
integration suite's get_dest_received_dir); rsync copies the source tree
contents directly into dest_dir.
"""
if tool == "rsync":
return dest_dir
return os.path.join(dest_dir, os.path.abspath(source_dir).lstrip(os.sep))
def run_benchmark(source_dir, dest_dir, configs, runs, profile_name,
measure_bytes, verify=True, warm=False, mutation=None,
progress=None):
"""Run benchmark for all configs, returns list of results."""
is_limited = profile_name != "unlimited"
has_rsync = any(c["tool"] == "rsync" for c in configs)
@@ -298,7 +515,10 @@ def run_benchmark(source_dir, dest_dir, configs, runs, profile_name, progress=No
results = []
for config in configs:
times = []
invalid = 0
for run_idx in range(runs):
if warm:
revert_incremental_changes(source_dir, mutation)
if os.path.exists(dest_dir):
shutil.rmtree(dest_dir)
os.makedirs(dest_dir, exist_ok=True)
@@ -306,15 +526,33 @@ def run_benchmark(source_dir, dest_dir, configs, runs, profile_name, progress=No
port = find_free_port()
server = None
try:
if config["tool"] == "fastsync":
if config["tool"] == "fastsync" or warm:
server = subprocess.Popen(
SERVER_CMD + ["-p", str(port)],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
)
wait_for_port(port)
if warm:
seed = run_transfer(config, source_dir, dest_dir, port, rsync_daemon)
if seed is None:
invalid += 1
sys.stderr.write(" warm-mode seeding failed; run not counted\n")
continue
reapply_incremental_changes(source_dir, mutation)
t = run_transfer(config, source_dir, dest_dir, port, rsync_daemon)
if t is not None:
if t is None:
invalid += 1
elif verify:
root = expected_received_root(dest_dir, source_dir, config["tool"])
ok, detail = verify_transfer(source_dir, root)
if ok:
times.append(t)
else:
invalid += 1
sys.stderr.write(f" verification FAILED ({detail}); run not counted\n")
else:
times.append(t)
finally:
if server:
@@ -327,15 +565,22 @@ def run_benchmark(source_dir, dest_dir, configs, runs, profile_name, progress=No
"config": config["name"],
"tool": config["tool"],
"profile": profile_name,
"warm": warm,
"runs": len(times),
"invalid": invalid,
"times": [round(t, 4) for t in times],
}
if times:
entry["p50"] = round(statistics.median(times), 4)
entry["p95"] = round(sorted(times)[int(len(times) * 0.95)], 4) if len(times) > 1 else entry["p50"]
p50 = percentile(times, 50)
p95 = percentile(times, 95)
entry["p50"] = round(p50, 4)
entry["p95"] = round(p95, 4)
entry["min"] = round(min(times), 4)
entry["max"] = round(max(times), 4)
entry["stdev"] = round(statistics.stdev(times), 4) if len(times) > 1 else 0.0
if measure_bytes:
entry["throughput_mbps"] = round(
(measure_bytes / (1024 * 1024)) / p50, 3)
results.append(entry)
return results
finally:
@@ -345,44 +590,59 @@ def run_benchmark(source_dir, dest_dir, configs, runs, profile_name, progress=No
netem_reset()
def print_table(results, total_bytes, random_ratio):
def print_table(results, measure_bytes, stats, warm):
"""Print results as a human-readable table grouped by profile."""
profiles = {}
for r in results:
profiles.setdefault(r["profile"], []).append(r)
total = stats["total_bytes"]
comp_pct = stats["compressible_bytes"] / total * 100 if total else 0
rand_pct = stats["random_bytes"] / total * 100 if total else 0
for profile, entries in profiles.items():
params = NETWORK_PROFILES.get(profile, {})
print(f"\n{'=' * 85}")
print(f"\n{'=' * 95}")
print(f" Profile: {profile.upper()}")
if params.get("rate"):
print(f" Network: {params['rate']}, {params['delay']} +/- {params['jitter']}, loss {params['loss']}")
else:
print(f" Network: unlimited")
print(f" Data: {total_bytes / (1024*1024):.1f} MB ({random_ratio*100:.0f}% random, {(1-random_ratio)*100:.0f}% compressible)")
print(f"{'=' * 85}")
print(f" Data: {total / (1024*1024):.1f} MB "
f"({rand_pct:.0f}% random, {comp_pct:.0f}% compressible actual)")
if warm:
print(f" Mode: warm (incremental) — measured {measure_bytes / (1024*1024):.2f} MB "
f"changed after an untimed full seed")
else:
print(" Mode: cold (full copy)")
print(f"{'=' * 95}")
fs_entries = [e for e in entries if e.get("tool") == "fastsync"]
rsync_entries = [e for e in entries if e.get("tool") == "rsync"]
header = (f" {'Config':<38} {'p50':>8} {'p95':>8} {'min':>8} {'max':>8} "
f"{'stdev':>8} {'MB/s':>9} {'runs':>5} {'bad':>4}")
rule = (f" {'-' * 38} {'-' * 8} {'-' * 8} {'-' * 8} {'-' * 8} "
f"{'-' * 8} {'-' * 9} {'-' * 5} {'-' * 4}")
if fs_entries:
print(f"\n FastSync:")
print(f" {'Config':<25} {'p50':>8} {'p95':>8} {'min':>8} {'max':>8} {'stdev':>8} {'runs':>5}")
print(f" {'-' * 25} {'-' * 8} {'-' * 8} {'-' * 8} {'-' * 8} {'-' * 8} {'-' * 5}")
print(header)
print(rule)
for e in sorted(fs_entries, key=lambda x: x.get("p50", 999)):
_print_entry(e)
if rsync_entries:
print(f"\n rsync:")
print(f" {'Config':<25} {'p50':>8} {'p95':>8} {'min':>8} {'max':>8} {'stdev':>8} {'runs':>5}")
print(f" {'-' * 25} {'-' * 8} {'-' * 8} {'-' * 8} {'-' * 8} {'-' * 8} {'-' * 5}")
print(header)
print(rule)
for e in sorted(rsync_entries, key=lambda x: x.get("p50", 999)):
_print_entry(e)
if params.get("rate_bps") and fs_entries and rsync_entries:
fs_best = min((e["p50"] for e in fs_entries if "p50" in e), default=None)
rsync_best = min((e["p50"] for e in rsync_entries if "p50" in e), default=None)
theoretical = total_bytes / params["rate_bps"]
theoretical = measure_bytes / params["rate_bps"]
if fs_best and rsync_best:
print(f"\n Theoretical max (line rate): {theoretical:.4f}s")
print(f" FastSync best: {fs_best:.4f}s ({theoretical/fs_best:.2f}x vs line rate)")
@@ -392,10 +652,43 @@ def print_table(results, total_bytes, random_ratio):
def _print_entry(e):
if "p50" in e:
print(f" {e['config']:<25} {e['p50']:>7.4f}s {e['p95']:>7.4f}s "
f"{e['min']:>7.4f}s {e['max']:>7.4f}s {e['stdev']:>7.4f} {e['runs']:>5}")
tp = f"{e['throughput_mbps']:.2f}" if "throughput_mbps" in e else "N/A"
print(f" {e['config']:<38} {e['p50']:>7.4f}s {e['p95']:>7.4f}s "
f"{e['min']:>7.4f}s {e['max']:>7.4f}s {e['stdev']:>7.4f} "
f"{tp:>9} {e['runs']:>5} {e.get('invalid', 0):>4}")
else:
print(f" {e['config']:<25} {'N/A':>8} {'N/A':>8} {'N/A':>8} {'N/A':>8} {'N/A':>8} {e['runs']:>5}")
print(f" {e['config']:<38} {'N/A':>8} {'N/A':>8} {'N/A':>8} {'N/A':>8} "
f"{'N/A':>8} {'N/A':>9} {e['runs']:>5} {e.get('invalid', 0):>4}")
def configure_build_dirs(build_dir):
"""Install the selected build directory and derived binary paths."""
global BUILD_DIR, SERVER_CMD, CLIENT_CMD
if not os.path.isabs(build_dir):
build_dir = os.path.join(PROJECT_ROOT, build_dir)
BUILD_DIR = os.path.abspath(build_dir)
SERVER_CMD = [os.path.join(BUILD_DIR, "server"), "--allow-unauthenticated"]
CLIENT_CMD = [os.path.join(BUILD_DIR, "client")]
def build_project():
"""Configure (Release) and build into the dedicated bench build dir."""
if shutil.which("cmake") is None:
sys.stderr.write("cmake not found in PATH; cannot build\n")
sys.exit(1)
os.makedirs(BUILD_DIR, exist_ok=True)
configure = ["cmake", "-B", BUILD_DIR, "-S", PROJECT_ROOT,
"-DCMAKE_BUILD_TYPE=Release"]
result = subprocess.run(configure, capture_output=True, text=True)
if result.returncode != 0:
sys.stderr.write("CMake configure failed:\n" + result.stdout + result.stderr + "\n")
sys.exit(1)
jobs = str(os.cpu_count() or 1)
result = subprocess.run(["cmake", "--build", BUILD_DIR, "-j", jobs],
capture_output=True, text=True)
if result.returncode != 0:
sys.stderr.write("Build failed:\n" + result.stdout + result.stderr + "\n")
sys.exit(1)
def main():
@@ -412,12 +705,20 @@ Custom network limits (--delay/--jitter/--throughput) override profiles.
Data mix:
Default is ~75%% random/incompressible + ~25%% structured/compressible,
reflecting typical real-world file sets.
reflecting typical real-world file sets. The actual mix is measured and
reported. Transfers are verified (destination must match source) unless
--no-verify is given.
Warm mode:
--warm seeds the destination with an untimed full copy of a pristine base,
then measures only the incremental transfer after modifying a few files.
Examples:
%(prog)s --profiles wan --runs 5
%(prog)s --throughput 50mbit --delay 30ms --jitter 5ms
%(prog)s --random-ratio 0.5 --size-mb 100
%(prog)s --warm --runs 3 --no-rsync
%(prog)s --dry-run --size-mb 4 --random-ratio 0.25
""")
parser.add_argument("--runs", type=int, default=3,
help="Number of runs per config (default: 3)")
@@ -425,7 +726,8 @@ Examples:
choices=list(NETWORK_PROFILES.keys()),
help="Predefined network profiles (default: unlimited)")
parser.add_argument("--configs", nargs="+", default=None,
help="Custom FastSync config flags")
help="Custom FastSync config flags (shell-quoted, e.g. "
"\"-j -z --chunk-serialization\")")
parser.add_argument("--size-mb", type=int, default=25,
help="Test data size in MB (default: 25)")
parser.add_argument("--random-ratio", type=float, default=0.75,
@@ -440,6 +742,14 @@ Examples:
help="Custom packet loss (e.g. 1%%)")
parser.add_argument("--no-rsync", action="store_true",
help="Skip rsync comparison")
parser.add_argument("--no-verify", action="store_true",
help="Skip source/destination verification after each run")
parser.add_argument("--warm", action="store_true",
help="Incremental mode: seed dest first, measure only changes")
parser.add_argument("--build-dir", default=DEFAULT_BUILD_DIR,
help=f"Build directory (default: {DEFAULT_BUILD_DIR})")
parser.add_argument("--dry-run", action="store_true",
help="Only generate data and report its composition, then exit")
parser.add_argument("--progress", action="store_true",
help="Show progress bar with ETA")
parser.add_argument("--output", choices=["table", "json"], default="table",
@@ -448,12 +758,48 @@ Examples:
help="Don't clean up test data")
args = parser.parse_args()
# Build
print("Building...")
if os.system(f"cmake -B {BUILD_DIR} -S {PROJECT_ROOT} > /dev/null 2>&1") != 0:
print("CMake configure failed"); sys.exit(1)
if os.system(f"cmake --build {BUILD_DIR} -j$(nproc) > /dev/null 2>&1") != 0:
print("Build failed"); sys.exit(1)
if not 0.0 <= args.random_ratio <= 1.0:
parser.error("--random-ratio must be between 0.0 and 1.0")
if args.size_mb <= 0:
parser.error("--size-mb must be positive")
configure_build_dirs(args.build_dir)
# Generate data
source_dir = os.path.join(BENCH_DIR, "source")
dest_dir = os.path.join(BENCH_DIR, "dest")
stats = generate_bench_data(source_dir, args.size_mb, args.random_ratio)
total_bytes = stats["total_bytes"]
comp_pct = stats["compressible_bytes"] / total_bytes * 100 if total_bytes else 0
rand_pct = stats["random_bytes"] / total_bytes * 100 if total_bytes else 0
print(f"Generated {total_bytes / (1024*1024):.1f} MB in {stats['files']} files "
f"({rand_pct:.0f}% random, {comp_pct:.0f}% compressible actual)",
file=sys.stderr)
if args.dry_run:
print(f"size_mb={args.size_mb} random_ratio={args.random_ratio:.4f} "
f"total_bytes={stats['total_bytes']} "
f"compressible_bytes={stats['compressible_bytes']} "
f"random_bytes={stats['random_bytes']} files={stats['files']}")
if not args.keep_data:
shutil.rmtree(BENCH_DIR, ignore_errors=True)
return
# Warm mode: keep a pristine base copy, then mutate the live source.
base_dir = None
measure_bytes = total_bytes
mutation = None
if args.warm:
change_target = max(64 * 1024, min(int(total_bytes * 0.01), 4 * 1024 * 1024))
mutation = apply_incremental_changes(source_dir, change_target)
measure_bytes = mutation["changed"]
revert_incremental_changes(source_dir, mutation)
print(f"Warm mode: each run seeds a full copy, then measures "
f"{measure_bytes / (1024*1024):.3f} MB of add/change deltas", file=sys.stderr)
# Build (Release: benchmarking a debug build is meaningless)
print(f"Building (Release) into {BUILD_DIR}...", file=sys.stderr)
build_project()
# Determine active profile for display
has_custom_net = args.delay or args.jitter or args.throughput or args.loss
@@ -473,18 +819,10 @@ Examples:
else:
profiles_to_run = args.profiles or ["unlimited"]
# Generate data
source_dir = os.path.join(BENCH_DIR, "source")
dest_dir = os.path.join(BENCH_DIR, "dest")
total_bytes = generate_bench_data(source_dir, args.size_mb, args.random_ratio)
compressible_pct = (1 - args.random_ratio) * 100
random_pct = args.random_ratio * 100
print(f"Generated {total_bytes / (1024*1024):.1f} MB "
f"({random_pct:.0f}% random, {compressible_pct:.0f}% compressible)")
# Build config list
# Build config list (shlex so quoted/space-separated flags survive)
if args.configs:
fastsync_configs = [{"name": c, "flags": c.split(), "tool": "fastsync"} for c in args.configs]
fastsync_configs = [{"name": c, "flags": shlex.split(c), "tool": "fastsync"}
for c in args.configs]
else:
fastsync_configs = list(FASTSYNC_CONFIGS)
@@ -496,14 +834,21 @@ Examples:
total_runs = len(configs) * args.runs * len(profiles_to_run)
progress = Progress(total_runs, "Benchmarking") if args.progress else None
if progress:
print(f"Running {total_runs} transfers...")
print(f"Running {total_runs} transfers...", file=sys.stderr)
all_results = []
try:
for profile in profiles_to_run:
results = run_benchmark(source_dir, dest_dir, configs, args.runs, profile, progress)
results = run_benchmark(source_dir, dest_dir, configs, args.runs, profile,
measure_bytes, verify=not args.no_verify,
warm=args.warm, mutation=mutation,
progress=progress)
all_results.extend(results)
except RuntimeError as exc:
sys.stderr.write(f"error: {exc}\n")
sys.exit(1)
finally:
netem_reset()
if not args.keep_data:
shutil.rmtree(BENCH_DIR, ignore_errors=True)
@@ -511,7 +856,7 @@ Examples:
if args.output == "json":
print(json.dumps(all_results, indent=2))
else:
print_table(all_results, total_bytes, args.random_ratio)
print_table(all_results, measure_bytes, stats, args.warm)
print()
+10
View File
@@ -0,0 +1,10 @@
[pytest]
; Fast integration subset run on every pull request (see .gitea/workflows/ci.yaml).
markers =
ci: fast, representative integration tests run on the PR CI gate
setpriv: privilege-dependent tests (drop to an unprivileged user); excluded
from CI because their result depends on the runner/container uid and the
host mount permissions, but run locally as root
daemon_detach: real double-fork backgrounding path (--daemon without
--no-detach); slower/fragile, so it runs in the full suite but not the
fast PR gate
+34 -3
View File
@@ -3,11 +3,35 @@
}:
pkgs.mkShell {
# Development shell for FastSync. Provides the host-side toolchain needed to
# build, lint, unit-test, integration-test and benchmark the project.
# It deliberately does NOT build on entry: run the CMake commands in README.md
# (or use the CI Docker image for exact CI parity).
nativeBuildInputs = with pkgs; [
# build
gcc
cmake
gnumake
pkg-config
# lint / static analysis (matches CI)
clang-tools # clang-format
cppcheck
# tests
(python3.withPackages (ps: with ps; [ pytest pytest-xdist psutil ]))
openssh # SSH transport integration tests
# debugging
gdb
valgrind
# coverage
lcov
# benchmark tooling
rsync
iproute2 # tc/netem for network shaping
# misc
git
curl
nodejs
nixpkgs-fmt
docker
tea
];
@@ -15,14 +39,21 @@ pkgs.mkShell {
buildInputs = with pkgs; [
zstd
openssl
(python3.withPackages (ps: with ps; [ pytest ]))
];
# The CMake configure step fetches xxHash via FetchContent, which needs
# network access; NIX_ENFORCE_PURITY must be off so the sandbox does not block.
NIX_ENFORCE_PURITY = 0;
shellHook = ''
export NIX_ENFORCE_PURITY=0
cmake -B build
export PATH="$PWD/build:$PATH"
# Make an existing build tree available on PATH, but never build here.
if [ -d "$PWD/build" ]; then
export PATH="$PWD/build:$PATH"
fi
echo "FastSync dev shell ready."
echo " Build: cmake -B build -S . && cmake --build build -j\$(nproc)"
echo " Unit: ./build/tests"
echo " CI parity: docker run --rm --user \"\$(id -u):\$(id -g)\" -v \"\$PWD:/workspace\" -w /workspace gitea.tap-tap.win/taptap/fastsync-ci:v10 ..."
'';
}
+1515 -461
View File
File diff suppressed because it is too large. Load diff
+1024 -164
View File
File diff suppressed because it is too large. Load diff
+19 -2
View File
@@ -4,10 +4,27 @@
#include "chunk.h"
#include "config.h"
#include "transport_tcp.h"
#include <signal.h>
#include <stdbool.h>
int send_chunk(Client* client, Chunk* chunk, Config* config);
/* Set ONLY by the client's SIGINT/SIGTERM handler (async-signal-safe: the
* handler stores 1 and does nothing else). The send loops poll it via
* client_abort_pending() and, when set, best-effort send STATUS_ABORT so the
* receiver can clean up before the client exits. */
extern volatile sig_atomic_t client_abort_requested;
bool client_abort_pending(void);
/* Arm/disarm abort handling around the network phase. While disarmed, a
* SIGINT/SIGTERM takes the default action (immediate termination) so local-only
* modes are not left unresponsive. Defined in client_cli.c. */
void client_set_abort_armed(bool armed);
/* Both sender entry points BORROW `config` for the duration of the call; they
* never free it, and the caller retains ownership (freeing it with
* config_delete() once the call returns). */
int send_files(Config* config);
/* Takes ownership only when *config is set to NULL on return. */
int send_files_multithreaded(Config** config);
/* Phase 6 residual-batch (client-only). See client_send.c. */
int write_batch_from_source(const Config* config, const char* batch_path);
int apply_batch_to_dest(const Config* config, const char* batch_path, const char* dest_root);
#endif
+66 -47
View File
@@ -1,27 +1,57 @@
#include "client_validation.h"
#include "delay_updates.h"
#include "log.h"
#include "usage.h"
#include "utils.h"
#include <string.h>
#include <stdio.h>
/* Validate config after parsing. Returns true if valid. */
bool validate_config(const Config* config) {
if (!config->send_directory || !config->receive_root_directory) {
/* Phase 6 residual-batch modes relax the normal source+destination pair: the
batch driver is local and needs only what it consumes. --only-write-batch
emits a batch from the source (no destination, no server);
--read-batch applies a batch to the destination (no source, no server);
--write-batch runs the live transfer AND emits a batch, so it keeps the
full pair. */
bool write_batch = config->write_batch != NULL;
bool only_write_batch = config->only_write_batch != NULL;
bool read_batch = config->read_batch != NULL;
if ((write_batch && only_write_batch) || (write_batch && read_batch) ||
(only_write_batch && read_batch)) {
log_message(LOG_LEVEL_ERROR,
"--write-batch, --only-write-batch, and --read-batch are mutually exclusive");
return false;
}
/* A dry-run of a local batch apply is not meaningful: --read-batch bypasses
the client-side scan/server decision entirely, so dry-run would have no
wire state to report (and must not be used as a mutation escape hatch).
--only-write-batch likewise never contacts a receiver. --write-batch DOES
run a live transfer but additionally mutates the filesystem by emitting the
batch file, so a dry-run must not write it either. Reject all three up
front instead of silently ignoring --dry-run. */
if (config->dry_run && (read_batch || only_write_batch || write_batch)) {
log_message(LOG_LEVEL_ERROR,
"--dry-run cannot be combined with --read-batch, --only-write-batch, or "
"--write-batch; a dry-run must not mutate anything, including batch files");
return false;
}
if (read_batch) {
if (!config->receive_root_directory) {
log_message(LOG_LEVEL_ERROR, "--read-batch requires a destination directory");
print_usage();
return false;
}
} else if (only_write_batch) {
if (!config->send_directory) {
log_message(LOG_LEVEL_ERROR, "--only-write-batch requires a source directory");
print_usage();
return false;
}
} else if (!config->send_directory || !config->receive_root_directory) {
log_message(LOG_LEVEL_ERROR, "source and destination directories are required");
print_usage();
return false;
}
if (config_has_basis(config) && config->use_chunk_serialization) {
log_message(LOG_LEVEL_ERROR,
"--compare-dest/--copy-dest/--link-dest require per-file incremental checks and "
"cannot be combined with -s (chunk serialization)");
return false;
}
if (config->use_sendfile && (config->use_chunk_serialization || config->use_compression)) {
log_message(LOG_LEVEL_ERROR, "-f/--sendfile cannot be combined with -c (compression) or -s "
"(chunk serialization)");
return false;
}
if (config->compression_threads > 0 && !config->use_compression) {
log_message(LOG_LEVEL_ERROR, "--compress-threads requires compression (-c or -z)");
return false;
@@ -30,57 +60,46 @@ bool validate_config(const Config* config) {
log_message(LOG_LEVEL_ERROR, "-f/--sendfile is not supported with SSH transport");
return false;
}
if (config->use_incremental && config->use_chunk_serialization) {
log_message(LOG_LEVEL_ERROR, "--incremental is not supported with -s (chunk serialization)");
return false;
}
if (config->skip_compress_set && config->use_chunk_serialization) {
log_message(LOG_LEVEL_ERROR,
"--skip-compress cannot be combined with -s (chunk serialization)");
return false;
}
if (config->use_delta && !config->whole_file && !config->use_incremental) {
log_message(LOG_LEVEL_ERROR, "--delta requires --incremental");
return false;
}
if (config->use_delta && !config->whole_file && config->use_chunk_serialization) {
log_message(LOG_LEVEL_ERROR, "--delta cannot be combined with -s (chunk serialization)");
return false;
}
if (config->use_delta && !config->whole_file && config->use_sendfile) {
log_message(LOG_LEVEL_ERROR, "--delta cannot be combined with -f (sendfile)");
/* -4 and -6 are mutually exclusive: a socket address family cannot be both. */
if (config->ipv4 && config->ipv6) {
log_message(LOG_LEVEL_ERROR, "-4/--ipv4 and -6/--ipv6 are mutually exclusive");
return false;
}
if (config->log_file_format && !config->log_file) {
log_message(LOG_LEVEL_ERROR, "--log-file-format requires --log-file");
return false;
}
if (config->append || config->append_verify) {
fprintf(
stderr,
"Error: --append and --append-verify are not supported yet; refusing to ignore option\n");
return false;
}
if (config->use_tls) {
if (!config->tls_cert || !config->tls_key || !config->tls_ca) {
log_message(LOG_LEVEL_ERROR, "--tls requires --cert, --key, and --ca");
return false;
}
}
if (config->delay_updates && config->inplace) {
log_message(LOG_LEVEL_ERROR, "--delay-updates does not work with --inplace");
/* Daemon credentials (A7, protocol 2.19.0): a --password-file would send the
username in the clear and derive a SCRAM proof a network sniffer could
attack offline, so it is only allowed over TLS (which itself mandates a
verified --cert/--key/--ca set above) or to a loopback destination. A
remote plaintext daemon is refused here, before any network I/O. */
if (config->password_file && !config->use_tls && !utils_host_is_loopback(config->server_host)) {
log_message(LOG_LEVEL_ERROR, "sending daemon credentials to a non-local server requires --tls");
return false;
}
if (config->delay_updates && delay_updates_staging_name_conflict(config->backup_dir)) {
log_message(LOG_LEVEL_ERROR,
"--backup-dir is reserved when --delay-updates is active (used for the internal "
"staging directory)");
/* Every cross-field invariant the receiver enforces lives in one shared
predicate so the client and the server can never disagree. The client
reports the specific reason here, before any network I/O. */
const char* invariants_error = config_invariants_error(config);
if (invariants_error) {
log_message(LOG_LEVEL_ERROR, "%s", invariants_error);
return false;
}
if (!config_has_valid_delete_timing(config)) {
/* --protocol: FastSync has exactly one wire format, so the forced version
must equal the current PROTOCOL_VERSION exactly. Rejected here, before any
network I/O, rather than letting the server hit its own mismatch check. */
if (strcmp(config->version, PROTOCOL_VERSION) != 0) {
log_message(LOG_LEVEL_ERROR,
"--delete-before/--delete-during/--delete-delay/--delete-after select the delete "
"timing; at most one may be given and each implies --delete");
"--protocol must be %s (FastSync supports only its current wire "
"protocol version and cannot speak an older or virtual one)",
PROTOCOL_VERSION);
return false;
}
return true;
+351 -108
View File
@@ -10,10 +10,13 @@
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <sys/sysmacros.h>
#include <threads.h>
#include <unistd.h>
#include <limits.h>
#include "xattr.h"
typedef struct {
char* path;
int depth;
@@ -112,6 +115,12 @@ typedef struct {
char* path;
struct stat stats;
bool is_directory;
/* True when the entry should be carried through as a SYMLINK (is_symlink)
rather than a dereferenced file/directory. When true, `link_target` holds
the owned target string to transmit (sender-munged under --munge-links);
ownership transfers to the File built from this entry. */
bool is_symlink;
char* link_target;
/* True when the entry was pruned by a user selection rule (--filter/-C/per-dir
rules, the --exclude/--include layer, or --max-size/--min-size) rather than
skipped for another reason (unreadable, symlink policy, not applicable). */
@@ -165,6 +174,70 @@ static bool entry_passes_selection(const FileListSet* file_list, const FilterRul
return true;
}
/* Best-effort capture of the file's whitelisted xattrs (-X/-A). A failure to
* read xattrs is non-fatal: the file is transferred without them. */
static void scanner_capture_xattrs(const DirectoryScanner* scanner, File* file) {
if (!scanner || !file || !(scanner->options.preserve_xattrs || scanner->options.preserve_acls))
return;
file->xattrs = xattr_capture_path(file->path, scanner->options.preserve_acls);
}
/* Apply --hard-links (-H) detection to one regular File. On a sibling (a
* later member of an already-seen source inode) the File keeps the group id
* and the first member's wire path but carries NO data payload (size 0); the
* first member is left untouched (data present, link_first). Allocation
* failure is fatal: the scanner is marked failed. */
static void scanner_assign_hardlink(DirectoryScanner* scanner, HardLinkTable* table, File* file,
const struct stat* stats) {
if (!table || !file || !stats)
return;
int gid;
bool is_first;
char* first_path = NULL;
if (!hardlink_table_assign(table, file_wire_path(file), stats->st_dev, stats->st_ino, &gid,
&is_first, &first_path)) {
if (scanner)
scanner->failed = true;
return;
}
file->link_group = gid;
file->link_first = is_first;
if (!is_first) {
file->hardlink_target = first_path;
file->data->size = 0;
} else {
free(first_path);
}
}
/* Phase 4 special/devices: detect a device (char/block), FIFO or socket entry
and, when the matching --devices/--specials flag asks it be preserved,
convert the File into a node to recreate (is_special, empty payload) with its
device rdev captured from the source stat. When the entry is not preserved
(or --copy-devices instead copies its content as an ordinary regular file)
the File is left as a normal data file. Returns true when converted. */
static bool scanner_prepare_special(bool preserve_devices, bool preserve_specials, File* file,
const struct stat* stats) {
if (!file || !stats)
return false;
bool is_device = S_ISCHR(stats->st_mode) || S_ISBLK(stats->st_mode);
bool is_fifo = S_ISFIFO(stats->st_mode);
bool is_socket = S_ISSOCK(stats->st_mode);
if (!is_device && !is_fifo && !is_socket)
return false;
bool preserve = is_device ? preserve_devices : preserve_specials;
if (!preserve)
return false;
file->is_special = true;
file->data->size = 0;
file->data->data = NULL;
if (is_device) {
file->rdev_major = (int32_t)major(stats->st_rdev);
file->rdev_minor = (int32_t)minor(stats->st_rdev);
}
return true;
}
/* Append `rel` to the caller's exclusion sink, taking `mtx` when shared across
parallel worker threads. Returns false on allocation failure (list left
unchanged). */
@@ -190,10 +263,10 @@ static bool excluded_sink_append(ArrayList* list, mtx_t* mtx, const char* rel) {
how manifest keep entries are stored), so the receiver's walker prefixes
match the destination layout. An allocation failure is a fatal scan error. */
static void scanner_record_excluded(DirectoryScanner* scanner, const char* fs_path) {
if (!scanner->excluded_paths || !fs_path)
if (!scanner->options.excluded_paths || !fs_path)
return;
const char* rel = *fs_path == '/' ? fs_path + 1 : fs_path;
if (!excluded_sink_append(scanner->excluded_paths, scanner->excluded_mutex, rel))
if (!excluded_sink_append(scanner->options.excluded_paths, scanner->options.excluded_mutex, rel))
scanner->failed = true;
}
@@ -201,7 +274,7 @@ static void scanner_record_excluded(DirectoryScanner* scanner, const char* fs_pa
* context, returning the context used for this directory's entries. On a parse
* error the scanner is marked failed. Returns 0 on success, -1 on failure. */
static int open_directory_filter_context(DirectoryScanner* scanner, const FilterNode* inherited) {
if (!scanner->per_dir_filters) {
if (!scanner->options.per_dir_filters) {
scanner->current_node = (FilterNode*)inherited;
return 0;
}
@@ -211,7 +284,10 @@ static int open_directory_filter_context(DirectoryScanner* scanner, const Filter
filter_file_read(scanner->current_path, scanner->current_rel ? scanner->current_rel : "",
&exists, err, sizeof(err));
if (!own) {
log_message(LOG_LEVEL_ERROR, "invalid .rsync-filter in %s: %s", scanner->current_path, err);
char* escaped_path = output_escape(scanner->current_path, log_get_8_bit_output());
log_message(LOG_LEVEL_ERROR, "invalid .rsync-filter in %s: %s",
escaped_path ? escaped_path : "<allocation failed>", err);
free(escaped_path);
scanner->failed = true;
return -1;
}
@@ -235,6 +311,8 @@ static int scanner_inspect_entry(const ScannerOptions* options, const char* sour
const char* containing_dir, const char* name,
ScannerEntry* entry) {
entry->excluded = false;
entry->is_symlink = false;
entry->link_target = NULL;
entry->path = path_cat(containing_dir, name);
if (!entry->path)
return -1;
@@ -245,38 +323,85 @@ static int scanner_inspect_entry(const ScannerOptions* options, const char* sour
return 0;
}
bool is_symlink = S_ISLNK(link_stats.st_mode);
if (is_symlink && !options->follow_symlinks && !options->copy_links && !options->safe_links &&
!options->copy_unsafe_links)
if (!is_symlink)
goto regular;
/* Symlink: choose between dereferencing (---copy-links / --safe-links /
--copy-unsafe-links, plus -k for symlinks-to-directories) and carrying the
link through as a symlink (-l, and -k for symlinks-to-files). No link
option means the symlink is skipped entirely (pre-existing behavior). */
const bool any_link_option = options->follow_symlinks || options->copy_links ||
options->safe_links || options->copy_unsafe_links ||
options->copy_dirlinks;
if (!any_link_option)
goto skip;
if (is_symlink && options->safe_links) {
char link_target[4096];
ssize_t length = readlink(entry->path, link_target, sizeof(link_target) - 1);
if (length < 0)
goto skip;
link_target[length] = '\0';
char link_target[4096];
ssize_t length = readlink(entry->path, link_target, sizeof(link_target) - 1);
if (length < 0)
goto skip;
link_target[length] = '\0';
if (options->safe_links) {
if (link_target[0] == '/' || !safe_relative_link(source_root, containing_dir, link_target))
goto skip;
}
if (is_symlink && options->copy_unsafe_links && !options->copy_links) {
char link_target[4096];
ssize_t length = readlink(entry->path, link_target, sizeof(link_target) - 1);
if (length < 0)
goto skip;
link_target[length] = '\0';
if (options->copy_unsafe_links && !options->copy_links) {
if (link_target[0] != '/')
goto skip;
}
if (is_symlink && options->follow_symlinks && !options->copy_links)
entry->stats = link_stats;
else if (stat(entry->path, &entry->stats) != 0)
goto skip;
bool emit_symlink = false;
if (options->copy_links) {
emit_symlink = false; /* --copy-links dereferences every referent */
} else if (options->safe_links || options->copy_unsafe_links) {
emit_symlink = false; /* preserve pre-existing dereference behavior */
} else if (options->copy_dirlinks) {
struct stat ref;
if (stat(entry->path, &ref) == 0 && S_ISDIR(ref.st_mode))
emit_symlink = false; /* -k: symlink to a directory recurses as a dir */
else
emit_symlink = true; /* -k: symlink to a file stays a symlink */
} else if (options->follow_symlinks) {
emit_symlink = true; /* -l: copy symlink as symlink */
}
entry->is_directory = S_ISDIR(entry->stats.st_mode);
if (!emit_symlink) {
if (stat(entry->path, &entry->stats) != 0)
goto skip;
entry->is_directory = S_ISDIR(entry->stats.st_mode);
if (entry->is_directory)
return 1;
goto apply_filters;
}
/* Carry the link as a symlink. --munge-links containment: a target that
could escape the receive root (absolute or containing "..") is never
transmitted -- the entry is merely skipped ("contained"). */
if (link_target[0] == '\0' ||
(options->munge_links && !file_symlink_target_contained(link_target)))
goto skip;
entry->is_symlink = true;
entry->stats = link_stats;
entry->is_directory = false;
entry->link_target =
options->munge_links ? file_symlink_munge(link_target) : str_dup(link_target);
if (!entry->link_target)
goto skip;
goto apply_filters;
regular:
/* Not a symlink: the lstat() above already described this entry, and lstat
and stat are identical for every non-symlink, so reuse that result instead
of issuing a redundant stat() on the scanner hot path. stat() is still
used on the dereference paths above/below for actual symlinks (copy-links,
safe/copy-unsafe links, and -k symlinks-to-directories). */
entry->stats = link_stats;
entry->is_directory = S_ISDIR(link_stats.st_mode);
if (entry->is_directory)
return 1;
apply_filters:
for (int i = 0; i < options->exclude_count; i++)
if (glob_match(options->exclude_patterns[i], name)) {
entry->excluded = true;
@@ -302,6 +427,8 @@ static int scanner_inspect_entry(const ScannerOptions* options, const char* sour
skip:
free(entry->path);
entry->path = NULL;
free(entry->link_target);
entry->link_target = NULL;
return 0;
}
@@ -312,6 +439,11 @@ DirectoryScanner* directory_scanner_create_with_options(const char* root_directo
DirectoryScanner* scanner = calloc(1, sizeof(DirectoryScanner));
if (scanner == NULL)
return NULL;
/* One copy of the scan inputs; normalize chunk_size as the old field-by-field
copy did. */
scanner->options = *options;
if (scanner->options.chunk_size == 0)
scanner->options.chunk_size = DESIRED_CHUNK_SIZE;
scanner->directories = queue_create(100, dir_entry_destroy);
if (!scanner->directories) {
free(scanner);
@@ -319,22 +451,7 @@ DirectoryScanner* directory_scanner_create_with_options(const char* root_directo
}
scanner->current_dir = NULL;
scanner->current_path = NULL;
scanner->use_metadata = options->use_metadata;
scanner->chunk_size = options->chunk_size > 0 ? options->chunk_size : DESIRED_CHUNK_SIZE;
scanner->exclude_patterns = options->exclude_patterns;
scanner->exclude_count = options->exclude_count;
scanner->include_patterns = options->include_patterns;
scanner->include_count = options->include_count;
scanner->max_size = options->max_size;
scanner->min_size = options->min_size;
scanner->max_depth = options->max_depth;
scanner->current_depth = 0;
scanner->follow_symlinks = options->follow_symlinks;
scanner->copy_links = options->copy_links;
scanner->safe_links = options->safe_links;
scanner->copy_unsafe_links = options->copy_unsafe_links;
scanner->checksum = options->checksum;
scanner->one_file_system = options->one_file_system;
scanner->failed = false;
scanner->root_path = str_dup(root_directory);
if (!scanner->root_path) {
@@ -346,23 +463,14 @@ DirectoryScanner* directory_scanner_create_with_options(const char* root_directo
scanner->at_seed_dir = true;
scanner->seed_node = NULL;
scanner->current_node = NULL;
scanner->file_list = options->file_list;
scanner->base_filters = options->base_filters;
scanner->per_dir_filters = options->per_dir_filters;
scanner->excluded_paths = options->excluded_paths;
scanner->excluded_mutex = options->excluded_mutex;
scanner->ignore_io_errors = options->ignore_io_errors;
scanner->ignore_missing_args = options->ignore_missing_args;
scanner->io_error = false;
scanner->dirs_mode = options->dirs;
scanner->relative_mode = options->relative && options->file_list != NULL;
scanner->prune_empty_dirs = options->prune_empty_dirs;
scanner->dirs_root_emitted = false;
scanner->list_index = 0;
scanner->dirs_batch = NULL;
scanner->dirs_batch_size = 0;
scanner->filter_nodes = NULL;
if (scanner->base_filters || scanner->per_dir_filters) {
if (scanner->options.base_filters || scanner->options.per_dir_filters) {
scanner->filter_nodes = array_list_create(filter_node_destroy);
if (!scanner->filter_nodes) {
free(scanner->root_path);
@@ -371,7 +479,7 @@ DirectoryScanner* directory_scanner_create_with_options(const char* root_directo
return NULL;
}
}
if (scanner->one_file_system) {
if (scanner->options.one_file_system) {
struct stat root_stats;
if (stat(root_directory, &root_stats) != 0) {
log_perror("Could not stat source directory");
@@ -453,17 +561,78 @@ void directory_scanner_destroy(DirectoryScanner* scanner) {
static Chunk* chunk_data_to_chunk(ArrayList* chunk_data) {
void** chunk_items = array_list_to_array(chunk_data);
if (!chunk_items)
if (!chunk_items) {
array_list_delete(chunk_data);
return NULL;
}
Chunk* chunk = chunk_create((File**)chunk_items, chunk_data->size);
free(chunk_items);
if (!chunk)
if (!chunk) {
array_list_delete(chunk_data);
return NULL;
}
chunk_data->item_destroyer = NULL;
array_list_delete(chunk_data);
return chunk;
}
/* P7 Wave D: append one traversed source directory's captured metadata to the
* shared pending-directory-time list. The File carries no payload; only the
* wire path (absolute fs path normally, the bare relative path under
* -R + --files-from) and its metadata are used, and the sender transmits them
* in trailing STATUS_DIR_TIMES frame(s). `mutex` (optional) serializes the
* append for the parallel scanner's shared workers. An unstattable or
* non-directory path is silently skipped (the transfer is unaffected); an
* allocation failure is fatal and reported to the caller. */
static bool scanner_capture_dir_time(ArrayList* dir_entries, mtx_t* mutex, const char* root_path,
const char* fs_path, bool relative_mode, bool preserve_atimes,
bool preserve_crtimes) {
if (!dir_entries || !root_path || !fs_path)
return true;
struct stat st;
if (stat(fs_path, &st) != 0 || !S_ISDIR(st.st_mode))
return true;
char* rel = scanner_path_relative(root_path, fs_path);
if (!rel)
return true;
if (relative_mode && rel[0] == '\0') {
/* -R + --files-from: the transfer root itself has no bare relative wire
path (matches the -R scan, which never emits the root). */
free(rel);
return true;
}
File* file = file_create(fs_path);
if (!file) {
free(rel);
return false;
}
file->is_dir = true;
file->metadata = file_metadata_create(fs_path, &st, preserve_atimes, preserve_crtimes);
if (!file->metadata) {
free(rel);
file_destroy(file);
return false;
}
if (relative_mode) {
file->send_path = rel;
rel = NULL;
}
free(rel);
bool added;
if (mutex) {
mtx_lock(mutex);
added = array_list_add(dir_entries, file);
mtx_unlock(mutex);
} else {
added = array_list_add(dir_entries, file);
}
if (!added) {
file_destroy(file);
return false;
}
return true;
}
/* Open the next queued directory and set up its filter context. Returns 1 when
a directory is open, 0 when the queue is exhausted, and -1 on a fatal error.
A directory that cannot be opened is an I/O error: it is recorded on the
@@ -516,7 +685,7 @@ static int open_next_directory(DirectoryScanner* scanner) {
scanner->current_rel = NULL;
free(scanner->current_path);
scanner->current_path = NULL;
if (!scanner->ignore_io_errors || is_root_seed) {
if (!scanner->options.ignore_io_errors || is_root_seed) {
scanner->failed = true;
return -1;
}
@@ -530,6 +699,18 @@ static int open_next_directory(DirectoryScanner* scanner) {
scanner->current_path = NULL;
return -1;
}
if (scanner->options.capture_dir_times &&
!scanner_capture_dir_time(scanner->options.dir_entries, scanner->options.dir_entries_mutex,
scanner->root_path, scanner->current_path, scanner->relative_mode,
scanner->options.preserve_atimes,
scanner->options.preserve_crtimes)) {
closedir(scanner->current_dir);
scanner->current_dir = NULL;
free(scanner->current_path);
scanner->current_path = NULL;
scanner->failed = true;
return -1;
}
return 1;
}
return 0;
@@ -563,8 +744,9 @@ static File* dirs_root_dir_file(DirectoryScanner* scanner) {
return NULL;
}
file->is_dir = true;
if (scanner->use_metadata) {
file->metadata = file_metadata_create(&st);
if (scanner->options.use_metadata) {
file->metadata = file_metadata_create(scanner->root_path, &st, scanner->options.preserve_atimes,
scanner->options.preserve_crtimes);
if (!file->metadata) {
file_destroy(file);
scanner->failed = true;
@@ -597,12 +779,20 @@ static File* dirs_file_for_entry(DirectoryScanner* scanner, const char* entry) {
missing argument and is skipped here, exactly as the recursive scan skips
nothing (missing entries never appear there). Without the flags it stays
a hard pre-transfer error. */
if (scanner->ignore_missing_args) {
log_info_message(LOG_INFO_MISC, "skipping missing --files-from entry '%s'", entry);
if (scanner->options.ignore_missing_args) {
char* escaped_entry = output_escape(entry, log_get_8_bit_output());
log_info_message(LOG_INFO_MISC, "skipping missing --files-from entry '%s'",
escaped_entry ? escaped_entry : "<allocation failed>");
free(escaped_entry);
free(abs_path);
return NULL;
}
log_message(LOG_LEVEL_ERROR, "--dirs listed entry is not present under the source: %s", entry);
{
char* escaped_entry = output_escape(entry, log_get_8_bit_output());
log_message(LOG_LEVEL_ERROR, "--dirs listed entry is not present under the source: %s",
escaped_entry ? escaped_entry : "<allocation failed>");
free(escaped_entry);
}
free(abs_path);
scanner->failed = true;
return NULL;
@@ -611,8 +801,8 @@ static File* dirs_file_for_entry(DirectoryScanner* scanner, const char* entry) {
if (S_ISLNK(link_stats.st_mode)) {
/* A symlink is transferred (following its referent) only when a link
resolution option is active, mirroring the regular scanner. */
bool resolve = scanner->follow_symlinks || scanner->copy_links || scanner->safe_links ||
scanner->copy_unsafe_links;
bool resolve = scanner->options.follow_symlinks || scanner->options.copy_links ||
scanner->options.safe_links || scanner->options.copy_unsafe_links;
if (!resolve || stat(abs_path, &effective) != 0) {
free(abs_path);
return NULL;
@@ -640,8 +830,9 @@ static File* dirs_file_for_entry(DirectoryScanner* scanner, const char* entry) {
return NULL;
}
}
if (scanner->use_metadata) {
file->metadata = file_metadata_create(&effective);
if (scanner->options.use_metadata) {
file->metadata = file_metadata_create(file->path, &effective, scanner->options.preserve_atimes,
scanner->options.preserve_crtimes);
if (!file->metadata) {
file_destroy(file);
scanner->failed = true;
@@ -672,18 +863,18 @@ static bool dirs_source_dir_is_empty(const char* path) {
/* The next File from the --dirs generator, or NULL when exhausted. */
static File* dirs_next_file(DirectoryScanner* scanner) {
if (!scanner->file_list) {
if (!scanner->options.file_list) {
if (scanner->dirs_root_emitted)
return NULL;
scanner->dirs_root_emitted = true;
/* --prune-empty-dirs: a physically empty source directory's explicit entry
would only create an empty destination directory, so it is omitted. */
if (scanner->prune_empty_dirs && dirs_source_dir_is_empty(scanner->root_path))
if (scanner->options.prune_empty_dirs && dirs_source_dir_is_empty(scanner->root_path))
return NULL;
return dirs_root_dir_file(scanner);
}
while (scanner->list_index < scanner->file_list->count) {
const char* entry = scanner->file_list->entries[scanner->list_index++];
while (scanner->list_index < scanner->options.file_list->count) {
const char* entry = scanner->options.file_list->entries[scanner->list_index++];
File* file = dirs_file_for_entry(scanner, entry);
if (scanner->failed)
return NULL;
@@ -710,7 +901,14 @@ static Chunk* dirs_flush_batch(DirectoryScanner* scanner) {
}
static Chunk* directory_scanner_next_dirs(DirectoryScanner* scanner) {
while (scanner->dirs_batch == NULL || scanner->dirs_batch_size <= scanner->chunk_size) {
while (scanner->dirs_batch == NULL || scanner->dirs_batch_size <= scanner->options.chunk_size) {
if (scanner->options.stop_condition &&
stop_condition_reached(scanner->options.stop_condition)) {
Chunk* leftover = dirs_flush_batch(scanner);
if (leftover)
chunk_destroy(leftover);
return NULL;
}
if (!scanner->dirs_batch) {
scanner->dirs_batch = array_list_create(file_destroy);
if (!scanner->dirs_batch) {
@@ -744,7 +942,7 @@ static Chunk* directory_scanner_next_dirs(DirectoryScanner* scanner) {
}
Chunk* directory_scanner_next(DirectoryScanner* scanner) {
if (scanner && scanner->dirs_mode)
if (scanner && scanner->options.dirs)
return directory_scanner_next_dirs(scanner);
ArrayList* chunk_data = array_list_create(file_destroy);
if (!chunk_data) {
@@ -754,6 +952,11 @@ Chunk* directory_scanner_next(DirectoryScanner* scanner) {
unsigned long long chunk_data_size = 0;
while (1) {
if (scanner->options.stop_condition &&
stop_condition_reached(scanner->options.stop_condition)) {
array_list_delete(chunk_data);
return NULL;
}
if (scanner->current_dir == NULL) {
int ret = open_next_directory(scanner);
if (ret == 0)
@@ -774,32 +977,9 @@ Chunk* directory_scanner_next(DirectoryScanner* scanner) {
if (strcmp(entry->d_name, ".") == 0 || strcmp(entry->d_name, "..") == 0)
continue;
ScannerOptions options = {
.use_metadata = scanner->use_metadata,
.chunk_size = scanner->chunk_size,
.exclude_patterns = scanner->exclude_patterns,
.exclude_count = scanner->exclude_count,
.include_patterns = scanner->include_patterns,
.include_count = scanner->include_count,
.max_size = scanner->max_size,
.min_size = scanner->min_size,
.max_depth = scanner->max_depth,
.num_threads = 0,
.follow_symlinks = scanner->follow_symlinks,
.copy_links = scanner->copy_links,
.safe_links = scanner->safe_links,
.copy_unsafe_links = scanner->copy_unsafe_links,
.checksum = scanner->checksum,
.one_file_system = scanner->one_file_system,
.file_list = scanner->file_list,
.base_filters = scanner->base_filters,
.per_dir_filters = scanner->per_dir_filters,
.dirs = false,
.relative = false,
};
ScannerEntry inspected;
int inspection = scanner_inspect_entry(&options, scanner->current_path, scanner->current_path,
entry->d_name, &inspected);
int inspection = scanner_inspect_entry(&scanner->options, scanner->current_path,
scanner->current_path, entry->d_name, &inspected);
if (inspection < 0) {
scanner->failed = true;
break;
@@ -831,16 +1011,17 @@ Chunk* directory_scanner_next(DirectoryScanner* scanner) {
scanner->failed = true;
break;
}
bool passes_selection =
entry_passes_selection(scanner->file_list, scanner->base_filters, scanner->current_node,
rel, entry->d_name, is_dir, scanner->per_dir_filters);
bool passes_selection = entry_passes_selection(
scanner->options.file_list, scanner->options.base_filters, scanner->current_node, rel,
entry->d_name, is_dir, scanner->options.per_dir_filters);
if (!passes_selection) {
/* --files-from subset pruning is not a filter exclusion: its delete
semantics stay keep-set-only (an unlisted source path is treated as
absent, so its destination mirror is a deletable extra). A rule-based
exclusion is recorded as a protected prefix. -R + --files-from bare
wire paths are never recorded (see ScannerOptions.excluded_paths). */
bool files_from_prune = scanner->file_list && !file_list_affects(scanner->file_list, rel);
bool files_from_prune =
scanner->options.file_list && !file_list_affects(scanner->options.file_list, rel);
if (!files_from_prune && !scanner->relative_mode)
scanner_record_excluded(scanner, cur_path);
}
@@ -861,12 +1042,13 @@ Chunk* directory_scanner_next(DirectoryScanner* scanner) {
if (is_dir) {
free(rel_copy);
if (!scanner_same_filesystem(scanner->one_file_system, scanner->root_dev, stats.st_dev)) {
if (!scanner_same_filesystem(scanner->options.one_file_system, scanner->root_dev,
stats.st_dev)) {
free(cur_path);
continue;
}
int next_depth = scanner->current_depth + 1;
if (scanner->max_depth <= 0 || next_depth < scanner->max_depth) {
if (scanner->options.max_depth <= 0 || next_depth < scanner->options.max_depth) {
DirEntry* de = dir_entry_create(cur_path, next_depth, scanner->current_node);
if (!de || !queue_enqueue(scanner->directories, de)) {
dir_entry_destroy(de);
@@ -875,7 +1057,8 @@ Chunk* directory_scanner_next(DirectoryScanner* scanner) {
}
free(cur_path);
} else {
if (scanner->max_depth > 0 && scanner->current_depth + 1 > scanner->max_depth) {
if (scanner->options.max_depth > 0 &&
scanner->current_depth + 1 > scanner->options.max_depth) {
free(rel_copy);
free(cur_path);
continue;
@@ -884,22 +1067,39 @@ Chunk* directory_scanner_next(DirectoryScanner* scanner) {
free(cur_path);
if (file == NULL) {
free(rel_copy);
free(inspected.link_target);
inspected.link_target = NULL;
scanner->failed = true;
continue;
}
file->data->size = stats.st_size;
if (inspected.is_symlink) {
file->is_symlink = true;
file->symlink_target = inspected.link_target;
inspected.link_target = NULL;
} else {
file->data->size = stats.st_size;
}
if (scanner->relative_mode) {
file->send_path = rel_copy;
rel_copy = NULL;
}
if (scanner->use_metadata)
file->metadata = file_metadata_create(&stats);
if (scanner->use_metadata && !file->metadata) {
/* --devices/--specials: a device/FIFO/socket entry marked for preservation
becomes a node to recreate (is_special, no data, rdev captured). */
scanner_prepare_special(scanner->options.preserve_devices, scanner->options.preserve_specials,
file, &stats);
if (scanner->options.hardlinks && S_ISREG(stats.st_mode))
scanner_assign_hardlink(scanner, scanner->options.hardlinks, file, &stats);
if (scanner->options.use_metadata)
file->metadata = file_metadata_create(file->path, &stats, scanner->options.preserve_atimes,
scanner->options.preserve_crtimes);
if (scanner->options.use_metadata && !file->metadata) {
free(rel_copy);
file_destroy(file);
scanner->failed = true;
break;
}
if (!(file->link_group != 0 && !file->link_first))
scanner_capture_xattrs(scanner, file);
if (!array_list_add(chunk_data, file)) {
free(rel_copy);
file_destroy(file);
@@ -907,7 +1107,7 @@ Chunk* directory_scanner_next(DirectoryScanner* scanner) {
break;
}
chunk_data_size += file->data->size;
if (chunk_data_size > scanner->chunk_size) {
if (chunk_data_size > scanner->options.chunk_size) {
free(rel_copy);
Chunk* result = chunk_data_to_chunk(chunk_data);
if (!result)
@@ -971,7 +1171,7 @@ static int parallel_worker_thread(void* arg) {
free(ds->root_path);
ds->root_path = str_dup(wa->root_dir);
ds->seed_node = wa->ps->root_filter_node;
ds->excluded_mutex = &wa->ps->result_mutex;
ds->options.excluded_mutex = &wa->ps->result_mutex;
Chunk* chunk;
while ((chunk = directory_scanner_next(ds)) != NULL) {
if (!queue_enqueue_multithreaded_cancel(wa->ps->result_queue, chunk, &wa->ps->result_mutex,
@@ -1199,22 +1399,53 @@ static void scan_root_entry(const ScannerOptions* options, const FilterNode* roo
free(cur_path);
if (!file) {
free(rel);
free(inspected.link_target);
inspected.link_target = NULL;
ps->failed = true;
return;
}
file->data->size = st.st_size;
if (inspected.is_symlink) {
file->is_symlink = true;
file->symlink_target = inspected.link_target;
inspected.link_target = NULL;
} else {
file->data->size = st.st_size;
}
if (use_rel) {
file->send_path = rel;
rel = NULL;
}
scanner_prepare_special(options->preserve_devices, options->preserve_specials, file, &st);
if (options->hardlinks && S_ISREG(st.st_mode)) {
int gid;
bool is_first;
char* first_path = NULL;
if (!hardlink_table_assign((HardLinkTable*)options->hardlinks, file_wire_path(file), st.st_dev,
st.st_ino, &gid, &is_first, &first_path)) {
ps->failed = true;
} else {
file->link_group = gid;
file->link_first = is_first;
if (!is_first) {
file->hardlink_target = first_path;
file->data->size = 0;
} else {
free(first_path);
}
}
}
if (options->use_metadata)
file->metadata = file_metadata_create(&st);
file->metadata =
file_metadata_create(file->path, &st, options->preserve_atimes, options->preserve_crtimes);
if (options->use_metadata && !file->metadata) {
free(rel);
file_destroy(file);
ps->failed = true;
return;
}
if ((options->preserve_xattrs || options->preserve_acls) &&
!(file->link_group != 0 && !file->link_first))
file->xattrs = xattr_capture_path(file->path, options->preserve_acls);
if (!array_list_add(root_files, file)) {
free(rel);
file_destroy(file);
@@ -1392,6 +1623,18 @@ ParallelScanner* parallel_scanner_create_with_options(const char* root_directory
parallel_scanner_destroy(ps);
return NULL;
}
/* P7 Wave D: the parallel scanner never runs a DirectoryScanner over the
transfer root itself (it hands the root's immediate subdirectories to
workers), so capture the root's directory time here. */
if (options->capture_dir_times &&
!scanner_capture_dir_time(options->dir_entries, options->dir_entries_mutex, root_directory,
root_directory, options->relative && options->file_list != NULL,
options->preserve_atimes, options->preserve_crtimes)) {
array_list_delete(root_files);
array_list_delete(subdirs);
parallel_scanner_destroy(ps);
return NULL;
}
unsigned long long cs = options->chunk_size > 0 ? options->chunk_size : DESIRED_CHUNK_SIZE;
ps->initial_chunk = batch_files(root_files, cs, ps->result_queue, &ps->failed);
+55 -30
View File
@@ -4,16 +4,30 @@
#include "chunk.h"
#include "file_list.h"
#include "filter.h"
#include "hardlink.h"
#include "protocol.h"
#include "queue.h"
#include "stop_condition.h"
#include <dirent.h>
#include <stdbool.h>
#include <stdatomic.h>
#include <sys/types.h>
#include <threads.h>
/* Upper bound on the configurable parallel scanner worker count (--threads=N):
* keeps one transfer from spawning an unbounded pool on a very large machine. */
#define MAX_SCANNER_THREADS 256
typedef struct {
bool use_metadata;
/* Phase 4 metadata capture: -U/--atimes and -N/--crtimes tell the scanner to
* capture the source access / birth time into each entry's FileMetadata. */
bool preserve_atimes;
bool preserve_crtimes;
/* Phase 4 xattrs: when preserve_xattrs || preserve_acls is set the scanner
* captures each regular file's whitelisted xattr set onto the File. */
bool preserve_xattrs;
bool preserve_acls;
unsigned long long chunk_size;
char** exclude_patterns;
int exclude_count;
@@ -27,8 +41,22 @@ typedef struct {
bool copy_links;
bool safe_links;
bool copy_unsafe_links;
/* Phase 4 symlink-trust sender options: -k/--copy-dirlinks (dereference a
* symlink to a directory as a directory, keeping symlinks-to-files as
* symlinks) and --munge-links (rewrite each transmitted symlink target with a
* marker; escaping targets are never transmitted). Both are client/sender
* side only and never serialized to the wire (keep_dirlinks is the
* receiver-side counterpart). */
bool copy_dirlinks;
bool munge_links;
bool checksum;
bool one_file_system;
/* Phase 4 special/devices: whether device nodes (--devices) and special files
* (--specials) are preserved via recreation, and whether --copy-devices
* copies a device's content as an ordinary regular file. */
bool preserve_devices;
bool preserve_specials;
bool copy_devices;
/* Phase 2 (files-from / filter layer). All pointers are shared read-only
* across scanner instances and worker threads; ownership stays with the
* caller (client_send). */
@@ -64,6 +92,28 @@ typedef struct {
* instead of failing (the --dirs generator is the only scanner path that
* observes a listed-but-missing entry). */
bool ignore_missing_args;
/* --hard-links (-H): shared, mutable (mutex-guarded) link-group detection
* table, NULL when -H is off. Owned by the caller (client_send), shared
* read-only here; the parallel scanner passes it unchanged to every worker so
* one table detects every group across all subdirectories. */
HardLinkTable* hardlinks;
/* Phase 6: optional sender stop deadline. When non-NULL the scanner checks
* it at natural loop boundaries and stops emitting chunks once reached
* (without marking the scan as failed), so a busy scan itself stops early.
* Client-only, never serialized to the wire. */
const StopCondition* stop_condition;
/* P7 Wave D (protocol 2.17.0): directory-time capture sink. When
* `capture_dir_times` is true the recursive scan appends one is_dir File
* (with metadata, no payload) per source directory it traverses to
* `dir_entries`, so the sender can transmit trailing STATUS_DIR_TIMES
* frame(s) and the receiver can apply directory mtimes AFTER all children
* are written. `dir_entries_mutex` (optional) guards the list
* for the parallel scanner's shared worker threads; the caller owns both.
* The --dirs generator does not use this (its directory entries carry their
* metadata inline through STATUS_MKDIR). */
bool capture_dir_times;
ArrayList* dir_entries;
mtx_t* dir_entries_mutex;
} ScannerOptions;
/* Internal per-scanner filter state. FilterNode chains represent the ordered
@@ -71,25 +121,14 @@ typedef struct {
typedef struct FilterNode FilterNode;
typedef struct {
/* Scan inputs, copied once at create time. Everything that is also a
ScannerOptions field lives here (with the normalized chunk_size); only
scanner-owned bookkeeping stays as direct members below. */
ScannerOptions options;
Queue* directories;
DIR* current_dir;
char* current_path;
bool use_metadata;
unsigned long long chunk_size;
char** exclude_patterns;
int exclude_count;
char** include_patterns;
int include_count;
unsigned long long max_size;
unsigned long long min_size;
int max_depth;
int current_depth;
bool follow_symlinks;
bool copy_links;
bool safe_links;
bool copy_unsafe_links;
bool checksum;
bool one_file_system;
dev_t root_dev;
bool failed;
/* Phase 2 (files-from / filter layer). */
@@ -99,27 +138,13 @@ typedef struct {
FilterNode* seed_node; /* inherited context of the seed dir, or NULL */
FilterNode* current_node; /* filter context of the open directory */
ArrayList* filter_nodes; /* owned FilterNode arena (may be NULL) */
const FileListSet* file_list;
const FilterRuleList* base_filters;
bool per_dir_filters;
/* --dirs / -R state for the directory-entry generator (dirs_mode replaces
/* --dirs / -R state for the directory-entry generator (options.dirs replaces
the recursive scan). */
bool dirs_mode;
bool relative_mode; /* file_list && relative: send bare relative wire paths */
bool prune_empty_dirs;
bool dirs_root_emitted;
int list_index;
ArrayList* dirs_batch; /* owned when non-NULL */
unsigned long long dirs_batch_size;
/* Excluded-path sink (see ScannerOptions). `excluded_mutex` is shared across
parallel worker threads. */
ArrayList* excluded_paths;
mtx_t* excluded_mutex;
/* --ignore-errors: continue past unreadable directories (records io_error). */
bool ignore_io_errors;
/* --ignore-missing-args: --dirs listed-but-missing entries are skipped, not
fatal (see ScannerOptions.ignore_missing_args). */
bool ignore_missing_args;
/* A directory could not be opened (I/O error, e.g. EACCES). With
--ignore-errors the scan continues past it and the caller decides what to
do; `failed` is reserved for fatal errors that always abort the scan. */
+144 -23
View File
@@ -2,6 +2,7 @@
#include <stdio.h>
#include <delta.h>
#include <chunk.h>
#include "scanner.h"
void print_usage(void) {
printf("Usage:\n");
@@ -11,18 +12,48 @@ void print_usage(void) {
printf("Destination formats:\n");
printf(" user@host:/path SSH transport (rsync-style)\n");
printf(" host:/path SSH transport (current user)\n");
printf(" host::module/path Daemon TCP transport (fastsync-server --daemon);\n");
printf(" module names a server-side module, path is relative\n");
printf(" within it (connect with --server-port)\n");
printf(" /local/path TCP transport (requires server on localhost:8080)\n");
printf("\n");
printf("Options:\n");
printf(" -c [level] Enable compression (level 1-22, default 5)\n");
printf(" -z [level] Alias for -c\n");
printf(" -a, --archive Archive mode (-c -m -M)\n");
printf(" -c, --checksum Verify content by checksum instead of size+mtime\n");
printf(" -z, --compress [level] Enable compression (level 1-22, default 5)\n");
printf(" -a, --archive rsync archive mode (-rlptgoD): links, metadata,\n");
printf(" devices and specials (not compression/multithreading)\n");
printf(" -n, --dry-run Show what would be transferred\n");
printf(" --remove-source-files Remove regular source files after successful transfer\n");
printf(" -p <port> SSH port (default: 22)\n");
printf(" -p, --perms Preserve permission bits (part of the metadata bundle)\n");
printf(" --ssh-port <port> SSH port (default: 22)\n");
printf(" -e, --rsh <command> Remote shell to launch on the client for the SSH\n");
printf(" transport (default: ssh). The command may include\n");
printf(" arguments, e.g. -e \"ssh -p 2222\"\n");
printf(" --rsync-path <path> Alias for --fastsync-server-path (path to the\n");
printf(" fastsync server binary on the remote side)\n");
printf(" --blocking-io Leave the SSH transport socket without read/write\n");
printf(" timeouts so it blocks naturally\n");
printf(" --outbuf=MODE stdout/stderr buffering: N (none/unbuffered),\n");
printf(" L (line-buffered), or B (block-buffered, default)\n");
printf(" --progress Show transfer progress\n");
printf(" -P Partial mode with progress (retention incomplete)\n");
printf(" -8, --8-bit-output Leave high-bit characters unescaped in output\n");
printf(" --iconv=LOCAL[,REMOTE] Convert file-NAME charsets at the wire boundary:\n");
printf(" LOCAL is the charset of our file names, REMOTE is the\n");
printf(" remote side's charset (defaults to LOCAL). Names are\n");
printf(" converted before transmission and back on receipt; a\n");
printf(" name that cannot be represented in the target charset\n");
printf(" fails that transfer cleanly (rsync-compatible)\n");
printf(" --protocol=NUM Force the wire protocol version (must equal the current\n");
printf(" PROTOCOL_VERSION; FastSync cannot speak older/virtual\n");
printf(" wire formats)\n");
printf(" --write-batch=FILE Run the normal live transfer AND also emit a\n");
printf(" self-contained batch file of the whole source tree\n");
printf(" (implies the single-threaded transfer path)\n");
printf(" --only-write-batch=FILE\n");
printf(" Emit the batch file only (no destination, no server)\n");
printf(" --read-batch=FILE Apply the batch file to the destination (no source, no\n");
printf(" server); takes only the destination as an argument\n");
printf(" --delete Delete files on receiver not in source\n");
printf(" (default timing: delete only after the whole\n");
printf(" transfer has succeeded)\n");
@@ -52,9 +83,8 @@ void print_usage(void) {
printf(" entry's destination mirror receiver-side. Independent of\n");
printf(" --delete (it does not imply --delete; a non-empty directory\n");
printf(" mirror is removed only with --force or --delete)\n");
printf(" --prune-empty-dirs Do not transfer empty directory entries (--dirs mode);\n");
printf(" recursive transfers never send empty dirs. rsync's -m\n");
printf(" short form stays FastSync multithreading\n");
printf(" -m, --prune-empty-dirs Do not transfer empty directory entries (--dirs mode);\n");
printf(" recursive transfers never send empty dirs\n");
printf(" Note: each timing flag implies --delete. Combining a timing flag with\n");
printf(" --no-delete (in either order) is rejected as a config error.\n");
printf(" --ignore-existing Skip files that already exist on receiver\n");
@@ -77,8 +107,8 @@ void print_usage(void) {
printf(" --files-from <file> Read the source file list from FILE (paths relative to the "
"source root)\n");
printf(" -0, --from0 Entries in --files-from are NUL-delimited\n");
printf(" --filter=RULE rsync-style filter rule (+/- include/exclude; repeatable; the\n");
printf(" rsync -f short form conflicts with FastSync sendfile -f)\n");
printf(" -f, --filter=RULE rsync-style filter rule (+/- include/exclude; repeatable;\n");
printf(" both --filter=RULE and the -f RULE / -f=RULE short forms work)\n");
printf(" -C, --cvs-exclude Auto-ignore common CVS/SCM files (.git/, .svn/, *.o, *~, ...)\n");
printf(" -F Apply per-directory .rsync-filter files during the scan\n");
printf(" --max-size <n> Skip files larger than n bytes\n");
@@ -97,8 +127,12 @@ void print_usage(void) {
printf(" into the destination instead of transferring its data\n");
printf(" --link-dest <dir> Like --copy-dest, but hard-links the unchanged file from DIR\n");
printf(" into the destination (repeatable; earlier DIRs win)\n");
printf(" --checksum-choice, --cc <alg> Checksum algorithm (not supported yet; xxHash64 is "
"used)\n");
printf(" --checksum-choice, --cc <alg> Whole-file checksum algorithm for --incremental/\n");
printf(" --checksum compares (xxh64/xxhash or md5; default xxh64 with\n");
printf(" seed 0). The seed comes from --checksum-seed\n");
printf(" --checksum-seed <num> Seed for the whole-file xxHash64 digest (and the delta\n");
printf(" block strong hash, low 32 bits); md5 ignores the seed. The\n");
printf(" digest algorithm and seed must match on sender and receiver\n");
printf(" --delta Delta transfer for changed files (requires --incremental)\n");
printf(" -W, --whole-file Transfer changed files without delta processing\n");
printf(" -y, --fuzzy Use a similar-named file already in the destination\n");
@@ -107,14 +141,18 @@ void print_usage(void) {
printf(" --incremental and --delta; inert with --whole-file,\n");
printf(" --no-delta, or --no-incremental)\n");
printf(" --no-fuzzy Disable --fuzzy\n");
printf(" --delta-block <n> Delta block size in bytes (default: %d)\n",
DELTA_BLOCK_SIZE_DEFAULT);
printf(" --delta-block <n>, --block-size <n>\n");
printf(" Delta block size in bytes (default: %d)\n", DELTA_BLOCK_SIZE_DEFAULT);
printf(" --delta-max <n> Max file size for delta transfer (default: %llu)\n",
DELTA_MAX_FILE_SIZE);
printf(" -m Enable multithreading\n");
printf(" -s Enable chunk serialization\n");
printf(" --secluded-args Accept rsync compatibility option (no effect)\n");
printf(" -f Enable sendfile (TCP only, not with -c or -s)\n");
printf(" -j, --threads[=N] Enable the multithreaded scanner/loader/sender\n");
printf(" pipeline; N (1-%d) sets the parallel scanner worker\n",
MAX_SCANNER_THREADS);
printf(" count (bare -j/--threads uses the default)\n");
printf(" --chunk-serialization Enable chunk serialization (long form only)\n");
printf(" -s, --secluded-args Protect-args compatibility option (no effect; remote\n");
printf(" SSH argv is already built injection-safe)\n");
printf(" --sendfile Enable sendfile zero-copy (TCP only; long form only)\n");
printf(" --compress-choice <alg> Compression algorithm (default: zstd)\n");
printf(" --zc <alg> Alias for --compress-choice\n");
printf(" -v, --verbose Enable debug logging\n");
@@ -122,23 +160,79 @@ void print_usage(void) {
printf(" --debug=FLAGS Fine-grained debug logging (use --debug=help for flags)\n");
printf(" --info=FLAGS Fine-grained info: copy,misc,skip,stats,all,none\n");
printf(" none suppresses info even with --verbose\n");
printf(" -M, --preserve Preserve file metadata\n");
printf(" --preserve Preserve file metadata (long form only)\n");
printf(" -E, --executability Preserve executable permission bits\n");
printf(" -X, --xattrs Preserve user extended attributes (user.* only;\n");
printf(" privileged security.*/trusted.* namespaces are\n");
printf(" never captured or applied)\n");
printf(" -A, --acls Preserve POSIX ACLs (the system.posix_acl_* xattrs;\n");
printf(" setting an ACL the receiver is not permitted to\n");
printf(" set is warned and skipped, never fatal)\n");
printf(" --fake-super Store the source uid/gid/mode/mtime in a reserved\n");
printf(" user.fastsync.stat xattr on each written file and\n");
printf(" re-apply it (fd-relative) on a privileged run; the\n");
printf(" recording format diverges from rsync's user.rsync.%%stat%%\n");
printf(" --super Permit the receiver to attempt super-user activities\n");
printf(" (char/block device-node creation, --write-devices)\n");
printf(" within the confined receive root. Never elevates\n");
printf(" privileges and never bypasses confinement; ownership\n");
printf(" is still applied only with an explicit identity flag\n");
printf(" (--numeric-ids/--chown/--usermap/--groupmap/--copy-as)\n");
printf(" --no-super Forbid those super-user activities even when the\n");
printf(" receiver is running as root\n");
printf(" --chmod <changes> Modify transferred permissions (rsync syntax)\n");
printf(" --numeric-ids Do not map uid/gid by name: use the source numeric\n");
printf(" ids directly when applying ownership\n");
printf(" --usermap=MAP Map usernames when applying ownership: comma-separated\n");
printf(" FROM:TO rules, first match wins. FROM/TO are names\n");
printf(" (resolved on the source machine), * (match any /\n");
printf(" current user), or @N numeric ids. e.g. *:nobody\n");
printf(" --groupmap=MAP Map group names when applying ownership (same syntax)\n");
printf(" --chown=USER:GROUP Override the ownership of transferred files. Forms:\n");
printf(" USER:GROUP, USER (owner only), :GROUP (group only); a\n");
printf(" value of * means the current/root user as appropriate.\n");
printf(" Names resolve on the source machine; @N for numerics.\n");
printf(" (Metadata is enabled with --preserve; -M now means\n");
printf(" rsync's --remote-option.)\n");
printf(" --copy-as=USER[:GROUP] Force every written entry (files, dirs, symlinks\n");
printf(" and special nodes) to USER[:GROUP], resolved on the\n");
printf(" source machine like --chown. Requires a privileged\n");
printf(" (root) receiver and implies --preserve; an\n");
printf(" unprivileged receiver refuses the transfer. Never\n");
printf(" switches process credentials (safe-subset; see\n");
printf(" RSYNC_COMPAT.md). A daemon refuses it.\n");
printf(" --chunk-size <n> Chunk size in bytes (default: %d)\n", DEFAULT_CHUNK_SIZE);
printf(" --source-dir <path> Source directory\n");
printf(" --dest-dir <path> Destination directory\n");
printf(" --save-to-disk Write received files to disk\n");
printf(" --server-host <ip> Server IP address (default: 127.0.0.1)\n");
printf(" --server-port <n> Server port (default: 8080)\n");
printf(" --port <n> Alias for --server-port\n");
printf(" --password-file <f> Authenticate a host::module/path daemon destination.\n");
printf(" The file's first user:password line supplies the\n");
printf(" username and password (only a SHA-256 digest of the\n");
printf(" password is sent; keep the file mode 0600)\n");
printf(" --no-motd Suppress display of the daemon's MOTD (the server\n");
printf(" still sends it; the client just does not show it)\n");
printf(" --bwlimit <KB/s> Bandwidth limit in kilobytes per second\n");
printf(" --tls Enable TLS encryption\n");
printf(" --cert <path> TLS certificate file (PEM)\n");
printf(" --key <path> TLS private key file (PEM)\n");
printf(" --ca <path> TLS CA certificate file (PEM)\n");
printf(" --timeout <sec> I/O timeout in seconds (default: 30)\n");
printf(" -T <sec> Alias for --timeout\n");
printf(" --timeout <sec> I/O timeout in seconds (default: 30; long form only)\n");
printf(" --contimeout <sec> Connection timeout in seconds (default: 10)\n");
printf(" --stop-after=MINS Stop the transfer after MINS minutes (a positive\n");
printf(" integer); whatever was already transferred is kept\n");
printf(" --stop-at=TIME Stop at an absolute time: HH:MM, HH:MM:SS, or\n");
printf(" now+N[smhd] (a time already in the past stops the\n");
printf(" transfer immediately; client-only). An early stop\n");
printf(" skips the late --delete keep-set so it cannot delete\n");
printf(" source mirrors that were not yet scanned\n");
printf(" --address <ip> Bind the outgoing client socket to this source address\n");
printf(" -4, --ipv4 Force IPv4 for destination resolution\n");
printf(" -6, --ipv6 Force IPv6 for destination resolution\n");
printf(" --sockopts=OPTS Comma-separated OPT=VAL socket options applied before connect:\n");
printf(" TCP_NODELAY, SO_KEEPALIVE, SO_RCVBUF, SO_SNDBUF, SO_REUSEADDR\n");
printf(" --backup Backup existing files before overwriting\n");
printf(" --backup-dir <dir> Directory for backups (requires --backup)\n");
printf(" --suffix <str> Backup suffix (default: ~)\n");
@@ -154,17 +248,44 @@ void print_usage(void) {
printf(" --stderr=MODE Route logging to stderr: errors or all\n");
printf(" --partial Keep partial files on interrupted transfer\n");
printf(" --partial-dir <dir> Directory for partial files\n");
printf(" --temp-dir <dir> Scratch dir for temp files before atomic install\n");
printf(" -T, --temp-dir <dir> Scratch dir for temp files before atomic install\n");
printf(" --fastsync-server-path <path>\n");
printf(" Path to fastsync-server on remote (default: fastsync-server)\n");
printf(
" --old-args Disable safe SSH command argument quoting (legacy compatibility)\n");
printf(" --old-args Accepted for rsync CLI compatibility; no effect (the\n");
printf(" remote server path is always safely quoted now)\n");
printf(" -M, --remote-option=OPT Append OPT to the REMOTE server invocation over SSH\n");
printf(" (repeatable; each value is single-quote-escaped on the remote\n");
printf(" command line; empty values and values with control characters\n");
printf(" are rejected; -M OPT, -M=OPT and --remote-option=OPT work)\n");
printf(" --trust-sender Trust the remote sender's file list: the receiver skips its\n");
printf(" own up-front path-traversal/containment re-validation of the\n");
printf(" incoming file list (fewer checks, faster, potentially unsafe).\n");
printf(" Local receiver policy: never sent to the peer, off by default\n");
printf(" -l, --links Copy symlinks as symlinks\n");
printf(" --copy-links Transform symlinks into referent files\n");
printf(" --safe-links Skip symlinks that point outside transfer tree\n");
printf(" --copy-unsafe-links Only transform unsafe symlinks into referent files\n");
printf(" -k, --copy-dirlinks Transform symlinks to directories into real dirs\n");
printf(" -K, --keep-dirlinks Keep an existing symlink-to-dir as that dir\n");
printf(" --munge-links Munge symlink targets on the wire (sender)\n");
printf(" -H, --hard-links Preserve hard-link relationships across the transfer\n");
printf(" -S, --sparse Handle sparse files efficiently\n");
printf(
" -D Preserve device and special files (implies --devices --specials)\n");
printf(
" --devices Recreate device nodes on the destination (privileged; skipped when\n");
printf(" the receiver lacks CAP_MKNOD)\n");
printf(" --specials Recreate special files (FIFOs) on the destination (sockets "
"skipped)\n");
printf(" --copy-devices Copy a source device's content as a regular file instead\n");
printf(" --write-devices Write received data into an existing destination device node\n");
printf(" --inplace Update files in-place (no temp+rename)\n");
printf(
" --preallocate Allocate destination file space up front (fail-fast on full disk)\n");
printf(" --append Resume a shorter destination by appending only its tail\n");
printf(" (prefix is not verified; requires --incremental)\n");
printf(" --append-verify Like --append, but verifies the retained prefix checksum\n");
printf(" before appending (falls back to a full transfer on mismatch)\n");
printf(" --fsync Fsync every written file before publication\n");
printf(" --compress-level <n> Compression level (default: 5)\n");
printf(" --zl <n> Alias for --compress-level\n");
+201 -10
View File
@@ -1,8 +1,10 @@
#include "receiver.h"
#include "charset.h"
#include "chunk.h"
#include "config.h"
#include "delay_updates.h"
#include "file.h"
#include "file_receive.h"
#include "log.h"
#include "metadata.h"
@@ -10,6 +12,7 @@
#include "utils.h"
#include <stdlib.h>
#include <sys/stat.h>
#include <time.h>
bool receiver_outcomes_append(ReceiverOutcomes* outcomes, unsigned char code) {
if (!outcomes)
@@ -72,13 +75,32 @@ static bool receiver_process_chunk(Chunk* chunk, const ReceiverSink* sink) {
return true;
}
/* P7 Wave D: read one STATUS_DIR_TIMES frame (a count followed by that many
* (path, metadata) directory entries) and route every entry through the regular
* store_file sink. A dir-time entry is RECORD-ONLY (file->dir_time_only): the
* sink accumulates its metadata for end-of-transfer application but creates
* nothing, so an empty/pruned source directory is never resurrected. A large
* tree arrives as repeated frames, each bounded by MAX_MANIFEST_ENTRIES; a
* malformed count or entry is a hard error. */
static bool receiver_process_dir_times(int fd, const Config* config, const ReceiverSink* sink) {
int count;
if (!receive_int(fd, &count) || count < 0 || count > MAX_MANIFEST_ENTRIES)
return false;
for (int i = 0; i < count; i++) {
File* dir = file_receive_dir_time(fd, config);
if (!dir || !sink->store_file(dir, sink->context))
return false;
}
return true;
}
static bool receiver_process_batch(Config* config, int file_descriptor) {
int count;
if (config->checksum || !receive_int(file_descriptor, &count) || count < 0 ||
count > MAX_MANIFEST_ENTRIES)
return false;
for (int i = 0; i < count; i++) {
char* check_path = receive_str(file_descriptor);
char* check_path = receive_wire_str(file_descriptor);
if (!check_path)
return false;
unsigned long long check_size;
@@ -92,7 +114,11 @@ static bool receiver_process_batch(Config* config, int file_descriptor) {
send_status(file_descriptor, STATUS_ERROR);
return false;
}
if (!utils_valid_batch_path(check_path)) {
/* --trust-sender: accept a ``..``/absolute check path (a trusted sender's
odd-but-legit entry) and defer containment to the secure stat below;
an empty path is still always rejected. */
if (check_path[0] == '\0' ||
(!file_get_trust_sender() && !utils_valid_batch_path(check_path))) {
free(check_path);
send_status(file_descriptor, STATUS_ERROR);
return false;
@@ -128,6 +154,93 @@ static bool receiver_process_batch(Config* config, int file_descriptor) {
return true;
}
/* ---- Anti-slowloris connection bounds ----
* A legitimate transfer either streams data frames continuously or, when it
* must pause, sends STATUS_KEEPALIVE so the peer sees the connection is alive.
* An attacker can therefore squat on a connection slot indefinitely by sending
* only keepalives under the per-message timeout. Two CLOCK_MONOTONIC bounds
* defeat that without ever punishing a real transfer:
*
* MAX_SESSION_IDLE_SEC (1 h): the longest a stream may make no forward
* progress. Data/status frames count as progress and refresh the timer;
* keepalives do not. One hour is far longer than any real pause between
* data frames, yet small enough to reap a slowloris well before the 24 h
* session cap.
*
* MAX_SESSION_WALL_SEC (24 h): an absolute ceiling on one connection's
* lifetime as defense-in-depth against a trickle of progress frames that
* resets the idle timer just below its limit. Larger than any plausible
* single transfer while still bounding resource occupancy.
*
* Both are wall-clock deltas, so the per-message poll timeout (60 s by default,
* or --timeout) can never fool them, and both the single-threaded and the -m
* receiver paths (receiver_process_pending) share the same logic. */
#define MAX_SESSION_IDLE_SEC 3600u
#define MAX_SESSION_WALL_SEC 86400u
static unsigned int g_max_session_idle_sec = MAX_SESSION_IDLE_SEC;
static unsigned int g_max_session_wall_sec = MAX_SESSION_WALL_SEC;
void receiver_set_time_limits(unsigned int idle_sec, unsigned int wall_sec) {
g_max_session_idle_sec = idle_sec;
g_max_session_wall_sec = wall_sec;
}
void receiver_reset_time_limits(void) {
g_max_session_idle_sec = MAX_SESSION_IDLE_SEC;
g_max_session_wall_sec = MAX_SESSION_WALL_SEC;
}
bool receiver_time_limit_exceeded(const struct timespec* session_start,
const struct timespec* last_progress,
const struct timespec* now) {
if (!session_start || !last_progress || !now)
return false;
if (now->tv_sec - session_start->tv_sec >= (time_t)g_max_session_wall_sec)
return true;
if (now->tv_sec - last_progress->tv_sec >= (time_t)g_max_session_idle_sec)
return true;
return false;
}
/* A frame proves forward progress only when it cannot be fabricated for free.
* KEEPALIVE/ABORT are pure liveness, and CHECK_BATCH/DIR_TIMES may carry zero
* entries, so a peer must not be able to hold a connection slot forever by
* merely emitting empty frames. */
static bool status_counts_as_progress(Status status) {
switch (status) {
case STATUS_KEEPALIVE:
case STATUS_ABORT:
case STATUS_CHECK_BATCH:
case STATUS_DIR_TIMES:
return false;
default:
return true;
}
}
/* Refresh the progress timestamp for a forward-moving frame and enforce the
* bounds above. Returns false when the connection must be dropped; the
* terminal STATUS_ERROR is sent only when the sink owns error reporting (the
* -m sink sets send_error=false so the main thread emits exactly one). */
static bool receiver_note_status(const struct timespec* session_start,
struct timespec* last_progress, Status status, int file_descriptor,
const ReceiverSink* sink) {
struct timespec now;
if (clock_gettime(CLOCK_MONOTONIC, &now) != 0)
now = *last_progress;
if (status_counts_as_progress(status))
*last_progress = now;
if (!receiver_time_limit_exceeded(session_start, last_progress, &now))
return true;
log_message(LOG_LEVEL_ERROR,
"Receive session exceeded its time bound (idle %us / total %us); aborting connection",
g_max_session_idle_sec, g_max_session_wall_sec);
if (!sink || sink->send_error)
send_status(file_descriptor, STATUS_ERROR);
return false;
}
int receiver_process(Config* config, int file_descriptor, const ReceiverSink* sink) {
return receiver_process_pending(config, file_descriptor, sink, NULL);
}
@@ -147,6 +260,15 @@ int receiver_process_pending(Config* config, int file_descriptor, const Receiver
Status status;
if (!receive_status(file_descriptor, &status))
return -1;
/* Wall-clock (=CLOCK_MONOTONIC) anti-slowloris bookkeeping. session_start is
* fixed for the whole connection; last_progress is refreshed by every frame
* that is not a keepalive/abort. */
struct timespec session_start;
struct timespec last_progress;
clock_gettime(CLOCK_MONOTONIC, &session_start);
last_progress = session_start;
if (!receiver_note_status(&session_start, &last_progress, status, file_descriptor, sink))
return -1;
bool early_delete = config_delete_timing_early(config);
/* Parked keep-set for the late/commit timing. Every exit path below frees it
exactly once; the only exception is the successful FINISHED handoff, which
@@ -154,7 +276,8 @@ int receiver_process_pending(Config* config, int file_descriptor, const Receiver
DeleteManifest* deferred_manifest = NULL;
while (status == STATUS_NEXT || status == STATUS_CHUNK || status == STATUS_CHECK ||
status == STATUS_KEEPALIVE || status == STATUS_ABORT || status == STATUS_CHECK_BATCH ||
status == STATUS_MKDIR || status == STATUS_MANIFEST) {
status == STATUS_MKDIR || status == STATUS_MANIFEST || status == STATUS_HARDLINK ||
status == STATUS_SYMLINK || status == STATUS_SPECIAL || status == STATUS_DIR_TIMES) {
if (status == STATUS_KEEPALIVE) {
if (!send_status(file_descriptor, STATUS_KEEPALIVE))
goto fail;
@@ -165,10 +288,19 @@ int receiver_process_pending(Config* config, int file_descriptor, const Receiver
goto fail;
}
if (status == STATUS_CHECK) {
bool skipped;
File* file = receive_incremental_check(file_descriptor, config, &skipped);
if (!skipped && (!file || !sink->store_file(file, sink->context)))
bool skipped = false;
bool would_transfer = false;
File* file = receive_incremental_check_ex(file_descriptor, config, &skipped, &would_transfer);
if (config->dry_run) {
/* Server-contacting --dry-run: the reply has already been sent
(STATUS_OK = up to date, STATUS_DRY_RUN_TRANSFER = would transfer) and
nothing may be stored. Both flags false means a genuine protocol
error (STATUS_ERROR already sent or sent by receive_error below). */
if (!skipped && !would_transfer)
goto receive_error;
} else if (!skipped && (!file || !sink->store_file(file, sink->context))) {
goto receive_error;
}
} else if (status == STATUS_CHUNK) {
Chunk* chunk = receive_chunk_data(file_descriptor, config);
if (!chunk || !receiver_process_chunk(chunk, sink))
@@ -178,13 +310,37 @@ int receiver_process_pending(Config* config, int file_descriptor, const Receiver
goto fail;
goto next_status;
} else if (status == STATUS_MKDIR) {
File* dir = file_receive_directory(file_descriptor);
File* dir = file_receive_directory(file_descriptor, config);
if (!dir || !sink->store_file(dir, sink->context))
goto receive_error;
} else if (status == STATUS_DIR_TIMES) {
if (!receiver_process_dir_times(file_descriptor, config, sink))
goto receive_error;
} else if (status == STATUS_HARDLINK) {
File* file = file_receive_hardlink(file_descriptor);
if (!file || !sink->store_file(file, sink->context))
goto receive_error;
} else if (status == STATUS_SYMLINK) {
File* sym = file_receive_symlink(file_descriptor, config);
if (!sym || !sink->store_file(sym, sink->context))
goto receive_error;
} else if (status == STATUS_SPECIAL) {
File* file = file_receive_special(file_descriptor);
if (!file || !sink->store_file(file, sink->context))
goto receive_error;
} else if (status == STATUS_MANIFEST) {
DeleteManifest* manifest = receive_manifest_entries(file_descriptor);
if (!manifest)
goto fail; /* receive_manifest_entries already sent STATUS_ERROR */
if (config->dry_run) {
/* Server-contacting --dry-run mutates nothing, so a keep-set manifest
is consumed and discarded. The early-delete mode still needs its ACK
so a sender blocked on the delete handshake is not left hanging. */
delete_manifest_free(manifest);
if (early_delete && !send_status(file_descriptor, STATUS_OK))
goto fail;
goto next_status;
}
if (early_delete) {
/* --delete-before / --delete-during: the manifest is authoritative the
moment it arrives, before any file data. Delete now and acknowledge
@@ -230,6 +386,8 @@ int receiver_process_pending(Config* config, int file_descriptor, const Receiver
next_status:
if (!receive_status(file_descriptor, &status))
goto receive_error;
if (!receiver_note_status(&session_start, &last_progress, status, file_descriptor, sink))
goto fail;
}
if (status != STATUS_FINISHED) {
log_message(LOG_LEVEL_ERROR, "Did not receive FINISHED Status");
@@ -292,20 +450,42 @@ receive_error:
typedef struct {
Config* config;
ReceiverOutcomes outcomes;
/* P7 Wave D: directory metadata accumulated during the stream, applied only
after the whole transfer (and its delete/publication phases) has run so a
child write never clobbers a directory mtime. */
DirTimeList dir_times;
} ReceiverSaveContext;
static bool receiver_save_file(File* file, void* context_pointer) {
ReceiverSaveContext* context = context_pointer;
FileSaveResult result = FILE_SAVE_ERROR;
if (!context->config->save_to_disk) {
if (context->config->dry_run) {
/* Defense in depth: a dry-run receiver mutates nothing even if a data
frame reaches the sink (the sender is not supposed to send one). */
result = FILE_SAVE_SKIPPED;
} else if (!context->config->save_to_disk) {
/* Nothing is stored; report the file as not-written so a
--remove-source-files sender keeps its source. */
result = FILE_SAVE_SKIPPED;
} else {
result = file_save_to_disk_full(context->config->receive_root_directory, file, context->config);
}
if (result != FILE_SAVE_ERROR && context->config->remove_source_files && !file->is_dir &&
!file->skip && !receiver_outcomes_append(&context->outcomes, (unsigned char)result)) {
/* A directory's times are deferred, never applied inline: collect the
metadata now and apply it at the end. -O/--omit-dir-times is honored by
dir_time_list_apply's caller (see receiver_send_success_frame). */
if (result != FILE_SAVE_ERROR && file->is_dir && file->metadata &&
dir_times_should_capture(context->config) &&
!dir_time_list_add(&context->dir_times, file->path, file->metadata)) {
file_destroy(file);
return false;
}
/* A dry-run receiver mutates nothing AND records no per-file outcomes: a
hostile dry-run client that streamed data frames anyway must not be able to
grow `outcomes` without bound (receiver_outcomes_append reallocs uncharged)
or force a per-frame ack. */
if (!context->config->dry_run && result != FILE_SAVE_ERROR &&
context->config->remove_source_files && !file->is_dir && !file->is_special && !file->skip &&
!receiver_outcomes_append(&context->outcomes, (unsigned char)result)) {
file_destroy(file);
return false;
}
@@ -315,6 +495,10 @@ static bool receiver_save_file(File* file, void* context_pointer) {
static bool receiver_send_success_frame(int fd, void* context_pointer) {
ReceiverSaveContext* context = context_pointer;
/* Server-contacting --dry-run: nothing was staged or written, so there is
nothing to publish and no directory times to stamp. */
if (context->config->dry_run)
return receiver_send_final_success(fd, context->config, &context->outcomes);
/* --delay-updates: the whole protocol stream (including manifest/delete
handling, which ran inside receiver_process) has succeeded and every
staged file was fully written. Publish them atomically now, before the
@@ -326,15 +510,22 @@ static bool receiver_send_success_frame(int fd, void* context_pointer) {
return false;
}
}
/* P7 Wave D: every child is now written and the delete / --delay-updates
phases have committed, so it is finally safe to stamp directory times.
This runs after the deferred deletion because receiver_process commits it
before calling this success frame. */
dir_time_list_apply(&context->dir_times, context->config->receive_root_directory);
return receiver_send_final_success(fd, context->config, &context->outcomes);
}
int receiver_receive_files(Config* config, int file_descriptor) {
ReceiverSaveContext context = {.config = config, .outcomes = {0}};
dir_time_list_init(&context.dir_times);
ReceiverSink sink = {receiver_save_file, &context, true, true, receiver_send_success_frame};
int ret = receiver_process(config, file_descriptor, &sink);
if (ret != 0 && config->delay_updates && config->delay_context)
delay_updates_cleanup(config->delay_context);
receiver_outcomes_destroy(&context.outcomes);
dir_time_list_free(&context.dir_times);
return ret;
}
+20
View File
@@ -4,6 +4,8 @@
#include "config.h"
#include "file.h"
#include "file_receive.h"
#include <stdbool.h>
#include <time.h>
typedef bool (*ReceiverFileSink)(File* file, void* context);
@@ -45,4 +47,22 @@ int receiver_process_pending(Config* config, int file_descriptor, const Receiver
DeleteManifest** pending_manifest);
int receiver_receive_files(Config* config, int file_descriptor);
/* ---- Connection time bounds (anti-slowloris) ----
* receiver_process_pending() aborts a connection that makes no forward progress
* (only STATUS_KEEPALIVE/STATUS_ABORT frames) beyond a wall-clock idle limit,
* and enforces a hard cap on the whole session. Both are CLOCK_MONOTONIC
* deltas, independent of the per-message poll deadline, so a 60 s (or
* --timeout) receive window can never reset them. Defaults are deliberately
* generous (see MAX_SESSION_IDLE_SEC / MAX_SESSION_WALL_SEC in receiver.c). */
/* Test seam: override the idle/session wall-clock limits (0 = abort on the
* next status). Always restore with receiver_reset_time_limits(). */
void receiver_set_time_limits(unsigned int idle_sec, unsigned int wall_sec);
void receiver_reset_time_limits(void);
/* Pure predicate over explicit monotonic timestamps, exposed so the bound is
* unit-testable without sleeping. True when either the idle or the overall
* session limit has elapsed. */
bool receiver_time_limit_exceeded(const struct timespec* session_start,
const struct timespec* last_progress, const struct timespec* now);
#endif
+258
View File
@@ -0,0 +1,258 @@
#include "receiver_pipeline.h"
#include "log.h"
#include "protocol.h"
#include "queue.h"
#include "utils.h"
#include <stdlib.h>
#include <string.h>
#include <threads.h>
PipelineContextReceiver* pipeline_context_receiver_create(Config* config, Queue* queue,
int file_descriptor, SSL* ssl) {
PipelineContextReceiver* context = malloc(sizeof(PipelineContextReceiver));
if (context == NULL)
return NULL;
context->config = config;
context->queue = queue;
context->file_descriptor = file_descriptor;
context->ssl = ssl;
context->outcomes.entries = NULL;
context->outcomes.count = 0;
context->outcomes.capacity = 0;
dir_time_list_init(&context->dir_times);
protocol_session_init(&context->session, file_descriptor, file_descriptor);
protocol_session_set_ssl(&context->session, ssl);
context->receiver_done = false;
context->queued_bytes = 0;
context->max_queue_bytes = 0;
context->deferred_manifest = NULL;
atomic_init(&context->cancelled, false);
int init = 0;
if (mtx_init(&context->mutex, mtx_plain) != thrd_success)
goto fail;
init++;
if (cnd_init(&context->condition_not_full) != thrd_success)
goto fail;
init++;
if (cnd_init(&context->condition_not_empty) != thrd_success)
goto fail;
// cppcheck-suppress unreadVariable
init++;
return context;
fail:
log_perror("Error initializing synchronization objects");
if (init >= 3)
cnd_destroy(&context->condition_not_empty);
if (init >= 2)
cnd_destroy(&context->condition_not_full);
if (init >= 1)
mtx_destroy(&context->mutex);
free(context);
return NULL;
}
void pipeline_context_receiver_destroy(PipelineContextReceiver* context) {
config_delete(context->config);
if (context->deferred_manifest)
delete_manifest_free(context->deferred_manifest);
queue_destroy(context->queue);
receiver_outcomes_destroy(&context->outcomes);
dir_time_list_free(&context->dir_times);
mtx_destroy(&context->mutex);
cnd_destroy(&context->condition_not_full);
cnd_destroy(&context->condition_not_empty);
free(context);
}
void pipeline_context_receiver_set_queue_byte_limit(PipelineContextReceiver* context,
size_t max_bytes) {
if (context == NULL)
return;
mtx_lock(&context->mutex);
context->max_queue_bytes = max_bytes;
context->queued_bytes = 0;
cnd_broadcast(&context->condition_not_full);
mtx_unlock(&context->mutex);
}
void pipeline_context_receiver_note_bytes_released(PipelineContextReceiver* context,
size_t released_bytes) {
if (context == NULL || context->max_queue_bytes == 0 || released_bytes == 0)
return;
mtx_lock(&context->mutex);
if (released_bytes >= context->queued_bytes)
context->queued_bytes = 0;
else
context->queued_bytes -= released_bytes;
cnd_signal(&context->condition_not_full);
mtx_unlock(&context->mutex);
}
bool pipeline_context_receiver_enqueue_file(PipelineContextReceiver* context, File* file) {
if (context == NULL || file == NULL)
return false;
size_t file_bytes = file->data ? file->data->size : 0;
mtx_lock(&context->mutex);
while (!atomic_load(&context->cancelled)) {
bool blocked_by_count = queue_is_full(context->queue);
bool blocked_by_budget = false;
if (context->max_queue_bytes > 0) {
size_t budget = context->max_queue_bytes;
size_t used = context->queued_bytes;
if (used >= budget) {
blocked_by_budget = true;
} else if (file_bytes > budget - used) {
/* A single payload larger than the whole budget (not possible with
the per-file receive cap) is only admitted to an empty pipeline so
the wait can never deadlock. */
blocked_by_budget = used != 0;
}
}
if (!blocked_by_count && !blocked_by_budget)
break;
cnd_wait(&context->condition_not_full, &context->mutex);
}
if (atomic_load(&context->cancelled)) {
mtx_unlock(&context->mutex);
file_destroy(file);
return false;
}
if (!queue_enqueue(context->queue, file)) {
mtx_unlock(&context->mutex);
file_destroy(file);
return false;
}
context->queued_bytes += file_bytes;
cnd_signal(&context->condition_not_empty);
mtx_unlock(&context->mutex);
return true;
}
static bool receiver_enqueue_file(File* file, void* context_pointer) {
PipelineContextReceiver* context = (PipelineContextReceiver*)context_pointer;
return pipeline_context_receiver_enqueue_file(context, file);
}
static void receiver_thread_fail(PipelineContextReceiver* context) {
mtx_lock(&context->mutex);
atomic_store(&context->cancelled, true);
context->receiver_done = true;
cnd_broadcast(&context->condition_not_empty);
cnd_broadcast(&context->condition_not_full);
mtx_unlock(&context->mutex);
}
int receive_thread(void* pipeline_context) {
PipelineContextReceiver* context = (PipelineContextReceiver*)pipeline_context;
protocol_session_bind(&context->session);
mtx_lock(&context->mutex);
int file_descriptor = context->file_descriptor;
const Config* config = context->config;
mtx_unlock(&context->mutex);
ReceiverSink sink = {receiver_enqueue_file, context, false, false, NULL};
if (receiver_process_pending((Config*)config, file_descriptor, &sink,
&context->deferred_manifest) != 0) {
receiver_thread_fail(context);
protocol_session_unbind();
return thrd_error;
}
mtx_lock(&context->mutex);
context->receiver_done = true;
cnd_signal(&context->condition_not_empty);
mtx_unlock(&context->mutex);
protocol_session_unbind();
return thrd_success;
}
int write_thread(void* pipeline_context) {
PipelineContextReceiver* context = (PipelineContextReceiver*)pipeline_context;
protocol_session_bind(&context->session);
mtx_lock(&context->mutex);
bool save_to_disk = context->config->save_to_disk;
char* root_directory = str_dup(context->config->receive_root_directory);
mtx_unlock(&context->mutex);
if (save_to_disk && !root_directory) {
mtx_lock(&context->mutex);
atomic_store(&context->cancelled, true);
context->receiver_done = true;
cnd_broadcast(&context->condition_not_full);
cnd_broadcast(&context->condition_not_empty);
mtx_unlock(&context->mutex);
protocol_session_unbind();
return thrd_error;
}
while (true) {
File* file =
queue_dequeue_multithreaded(context->queue, &context->mutex, &context->condition_not_empty,
&context->condition_not_full, &context->receiver_done);
if (file == NULL) {
free(root_directory);
protocol_session_unbind();
return thrd_success;
}
size_t file_bytes = file->data ? file->data->size : 0;
FileSaveResult result = FILE_SAVE_SKIPPED;
/* Server-contacting --dry-run: never write. The receiver thread does not
enqueue anything on the dry-run path, but this keeps the writer thread
provably mutation-free if a data frame ever reached it. */
bool dry_run = context->config->dry_run;
if (save_to_disk && !dry_run) {
result = file_save_to_disk_full(root_directory, file, context->config);
if (result == FILE_SAVE_ERROR) {
file_destroy(file);
pipeline_context_receiver_note_bytes_released(context, file_bytes);
mtx_lock(&context->mutex);
atomic_store(&context->cancelled, true);
context->receiver_done = true;
cnd_broadcast(&context->condition_not_full);
cnd_broadcast(&context->condition_not_empty);
mtx_unlock(&context->mutex);
free(root_directory);
protocol_session_unbind();
return thrd_error;
}
}
/* P7 Wave D: a directory's times are never applied inline (a later child
write would clobber them); accumulate the metadata here and let the
caller apply it once every writer has drained. */
if (!dry_run && result != FILE_SAVE_ERROR && file->is_dir && file->metadata &&
dir_times_should_capture(context->config) &&
!dir_time_list_add(&context->dir_times, file->path, file->metadata)) {
file_destroy(file);
pipeline_context_receiver_note_bytes_released(context, file_bytes);
mtx_lock(&context->mutex);
atomic_store(&context->cancelled, true);
context->receiver_done = true;
cnd_broadcast(&context->condition_not_full);
cnd_broadcast(&context->condition_not_empty);
mtx_unlock(&context->mutex);
free(root_directory);
protocol_session_unbind();
return thrd_error;
}
/* Record the per-file outcome so a --remove-source-files sender learns
which sources were actually written versus skipped on the receiver.
Explicit directory entries and recreated device/special nodes have no
source and are never acknowledged (mirrors receiver.c). */
if (!dry_run && context->config->remove_source_files && !file->is_dir && !file->is_special &&
!file->skip && !receiver_outcomes_append(&context->outcomes, (unsigned char)result)) {
file_destroy(file);
pipeline_context_receiver_note_bytes_released(context, file_bytes);
mtx_lock(&context->mutex);
atomic_store(&context->cancelled, true);
context->receiver_done = true;
cnd_broadcast(&context->condition_not_full);
cnd_broadcast(&context->condition_not_empty);
mtx_unlock(&context->mutex);
free(root_directory);
protocol_session_unbind();
return thrd_error;
}
file_destroy(file);
pipeline_context_receiver_note_bytes_released(context, file_bytes);
}
}
+68
View File
@@ -0,0 +1,68 @@
#ifndef RECEIVER_PIPELINE_H
#define RECEIVER_PIPELINE_H
#include <stdatomic.h>
#include <stdbool.h>
#include <threads.h>
#include "config.h"
#include "file.h"
#include "file_receive.h"
#include "protocol.h"
#include "queue.h"
#include "receiver.h"
#include <openssl/ssl.h>
typedef struct PipelineContextReceiver {
Queue* queue;
Config* config;
int file_descriptor;
SSL* ssl;
ProtocolSession session;
ReceiverOutcomes outcomes;
mtx_t mutex;
cnd_t condition_not_full;
cnd_t condition_not_empty;
bool receiver_done;
atomic_bool cancelled;
/* Aggregate payload bytes that have been received but not yet released by
the disk writer (queued or in the writer's hand). Guarded by `mutex`.
When `max_queue_bytes` is non-zero the receiver blocks before enqueuing
once this total would exceed it, so decompressed/copied file payloads
buffered ahead of a slow disk writer respect the per-connection memory
budget instead of growing without bound. */
size_t queued_bytes;
size_t max_queue_bytes;
/* Keep-set manifest for the commit-style (late) deletion
(--delete/--delete-after/--delete-delay). receive_thread parses the whole
protocol stream but hands the manifest here instead of deleting while the
disk writer may still be draining; the caller (server.c) commits the
deletion after both threads have joined, so no extra is removed unless the
transfer truly succeeded. NULL in the early delete modes (which delete at
the manifest). */
DeleteManifest* deferred_manifest;
/* P7 Wave D: directory metadata collected by write_thread from received
directory entries. Only write_thread mutates it (before it joins); the
caller (server.c) applies it after the delete/delay-updates phase. */
DirTimeList dir_times;
} PipelineContextReceiver;
PipelineContextReceiver* pipeline_context_receiver_create(Config* config, Queue* queue_receiver,
int file_descriptor, SSL* ssl);
void pipeline_context_receiver_destroy(PipelineContextReceiver* context);
/* Bound the bytes buffered ahead of the disk writer (see max_queue_bytes). */
void pipeline_context_receiver_set_queue_byte_limit(PipelineContextReceiver* context,
size_t max_bytes);
/* Blocking enqueue used by the receive pipeline sink. Blocks while the queue
is full by element count or when adding `file` would push queued_bytes over
the configured byte limit; waits until the disk writer releases bytes.
Takes ownership of `file` on success and destroys it on failure/cancel. */
bool pipeline_context_receiver_enqueue_file(PipelineContextReceiver* context, File* file);
/* Account for `released_bytes` of payload memory that has been freed by the
disk writer, unblocking a receiver that is waiting on the byte limit. */
void pipeline_context_receiver_note_bytes_released(PipelineContextReceiver* context,
size_t released_bytes);
int receive_thread(void* pipeline_context);
int write_thread(void* pipeline_context);
#endif
+1133 -179
View File
File diff suppressed because it is too large. Load diff
+305
View File
@@ -0,0 +1,305 @@
#include "server_cli.h"
#include "charset.h"
#include "credentials.h"
#include "utils.h"
#include <limits.h>
#include <stdarg.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/socket.h>
static void set_error(char* err, size_t err_size, const char* fmt, ...) {
if (!err || err_size == 0)
return;
va_list args;
va_start(args, fmt);
vsnprintf(err, err_size, fmt, args);
va_end(args);
}
void server_cli_options_default(ServerCliOptions* opts) {
if (!opts)
return;
memset(opts, 0, sizeof(*opts));
opts->destination_root = ".";
opts->port = 8080;
opts->bind_family = AF_UNSPEC;
}
static bool arg_is(const char* arg, const char* name) {
return strcmp(arg, name) == 0;
}
/* Match "--opt" against "--opt=value" / separate-value forms; on the "=" form
* *value receives the inline value. Returns true when the argument is the
* named option in either form. */
static bool arg_has_value(const char* arg, const char* name, const char** value) {
if (strcmp(arg, name) == 0)
return true; /* separate form; caller takes the next argv slot */
size_t name_len = strlen(name);
if (strncmp(arg, name, name_len) == 0 && arg[name_len] == '=') {
*value = arg + name_len + 1;
return true;
}
return false;
}
static int parse_port_arg(const char* value, int* port, char* err, size_t err_size) {
char* end;
long p = strtol(value, &end, 10);
if (*end != '\0' || p <= 0 || p > 65535) {
char* escaped = output_escape(value, false);
set_error(err, err_size, "invalid port '%s' (must be 1-65535)",
escaped ? escaped : "<allocation failed>");
free(escaped);
return -1;
}
*port = (int)p;
return 0;
}
int server_cli_parse(int argc, char* argv[], ServerCliOptions* opts, char* err, size_t err_size) {
if (err && err_size)
err[0] = '\0';
server_cli_options_default(opts);
for (int i = 1; i < argc; i++) {
const char* inline_value = NULL;
if (arg_is(argv[i], "--help")) {
opts->show_help = true;
return 1;
} else if (arg_is(argv[i], "--stdio")) {
opts->stdio_mode = true;
} else if (arg_is(argv[i], "--daemon")) {
opts->daemon_mode = true;
} else if (arg_is(argv[i], "--no-detach")) {
opts->no_detach = true;
} else if (arg_is(argv[i], "-v") || arg_is(argv[i], "--verbose")) {
opts->verbose = true;
} else if (arg_is(argv[i], "--tls")) {
opts->use_tls = true;
} else if (arg_is(argv[i], "--cert")) {
if (i + 1 >= argc) {
set_error(err, err_size, "missing argument for --cert");
return -1;
}
opts->tls_cert = argv[++i];
} else if (arg_is(argv[i], "--key")) {
if (i + 1 >= argc) {
set_error(err, err_size, "missing argument for --key");
return -1;
}
opts->tls_key = argv[++i];
} else if (arg_is(argv[i], "--ca")) {
if (i + 1 >= argc) {
set_error(err, err_size, "missing argument for --ca");
return -1;
}
opts->tls_ca = argv[++i];
} else if (arg_is(argv[i], "--client-cn")) {
if (i + 1 >= argc) {
set_error(err, err_size, "missing argument for --client-cn");
return -1;
}
opts->client_cn = argv[++i];
} else if (arg_is(argv[i], "--destination-root")) {
if (i + 1 >= argc) {
set_error(err, err_size, "missing argument for --destination-root");
return -1;
}
opts->destination_root = argv[++i];
opts->destination_root_set = true;
} else if (arg_has_value(argv[i], "--password-file", &inline_value)) {
if (!inline_value) {
if (i + 1 >= argc) {
set_error(err, err_size, "missing argument for --password-file");
return -1;
}
inline_value = argv[++i];
}
opts->password_file = inline_value;
} else if (arg_has_value(argv[i], "--early-input", &inline_value)) {
if (!inline_value) {
if (i + 1 >= argc) {
set_error(err, err_size, "missing argument for --early-input");
return -1;
}
inline_value = argv[++i];
}
opts->early_input_file = inline_value;
} else if (arg_has_value(argv[i], "--hash-credentials", &inline_value)) {
if (!inline_value) {
if (i + 1 >= argc) {
set_error(err, err_size, "missing argument for --hash-credentials");
return -1;
}
inline_value = argv[++i];
}
opts->hash_credentials_file = inline_value;
} else if (arg_has_value(argv[i], "--iterations", &inline_value)) {
if (!inline_value) {
if (i + 1 >= argc) {
set_error(err, err_size, "missing argument for --iterations");
return -1;
}
inline_value = argv[++i];
}
char* end = NULL;
long n = strtol(inline_value, &end, 10);
if (!end || *end != '\0' || n < (long)CREDENTIAL_MIN_ITERS ||
n > (long)CREDENTIAL_MAX_ITERS) {
set_error(err, err_size, "--iterations must be in [%u,%u], got '%s'", CREDENTIAL_MIN_ITERS,
CREDENTIAL_MAX_ITERS, inline_value);
return -1;
}
opts->hash_iterations = (uint32_t)n;
opts->hash_iterations_set = true;
} else if (arg_is(argv[i], "--address")) {
if (i + 1 >= argc) {
set_error(err, err_size, "missing argument for --address");
return -1;
}
opts->bind_address = argv[++i];
} else if (arg_is(argv[i], "-4") || arg_is(argv[i], "--ipv4")) {
if (opts->bind_family == AF_INET6) {
set_error(err, err_size, "--ipv4 and --ipv6 are mutually exclusive");
return -1;
}
opts->bind_family = AF_INET;
} else if (arg_is(argv[i], "-6") || arg_is(argv[i], "--ipv6")) {
if (opts->bind_family == AF_INET) {
set_error(err, err_size, "--ipv4 and --ipv6 are mutually exclusive");
return -1;
}
opts->bind_family = AF_INET6;
} else if (arg_is(argv[i], "--allow-delete")) {
opts->allow_delete = true;
} else if (arg_is(argv[i], "--trust-sender")) {
opts->trust_sender = true;
} else if (arg_is(argv[i], "--no-super")) {
opts->no_super = true;
} else if (arg_is(argv[i], "--allow-super")) {
opts->allow_super = true;
} else if (arg_is(argv[i], "--allow-unauthenticated")) {
opts->allow_unauthenticated = true;
} else if (arg_has_value(argv[i], "--iconv", &inline_value)) {
if (!inline_value) {
if (i + 1 >= argc) {
set_error(err, err_size, "missing argument for --iconv");
return -1;
}
inline_value = argv[++i];
}
opts->iconv_spec = inline_value;
} else if (arg_is(argv[i], "-p")) {
if (i + 1 >= argc) {
set_error(err, err_size, "missing argument for -p");
return -1;
}
opts->port_set = true;
if (parse_port_arg(argv[++i], &opts->port, err, err_size) != 0)
return -1;
} else {
if (arg_has_value(argv[i], "--config", &inline_value)) {
if (!inline_value) {
if (i + 1 >= argc) {
set_error(err, err_size, "missing argument for --config");
return -1;
}
inline_value = argv[++i];
}
opts->config_path = inline_value;
} else if (arg_has_value(argv[i], "--dparam", &inline_value)) {
if (!inline_value) {
if (i + 1 >= argc) {
set_error(err, err_size, "missing argument for --dparam");
return -1;
}
inline_value = argv[++i];
}
const char** grown =
realloc((char**)opts->dparams, (size_t)(opts->dparam_count + 1) * sizeof(const char*));
if (!grown) {
set_error(err, err_size, "out of memory parsing --dparam");
return -1;
}
opts->dparams = grown;
opts->dparams[opts->dparam_count++] = inline_value;
} else if (argv[i][0] == '-') {
char* escaped = output_escape(argv[i], false);
set_error(err, err_size, "unknown option: %s", escaped ? escaped : "<allocation failed>");
free(escaped);
return -1;
} else {
set_error(err, err_size, "unexpected argument '%s'", argv[i]);
return -1;
}
}
}
/* Cross-mode validation. */
if (opts->stdio_mode && opts->daemon_mode) {
set_error(err, err_size, "--stdio and --daemon are mutually exclusive");
return -1;
}
if (opts->daemon_mode && opts->destination_root_set) {
set_error(err, err_size,
"--destination-root cannot be combined with --daemon (module paths "
"replace it)");
return -1;
}
if (!opts->daemon_mode &&
(opts->config_path != NULL || opts->dparam_count > 0 || opts->no_detach ||
opts->password_file != NULL || opts->early_input_file != NULL)) {
set_error(err, err_size,
"--config, --dparam, --no-detach, --password-file, and --early-input require "
"--daemon");
return -1;
}
if (opts->hash_credentials_file != NULL && (opts->daemon_mode || opts->stdio_mode)) {
set_error(err, err_size, "--hash-credentials cannot be combined with --daemon or --stdio");
return -1;
}
if (opts->allow_super && opts->no_super) {
set_error(err, err_size, "--allow-super and --no-super are mutually exclusive");
return -1;
}
if (opts->allow_super && opts->daemon_mode) {
set_error(err, err_size,
"--allow-super is for a locally-launched standalone TCP server; daemon modules opt "
"in per module with 'client owner = yes'");
return -1;
}
/* --stdio is the SSH transport: the remote server argv is composed by the
* CLIENT (directly and via --remote-option), so a client could otherwise pass
* --allow-super to a root --stdio receiver and defeat the C3 secure default.
* Never honor it there; the super mode stays forced OFF. An operator who
* must keep the historical permissive behavior over SSH has to launch the
* receiver through a forced command, not via client-composed argv. */
if (opts->allow_super && opts->stdio_mode) {
set_error(err, err_size,
"--allow-super is not accepted with --stdio (the remote argv is client-composed; "
"use a forced command if the default must hold)");
return -1;
}
if (opts->hash_iterations_set && opts->hash_credentials_file == NULL) {
set_error(err, err_size, "--iterations requires --hash-credentials");
return -1;
}
/* --iconv: reject a malformed CONVERT_SPEC or an unsupported charset name at
startup (a probe iconv_open is attempted). */
if (opts->iconv_spec != NULL && !charset_spec_valid(opts->iconv_spec)) {
set_error(err, err_size, "--iconv requires LOCAL[,REMOTE] charset names supported by iconv");
return -1;
}
return 0;
}
void server_cli_options_free(ServerCliOptions* opts) {
if (!opts)
return;
free((char**)opts->dparams);
opts->dparams = NULL;
opts->dparam_count = 0;
}
+79
View File
@@ -0,0 +1,79 @@
#ifndef SERVER_CLI_H
#define SERVER_CLI_H
#include <stdbool.h>
#include <stddef.h>
#include <stdint.h>
/* Parsed fastsync-server command line. All string members are borrowed
* pointers into the original argv (valid for the life of the argv array the
* caller passed to server_cli_parse); dparams points at the raw --dparam
* argument strings. No member owns heap memory. */
typedef struct ServerCliOptions {
bool stdio_mode; /* --stdio */
bool daemon_mode; /* --daemon */
bool no_detach; /* --no-detach */
bool verbose; /* -v / --verbose */
bool show_help; /* --help */
bool use_tls; /* --tls */
const char* tls_cert; /* --cert */
const char* tls_key; /* --key */
const char* tls_ca; /* --ca */
const char* client_cn; /* --client-cn */
bool destination_root_set; /* an explicit --destination-root was given */
const char* destination_root; /* --destination-root value ("." if unset) */
bool port_set; /* an explicit -p was given */
int port; /* -p value (default 8080 when unset) */
const char* config_path; /* --config value, or NULL */
const char* password_file; /* --password-file value, or NULL (daemon) */
const char* early_input_file; /* --early-input value, or NULL (daemon) */
/* --hash-credentials=FILE: read `user:password` lines from FILE and print
* new-format credential-store lines to stdout, then exit. Standalone mode
* (mutually exclusive with --daemon/--stdio). */
const char* hash_credentials_file;
bool hash_iterations_set; /* an explicit --iterations was given */
uint32_t hash_iterations; /* --iterations value (default CREDENTIAL_DEFAULT_ITERS) */
const char** dparams; /* raw --dparam override strings */
int dparam_count;
const char* bind_address; /* --address */
int bind_family; /* AF_UNSPEC / AF_INET / AF_INET6 */
bool allow_delete; /* --allow-delete */
bool trust_sender; /* --trust-sender */
bool allow_unauthenticated; /* --allow-unauthenticated */
/* --no-super: operator veto forcing SUPER_MODE_OFF for every connection, so
* the receiver never attempts super-user activities (ownership application,
* device-node creation) even when running as root. Applies to --stdio and
* --daemon alike; also makes the server refuse any client --copy-as. */
bool no_super; /* --no-super */
/* --allow-super: locally-launched standalone TCP listener opt-in that keeps
* the historical permissive behavior for a PRIVILEGED (root) receiver.
* Without it a root standalone server forces SUPER_MODE_OFF, so a client
* --devices / --write-devices / --super / ownership request cannot make it
* create device nodes, write raw devices, or apply client-chosen ownership.
* It is rejected for --stdio: that path's remote argv is composed by the
* client (directly and via --remote-option), so it must never opt a root
* receiver back into super mode. Non-root receivers are unaffected (the
* kernel refuses the confined attempts). The daemon path instead uses the
* per-module `client owner = yes` opt-in. */
bool allow_super; /* --allow-super */
/* --iconv=CONVERT_SPEC: the server's own LOCAL charset declaration. The
* client's full spec rides the wire config frame anyway; when the server is
* started with its own --iconv, its LOCAL half overrides the local charset
* the client assumed so the server converts received names to ITS charset.
* Borrowed pointer into argv (never owns heap). */
const char* iconv_spec; /* --iconv value, or NULL */
} ServerCliOptions;
/* Parse argc/argv into *opts. Zero-initialize *opts before calling (or use
* server_cli_options_default). Returns:
* 1 -- --help was requested (opts->show_help set; caller prints usage).
* 0 -- parsed successfully.
* -1 -- invalid arguments (err is filled with the reason).
*/
void server_cli_options_default(ServerCliOptions* opts);
int server_cli_parse(int argc, char* argv[], ServerCliOptions* opts, char* err, size_t err_size);
/* Release the only heap the parsed options own (the dparams pointer array; the
* strings it points at are borrowed from argv and are not freed). Safe to
* call on a zero-initialized/defaulted struct. */
void server_cli_options_free(ServerCliOptions* opts);
#endif
+3
View File
@@ -1,6 +1,7 @@
#include "log.h"
#include "array_list.h"
#include "protocol.h"
#include <limits.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
@@ -39,6 +40,8 @@ void array_list_delete(ArrayList* array_list) {
static bool array_list_extend(ArrayList* array_list) {
if (array_list == NULL)
return false;
if (array_list->capacity > INT_MAX / 2)
return false;
int new_capacity = array_list->capacity * 2;
if (new_capacity == 0)
new_capacity = INITIAL_ARRAY_SIZE;
+160
View File
@@ -0,0 +1,160 @@
#include "batch.h"
#include "data.h"
#include "file.h"
#include "file_receive.h"
#include "log.h"
#include <errno.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
/* Serialization metadata mode for the batch stream, captured from the config at
* batch_write_header time. The header persists it into the file so a batch is
* self-describing: batch_read_apply re-reads it from the file (not from the
* reading config), so a batch written with -M is applied identically by an
* invoking process regardless of its own -M setting. The batch driver is a
* single sequential scan pass within one thread, so this module-level flag is
* safe. */
static bool batch_metadata_mode = false;
static bool write_all_bytes(int fd, const void* data, size_t size) {
const unsigned char* p = (const unsigned char*)data;
size_t done = 0;
while (done < size) {
ssize_t n = write(fd, p + done, size - done);
if (n < 0 && errno == EINTR)
continue;
if (n <= 0)
return false;
done += (size_t)n;
}
return true;
}
bool batch_write_header(int fd, const Config* config) {
if (fd < 0)
return false;
batch_metadata_mode = config != NULL && config->use_metadata;
if (!write_all_bytes(fd, BATCH_MAGIC, BATCH_MAGIC_LEN))
return false;
unsigned char version = BATCH_FORMAT_VERSION;
if (!write_all_bytes(fd, &version, 1))
return false;
unsigned char mode = batch_metadata_mode ? 1 : 0;
return write_all_bytes(fd, &mode, 1);
}
bool batch_write_chunk(int fd, Chunk* chunk) {
if (fd < 0 || chunk == NULL)
return false;
Data* serialized = chunk_serialize(chunk, batch_metadata_mode);
if (serialized == NULL)
return false;
bool ok = false;
unsigned long long length = (unsigned long long)serialized->size;
if (length > BATCH_MAX_RECORD) {
log_message(LOG_LEVEL_ERROR, "batch: record size %llu exceeds the %llu-byte cap", length,
(unsigned long long)BATCH_MAX_RECORD);
} else if (write_all_bytes(fd, &length, sizeof(length)) &&
(length == 0 || write_all_bytes(fd, serialized->data, (size_t)length))) {
ok = true;
}
data_destroy(serialized);
return ok;
}
/* Read exactly `size` bytes. Returns true on success. On reaching EOF, sets
* *clean_eof only when no bytes had been read yet (a clean boundary) and returns
* that value, so a truncated record (EOF mid-read) yields false. */
static bool read_exact(int fd, void* data, size_t size, bool* clean_eof) {
unsigned char* p = (unsigned char*)data;
size_t done = 0;
while (done < size) {
ssize_t n = read(fd, p + done, size - done);
if (n < 0 && errno == EINTR)
continue;
if (n == 0) {
if (clean_eof)
*clean_eof = done == 0;
return done == 0;
}
if (n < 0)
return false;
done += (size_t)n;
}
if (clean_eof)
*clean_eof = false;
return true;
}
int batch_read_apply(int fd, const Config* config, const char* dest_root) {
if (fd < 0 || dest_root == NULL || dest_root[0] == '\0')
return -1;
char magic[BATCH_MAGIC_LEN];
bool eof = false;
if (!read_exact(fd, magic, BATCH_MAGIC_LEN, &eof) || eof ||
memcmp(magic, BATCH_MAGIC, BATCH_MAGIC_LEN) != 0) {
log_message(LOG_LEVEL_ERROR, "batch: malformed header (bad magic)");
return -1;
}
unsigned char version;
if (!read_exact(fd, &version, 1, &eof) || eof || version != BATCH_FORMAT_VERSION) {
log_message(LOG_LEVEL_ERROR, "batch: malformed header (bad or missing format version)");
return -1;
}
unsigned char mode;
if (!read_exact(fd, &mode, 1, &eof) || eof || (mode != 0 && mode != 1)) {
log_message(LOG_LEVEL_ERROR, "batch: malformed header (bad metadata flag)");
return -1;
}
bool use_metadata = mode == 1;
while (1) {
unsigned long long length;
if (!read_exact(fd, &length, sizeof(length), &eof)) {
log_message(LOG_LEVEL_ERROR, "batch: truncated length prefix");
return -1;
}
if (eof)
break; /* clean end of stream */
if (length == 0 || length > BATCH_MAX_RECORD) {
log_message(LOG_LEVEL_ERROR, "batch: rejected record length %llu (valid range 1..%llu)",
length, (unsigned long long)BATCH_MAX_RECORD);
return -1;
}
char* record = (char*)malloc((size_t)length);
if (record == NULL) {
log_message(LOG_LEVEL_ERROR, "batch: could not allocate a %llu-byte record", length);
return -1;
}
if (!read_exact(fd, record, (size_t)length, &eof) || eof) {
log_message(LOG_LEVEL_ERROR, "batch: truncated chunk record");
free(record);
return -1;
}
Data* data = data_create(record, (size_t)length);
if (data == NULL)
return -1; /* data_create frees `record` on failure */
Chunk* chunk = chunk_deserialize(data, use_metadata);
data_destroy(data);
if (chunk == NULL) {
log_message(LOG_LEVEL_ERROR, "batch: rejected malformed chunk record");
return -1;
}
for (int i = 0; i < chunk->element_count; i++) {
File* file = chunk->items[i];
chunk->items[i] = NULL;
if (file == NULL)
continue;
FileSaveResult result = file_save_to_disk_full(dest_root, file, config);
file_destroy(file);
if (result == FILE_SAVE_ERROR) {
chunk_destroy(chunk);
return -1;
}
}
chunk_destroy(chunk);
}
return 0;
}
+28
View File
@@ -0,0 +1,28 @@
#ifndef BATCH_H
#define BATCH_H
#include "chunk.h"
#include "config.h"
/* Phase 6 residual-batch codec. A residual batch is a self-contained
* single-file record of a whole source tree: a magic+format-version header
* followed by length-prefixed chunk blobs (each built with chunk_serialize),
* byte-identical by construction. The batch is a client-only driver feature:
* it never crosses the wire, so there is no PROTOCOL_VERSION bump and no server
* change. */
#define BATCH_MAGIC "FSTRESBATCH"
#define BATCH_MAGIC_LEN 11
#define BATCH_FORMAT_VERSION 1
/* Max size of a single length-prefixed record (a whole serialized chunk,
* which can span several files). A single source file near the 64 MB wire
* limit plus per-file headers can produce a record slightly over 64 MB, so a
* large file just under the wire cap may be refused by the batch writer; this
* is documented upstream and the failure is clean (the partial batch is
* unlinked), never a truncated/corrupt batch. */
#define BATCH_MAX_RECORD (64ULL * 1024 * 1024)
bool batch_write_header(int fd, const Config* config);
bool batch_write_chunk(int fd, Chunk* chunk);
int batch_read_apply(int fd, const Config* config, const char* dest_root);
#endif
+384
View File
@@ -0,0 +1,384 @@
#include "charset.h"
#include "log.h"
#include "protocol.h"
#include "utils.h"
#include <errno.h>
#include <iconv.h>
#include <stdlib.h>
#include <string.h>
typedef struct {
iconv_t cd;
} CharsetConversion;
/* Process-wide wire conversion descriptor (one direction per process: a client
* only sends, a server only receives). CONCURRENCY CONTRACT: iconv_t is not
* guaranteed thread-safe, so every conversion MUST run on a single thread at a
* time. This holds today -- on the client the conversions run on the sender
* thread (in the -m pipeline chunk_serialize/send happen on the sender thread
* only), on the server on the receive-loop thread; the descriptor is
* initialized on one thread before any transfer thread spawns and torn down
* (charset_wire_free) only after all threads have joined. Do not add a
* concurrent conversion path (e.g. parallel chunk serialization) without
* guarding access with a mutex. */
static CharsetConversion* g_wire_conv;
/* Grow *buf to double capacity, freeing it on failure. realloc preserves the
* already-written prefix, so the caller only tracks its write offset. */
static bool grow_charset_buffer(char** buf, size_t* cap) {
size_t new_cap = *cap * 2;
if (new_cap <= *cap) {
free(*buf);
*buf = NULL;
return false;
}
char* grown = realloc(*buf, new_cap);
if (!grown) {
free(*buf);
*buf = NULL;
return false;
}
*buf = grown;
*cap = new_cap;
return true;
}
/* Throw away any pending shift state so a subsequent conversion starts clean.
* The flush output is discarded; for the stateless single-byte/UTF charsets
* this feature targets it is a no-op. */
static void charset_conversion_reset(const CharsetConversion* conv) {
char scratch[64];
char* sp = scratch;
size_t sl = sizeof(scratch);
(void)iconv(conv->cd, NULL, NULL, &sp, &sl);
}
int charset_spec_parse(const char* spec, char** local_out, char** remote_out) {
if (!local_out || !remote_out)
return -1;
*local_out = NULL;
*remote_out = NULL;
if (!spec || spec[0] == '\0')
return -1;
char* dup = str_dup(spec);
if (!dup)
return -1;
char* comma = strchr(dup, ',');
if (comma) {
if (comma == dup || comma[1] == '\0') {
free(dup);
return -1;
}
*comma = '\0';
*local_out = str_dup(dup);
*remote_out = str_dup(comma + 1);
free(dup);
} else {
*local_out = str_dup(dup);
*remote_out = str_dup(dup);
free(dup);
}
if (!*local_out || !*remote_out) {
free(*local_out);
free(*remote_out);
*local_out = NULL;
*remote_out = NULL;
return -1;
}
return 0;
}
void* charset_conversion_open(const char* from_charset, const char* to_charset) {
if (!from_charset || !to_charset)
return NULL;
iconv_t cd = iconv_open(to_charset, from_charset);
if (cd == (iconv_t)-1)
return NULL;
CharsetConversion* conv = malloc(sizeof(CharsetConversion));
if (!conv) {
iconv_close(cd);
return NULL;
}
conv->cd = cd;
return conv;
}
void charset_conversion_close(void* conversion) {
if (!conversion)
return;
CharsetConversion* conv = (CharsetConversion*)conversion;
iconv_close(conv->cd);
free(conv);
}
/* Probe a single conversion direction: the from/to charsets both open AND a
* representative ASCII name converts to a byte string containing no embedded
* NUL (so a target charset like UTF-16 that emits NUL bytes for ordinary ASCII
* names is rejected up front -- such an output would be silently truncated by
* the C-string wire helpers). */
static bool direction_probe_valid(const char* from, const char* to) {
if (!from || !to)
return false;
void* conv = charset_conversion_open(from, to);
if (!conv)
return false;
bool ok = true;
char input = 'a';
char* in_ptr = &input;
size_t in_left = 1;
char out_buf[64];
char* out_ptr = out_buf;
size_t out_left = sizeof(out_buf);
if (iconv(((CharsetConversion*)conv)->cd, &in_ptr, &in_left, &out_ptr, &out_left) == (size_t)-1)
ok = false;
char flush_buf[64];
char* flush_ptr = flush_buf;
size_t flush_left = sizeof(flush_buf);
if (ok &&
iconv(((CharsetConversion*)conv)->cd, NULL, NULL, &flush_ptr, &flush_left) == (size_t)-1)
ok = false;
size_t produced = (size_t)(out_ptr - out_buf);
if (ok && produced > 0 && memchr(out_buf, '\0', produced) != NULL)
ok = false;
charset_conversion_close(conv);
return ok;
}
bool charset_pair_valid(const char* local, const char* remote) {
/* Both ends convert in opposite directions with the same two charsets, so a
* valid spec must open (and be NUL-free) in BOTH directions: the sender
* opens local->remote, the receiver opens remote->local. */
return direction_probe_valid(local, remote) && direction_probe_valid(remote, local);
}
bool charset_spec_valid(const char* spec) {
if (!spec)
return true;
char* local;
char* remote;
if (charset_spec_parse(spec, &local, &remote) != 0)
return false;
bool ok = charset_pair_valid(local, remote);
free(local);
free(remote);
return ok;
}
bool charset_spec_valid_direction(const char* from_charset, const char* to_charset) {
return direction_probe_valid(from_charset, to_charset);
}
/* The receiver's real conversion is wire(client REMOTE) -> server-local (the
* server's own --iconv LOCAL half, or the client's LOCAL half when the server
* has no --iconv). A dedicated pre-ack check so an impossible direction is
* rejected before the connection instead of refusing mid-transfer. */
bool charset_wire_receiver_spec_valid(const char* spec, const char* server_spec) {
if (!spec)
return true;
char* local;
char* remote;
if (charset_spec_parse(spec, &local, &remote) != 0)
return false;
const char* wire = remote;
const char* target_local = local;
char* server_local = NULL;
char* server_remote = NULL;
if (server_spec) {
if (charset_spec_parse(server_spec, &server_local, &server_remote) != 0) {
free(local);
free(remote);
return false;
}
target_local = server_local;
}
bool ok = charset_spec_valid_direction(wire, target_local);
free(server_local);
free(server_remote);
free(local);
free(remote);
return ok;
}
char* charset_convert(const void* conversion, const char* in, int* err_out) {
if (!conversion || !in)
return NULL;
const CharsetConversion* conv = (const CharsetConversion*)conversion;
size_t in_len = strlen(in);
size_t cap = in_len + 16;
char* out = malloc(cap);
if (!out)
return NULL;
size_t in_left = in_len;
char* in_ptr = (char*)in;
size_t out_used = 0;
while (in_left > 0) {
char* out_ptr = out + out_used;
size_t out_left = cap - out_used;
if (iconv(conv->cd, &in_ptr, &in_left, &out_ptr, &out_left) == (size_t)-1) {
if (errno != E2BIG) {
if (err_out)
*err_out = errno;
charset_conversion_reset(conv);
free(out);
return NULL;
}
/* Output exhausted but input remains. E2BIG does not roll the output
pointer back: the bytes iconv already emitted before the failure must
be preserved, so advance out_used before growing. */
out_used = (size_t)(out_ptr - out);
if (!grow_charset_buffer(&out, &cap))
return NULL;
continue;
}
out_used = (size_t)(out_ptr - out);
}
/* Flush any pending shift state (a no-op for the stateless single-byte and
UTF charsets this feature targets, but keeps the descriptor clean). */
for (;;) {
char* out_ptr = out + out_used;
size_t out_left = cap - out_used;
if (iconv(conv->cd, NULL, NULL, &out_ptr, &out_left) == (size_t)-1) {
if (errno != E2BIG) {
if (err_out)
*err_out = errno;
charset_conversion_reset(conv);
free(out);
return NULL;
}
out_used = (size_t)(out_ptr - out);
if (!grow_charset_buffer(&out, &cap))
return NULL;
continue;
}
out_used = (size_t)(out_ptr - out);
break;
}
/* A successful iconv call may legitimately consume the whole buffer (output
exactly fills cap), leaving no room for the terminator: guarantee headroom
before the final write. */
if (out_used >= cap && !grow_charset_buffer(&out, &cap))
return NULL;
/* Defense in depth: a target charset that emits embedded NUL bytes would
truncate at the first NUL in the C-string wire helpers; fail cleanly
(validation already rejects such charsets up front). */
if (memchr(out, '\0', out_used) != NULL) {
if (err_out)
*err_out = EILSEQ;
charset_conversion_reset(conv);
free(out);
return NULL;
}
out[out_used] = '\0';
return out;
}
bool charset_wire_init_sender(const char* spec) {
charset_wire_free();
if (!spec)
return true;
char* local;
char* remote;
if (charset_spec_parse(spec, &local, &remote) != 0)
return false;
void* conv = charset_conversion_open(local, remote);
free(local);
free(remote);
if (!conv)
return false;
g_wire_conv = (CharsetConversion*)conv;
return true;
}
bool charset_wire_init_receiver(const char* spec, const char* server_spec) {
charset_wire_free();
if (!spec)
return true;
char* local;
char* remote;
if (charset_spec_parse(spec, &local, &remote) != 0)
return false;
/* The wire charset is the client spec's REMOTE half; the local charset is
* the client spec's LOCAL half unless the server was itself started with
* --iconv naming a different local charset (the server halves above never
* travel, so the server's own flag is the only way its local charset can
* differ from what the client assumed). */
const char* wire = remote;
const char* target_local = local;
char* server_local = NULL;
char* server_remote = NULL;
if (server_spec) {
if (charset_spec_parse(server_spec, &server_local, &server_remote) != 0) {
free(local);
free(remote);
return false;
}
target_local = server_local;
}
void* conv = charset_conversion_open(wire, target_local);
free(server_local);
free(server_remote);
free(local);
free(remote);
if (!conv)
return false;
g_wire_conv = (CharsetConversion*)conv;
return true;
}
void charset_wire_free(void) {
if (g_wire_conv) {
charset_conversion_close(g_wire_conv);
g_wire_conv = NULL;
}
}
bool charset_wire_active(void) {
return g_wire_conv != NULL;
}
char* charset_wire_apply(const char* path) {
if (!g_wire_conv)
return str_dup(path);
return charset_convert(g_wire_conv, path, NULL);
}
static void charset_convert_failure_log(const char* path) {
char* escaped = output_escape(path, false);
log_message(LOG_LEVEL_ERROR, "--iconv: cannot convert file name '%s' to the target charset",
escaped ? escaped : "<unprintable>");
free(escaped);
}
bool send_wire_str(int file_descriptor, const char* local_path) {
if (!g_wire_conv)
return send_str(file_descriptor, local_path);
char* wire = charset_wire_apply(local_path);
if (!wire) {
charset_convert_failure_log(local_path);
return false;
}
bool ok = send_str(file_descriptor, wire);
free(wire);
return ok;
}
char* receive_wire_str(int file_descriptor) {
char* raw = receive_str(file_descriptor);
if (!raw)
return NULL;
if (!g_wire_conv)
return raw;
char* local = charset_convert(g_wire_conv, raw, NULL);
if (!local) {
charset_convert_failure_log(raw);
free(raw);
return NULL;
}
free(raw);
return local;
}
+85
View File
@@ -0,0 +1,85 @@
#ifndef CHARSET_H
#define CHARSET_H
#include <stdbool.h>
#include <stddef.h>
/* --iconv=CONVERT_SPEC file-name charset conversion (rsync compatibility).
*
* CONVERT_SPEC is "LOCAL[,REMOTE]": LOCAL is the charset of our own file
* names, REMOTE is the charset of the remote side's file names and defaults
* to LOCAL when the comma half is omitted. The sender converts every local
* path from LOCAL to REMOTE before it goes on the wire; the receiver converts
* every received path back from REMOTE to LOCAL. A NULL/disabled spec means
* identity with zero overhead (the common path never consults iconv).
*
* All helpers are friendly to the strict cold path: the wire conversion state
* is process-global (one direction per process -- a client only sends, a
* server only receives) and is initialized once, before any path is
* serialized, so conversion compiles to a single non-NULL check when disabled.
*/
/* Parse CONVERT_SPEC into malloc'd LOCAL and REMOTE charset names (caller
* frees both). REMOTE is a separate copy of LOCAL when no comma is present.
* Returns 0 on success, -1 on a malformed spec (empty halves / missing value /
* allocation failure); nothing is allocated on the -1 path. Both output
* pointers are REQUIRED (non-NULL). */
int charset_spec_parse(const char* spec, char** local_out, char** remote_out);
/* True when a CONVERT_SPEC is well-formed AND its charsets are usable for this
* feature: each pair opens in a probe iconv_open in BOTH directions (a sender
* converts local->remote, the receiver converts remote->local) and converting
* a representative ASCII name emits no embedded NUL byte (a UTF-16-style NUL
* emitter would be silently truncated by the C-string wire helpers). A typo'd
* charset name is therefore rejected at startup, not mid-run. NULL (iconv
* disabled) is always valid. */
bool charset_spec_valid(const char* spec);
/* Probe a concrete from->to conversion pair without keeping the descriptor:
* both charsets open AND a representative ASCII name converts with no embedded
* NUL. Used for direction-specific validation (e.g. the receiver's exact
* wire->local direction including a server-side charset override). */
bool charset_spec_valid_direction(const char* from_charset, const char* to_charset);
bool charset_pair_valid(const char* local, const char* remote);
/* One-shot conversion of a NUL-terminated input to a malloc'd NUL-terminated
* result, or NULL on failure. On failure *err_out (when non-NULL) receives
* the iconv errno (EILSEQ/EINVAL = the input is not representable in the
* target charset). The caller must free the result. */
char* charset_convert(const void* conversion, const char* in, int* err_out);
/* Open a conversion descriptor for direction from_charset -> to_charset.
* Returns NULL (errno = EINVAL) when a charset name is unsupported. Freed
* with charset_conversion_close. */
void* charset_conversion_open(const char* from_charset, const char* to_charset);
void charset_conversion_close(void* conversion);
/* Process-wide wire conversion. charset_wire_init_sender (client side) opens
* LOCAL->REMOTE; charset_wire_init_receiver (server side) opens
* wire(REMOTE)->server-local. server_spec is the server's own --iconv, whose
* LOCAL half may override the local charset the client assumed; NULL reuses
* the client spec's LOCAL half. Both return false on an unsupported spec.
* The state is freed with charset_wire_free. */
bool charset_wire_init_sender(const char* spec);
bool charset_wire_init_receiver(const char* spec, const char* server_spec);
void charset_wire_free(void);
bool charset_wire_active(void);
/* Pre-ack receiver-direction sanity (see charset_wire_init_receiver): true
* when the exact wire->server-local conversion the receiver will use (client
* spec's REMOTE half into the server's own LOCAL half, or the client's LOCAL
* half when the server has no --iconv) opens and produces NUL-free output. */
bool charset_wire_receiver_spec_valid(const char* spec, const char* server_spec);
/* Convert a path across the wire in the process direction. Returns a malloc'd
* string, or NULL when the name cannot be represented in the target charset. */
char* charset_wire_apply(const char* path);
/* Convenience wire string I/O: encode+send_str / receive_str+decode. Both
* return false/NULL (logging a clear --iconv error) on conversion failure, so
* an unconvertible path FAILS the transfer cleanly instead of silently sending
* a mangled name. */
bool send_wire_str(int file_descriptor, const char* local_path);
char* receive_wire_str(int file_descriptor);
#endif
+75
View File
@@ -0,0 +1,75 @@
#include "checksum.h"
#include <openssl/evp.h>
#include <string.h>
#include <strings.h>
/* delta.c owns the single XXH_IMPLEMENTATION that provides the xxHash symbols
* for the whole binary; this TU only needs the declarations. */
#include <xxhash.h>
bool checksum_digest(ChecksumAlgo algo, uint64_t seed, const void* data, size_t size, uint8_t* out,
size_t out_capacity, size_t* out_len) {
if (!out || !out_len || out_capacity < CHECKSUM_MAX_DIGEST_LEN)
return false;
if (data == NULL && size != 0)
return false;
if (algo == CHECKSUM_ALGO_XXH64) {
uint64_t digest = XXH64(data, size, seed);
memcpy(out, &digest, sizeof(digest));
*out_len = sizeof(digest);
return true;
}
if (algo == CHECKSUM_ALGO_MD5) {
/* md5 takes no seed; the caller's seed is deliberately ignored (documented
* in RSYNC_COMPAT.md). OpenSSL's one-shot EVP_Digest needs a non-NULL
* buffer even for an empty input, so map a NULL data + size==0 to an empty
* buffer. */
static const uint8_t empty = 0;
const void* input = data ? data : &empty;
unsigned int digest_len = 0;
if (EVP_Digest(input, size, out, &digest_len, EVP_md5(), NULL) != 1)
return false;
if (digest_len > out_capacity)
return false;
*out_len = digest_len;
return true;
}
return false;
}
int checksum_algo_from_name(const char* name) {
if (!name)
return -1;
if (strcasecmp(name, "xxh64") == 0 || strcasecmp(name, "xxhash") == 0)
return (int)CHECKSUM_ALGO_XXH64;
if (strcasecmp(name, "md5") == 0)
return (int)CHECKSUM_ALGO_MD5;
return -1;
}
const char* checksum_algo_name(ChecksumAlgo algo) {
switch (algo) {
case CHECKSUM_ALGO_XXH64:
return "xxh64";
case CHECKSUM_ALGO_MD5:
return "md5";
}
return "<unknown>";
}
bool checksum_algo_valid(int algo) {
return algo == (int)CHECKSUM_ALGO_XXH64 || algo == (int)CHECKSUM_ALGO_MD5;
}
uint8_t checksum_digest_len(ChecksumAlgo algo) {
switch (algo) {
case CHECKSUM_ALGO_XXH64:
return 8;
case CHECKSUM_ALGO_MD5:
return 16;
}
return 0;
}
+45
View File
@@ -0,0 +1,45 @@
#ifndef CHECKSUM_H
#define CHECKSUM_H
#include <stdbool.h>
#include <stddef.h>
#include <stdint.h>
/* Whole-file content-digest algorithms selectable with --checksum-choice and
* seeded with --checksum-seed. The ids are the values actually placed on the
* wire (config frame), so they must be kept stable and validated on receive.
* CHECKSUM_ALGO_XXH64 == 0 is the default and is byte-for-byte what FastSync
* computed before these options existed (xxHash64 with seed 0). */
typedef enum { CHECKSUM_ALGO_XXH64 = 0, CHECKSUM_ALGO_MD5 = 1 } ChecksumAlgo;
/* md5 digest is 16 bytes, the longest supported. */
#define CHECKSUM_MAX_DIGEST_LEN 16
/* Compute the whole-file digest of the first `size` bytes of `data`.
*
* - CHECKSUM_ALGO_XXH64: xxHash64(data, size, seed) (full 64-bit seed).
* - CHECKSUM_ALGO_MD5: md5(data, size) via OpenSSL EVP.
* md5 has no seed, so `seed` is ignored (documented).
* - `size == 0` hashes the empty input (plus its seed), not a NULL input.
*
* Writes up to `out_capacity` bytes into `out`, storing the digest length in
* *out_len. Returns false on NULL out* or when the digest would not fit.
* Never writes more than CHECKSUM_MAX_DIGEST_LEN bytes. */
bool checksum_digest(ChecksumAlgo algo, uint64_t seed, const void* data, size_t size, uint8_t* out,
size_t out_capacity, size_t* out_len);
/* Resolve a --checksum-choice string (case-insensitive) to an algorithm id.
* Accepts "xxh64" and "xxhash" (both map to CHECKSUM_ALGO_XXH64, rsync's
* xxhash spelling) and "md5". Returns -1 for any unsupported name. */
int checksum_algo_from_name(const char* name);
/* Canonical name of an algorithm (used in CLI error messages). */
const char* checksum_algo_name(ChecksumAlgo algo);
/* True when `algo` is a supported id (used by config receive validation). */
bool checksum_algo_valid(int algo);
/* Digest length in bytes for an algorithm (xxx64 = 8, md5 = 16). */
uint8_t checksum_digest_len(ChecksumAlgo algo);
#endif /* CHECKSUM_H */
+247 -79
View File
@@ -1,11 +1,13 @@
#include <stddef.h>
#include <stdint.h>
#include <limits.h>
#include <stdatomic.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "array_list.h"
#include "charset.h"
#include "chunk.h"
#include "compression.h"
#include "data.h"
@@ -19,6 +21,33 @@
#define MAX_FILE_DATA_SIZE (64ULL * 1024 * 1024)
#define MAX_FILES_PER_CHUNK 65536U
/* Reserve `charge` against `session`'s connection budget. This mirrors the
static protocol_reserve_memory() in protocol.c: the receive-side call sites
only have the Data.owner pointer (a ProtocolSession*), and protocol.c is out
of scope for this fix, so the same atomic CAS accounting is reproduced here.
The matching release always goes through data_destroy()'s Data.owner path. */
static bool chunk_session_reserve(ProtocolSession* session, size_t charge) {
unsigned long long allocated = atomic_load(&session->total_allocated_bytes);
while (true) {
if (allocated > MAX_CONNECTION_MEMORY ||
(unsigned long long)charge > MAX_CONNECTION_MEMORY - allocated)
return false;
if (atomic_compare_exchange_weak(&session->total_allocated_bytes, &allocated,
allocated + (unsigned long long)charge))
return true;
}
}
bool data_charge_session(Data* data, ProtocolSession* session, size_t charge) {
if (!data || charge == 0 || session == NULL)
return true;
if (!chunk_session_reserve(session, charge))
return false;
data->owner = session;
data->protocol_charge = charge;
return true;
}
Chunk* chunk_create(File** items, int element_count) {
if (element_count < 0 || (element_count > 0 && items == NULL))
return NULL;
@@ -63,9 +92,22 @@ void chunk_destroy(void* item) {
free(chunk);
}
/* --iconv: a chunk blob carries wire-charset path/target bytes. Encode the
* sender-side path (a no-op copy when iconv is disabled) so the blob is in the
* same charset as every other wire string. */
static char* chunk_encode_wire(const char* path) {
if (!charset_wire_active())
return str_dup(path);
return charset_wire_apply(path);
}
static unsigned long long per_file_serialize_size(File* file, bool use_metadata) {
unsigned long long size = sizeof(size_t);
size_t path_len = strlen(file_wire_path(file));
char* wire_path = chunk_encode_wire(file_wire_path(file));
if (!wire_path)
return 0;
size_t path_len = strlen(wire_path);
free(wire_path);
unsigned long long metadata_size =
use_metadata ? sizeof(int) + (file->metadata ? FILE_METADATA_WIRE_SIZE : 0) : 0;
if ((unsigned long long)path_len > ULLONG_MAX - size)
@@ -74,16 +116,39 @@ static unsigned long long per_file_serialize_size(File* file, bool use_metadata)
if (metadata_size > ULLONG_MAX - size)
return 0;
size += metadata_size;
/* Entry type marker: 0 = regular file, 1 = explicit directory entry. */
/* Entry type marker: 0 = regular file, 1 = explicit directory entry,
2 = symlink entry (carries its target string), 3 = special/device node
(recreated by the receiver). */
if (sizeof(int) > ULLONG_MAX - size)
return 0;
size += sizeof(int);
/* A special node also carries its rdev major/minor. */
if (file->is_special) {
if (2 * sizeof(int32_t) > ULLONG_MAX - size)
return 0;
size += 2 * sizeof(int32_t);
}
if (sizeof(size_t) > ULLONG_MAX - size)
return 0;
size += sizeof(size_t);
if ((unsigned long long)file->data->size > ULLONG_MAX - size)
return 0;
return size + file->data->size;
size += file->data->size;
/* Symlink entries append the target string (length-prefixed). */
if (file->is_symlink) {
char* wire_target = chunk_encode_wire(file->symlink_target ? file->symlink_target : "");
if (!wire_target)
return 0;
size_t target_len = strlen(wire_target);
free(wire_target);
if (sizeof(size_t) > ULLONG_MAX - size)
return 0;
size += sizeof(size_t);
if ((unsigned long long)target_len > ULLONG_MAX - size)
return 0;
size += target_len;
}
return size;
}
Data* chunk_serialize(Chunk* chunk, bool use_metadata) {
@@ -109,17 +174,31 @@ Data* chunk_serialize(Chunk* chunk, bool use_metadata) {
char* data_pointer = data->data;
for (int i = 0; i < chunk->element_count; i++) {
File* file = chunk->items[i];
const char* wire_path = file_wire_path(file);
char* wire_path = chunk_encode_wire(file_wire_path(file));
if (wire_path == NULL) {
data_destroy(data);
return NULL;
}
size_t path_len = strlen(wire_path);
memcpy(data_pointer, &path_len, sizeof(size_t));
data_pointer += sizeof(size_t);
memcpy(data_pointer, wire_path, path_len);
data_pointer += path_len;
free(wire_path);
int entry_type = file->is_dir ? 1 : 0;
int entry_type = file->is_dir ? 1 : (file->is_symlink ? 2 : (file->is_special ? 3 : 0));
memcpy(data_pointer, &entry_type, sizeof(int));
data_pointer += sizeof(int);
if (file->is_special) {
int32_t special_major = file->rdev_major;
int32_t special_minor = file->rdev_minor;
memcpy(data_pointer, &special_major, sizeof(special_major));
data_pointer += sizeof(special_major);
memcpy(data_pointer, &special_minor, sizeof(special_minor));
data_pointer += sizeof(special_minor);
}
if (use_metadata)
metadata_to_buf(&data_pointer, file->metadata);
@@ -129,6 +208,21 @@ Data* chunk_serialize(Chunk* chunk, bool use_metadata) {
if (file_data_size > 0)
memcpy(data_pointer, file->data->data, file_data_size);
data_pointer += file_data_size;
if (file->is_symlink) {
char* wire_target = chunk_encode_wire(file->symlink_target ? file->symlink_target : "");
if (wire_target == NULL) {
data_destroy(data);
return NULL;
}
size_t target_len = strlen(wire_target);
memcpy(data_pointer, &target_len, sizeof(size_t));
data_pointer += sizeof(size_t);
if (target_len > 0)
memcpy(data_pointer, wire_target, target_len);
data_pointer += target_len;
free(wire_target);
}
}
return data;
}
@@ -141,17 +235,20 @@ Chunk* chunk_deserialize(Data* data, bool use_metadata) {
return NULL;
char* data_pointer = data->data;
size_t remaining_size = data->size;
/* The element currently being parsed is owned by `files` only after the
* array_list_add() at the end of the iteration; until then the error
* epilogue destroys it directly. Keeping this one pointer nulled after the
* hand-off makes the single cleanup path correct for every failure. */
File* file = NULL;
while (remaining_size > 0) {
if ((unsigned int)files->size >= MAX_FILES_PER_CHUNK) {
log_message(LOG_LEVEL_ERROR, "Chunk contains too many files");
array_list_delete(files);
return NULL;
goto error;
}
if (remaining_size < sizeof(size_t)) {
log_message(LOG_LEVEL_ERROR, "Invalid chunk format: not enough data for path length");
array_list_delete(files);
return NULL;
goto error;
}
size_t path_len;
@@ -161,95 +258,117 @@ Chunk* chunk_deserialize(Data* data, bool use_metadata) {
if (path_len > SIZE_MAX - 1 || remaining_size < path_len) {
log_message(LOG_LEVEL_ERROR, "Invalid chunk format: not enough data for path");
array_list_delete(files);
return NULL;
goto error;
}
if (path_len == SIZE_MAX) {
array_list_delete(files);
return NULL;
}
char* path = protocol_alloc(path_len + 1);
if (path == NULL) {
log_perror("Could not allocate memory for file path");
array_list_delete(files);
return NULL;
goto error;
}
memcpy(path, data_pointer, path_len);
path[path_len] = '\0';
if (memchr(path, '\0', path_len) != NULL) {
free(path);
array_list_delete(files);
return NULL;
goto error;
}
data_pointer += path_len;
remaining_size -= path_len;
if (path_len == 0 || has_path_traversal(path)) {
/* --iconv: the blob holds the wire charset; translate it to the receiver's
local charset before validation and creation so the destination gets the
local name. A name that cannot be decoded fails the file cleanly. */
if (charset_wire_active()) {
char* local_path = charset_wire_apply(path);
free(path);
array_list_delete(files);
return NULL;
if (local_path == NULL) {
log_message(LOG_LEVEL_ERROR,
"--iconv: received chunk file name cannot be converted to the local charset");
goto error;
}
path = local_path;
path_len = strlen(path);
}
File* file = file_create(path);
free(path);
if (file == NULL) {
array_list_delete(files);
return NULL;
if (path_len == 0 || has_path_traversal(path)) {
free(path);
goto error;
}
file = file_create(path);
free(path);
if (file == NULL)
goto error;
if (remaining_size < sizeof(int)) {
log_message(LOG_LEVEL_ERROR, "Invalid chunk format: not enough data for entry type");
file_destroy(file);
array_list_delete(files);
return NULL;
goto error;
}
int entry_type;
memcpy(&entry_type, data_pointer, sizeof(int));
if (entry_type != 0 && entry_type != 1) {
if (entry_type != 0 && entry_type != 1 && entry_type != 2 && entry_type != 3) {
log_message(LOG_LEVEL_ERROR, "Invalid chunk format: bad entry type");
file_destroy(file);
array_list_delete(files);
return NULL;
goto error;
}
file->is_dir = entry_type == 1;
file->is_symlink = entry_type == 2;
file->is_special = entry_type == 3;
data_pointer += sizeof(int);
remaining_size -= sizeof(int);
if (file->is_special) {
if (remaining_size < 2 * (int32_t)sizeof(int32_t)) {
log_message(LOG_LEVEL_ERROR, "Invalid chunk format: not enough data for special rdev");
goto error;
}
int32_t special_major, special_minor;
memcpy(&special_major, data_pointer, sizeof(special_major));
data_pointer += sizeof(special_major);
memcpy(&special_minor, data_pointer, sizeof(special_minor));
data_pointer += sizeof(special_minor);
remaining_size -= 2 * sizeof(int32_t);
/* Reject an out-of-range/negative rdev here as a malformed chunk (the
same 0xffff / 0x00ffffff bounds file_special_rdev_valid uses), so a
bogus large-but-positive rdev is refused cleanly instead of being
deferred to the creation site where it would abort after the frame. */
if (special_major < 0 || special_minor < 0 || special_major > 0xffff ||
special_minor > 0x00ffffff) {
log_message(LOG_LEVEL_ERROR, "Invalid chunk format: out-of-range special rdev");
goto error;
}
file->rdev_major = special_major;
file->rdev_minor = special_minor;
}
if (use_metadata) {
if (remaining_size < sizeof(int)) {
log_message(LOG_LEVEL_ERROR, "Invalid chunk format: not enough data for metadata");
file_destroy(file);
array_list_delete(files);
return NULL;
goto error;
}
// Peek at present flag to determine total size needed before reading
/* Peek at the present flag to determine the total record size before
decoding. metadata_from_buf() independently bounds-checks every read
against remaining_size, so a short body can never over-read. */
int present_flag;
memcpy(&present_flag, data_pointer, sizeof(int));
if ((present_flag != 0 && present_flag != 1) ||
(present_flag == 1 && remaining_size < sizeof(int) + FILE_METADATA_WIRE_SIZE)) {
log_message(LOG_LEVEL_ERROR, "Invalid chunk format: not enough data for metadata body");
file_destroy(file);
array_list_delete(files);
return NULL;
goto error;
}
file->metadata = metadata_from_buf(&data_pointer);
remaining_size -= sizeof(int);
file->metadata = metadata_from_buf((const uint8_t*)data_pointer, remaining_size);
size_t metadata_consumed = sizeof(int);
if (present_flag == 1) {
if (file->metadata == NULL) {
file_destroy(file);
array_list_delete(files);
return NULL;
}
remaining_size -= FILE_METADATA_WIRE_SIZE;
if (file->metadata == NULL)
goto error;
metadata_consumed += FILE_METADATA_WIRE_SIZE;
}
data_pointer += metadata_consumed;
remaining_size -= metadata_consumed;
}
if (remaining_size < sizeof(size_t)) {
log_message(LOG_LEVEL_ERROR, "Invalid chunk format: not enough data for data size");
file_destroy(file);
array_list_delete(files);
return NULL;
goto error;
}
size_t file_data_size;
@@ -259,63 +378,103 @@ Chunk* chunk_deserialize(Data* data, bool use_metadata) {
if (remaining_size < file_data_size) {
log_message(LOG_LEVEL_ERROR, "Invalid chunk format: not enough data for file content");
file_destroy(file);
array_list_delete(files);
return NULL;
goto error;
}
// Reject individual file data larger than the maximum allowed size.
if (file_data_size > MAX_FILE_DATA_SIZE) {
log_message(LOG_LEVEL_ERROR, "File data size %zu exceeds maximum %llu", file_data_size,
(unsigned long long)MAX_FILE_DATA_SIZE);
file_destroy(file);
array_list_delete(files);
return NULL;
goto error;
}
size_t allocation_size = file_data_size > 0 ? file_data_size : 1;
void* file_data = protocol_alloc(allocation_size);
if (file_data == NULL) {
log_perror("Could not allocate memory for file data");
file_destroy(file);
array_list_delete(files);
return NULL;
goto error;
}
memcpy(file_data, data_pointer, file_data_size);
Data* replacement = data_create(file_data, file_data_size);
if (replacement == NULL) {
file_destroy(file);
array_list_delete(files);
return NULL;
if (replacement == NULL)
goto error;
/* Charge the retained per-file copy to the connection budget (when the
inbound chunk carries an owning session) so the queued copies are not
held outside MAX_CONNECTION_MEMORY (B6). A NULL owner (e.g. a local
batch apply) leaves the copy uncharged. */
if (!data_charge_session(replacement, data->owner, allocation_size)) {
log_message(LOG_LEVEL_ERROR, "Per-connection memory limit exceeded for chunk file data");
data_destroy(replacement);
goto error;
}
data_destroy(file->data);
file->data = replacement;
data_pointer += file_data_size;
remaining_size -= file_data_size;
if (!array_list_add(files, file)) {
file_destroy(file);
array_list_delete(files);
return NULL;
if (file->is_symlink) {
if (remaining_size < sizeof(size_t)) {
log_message(LOG_LEVEL_ERROR, "Invalid chunk format: not enough data for symlink target");
goto error;
}
size_t target_len;
memcpy(&target_len, data_pointer, sizeof(size_t));
data_pointer += sizeof(size_t);
remaining_size -= sizeof(size_t);
if (target_len == 0 || remaining_size < target_len) {
log_message(LOG_LEVEL_ERROR, "Invalid chunk format: bad symlink target");
goto error;
}
char* target = protocol_alloc(target_len + 1);
if (!target) {
log_perror("Could not allocate memory for symlink target");
goto error;
}
memcpy(target, data_pointer, target_len);
target[target_len] = '\0';
if (memchr(target, '\0', target_len) != NULL) {
free(target);
goto error;
}
/* The symlink target also rides the wire charset; decode it to the local
charset like the path (a target is a path). */
if (charset_wire_active()) {
char* local_target = charset_wire_apply(target);
free(target);
if (local_target == NULL) {
log_message(LOG_LEVEL_ERROR,
"--iconv: received chunk symlink target cannot be converted to the local "
"charset");
goto error;
}
target = local_target;
}
file->symlink_target = target;
data_pointer += target_len;
remaining_size -= target_len;
}
if (!array_list_add(files, file))
goto error;
file = NULL;
}
File** file_array = (File**)array_list_to_array(files);
if (files->size > 0 && file_array == NULL) {
array_list_delete(files);
return NULL;
}
if (files->size > 0 && file_array == NULL)
goto error;
Chunk* chunk = chunk_create(file_array, files->size);
free(file_array);
if (chunk == NULL) {
array_list_delete(files);
return NULL;
}
if (chunk == NULL)
goto error;
files->item_destroyer = NULL;
array_list_delete(files);
return chunk;
error:
if (file)
file_destroy(file);
array_list_delete(files);
return NULL;
}
Data* chunk_compress(Chunk* chunk, int compression_level, bool use_metadata) {
@@ -344,12 +503,21 @@ Chunk* receive_chunk_data(int fd, const Config* config) {
}
Data* data_to_process = chunk_data;
if (config->use_compression) {
/* Preserve the inbound session across decompression so the (larger)
decompressed chunk is charged to the same connection budget; the
compressed buffer's own charge is released by data_destroy below. */
ProtocolSession* owner = chunk_data->owner;
data_to_process = data_decompress_limited(chunk_data, MAX_CHUNK_SIZE);
data_destroy(chunk_data);
if (data_to_process == NULL) {
log_message(LOG_LEVEL_ERROR, "Failed to decompress chunk");
return NULL;
}
if (!data_charge_session(data_to_process, owner, data_to_process->size)) {
log_message(LOG_LEVEL_ERROR, "Per-connection memory limit exceeded for decompressed chunk");
data_destroy(data_to_process);
return NULL;
}
}
// Reject chunks larger than the maximum allowed size to prevent OOM.
+11
View File
@@ -23,4 +23,15 @@ Data* chunk_compress_with_threads(Chunk* chunk, int compression_level, bool use_
int compression_threads);
Chunk* receive_chunk_data(int fd, const Config* config);
/* Charge `charge` retained bytes of `data` against `session`'s per-connection
* budget (MAX_CONNECTION_MEMORY), mirroring the protocol layer's accounting, and
* record them on `data` so data_destroy() returns the charge through the
* Data.owner path. Returns false (leaving `data` uncharged) when the ceiling
* would be exceeded. A NULL/zero-size charge or a NULL session is a no-op
* success. The receive-side decompression and chunk-copy paths know the owning
* session only through the Data.owner of the buffer they are processing, so
* this is the entry point that lets them participate in the connection budget
* without a session handle (B6). */
bool data_charge_session(Data* data, ProtocolSession* session, size_t charge);
#endif
+203 -52
View File
@@ -2,11 +2,12 @@
#include "data.h"
#include "log.h"
#include "protocol.h"
#include <stdlib.h>
#include <limits.h>
#include <stdint.h>
#include <stdlib.h>
#include <string.h>
#include <strings.h>
#include <threads.h>
#include <unistd.h>
#include <zstd.h>
@@ -16,10 +17,6 @@
static char* SKIP_COMPRESSION_EXTENSIONS[] = {".jpg", ".jpeg", ".png", ".gif", ".mp4", ".mkv",
".zip", ".gz", ".xz", ".zst", NULL};
bool compression_should_skip(const char* path) {
return compression_should_skip_with_suffixes(path, NULL, -1);
}
bool compression_should_skip_with_suffixes(const char* path, char* const* suffixes, int count) {
if (!path)
return false;
@@ -39,6 +36,96 @@ bool compression_should_skip_with_suffixes(const char* path, char* const* suffix
return false;
}
/* Per-thread cache of zstd contexts plus the grow-only compression scratch
* buffer. zstd contexts are stateful and not safe to share between threads,
* so each thread keeps its own (see compression_get_thread_ctx). The cache is
* stored in a C11 thread-specific storage slot whose destructor releases the
* contexts when the thread exits; this keeps LeakSanitizer clean for the
* short-lived sender/receiver/scanner worker threads without every worker
* entry point having to remember to call compression_free_thread_contexts().
* The main thread's slot is not torn down by tss at process exit, so an atexit
* hook releases it (and compression_free_thread_contexts allows eager
* release). */
typedef struct {
ZSTD_CCtx* cctx;
ZSTD_DCtx* dctx;
void* out_buf; /* reusable ZSTD_compressBound-sized output scratch */
size_t out_cap; /* bytes currently allocated for out_buf */
int level; /* compression level currently applied to cctx */
int workers; /* nbWorkers currently applied to cctx */
bool params_set;
bool cached; /* false when the TSS slot could not be used: caller owns */
} CompressionThreadCtx;
static once_flag compression_tls_once = ONCE_FLAG_INIT;
static tss_t compression_tls_key;
static bool compression_tls_ready;
static void compression_tls_make_key(void);
static void compression_ctx_free(CompressionThreadCtx* ctx) {
if (!ctx)
return;
if (ctx->cctx)
ZSTD_freeCCtx(ctx->cctx);
if (ctx->dctx)
ZSTD_freeDCtx(ctx->dctx);
free(ctx->out_buf);
free(ctx);
}
static void compression_tls_destructor(void* value) {
compression_ctx_free((CompressionThreadCtx*)value);
}
void compression_free_thread_contexts(void) {
call_once(&compression_tls_once, compression_tls_make_key);
if (!compression_tls_ready)
return;
CompressionThreadCtx* ctx = (CompressionThreadCtx*)tss_get(compression_tls_key);
if (!ctx)
return;
/* Clear the slot first so the thread-exit destructor cannot free it twice. */
tss_set(compression_tls_key, NULL);
compression_ctx_free(ctx);
}
static void compression_atexit_cleanup(void) {
compression_free_thread_contexts();
}
static void compression_tls_make_key(void) {
if (tss_create(&compression_tls_key, compression_tls_destructor) == thrd_success) {
compression_tls_ready = true;
atexit(compression_atexit_cleanup);
}
}
static CompressionThreadCtx* compression_get_thread_ctx(void) {
call_once(&compression_tls_once, compression_tls_make_key);
if (!compression_tls_ready) {
/* Extremely unlikely: fall back to an uncached context the caller frees. */
return (CompressionThreadCtx*)calloc(1, sizeof(CompressionThreadCtx));
}
CompressionThreadCtx* ctx = (CompressionThreadCtx*)tss_get(compression_tls_key);
if (ctx)
return ctx;
ctx = (CompressionThreadCtx*)calloc(1, sizeof(CompressionThreadCtx));
if (!ctx)
return NULL;
ctx->cached = true;
if (tss_set(compression_tls_key, ctx) != thrd_success)
ctx->cached = false;
return ctx;
}
/* Release an uncached context immediately; cached contexts are owned by the
* thread's TSS slot and freed on thread exit / compression_free_thread_contexts. */
static void compression_ctx_put(CompressionThreadCtx* ctx) {
if (ctx && !ctx->cached)
compression_ctx_free(ctx);
}
Data* data_compress(Data* data_to_compress, int compression_level) {
return data_compress_with_threads(data_to_compress, compression_level, 0);
}
@@ -50,68 +137,102 @@ Data* data_compress_with_threads(Data* data_to_compress, int compression_level,
return NULL;
log_message(LOG_LEVEL_DEBUG, "Starting to compress data");
size_t dst_size = ZSTD_compressBound(data_to_compress->size);
Data* compressed_data = data_create_empty(dst_size);
if (compressed_data == NULL)
return NULL;
ZSTD_CCtx* cctx = ZSTD_createCCtx();
if (!cctx) {
log_message(LOG_LEVEL_ERROR, "Failed to create ZSTD compression context");
data_destroy(compressed_data);
CompressionThreadCtx* ctx = compression_get_thread_ctx();
if (ctx == NULL) {
log_message(LOG_LEVEL_ERROR, "Failed to allocate ZSTD compression context");
return NULL;
}
Data* compressed_data = NULL;
size_t zret = ZSTD_CCtx_setParameter(cctx, ZSTD_c_compressionLevel, compression_level);
if (ZSTD_isError(zret)) {
log_message(LOG_LEVEL_ERROR, "Failed to set compression level: %s", ZSTD_getErrorName(zret));
ZSTD_freeCCtx(cctx);
data_destroy(compressed_data);
return NULL;
if (!ctx->cctx) {
ctx->cctx = ZSTD_createCCtx();
if (!ctx->cctx) {
log_message(LOG_LEVEL_ERROR, "Failed to create ZSTD compression context");
goto cleanup;
}
ctx->params_set = false;
}
/* Reset only the session: parameters (and any already-allocated zstd worker
* pool) stay attached to the context, so compressing the next file does not
* rebuild the pool. */
ZSTD_CCtx_reset(ctx->cctx, ZSTD_reset_session_only);
if (!ctx->params_set || ctx->level != compression_level) {
size_t zret = ZSTD_CCtx_setParameter(ctx->cctx, ZSTD_c_compressionLevel, compression_level);
if (ZSTD_isError(zret)) {
log_message(LOG_LEVEL_ERROR, "Failed to set compression level: %s", ZSTD_getErrorName(zret));
goto cleanup;
}
ctx->level = compression_level;
}
int available_threads = 0;
if (compression_threads > 0) {
long online_cpus = sysconf(_SC_NPROCESSORS_ONLN);
int available_threads = online_cpus > 0 && online_cpus < compression_threads
? (int)online_cpus
: compression_threads;
zret = ZSTD_CCtx_setParameter(cctx, ZSTD_c_nbWorkers, available_threads);
available_threads = online_cpus > 0 && online_cpus < compression_threads ? (int)online_cpus
: compression_threads;
}
if (!ctx->params_set || ctx->workers != available_threads) {
size_t zret = ZSTD_CCtx_setParameter(ctx->cctx, ZSTD_c_nbWorkers, available_threads);
if (ZSTD_isError(zret)) {
log_message(LOG_LEVEL_ERROR, "Failed to set compression threads: %s",
ZSTD_getErrorName(zret));
ZSTD_freeCCtx(cctx);
data_destroy(compressed_data);
return NULL;
goto cleanup;
}
ctx->workers = available_threads;
}
ctx->params_set = true;
if (available_threads > 0) {
/* Streaming compression needs the source size before threaded mode can end a frame. */
zret = ZSTD_CCtx_setPledgedSrcSize(cctx, data_to_compress->size);
size_t zret = ZSTD_CCtx_setPledgedSrcSize(ctx->cctx, data_to_compress->size);
if (ZSTD_isError(zret)) {
log_message(LOG_LEVEL_ERROR, "Failed to set compression source size: %s",
ZSTD_getErrorName(zret));
ZSTD_freeCCtx(cctx);
data_destroy(compressed_data);
return NULL;
goto cleanup;
}
}
if (ctx->out_cap < dst_size) {
void* grown = protocol_realloc(ctx->out_buf, dst_size);
if (grown == NULL) {
log_message(LOG_LEVEL_ERROR, "Failed to allocate compression buffer");
goto cleanup;
}
ctx->out_buf = grown;
ctx->out_cap = dst_size;
}
ZSTD_inBuffer input = {data_to_compress->data, data_to_compress->size, 0};
ZSTD_outBuffer output = {compressed_data->data, dst_size, 0};
ZSTD_outBuffer output = {ctx->out_buf, dst_size, 0};
size_t ret;
do {
ret = ZSTD_compressStream2(cctx, &output, &input, ZSTD_e_end);
ret = ZSTD_compressStream2(ctx->cctx, &output, &input, ZSTD_e_end);
if (ZSTD_isError(ret)) {
log_message(LOG_LEVEL_ERROR, "Compression failed: %s", ZSTD_getErrorName(ret));
ZSTD_freeCCtx(cctx);
data_destroy(compressed_data);
return NULL;
goto cleanup;
}
} while (ret > 0);
/* Hand off an exactly-sized copy; the scratch buffer stays cached so the next
* call does not reallocate a ZSTD_compressBound-sized block. */
compressed_data = data_create_empty(output.pos);
if (compressed_data == NULL) {
log_message(LOG_LEVEL_ERROR, "Failed to allocate compressed data");
goto cleanup;
}
if (output.pos > 0)
memcpy(compressed_data->data, ctx->out_buf, output.pos);
compressed_data->size = output.pos;
ZSTD_freeCCtx(cctx);
log_debug_message(LOG_DEBUG_UTIL, "Data succesfully compressed from %zu to %zu",
data_to_compress->size, compressed_data->size);
cleanup:
compression_ctx_put(ctx);
return compressed_data;
}
@@ -122,9 +243,13 @@ Data* data_decompress_limited(Data* compressed_data, size_t maximum_size) {
log_debug_message(LOG_DEBUG_UTIL, "Start to decompress data");
unsigned long long dst_size =
ZSTD_getFrameContentSize(compressed_data->data, compressed_data->size);
if (ZSTD_isError(dst_size)) {
log_message(LOG_LEVEL_ERROR, "Failed to get decompressed size: %s",
ZSTD_getErrorName(dst_size));
/* ZSTD_isError() is also true for ZSTD_CONTENTSIZE_ERROR and
* ZSTD_CONTENTSIZE_UNKNOWN (both are encoded near (size_t)-1), so test the
* sentinels explicitly instead of blanket-rejecting every error-ish value:
* only CONTENTSIZE_ERROR means an unreadable header, while CONTENTSIZE_UNKNOWN
* must reach the estimate fallback below. */
if (dst_size == ZSTD_CONTENTSIZE_ERROR) {
log_message(LOG_LEVEL_ERROR, "Failed to get decompressed size: invalid zstd frame");
return NULL;
}
@@ -144,20 +269,30 @@ Data* data_decompress_limited(Data* compressed_data, size_t maximum_size) {
return NULL;
}
ZSTD_DCtx* dctx = ZSTD_createDCtx();
if (!dctx) {
log_message(LOG_LEVEL_ERROR, "Failed to create ZSTD decompression context");
CompressionThreadCtx* ctx = compression_get_thread_ctx();
if (ctx == NULL) {
log_message(LOG_LEVEL_ERROR, "Failed to allocate ZSTD decompression context");
return NULL;
}
Data* uncompressed_data = NULL;
if (!ctx->dctx) {
ctx->dctx = ZSTD_createDCtx();
if (!ctx->dctx) {
log_message(LOG_LEVEL_ERROR, "Failed to create ZSTD decompression context");
goto cleanup;
}
}
/* Reset only the session; decompression parameters are sticky. */
ZSTD_DCtx_reset(ctx->dctx, ZSTD_reset_session_only);
size_t buf_size = (dst_size > 0) ? (size_t)dst_size : INITIAL_DECOMPRESS_BUF_SIZE;
if (buf_size > maximum_size)
buf_size = maximum_size;
Data* uncompressed_data = data_create_empty(buf_size);
uncompressed_data = data_create_empty(buf_size);
if (!uncompressed_data) {
log_message(LOG_LEVEL_ERROR, "Failed to allocate decompression buffer");
ZSTD_freeDCtx(dctx);
return NULL;
goto cleanup;
}
ZSTD_inBuffer input = {compressed_data->data, compressed_data->size, 0};
@@ -165,20 +300,20 @@ Data* data_decompress_limited(Data* compressed_data, size_t maximum_size) {
size_t ret;
do {
ret = ZSTD_decompressStream(dctx, &output, &input);
ret = ZSTD_decompressStream(ctx->dctx, &output, &input);
if (ZSTD_isError(ret)) {
log_message(LOG_LEVEL_ERROR, "Decompression failed: %s", ZSTD_getErrorName(ret));
ZSTD_freeDCtx(dctx);
data_destroy(uncompressed_data);
return NULL;
uncompressed_data = NULL;
goto cleanup;
}
if (ret > 0 && output.pos == output.size) {
if (buf_size >= hard_limit || buf_size > SIZE_MAX / 2) {
log_message(LOG_LEVEL_ERROR, "Decompressed data exceeds %llu bytes",
(unsigned long long)MAX_DECOMPRESSED_SIZE);
ZSTD_freeDCtx(dctx);
data_destroy(uncompressed_data);
return NULL;
uncompressed_data = NULL;
goto cleanup;
}
buf_size *= 2;
if (buf_size > hard_limit)
@@ -186,20 +321,36 @@ Data* data_decompress_limited(Data* compressed_data, size_t maximum_size) {
void* new_data = protocol_realloc(uncompressed_data->data, buf_size);
if (!new_data) {
log_message(LOG_LEVEL_ERROR, "Failed to grow decompression buffer");
ZSTD_freeDCtx(dctx);
data_destroy(uncompressed_data);
return NULL;
uncompressed_data = NULL;
goto cleanup;
}
uncompressed_data->data = new_data;
output.dst = new_data;
output.size = buf_size;
/* Re-attempt with the larger output buffer; the truncated-frame check
* below must not reject a complete frame that merely filled the previous
* buffer exactly. */
continue;
}
/* A positive hint with all input consumed means the frame is incomplete: a
* truncated stream would otherwise spin here forever (ZSTD_decompressStream
* keeps returning the same hint). Fail instead of burning CPU. */
if (ret != 0 && input.pos == input.size) {
log_message(LOG_LEVEL_ERROR,
"Truncated zstd frame: input exhausted with %zu bytes still expected", ret);
data_destroy(uncompressed_data);
uncompressed_data = NULL;
goto cleanup;
}
} while (ret > 0);
uncompressed_data->size = output.pos;
ZSTD_freeDCtx(dctx);
log_debug_message(LOG_DEBUG_UTIL, "Decompressed data successfully");
cleanup:
compression_ctx_put(ctx);
return uncompressed_data;
}
+8 -1
View File
@@ -11,7 +11,14 @@ Data* data_compress_with_threads(Data* data_to_compress, int compression_level,
int compression_threads);
Data* data_decompress(Data* compressed_data);
Data* data_decompress_limited(Data* compressed_data, size_t maximum_size);
bool compression_should_skip(const char* path);
bool compression_should_skip_with_suffixes(const char* path, char* const* suffixes, int count);
/* Release the calling thread's cached zstd contexts (compressor, decompressor
* and scratch buffer). The cache is thread-local and is also released
* automatically when a worker thread exits (via a C11 tss destructor) and for
* the main thread at process exit; this explicit entry point exists so tests
* and long-lived callers can drop the cache deterministically. Safe to call
* when no context has been created, and idempotent. */
void compression_free_thread_contexts(void);
#endif
+988 -347
View File
File diff suppressed because it is too large. Load diff
+794 -136
View File
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
+203
View File
@@ -0,0 +1,203 @@
#ifndef CREDENTIALS_H
#define CREDENTIALS_H
#include <stdbool.h>
#include <stddef.h>
#include <stdint.h>
#include <stdio.h>
/* Daemon password authentication (A7 remediation, protocol 2.19.0).
*
* FastSync authenticates a daemon connection with a SCRAM-SHA-256-style
* challenge/response handshake. The daemon stores only a salted PBKDF2
* verifier (never the password, and never a value that can be replayed as a
* bearer credential): the client proves knowledge of the password against a
* per-connection server nonce, and the server proves the same shared secret
* back. See credentials.c for the exact derivation.
*
* Server credential store format (--password-file and --early-input): one line
* per entry,
* user:$fastsync$1$pbkdf2-sha256$<iters>$<salt_b64>$<stored_key_b64>$<server_key_b64>
* with standard base64, a 16-byte salt and 32-byte keys, and iters in
* [CREDENTIAL_MIN_ITERS, CREDENTIAL_MAX_ITERS]. Blank lines and lines whose
* first non-space character is '#' or ';' are comments. The parser is STRICT:
* a malformed line fails the whole load so a typo can never silently change who
* may log in. A line holding the legacy (unsalted SHA-256 hex) secret is
* hard-rejected with an actionable "legacy" error; there is no auto-upgrade.
* Use `fastsync-server --hash-credentials` to generate new-format lines.
*
* Alongside the store, credentials_load maintains an exact-mode-0600
* `<store_path>.dummykey` sidecar holding the store-wide random dummy key. It
* is auto-created on first load and MUST be preserved across restarts: it makes
* the dummy challenge for an unknown user stable for the life of the store, so
* a daemon restart cannot be used as a username-enumeration oracle. A sidecar
* that is not an exact-mode-0600 regular file of exactly 32 bytes fails the load
* (fail closed); creation forces exact 0600 with fchmod (so a restrictive umask
* cannot leave the sidecar unreadable), and only a create/write/fsync/link or
* fchmod failure degrades to a transient per-run key with a warning. NOTE: the
* sidecar requires EXACT 0600, whereas the store / password files only reject
* group/other bits (a deliberate difference).
*
* Client --password-file format: the FIRST meaningful (non-comment, non-blank)
* line is `user:password`, holding the literal password. The client keeps it
* only for the duration of the handshake and wipes it at teardown; the file
* should be mode 0600 and readable only by its owner. */
/* Longest accepted credential-file line (excluding the trailing newline). */
#define CREDENTIAL_MAX_LINE 4096
/* Upper bound on a username in a credential file and on the wire. Kept well
* below MAX_STRING_SIZE so a wire username can never exhaust anything. */
#define CREDENTIAL_MAX_USER_LEN 256
/* Upper bound on a client-file password (before derivation). */
#define CREDENTIAL_MAX_PASSWORD_LEN 1024
/* SCRAM-SHA-256 parameters. Salt and client nonce sizes are fixed by the
* shared-auth-message framing; keys are always 32 bytes (SHA-256). */
#define CREDENTIAL_SALT_LEN 16
#define CREDENTIAL_NONCE_LEN 32
#define CREDENTIAL_KEY_LEN 32
#define CREDENTIAL_DEFAULT_ITERS 600000u
#define CREDENTIAL_MIN_ITERS 100000u
#define CREDENTIAL_MAX_ITERS 10000000u
/* Buffer size for the full AuthMessage (prefix + three length-prefixed fields).
* Worst case: 16 + 4 + 256 + 4 + 32 + 4 + 32. */
#define CREDENTIAL_AUTH_MESSAGE_MAX \
(16 + 4 + CREDENTIAL_MAX_USER_LEN + 4 + CREDENTIAL_NONCE_LEN + 4 + CREDENTIAL_NONCE_LEN)
typedef struct CredentialStore CredentialStore;
/* One resolved verifier. `found` is false for an unknown user or a user not on
* a module's auth list; the remaining fields then hold a deterministic dummy
* salt (HMAC of the store-wide dummy key over the username), the store-wide
* uniform iteration count (default for an empty store) and fixed dummy keys, so
* the server can run the same challenge/response math with no enumeration or
* timing oracle. */
typedef struct {
uint8_t salt[CREDENTIAL_SALT_LEN];
uint32_t iters;
uint8_t stored_key[CREDENTIAL_KEY_LEN];
uint8_t server_key[CREDENTIAL_KEY_LEN];
bool found;
} CredentialVerifier;
/* Load the daemon credential store.
*
* password_file and early_input_file are both NULL-or-path, matching the
* server's --password-file and --early-input options. A file that cannot be
* opened or that fails the strict grammar is a hard error (err filled, NULL
* returned) -- the daemon fails CLOSED rather than serving an auth-required
* module with a partial store. Both files may be NULL, which yields an empty
* store (every auth-required module then refuses connections). Every entry in
* the resulting store must agree on the iteration count; entries that disagree
* (within one file or across the two layered sources) are rejected. When both
* are given, the --early-input file is layered over --password-file: a duplicate
* username whose verifier matches is deduplicated; one whose verifier differs
* is an error (the two sources disagree), never a silent pick.
*
* The returned store is heap-owned; free it with credentials_free. */
CredentialStore* credentials_load(const char* password_file, const char* early_input_file,
char* err, size_t err_size);
/* Wipe every stored key/salt and free the store. */
void credentials_free(CredentialStore* store);
/* True when `user` is a single bounded token free of whitespace/control bytes
* (the rule applied to store users, client-file users and the module list). */
bool credentials_username_valid(const char* user);
/* Standard base64. encode writes NUL-terminated output to out (size out_sz).
* decode writes the raw bytes to out (capacity out_sz) and stores the length;
* the input must be a well-formed padded base64 string. Both return false on
* NULL arguments, a bad character/length, or insufficient output space. */
bool credentials_b64_encode(const uint8_t* in, size_t n, char* out, size_t out_sz);
bool credentials_b64_decode(const char* in, uint8_t* out, size_t out_sz, size_t* out_len);
/* Fill out[0..n) from the CSPRNG (RAND_bytes). Returns false on failure. */
bool credentials_random_bytes(uint8_t* out, size_t n);
/* Resolve `user` against the store AND the module's auth-user list. The list
* scan is a constant-time full-length comparison with no early break. On a
* miss, *out is filled with a dummy verifier (a deterministic per-username salt
* derived from the store's dummy key, the store-wide uniform iteration count,
* fixed dummy keys, found=false). Returns false on invalid arguments or an
* HMAC/crypto primitive failure. */
bool credentials_get_verifier(const CredentialStore* store, const char* user,
const char* const* module_users, int n, CredentialVerifier* out);
/* Derive the SCRAM keys from a plaintext password:
* K = PBKDF2-HMAC-SHA256(password, salt, iters, 32)
* ClientKey = HMAC-SHA256(K, "Client Key"); StoredKey = SHA256(ClientKey)
* ServerKey = HMAC-SHA256(K, "Server Key")
* Any of client_key/stored_key/server_key may be NULL when not needed.
* `iters` must lie in [CREDENTIAL_MIN_ITERS, CREDENTIAL_MAX_ITERS]. */
bool credentials_compute_keys(const char* password, const uint8_t salt[CREDENTIAL_SALT_LEN],
uint32_t iters, uint8_t client_key[CREDENTIAL_KEY_LEN],
uint8_t stored_key[CREDENTIAL_KEY_LEN],
uint8_t server_key[CREDENTIAL_KEY_LEN]);
/* Serialize the shared AuthMessage:
* "FastSync-Auth-v1" || be32(len(user)) || user
* || be32(32) || server_nonce
* || be32(32) || client_nonce
* out must hold at least CREDENTIAL_AUTH_MESSAGE_MAX bytes. *out_len receives
* the number of bytes written. */
bool credentials_build_auth_message(const char* user, const uint8_t* snonce, const uint8_t* cnonce,
uint8_t* out, size_t out_sz, size_t* out_len);
/* Client side: ClientProof = ClientKey XOR HMAC(StoredKey, AuthMessage), and
* the expected ServerSignature = HMAC(ServerKey, AuthMessage). */
bool credentials_client_proof(const uint8_t client_key[CREDENTIAL_KEY_LEN],
const uint8_t stored_key[CREDENTIAL_KEY_LEN],
const uint8_t server_key[CREDENTIAL_KEY_LEN], const uint8_t* auth_msg,
size_t msg_len, uint8_t proof[CREDENTIAL_KEY_LEN],
uint8_t server_sig[CREDENTIAL_KEY_LEN]);
/* Server side: recompute ClientSig' = HMAC(StoredKey, AuthMessage) and
* ClientKey' = proof XOR ClientSig', then accept iff v->found AND
* SHA256(ClientKey') equals StoredKey (constant-time over the 32-byte keys).
* Always computes server_sig_out = HMAC(ServerKey, AuthMessage). Returns the
* accept decision. */
bool credentials_verify_response(const CredentialVerifier* v, const char* user,
const uint8_t* snonce, const uint8_t* cnonce,
const uint8_t proof[CREDENTIAL_KEY_LEN],
uint8_t server_sig_out[CREDENTIAL_KEY_LEN]);
/* Derive a new-format store line for `user`/`password` and write it (without a
* trailing newline) into out. A random 16-byte salt is used. On failure err is
* filled. Used by --hash-credentials and by tests. */
bool credentials_hash_store_line(const char* user, const char* password, uint32_t iters, char* out,
size_t out_sz, char* err, size_t err_size);
/* Read `user:password` lines from `path` (the same no-group/other-bits check as
* the other secret files) and write one new-format store line per entry to
* `out`.
* Blank/comment lines are skipped; a malformed line fails the whole run.
* Returns 0 on success, -1 on error (err filled). Used by
* `--hash-credentials`. */
int credentials_hash_file(const char* path, uint32_t iters, FILE* out, char* err, size_t err_size);
/* Read the CLIENT-side secret file: the first meaningful line is
* `user:password` (the literal password). *user_out and *password_out are
* freshly allocated on success (password is plaintext -- the caller derives the
* proof and then burns/frees it); both are NULL on error. Returns 0 on
* success, -1 on failure (err filled: the path is named, never the credential
* itself). Only the line's trailing CR/LF are stripped: the password's bytes
* are otherwise preserved exactly, so a password with leading/trailing
* whitespace (after the ':') is kept usable. The username is trimmed of
* surrounding space/tabs. */
int credentials_read_secret_file(const char* path, char** user_out, char** password_out, char* err,
size_t err_size);
/* Constant-time equality over exactly len bytes. */
bool credentials_secure_equal(const char* a, const char* b, size_t len);
/* Overwrite secret[0..len) with zeros (best-effort wipe). */
void credentials_burn(char* secret, size_t len);
/* Number of entries currently in the store (tests/introspection). */
int credentials_store_size(const CredentialStore* store);
/* Whether the store contains an entry for `user` (tests/introspection). */
bool credentials_store_has(const CredentialStore* store, const char* user);
#endif
+794
View File
@@ -0,0 +1,794 @@
#include "daemon_conf.h"
#include "credentials.h"
#include "utils.h"
#include <arpa/inet.h>
#include <ctype.h>
#include <errno.h>
#include <limits.h>
#include <netinet/in.h>
#include <stdarg.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <strings.h>
/* ------------------------------------------------------------------ */
/* helpers */
/* ------------------------------------------------------------------ */
static void set_error(char* err, size_t err_size, const char* fmt, ...) {
if (!err || err_size == 0)
return;
va_list args;
va_start(args, fmt);
vsnprintf(err, err_size, fmt, args);
va_end(args);
}
/* Trim leading and trailing ASCII space/tab in place; returns the new start. */
static char* trim_ws(char* s) {
while (*s == ' ' || *s == '\t')
s++;
size_t len = strlen(s);
while (len > 0 && (s[len - 1] == ' ' || s[len - 1] == '\t'))
s[--len] = '\0';
return s;
}
/* Case-insensitive equality of a parsed key against a canonical key name. */
static bool key_equals(const char* key, const char* canonical) {
return strcasecmp(key, canonical) == 0;
}
static bool parse_bool_value(const char* value, bool* out) {
if (strcasecmp(value, "yes") == 0 || strcasecmp(value, "true") == 0 || strcmp(value, "1") == 0) {
*out = true;
return true;
}
if (strcasecmp(value, "no") == 0 || strcasecmp(value, "false") == 0 || strcmp(value, "0") == 0) {
*out = false;
return true;
}
return false;
}
/* Parse an IPv4/IPv6 CIDR "addr/prefix" into `bytes`/`*family`. Returns false
* for a malformed address, a missing/oversized prefix, or a prefix that does
* not fit the address family. */
static bool parse_cidr(const char* cidr, int* prefix_out, uint8_t* bytes, int* family_out) {
const char* slash = strchr(cidr, '/');
if (!slash)
return false;
size_t addr_len = (size_t)(slash - cidr);
if (addr_len == 0 || addr_len >= INET6_ADDRSTRLEN)
return false;
char addr[INET6_ADDRSTRLEN];
memcpy(addr, cidr, addr_len);
addr[addr_len] = '\0';
char* end = NULL;
long prefix = strtol(slash + 1, &end, 10);
if (end == slash + 1 || *end != '\0')
return false;
struct in_addr v4;
struct in6_addr v6;
if (inet_pton(AF_INET, addr, &v4) == 1) {
if (prefix < 0 || prefix > 32)
return false;
memcpy(bytes, &v4, sizeof(v4));
*prefix_out = (int)prefix;
*family_out = AF_INET;
return true;
}
if (inet_pton(AF_INET6, addr, &v6) == 1) {
if (prefix < 0 || prefix > 128)
return false;
memcpy(bytes, &v6, sizeof(v6));
*prefix_out = (int)prefix;
*family_out = AF_INET6;
return true;
}
return false;
}
/* A host pattern is valid when it is `*`, a valid IPv4/IPv6 literal, or a valid
* CIDR. Peer addresses reaching the matcher are always numeric, so hostname
* globs are rejected at parse time: accepting one would create a deny rule that
* silently never matches (fail-open). */
static bool host_pattern_valid(const char* pattern) {
if (!pattern || *pattern == '\0')
return false;
if (strcmp(pattern, "*") == 0)
return true;
if (strchr(pattern, '/')) {
uint8_t bytes[16];
int prefix;
int family;
return parse_cidr(pattern, &prefix, bytes, &family);
}
struct in_addr v4;
struct in6_addr v6;
return inet_pton(AF_INET, pattern, &v4) == 1 || inet_pton(AF_INET6, pattern, &v6) == 1;
}
/* Append every comma- and/or whitespace-separated host pattern in `value` to
* the heap-owned list (or replace the list when `replace` is set, which --dparam
* uses so an override can narrow access rather than only widen it). Returns
* false (err filled) on an invalid pattern or an allocation failure. */
static bool store_host_list(char*** list, int* count, const char* value, const char* key,
const char* module_name, bool replace, char* err, size_t err_size) {
if (replace) {
for (int i = 0; i < *count; i++)
free((*list)[i]);
free(*list);
*list = NULL;
*count = 0;
}
char* copy = str_dup(value);
if (!copy) {
if (module_name)
set_error(err, err_size, "out of memory parsing '%s' for module '%s'", key, module_name);
else
set_error(err, err_size, "out of memory parsing '%s'", key);
return false;
}
char* save = NULL;
int added = 0;
for (char* token = strtok_r(copy, ", \t", &save); token; token = strtok_r(NULL, ", \t", &save)) {
if (!host_pattern_valid(token)) {
if (module_name)
set_error(err, err_size, "module '%s': invalid host pattern '%s' in '%s'", module_name,
token, key);
else
set_error(err, err_size, "invalid host pattern '%s' in '%s'", token, key);
free(copy);
return false;
}
char** grown = realloc(*list, (size_t)(*count + 1) * sizeof(char*));
if (!grown) {
if (module_name)
set_error(err, err_size, "out of memory parsing '%s' for module '%s'", key, module_name);
else
set_error(err, err_size, "out of memory parsing '%s'", key);
free(copy);
return false;
}
*list = grown;
char* dup = str_dup(token);
if (!dup) {
if (module_name)
set_error(err, err_size, "out of memory parsing '%s' for module '%s'", key, module_name);
else
set_error(err, err_size, "out of memory parsing '%s'", key);
free(copy);
return false;
}
(*list)[(*count)++] = dup;
added++;
}
free(copy);
/* A present key with an empty (or separator-only) value would otherwise
* install a zero-length list, i.e. no ACL at all: a strict-parse config must
* never silently turn a restrictive directive into "allow everyone". */
if (added == 0) {
if (module_name)
set_error(err, err_size, "module '%s': '%s' must list at least one host pattern", module_name,
key);
else
set_error(err, err_size, "'%s' must list at least one host pattern", key);
return false;
}
return true;
}
/* Parse a `max connections` value: a positive integer (0/negative/garbage are
* rejected because they would silently disable the cap or admit nothing). */
static bool store_max_connections(int* slot, const char* value, const char* module_name, char* err,
size_t err_size) {
char* end = NULL;
errno = 0;
long n = strtol(value, &end, 10);
if (*value == '\0' || errno != 0 || *end != '\0' || n <= 0 || n > INT_MAX) {
if (module_name)
set_error(err, err_size,
"module '%s': invalid 'max connections' '%s' (must be a positive "
"integer)",
module_name, value);
else
set_error(err, err_size, "invalid 'max connections' '%s' (must be a positive integer)",
value);
return false;
}
*slot = (int)n;
return true;
}
/* Parse a non-negative concurrency cap where 0 means unlimited/disabled
* (per-module `max connections`, `max connections per host`,
* `auth lockout threshold`). Negative/garbage/oversized values are rejected. */
static bool store_optional_cap(int* slot, const char* value, int max_value, const char* key,
const char* module_name, char* err, size_t err_size) {
char* end = NULL;
errno = 0;
long n = strtol(value, &end, 10);
if (*value == '\0' || errno != 0 || *end != '\0' || n < 0 || n > max_value) {
if (module_name)
set_error(err, err_size, "module '%s': invalid '%s' '%s' (must be 0-%d)", module_name, key,
value, max_value);
else
set_error(err, err_size, "invalid '%s' '%s' (must be 0-%d)", key, value, max_value);
return false;
}
*slot = (int)n;
return true;
}
/* Parse an `auth failure delay` value: 0 (disabled) through the configured cap. */
static bool store_auth_failure_delay(int* slot, const char* value, char* err, size_t err_size) {
char* end = NULL;
errno = 0;
long n = strtol(value, &end, 10);
if (*value == '\0' || errno != 0 || *end != '\0' || n < 0 ||
n > DAEMON_CONF_MAX_AUTH_FAILURE_DELAY_MS) {
set_error(err, err_size, "invalid 'auth failure delay' '%s' (must be 0-%d milliseconds)", value,
DAEMON_CONF_MAX_AUTH_FAILURE_DELAY_MS);
return false;
}
*slot = (int)n;
return true;
}
bool daemon_module_name_valid(const char* name) {
if (!name || *name == '\0')
return false;
size_t len = strlen(name);
if (len > DAEMON_MAX_MODULE_NAME)
return false;
for (size_t i = 0; i < len; i++) {
unsigned char c = (unsigned char)name[i];
bool alnum = (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9');
if (!alnum && c != '.' && c != '_' && c != '-')
return false;
}
return true;
}
DaemonConf* daemon_conf_create(void) {
DaemonConf* conf = calloc(1, sizeof(DaemonConf));
if (!conf)
return NULL;
conf->global.port = DAEMON_CONF_DEFAULT_PORT;
conf->global.max_connections = DAEMON_CONF_DEFAULT_MAX_CONNECTIONS;
conf->global.auth_failure_delay_ms = DAEMON_CONF_DEFAULT_AUTH_FAILURE_DELAY_MS;
conf->global.max_connections_per_host = DAEMON_CONF_DEFAULT_MAX_CONNECTIONS_PER_HOST;
conf->global.auth_lockout_threshold = DAEMON_CONF_DEFAULT_AUTH_LOCKOUT_THRESHOLD;
conf->global.auth_lockout_duration_sec = DAEMON_CONF_DEFAULT_AUTH_LOCKOUT_DURATION_SEC;
return conf;
}
/* Free a heap-owned pattern list of `count` entries. */
static void free_string_list(char** list, int count) {
for (int i = 0; i < count; i++)
free(list[i]);
free(list);
}
void daemon_conf_free(DaemonConf* conf) {
if (!conf)
return;
free(conf->global.motd_file);
free(conf->global.address);
free_string_list(conf->global.hosts_allow, conf->global.hosts_allow_count);
free_string_list(conf->global.hosts_deny, conf->global.hosts_deny_count);
for (int i = 0; i < conf->module_count; i++) {
DaemonModule* m = &conf->modules[i];
free(m->name);
free(m->path);
for (int j = 0; j < m->auth_user_count; j++)
free(m->auth_users[j]);
free(m->auth_users);
free_string_list(m->hosts_allow, m->hosts_allow_count);
free_string_list(m->hosts_deny, m->hosts_deny_count);
}
free(conf->modules);
free(conf);
}
const DaemonModule* daemon_conf_find_module(const DaemonConf* conf, const char* name) {
if (!conf || !name)
return NULL;
for (int i = 0; i < conf->module_count; i++) {
if (strcmp(conf->modules[i].name, name) == 0)
return &conf->modules[i];
}
return NULL;
}
/* Replace *slot with a str_dup of value; returns false on allocation failure. */
static bool store_string(char** slot, const char* value) {
char* dup = str_dup(value);
if (!dup)
return false;
free(*slot);
*slot = dup;
return true;
}
static bool store_port(int* slot, const char* value, char* err, size_t err_size) {
char* end;
errno = 0;
long p = strtol(value, &end, 10);
if (errno != 0 || *end != '\0' || *value == '\0' || p <= 0 || p > 65535) {
set_error(err, err_size, "invalid port '%s' (must be 1-65535)", value);
return false;
}
*slot = (int)p;
return true;
}
/* Apply a global scalar key/value. Keys are case-insensitive. Returns false
* (err filled) on an unknown key or an invalid value. */
static bool apply_global_key(DaemonConf* conf, char* key, const char* value, bool replace_hosts,
char* err, size_t err_size) {
if (key_equals(key, "port"))
return store_port(&conf->global.port, value, err, err_size);
if (key_equals(key, "motd file")) {
if (!store_string(&conf->global.motd_file, value)) {
set_error(err, err_size, "out of memory parsing 'motd file'");
return false;
}
return true;
}
if (key_equals(key, "address")) {
if (!store_string(&conf->global.address, value)) {
set_error(err, err_size, "out of memory parsing 'address'");
return false;
}
return true;
}
if (key_equals(key, "max connections"))
return store_max_connections(&conf->global.max_connections, value, NULL, err, err_size);
if (key_equals(key, "max connections per host"))
return store_optional_cap(&conf->global.max_connections_per_host, value,
DAEMON_CONF_MAX_CONCURRENCY_LIMIT, "max connections per host", NULL,
err, err_size);
if (key_equals(key, "auth failure delay"))
return store_auth_failure_delay(&conf->global.auth_failure_delay_ms, value, err, err_size);
if (key_equals(key, "auth lockout threshold"))
return store_optional_cap(&conf->global.auth_lockout_threshold, value,
DAEMON_CONF_MAX_CONCURRENCY_LIMIT, "auth lockout threshold", NULL,
err, err_size);
if (key_equals(key, "auth lockout duration"))
return store_optional_cap(&conf->global.auth_lockout_duration_sec, value,
DAEMON_CONF_MAX_AUTH_LOCKOUT_DURATION_SEC, "auth lockout duration",
NULL, err, err_size);
if (key_equals(key, "hosts allow"))
return store_host_list(&conf->global.hosts_allow, &conf->global.hosts_allow_count, value,
"hosts allow", NULL, replace_hosts, err, err_size);
if (key_equals(key, "hosts deny"))
return store_host_list(&conf->global.hosts_deny, &conf->global.hosts_deny_count, value,
"hosts deny", NULL, replace_hosts, err, err_size);
set_error(err, err_size, "unknown global key '%s'", key);
return false;
}
/* Apply a module key/value to the currently-open module. Returns false (err
* filled) on an unknown module key or an invalid value. */
static bool apply_module_key(DaemonModule* module, char* key, char* value, char* err,
size_t err_size) {
if (key_equals(key, "path")) {
if (*value == '\0') {
set_error(err, err_size, "module '%s': 'path' must not be empty", module->name);
return false;
}
if (!store_string(&module->path, value)) {
set_error(err, err_size, "out of memory parsing 'path' for module '%s'", module->name);
return false;
}
return true;
}
if (key_equals(key, "read only")) {
bool parsed;
if (!parse_bool_value(value, &parsed)) {
set_error(err, err_size,
"module '%s': 'read only' must be yes/no (or true/false/1/0), got '%s'",
module->name, value);
return false;
}
module->read_only = parsed;
return true;
}
if (key_equals(key, "client owner")) {
bool parsed;
if (!parse_bool_value(value, &parsed)) {
set_error(err, err_size,
"module '%s': 'client owner' must be yes/no (or true/false/1/0), got '%s'",
module->name, value);
return false;
}
module->client_owner = parsed;
return true;
}
if (key_equals(key, "auth users")) {
char* list = str_dup(value);
if (!list) {
set_error(err, err_size, "out of memory parsing 'auth users' for module '%s'", module->name);
return false;
}
char* save = NULL;
int added = 0;
for (char* token = strtok_r(list, ",", &save); token; token = strtok_r(NULL, ",", &save)) {
const char* user = trim_ws(token);
if (*user == '\0')
continue;
if (!credentials_username_valid(user)) {
set_error(err, err_size, "module '%s': invalid 'auth users' entry '%s'", module->name,
user);
free(list);
return false;
}
char** grown =
realloc(module->auth_users, (size_t)(module->auth_user_count + 1) * sizeof(char*));
if (!grown) {
free(list);
set_error(err, err_size, "out of memory parsing 'auth users' for module '%s'",
module->name);
return false;
}
module->auth_users = grown;
char* dup = str_dup(user);
if (!dup) {
free(list);
set_error(err, err_size, "out of memory parsing 'auth users' for module '%s'",
module->name);
return false;
}
module->auth_users[module->auth_user_count++] = dup;
added++;
}
free(list);
/* An empty/separator-only value must not silently disable authentication:
* the key's presence is an explicit request for an allow-list. */
if (added == 0) {
set_error(err, err_size, "module '%s': 'auth users' must list at least one user",
module->name);
return false;
}
return true;
}
if (key_equals(key, "max connections"))
return store_optional_cap(&module->max_connections, value, DAEMON_CONF_MAX_CONCURRENCY_LIMIT,
"max connections", module->name, err, err_size);
if (key_equals(key, "hosts allow"))
return store_host_list(&module->hosts_allow, &module->hosts_allow_count, value, "hosts allow",
module->name, false, err, err_size);
if (key_equals(key, "hosts deny"))
return store_host_list(&module->hosts_deny, &module->hosts_deny_count, value, "hosts deny",
module->name, false, err, err_size);
set_error(err, err_size, "unknown key '%s' in module '%s'", key, module->name);
return false;
}
static bool module_open_valid(const DaemonModule* module, char* err, size_t err_size) {
if (module->path == NULL) {
set_error(err, err_size, "module '%s' has no 'path'", module->name);
return false;
}
return true;
}
/* Validate a [section] header line body (text between the brackets) and set
* *name to the module name. Returns false on a malformed header. */
static bool parse_section_name(char* body, const char** name_out, char* err, size_t err_size) {
char* name = trim_ws(body);
if (!daemon_module_name_valid(name)) {
set_error(err, err_size, "invalid module name '%s' (must be 1-%d chars of [A-Za-z0-9._-])",
name, DAEMON_MAX_MODULE_NAME);
return false;
}
*name_out = name;
return true;
}
/* Open (or switch to) a module section. Closes any previously open module
* (validating it has a path) and appends the new one. */
static int open_module(DaemonConf* conf, int* current_module, const char* name, char* err,
size_t err_size) {
if (*current_module >= 0) {
if (!module_open_valid(&conf->modules[*current_module], err, err_size))
return -1;
}
if (daemon_conf_find_module(conf, name)) {
set_error(err, err_size, "duplicate module '%s'", name);
return -1;
}
if (conf->module_count >= DAEMON_CONF_MAX_MODULES) {
set_error(err, err_size, "too many modules (limit %d); module '%s' rejected",
DAEMON_CONF_MAX_MODULES, name);
return -1;
}
DaemonModule* grown =
realloc(conf->modules, (size_t)(conf->module_count + 1) * sizeof(DaemonModule));
if (!grown) {
set_error(err, err_size, "out of memory adding module '%s'", name);
return -1;
}
conf->modules = grown;
memset(&conf->modules[conf->module_count], 0, sizeof(DaemonModule));
conf->modules[conf->module_count].name = str_dup(name);
if (!conf->modules[conf->module_count].name) {
set_error(err, err_size, "out of memory adding module '%s'", name);
return -1;
}
conf->module_count++;
*current_module = conf->module_count - 1;
return 0;
}
/* Split a "key = value" line (value pointer returned in *value, pointing into
* line). Returns false when there is no '='. */
static bool split_key_value(char* line, char** key, char** value) {
char* eq = strchr(line, '=');
if (!eq)
return false;
*eq = '\0';
*key = trim_ws(line);
*value = trim_ws(eq + 1);
return true;
}
/* Strip one layer of surrounding double quotes from a trimmed value. A value
* that starts with '"' but does not end with '"' is an error. */
static bool unquote_value(char* value, char* err, size_t err_size) {
size_t len = strlen(value);
if (len == 0 || value[0] != '"')
return true;
if (len < 2 || value[len - 1] != '"') {
set_error(err, err_size, "unterminated quoted value");
return false;
}
memmove(value, value + 1, len - 2);
value[len - 2] = '\0';
return true;
}
DaemonConf* daemon_conf_load(const char* path, char* err, size_t err_size) {
if (err && err_size)
err[0] = '\0';
if (!path) {
set_error(err, err_size, "no daemon config path");
return NULL;
}
FILE* fp = fopen(path, "r");
if (!fp) {
set_error(err, err_size, "cannot open daemon config '%s': %s", path, strerror(errno));
return NULL;
}
DaemonConf* conf = daemon_conf_create();
if (!conf) {
fclose(fp);
set_error(err, err_size, "out of memory allocating daemon config");
return NULL;
}
int current_module = -1;
int line_no = 0;
char line[DAEMON_CONF_MAX_LINE + 2];
bool ok = true;
while (ok && fgets(line, sizeof(line), fp)) {
line_no++;
size_t len = strlen(line);
if (len == DAEMON_CONF_MAX_LINE + 1 && line[len - 1] != '\n') {
/* The read stopped at the buffer edge without a newline and there is
* more file to come: the line exceeds the bound. */
if (!feof(fp)) {
set_error(err, err_size, "line %d exceeds the %d-byte limit", line_no,
DAEMON_CONF_MAX_LINE);
ok = false;
break;
}
}
if (len > 0 && line[len - 1] == '\n')
line[--len] = '\0';
if (len > 0 && line[len - 1] == '\r')
line[--len] = '\0';
char* cursor = line;
while (*cursor == ' ' || *cursor == '\t')
cursor++;
if (*cursor == '\0' || *cursor == '#' || *cursor == ';')
continue; /* blank or comment line */
if (*cursor == '[') {
char* close = strchr(cursor, ']');
if (!close) {
set_error(err, err_size, "line %d: unterminated module header", line_no);
ok = false;
break;
}
*close = '\0';
char* trailing = close + 1;
const char* rest = trim_ws(trailing);
if (*rest != '\0') {
set_error(err, err_size, "line %d: unexpected text after module header", line_no);
ok = false;
break;
}
const char* name = NULL;
if (!parse_section_name(cursor + 1, &name, err, err_size)) {
ok = false;
break;
}
if (open_module(conf, &current_module, name, err, err_size) != 0) {
ok = false;
break;
}
continue;
}
char* key;
char* value;
if (!split_key_value(cursor, &key, &value)) {
set_error(err, err_size, "line %d: expected 'key = value'", line_no);
ok = false;
break;
}
if (*key == '\0') {
set_error(err, err_size, "line %d: empty key", line_no);
ok = false;
break;
}
if (!unquote_value(value, err, err_size)) {
ok = false;
break;
}
if (current_module >= 0) {
if (!apply_module_key(&conf->modules[current_module], key, value, err, err_size)) {
ok = false;
break;
}
} else {
if (!apply_global_key(conf, key, value, false, err, err_size)) {
ok = false;
break;
}
}
}
if (ok && ferror(fp)) {
set_error(err, err_size, "error reading daemon config '%s': %s", path, strerror(errno));
ok = false;
}
fclose(fp);
if (ok && current_module >= 0 &&
!module_open_valid(&conf->modules[current_module], err, err_size)) {
ok = false;
}
if (!ok) {
daemon_conf_free(conf);
return NULL;
}
return conf;
}
int daemon_conf_apply_dparam(DaemonConf* conf, const char* assignment, char* err, size_t err_size) {
if (err && err_size)
err[0] = '\0';
if (!conf || !assignment || *assignment == '\0') {
set_error(err, err_size, "--dparam requires a KEY=VALUE override");
return -1;
}
char* copy = str_dup(assignment);
if (!copy) {
set_error(err, err_size, "out of memory parsing --dparam");
return -1;
}
char* eq = strchr(copy, '=');
if (!eq) {
free(copy);
set_error(err, err_size, "--dparam '%s' has no '=' (expected KEY=VALUE)", assignment);
return -1;
}
*eq = '\0';
char* key = trim_ws(copy);
const char* value = trim_ws(eq + 1);
if (*key == '\0') {
free(copy);
set_error(err, err_size, "--dparam '%s' has an empty key", assignment);
return -1;
}
if (*value == '\0') {
free(copy);
set_error(err, err_size, "--dparam '%s' has an empty value", assignment);
return -1;
}
bool ok = apply_global_key(conf, key, value, true, err, err_size);
free(copy);
return ok ? 0 : -1;
}
/* Compare the first `prefix` bits of two 16-byte address buffers. */
static bool bit_prefix_match(const uint8_t* a, const uint8_t* b, int prefix) {
int whole = prefix / 8;
if (whole > 0 && memcmp(a, b, (size_t)whole) != 0)
return false;
int remainder = prefix % 8;
if (remainder == 0)
return true;
uint8_t mask = (uint8_t)(0xffu << (8 - remainder));
return (a[whole] & mask) == (b[whole] & mask);
}
/* Case-insensitive glob match used for hostname patterns. Falls back to the
* shared case-sensitive matcher when an operand is too long for the stack
* buffers. */
static bool host_glob_match(const char* pattern, const char* str) {
char pbuf[256];
char sbuf[256];
size_t plen = strlen(pattern);
size_t slen = strlen(str);
if (plen >= sizeof(pbuf) || slen >= sizeof(sbuf))
return glob_match(pattern, str);
for (size_t i = 0; i <= plen; i++)
pbuf[i] = (char)tolower((unsigned char)pattern[i]);
for (size_t i = 0; i <= slen; i++)
sbuf[i] = (char)tolower((unsigned char)str[i]);
return glob_match(pbuf, sbuf);
}
bool daemon_host_pattern_match(const char* pattern, const char* peer_ip) {
if (!pattern || *pattern == '\0' || !peer_ip || *peer_ip == '\0')
return false;
if (strcmp(pattern, "*") == 0)
return true;
if (strchr(pattern, '/')) {
uint8_t pattern_bytes[16];
uint8_t peer_bytes[16];
int prefix = 0;
int family = AF_UNSPEC;
if (!parse_cidr(pattern, &prefix, pattern_bytes, &family))
return false;
if (inet_pton(family, peer_ip, peer_bytes) != 1)
return false;
return bit_prefix_match(pattern_bytes, peer_bytes, prefix);
}
struct in_addr pattern_v4;
struct in_addr peer_v4;
if (inet_pton(AF_INET, pattern, &pattern_v4) == 1)
return inet_pton(AF_INET, peer_ip, &peer_v4) == 1 && pattern_v4.s_addr == peer_v4.s_addr;
struct in6_addr pattern_v6;
struct in6_addr peer_v6;
if (inet_pton(AF_INET6, pattern, &pattern_v6) == 1)
return inet_pton(AF_INET6, peer_ip, &peer_v6) == 1 &&
memcmp(&pattern_v6, &peer_v6, sizeof(pattern_v6)) == 0;
/* Not a literal: a hostname/glob pattern. */
return host_glob_match(pattern, peer_ip);
}
bool daemon_hosts_allowed(const char* peer_ip, char* const* allow, int allow_count,
char* const* deny, int deny_count) {
if (!peer_ip)
return false;
for (int i = 0; i < deny_count; i++) {
if (daemon_host_pattern_match(deny[i], peer_ip))
return false;
}
if (allow_count > 0) {
for (int i = 0; i < allow_count; i++) {
if (daemon_host_pattern_match(allow[i], peer_ip))
return true;
}
return false;
}
return true;
}
bool daemon_hosts_restricted(char* const* allow, int allow_count, char* const* deny,
int deny_count) {
(void)allow;
(void)deny;
return allow_count > 0 || deny_count > 0;
}
+182
View File
@@ -0,0 +1,182 @@
#ifndef DAEMON_CONF_H
#define DAEMON_CONF_H
#include <stdbool.h>
#include <stddef.h>
/* FastSync-native daemon configuration (a FastSync analog of rsyncd.conf).
*
* This is the config the fastsync-server --daemon listener consumes. It is
* line-based with an implicit global section followed by zero or more
* [module] sections. The full grammar is documented in RSYNC_COMPAT.md
* ("Daemon Mode") and summarized below; the parser lives entirely in
* daemon_conf.c so it can be unit tested without any socket code.
*
* The parser is STRICT: an unknown key, a malformed line, a value that does
* not parse, a module without a `path`, or a line longer than
* DAEMON_CONF_MAX_LINE all fail the whole load with a clear, line-numbered
* error instead of being silently ignored. This keeps a typo from silently
* changing what a module serves.
*/
/* A daemon module's configured root is used exactly like the standalone
* server's --destination-root: the daemon confines every connection that
* selects this module to this path (file_open_secure_parent /
* has_path_traversal / path_is_within all keep the existing confinement, just
* per-module). There is never any client-chosen root: a module path always
* stays confined. A daemon REFUSES every client-chosen ownership / super-user
* request by default -- --numeric-ids, --chown, --usermap/--groupmap,
* --fake-super, --copy-as and an explicit --super -- because there is no
* per-module opt-in unless the operator adds one. An operator opts a single
* module in with `client owner = yes` (DaemonModule.client_owner), which allows
* that client to choose ownership within that module's root (the standalone/SSH
* server honors such requests for its single operator-authorized root). The
* operator-level --no-super veto additionally forces super-user activities off
* for every daemon connection, even an opted-in module. See server_module_gate
* in server.c and RSYNC_COMPAT.md.
*
* `auth_users` is honored by Wave B daemon authentication: a module that
* declares auth users accepts a connection only when the presented username is
* on this list AND verifies against the daemon's credential store
* (--password-file / --early-input). An auth-required module with no usable
* store refuses (fail closed) rather than falling open; see server.c. Auth is
* never bypassed by ignoring the list. */
typedef struct DaemonModule {
char* name; /* module name, as the client requests it */
char* path; /* module root (daemon-side authorized root) */
bool read_only; /* `read only = yes/no`; default no */
bool client_owner; /* `client owner = yes/no`; default no. Per-module opt-in
that lets this module's clients choose ownership
(--numeric-ids/--chown/--usermap/--groupmap/--fake-super/
--copy-as) and request explicit --super super-user
activities. Without it the daemon refuses all of them. */
char** auth_users; /* `auth users = a,b`; Wave B credential list */
int auth_user_count;
/* `max connections = N` (optional per-module cap). 0 means unlimited. The
* per-connection child records the selected module in the shared registry
* (daemon_limits.c) once the config frame names it, so the cap is enforced
* across all forked children; the parent reclaims the slot on SIGCHLD. */
int max_connections;
char** hosts_allow; /* `hosts allow = a,b`; host access allow patterns */
int hosts_allow_count;
char** hosts_deny; /* `hosts deny = a,b`; host access deny patterns */
int hosts_deny_count;
} DaemonModule;
/* Global (pre-module) scalar keys. `motd file` is parsed and stored but has
* no wire effect yet (MOTD display is Wave C). */
typedef struct DaemonConfGlobals {
int port; /* `port`, default DAEMON_CONF_DEFAULT_PORT (873) */
char* motd_file; /* `motd file`, may be NULL */
char* address; /* `address` (optional bind address), may be NULL */
int max_connections; /* `max connections`, default
DAEMON_CONF_DEFAULT_MAX_CONNECTIONS (100) */
int auth_failure_delay_ms; /* `auth failure delay`, milliseconds; default
DAEMON_CONF_DEFAULT_AUTH_FAILURE_DELAY_MS */
int max_connections_per_host; /* `max connections per host`, concurrent cap per
source IP; default
DAEMON_CONF_DEFAULT_MAX_CONNECTIONS_PER_HOST (0 =
unlimited) */
int auth_lockout_threshold; /* `auth lockout threshold`, failed attempts from
one source before lockout; default
DAEMON_CONF_DEFAULT_AUTH_LOCKOUT_THRESHOLD (0
disables) */
int auth_lockout_duration_sec; /* `auth lockout duration`, seconds; default
DAEMON_CONF_DEFAULT_AUTH_LOCKOUT_DURATION_SEC
(0 disables) */
char** hosts_allow; /* `hosts allow`; global host access allow patterns */
int hosts_allow_count;
char** hosts_deny; /* `hosts deny`; global host access deny patterns */
int hosts_deny_count;
} DaemonConfGlobals;
typedef struct DaemonConf {
DaemonConfGlobals global;
DaemonModule* modules;
int module_count;
} DaemonConf;
#define DAEMON_CONF_DEFAULT_PORT 873
/* Default global connection cap when `max connections` is absent. Matches the
* historical hardcoded listener value. */
#define DAEMON_CONF_DEFAULT_MAX_CONNECTIONS 100
/* Default `auth failure delay` in milliseconds (0 disables the throttle). */
#define DAEMON_CONF_DEFAULT_AUTH_FAILURE_DELAY_MS 500
/* Default `max connections per host` (0 = unlimited). */
#define DAEMON_CONF_DEFAULT_MAX_CONNECTIONS_PER_HOST 0
/* Default cross-process auth lockout: 10 failed attempts from one source lock
* it out for 300 s (0 disables either knob). */
#define DAEMON_CONF_DEFAULT_AUTH_LOCKOUT_THRESHOLD 10
#define DAEMON_CONF_DEFAULT_AUTH_LOCKOUT_DURATION_SEC 300
/* Upper bound on a `max connections per host` or `auth lockout threshold`
* value, so a typo cannot size the shared registry absurdly. */
#define DAEMON_CONF_MAX_CONCURRENCY_LIMIT 1000000
/* Upper bound on `auth lockout duration` (7 days). */
#define DAEMON_CONF_MAX_AUTH_LOCKOUT_DURATION_SEC 604800
/* Largest accepted `auth failure delay`, so a typo cannot pin a connection
* child in nanosleep for an absurd time. */
/* Bounded well below the socket I/O timeout so a failed-auth child cannot hold
* a connection slot for long enough to amplify connection-cap exhaustion. */
#define DAEMON_CONF_MAX_AUTH_FAILURE_DELAY_MS 5000
/* Upper bound on the number of [module] sections, so the shared registry's
* per-module counter array stays fixed-size. The parser rejects the next
* section past this bound. */
#define DAEMON_CONF_MAX_MODULES 256
/* Longest accepted config line (excluding the trailing newline). Longer lines
* are rejected rather than buffered unboundedly. */
#define DAEMON_CONF_MAX_LINE 4096
/* Upper bound on a module name. Kept far below MAX_STRING_SIZE so a wire
* module name can never exhaust anything by being long. */
#define DAEMON_MAX_MODULE_NAME 200
/* Allocate an empty daemon config with defaulted globals (port 873, no
* modules, no motd/address). Never fails for an allocation failure; callers
* must still NULL-check. */
DaemonConf* daemon_conf_create(void);
/* Parse `path` into a freshly allocated DaemonConf. Returns NULL on any error
* and fills `err` (err_size bytes) with a clear, line-numbered message. The
* returned object is heap-owned; free it with daemon_conf_free. */
DaemonConf* daemon_conf_load(const char* path, char* err, size_t err_size);
void daemon_conf_free(DaemonConf* conf);
/* Case-sensitive exact module lookup by name. Returns the module or NULL.
* Module names are matched exactly (rsync semantics). */
const DaemonModule* daemon_conf_find_module(const DaemonConf* conf, const char* name);
/* Module-name syntax check: non-empty, at most DAEMON_MAX_MODULE_NAME chars,
* and only [A-Za-z0-9._-]. Used by the config parser, the client's
* host::module/path destination parser, and (implicitly) by the daemon lookup
* (a name that fails this can never match a parsed module). */
bool daemon_module_name_valid(const char* name);
/* Parse one --dparam=KEY=VALUE (or "--dparam KEY=VALUE") override string and
* apply it to the global keys only. Keys are case-insensitive and limited to
* the global keys defined by the grammar (port, motd file, address,
* max connections, max connections per host, auth failure delay,
* auth lockout threshold, auth lockout duration, hosts allow, hosts deny).
* Returns 0 on success, -1 on error (err filled). */
int daemon_conf_apply_dparam(DaemonConf* conf, const char* assignment, char* err, size_t err_size);
/* Host access-control matching (pure; no I/O). `daemon_host_pattern_match`
* matches one configured pattern against a numeric peer IP string. Supported
* patterns: `*` (match anything), an IPv4/IPv6 literal, an IPv4/IPv6 CIDR
* (`10.0.0.0/8`, `2001:db8::/32`), or a glob (`*.example.com`) evaluated with
* the same matcher as file globs; a glob only matches a peer string of the
* same shape, so a numeric peer never matches a hostname glob. */
bool daemon_host_pattern_match(const char* pattern, const char* peer_ip);
/* rsync-like combined decision over a deny list and an allow list: a matching
* deny rejects (deny takes precedence); otherwise, when any allow entries
* exist, a peer that matches none is rejected; with no allow entries every
* peer not denied is accepted. An empty/unset pair returns true. */
bool daemon_hosts_allowed(const char* peer_ip, char* const* allow, int allow_count,
char* const* deny, int deny_count);
/* True when at least one allow or deny pattern is configured (i.e. an
* unprovable peer must fail closed rather than being treated as unrestricted). */
bool daemon_hosts_restricted(char* const* allow, int allow_count, char* const* deny,
int deny_count);
#endif
+494
View File
@@ -0,0 +1,494 @@
#include "daemon_limits.h"
#include "daemon_conf.h"
#include "log.h"
#include <arpa/inet.h>
#include <netinet/in.h>
#include <stdatomic.h>
#include <stdint.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include <time.h>
/* The two module-count bounds must agree: the daemon config parser never
* produces more than DAEMON_CONF_MAX_MODULES modules, so the shared registry's
* per-module counter array is sized from the same bound. */
_Static_assert(DAEMON_LIMITS_MAX_MODULES == DAEMON_CONF_MAX_MODULES,
"daemon_limits module bound must match daemon_conf");
/* Slot lifecycle states (stored in slot_state). */
enum {
SLOT_FREE = 0,
SLOT_CLAIMED = 1,
SLOT_REGISTERED = 2,
};
/* The registry header lives at the base of the shared mapping; the pointer
* fields point at the arrays carved out of the same mapping. Absolute pointers
* remain valid in a forked child because fork() clones the address space and
* mapping, so parent and child observe the same virtual addresses. */
struct DaemonLimitRegistry {
int max_slots;
int module_count;
int host_slots; /* power of two; 1 when no per-source tracking is needed */
int per_host_cap;
int lockout_threshold;
int lockout_duration_sec;
size_t map_size;
_Atomic long long host_full_warn; /* last "table full" warning epoch */
_Atomic int* slot_state;
_Atomic int* slot_pid;
_Atomic int* slot_module;
_Atomic int* slot_host; /* per-source table bucket, or -1 */
_Atomic int* module_active;
_Atomic uint64_t* host_key; /* 0 == empty bucket */
_Atomic int* host_active;
_Atomic int* host_fail;
_Atomic long long* host_until; /* epoch seconds the lockout expires */
_Atomic long long* host_last_use; /* epoch seconds the bucket was last touched */
};
static size_t round_up(size_t n, size_t align) {
return (n + align - 1) & ~(align - 1);
}
static size_t next_pow2(size_t n) {
size_t p = 1;
while (p < n)
p <<= 1;
return p;
}
/* Parse a numeric IPv4/IPv6 peer string into family + raw bytes. */
static bool parse_peer_ip(const char* peer_ip, int* family, unsigned char* bytes) {
if (!peer_ip || *peer_ip == '\0')
return false;
struct in_addr v4;
if (inet_pton(AF_INET, peer_ip, &v4) == 1) {
memcpy(bytes, &v4, sizeof(v4));
*family = AF_INET;
return true;
}
struct in6_addr v6;
if (inet_pton(AF_INET6, peer_ip, &v6) == 1) {
memcpy(bytes, &v6, sizeof(v6));
*family = AF_INET6;
return true;
}
return false;
}
uint64_t daemon_limits_host_hash(const char* peer_ip, bool* ok) {
if (ok)
*ok = false;
unsigned char bytes[16];
int family = AF_UNSPEC;
if (!parse_peer_ip(peer_ip, &family, bytes))
return 0;
uint64_t hash = 14695981039346656037ULL ^ (uint64_t)(uint32_t)family;
size_t length = family == AF_INET ? 4 : 16;
for (size_t i = 0; i < length; i++) {
hash ^= bytes[i];
hash *= 1099511628211ULL;
}
if (hash == 0)
hash = 0x9e3779b97f4a7c15ULL;
if (ok)
*ok = true;
return hash;
}
/* True when the registry must maintain per-source buckets: either the per-host
* cap is configured, or the auth lockout is (threshold AND duration > 0). A
* lockout threshold without a duration is a no-op, so it must not size or intern
* the table. create(), register() and the lockout paths all agree on this. */
static bool registry_tracks_hosts(const DaemonLimitRegistry* registry) {
return registry->per_host_cap > 0 ||
(registry->lockout_threshold > 0 && registry->lockout_duration_sec > 0);
}
/* Find the bucket holding `peer_ip`, or -1 when it has no entry. Finding a
* bucket refreshes its last-use time so the eviction policy sees it as live. */
static int host_lookup(DaemonLimitRegistry* registry, const char* peer_ip) {
bool ok = false;
uint64_t key = daemon_limits_host_hash(peer_ip, &ok);
if (!ok)
return -1;
size_t mask = (size_t)registry->host_slots - 1;
size_t start = (size_t)(key & mask);
for (size_t i = 0; i < (size_t)registry->host_slots; i++) {
size_t idx = (start + i) & mask;
uint64_t current = atomic_load_explicit(&registry->host_key[idx], memory_order_acquire);
if (current == key) {
atomic_store_explicit(&registry->host_last_use[idx], (long long)time(NULL),
memory_order_relaxed);
return (int)idx;
}
if (current == 0)
return -1; /* no tombstones: an empty bucket ends the probe chain */
}
return -1;
}
/* A bucket with no live connection may be repurposed: immediately when its
* lockout deadline has already passed (the review's "expired" case), or after an
* idle window when it holds no pending lockout. A bucket with a future lockout
* deadline is retained so the lockout actually lasts its configured duration. */
static bool host_bucket_reclaimable(DaemonLimitRegistry* registry, size_t idx, long long now) {
if (atomic_load_explicit(&registry->host_active[idx], memory_order_relaxed) != 0)
return false;
long long until = atomic_load_explicit(&registry->host_until[idx], memory_order_relaxed);
if (until != 0)
return until <= now;
long long last_use = atomic_load_explicit(&registry->host_last_use[idx], memory_order_relaxed);
/* A bucket whose key is published but whose last_use has not yet been stamped
* (last_use == 0) must be treated as live: reclaiming it here would steal a
* bucket a racing child just claimed. The claim path also stamps last_use
* before publishing the key, so this window cannot persist. */
return last_use != 0 && now - last_use >= DAEMON_LIMITS_HOST_EVICT_IDLE_SEC;
}
/* Emit at most one "per-source table full" warning per
* DAEMON_LIMITS_HOST_FULL_WARN_SEC across all forked children. Called from a
* normal (non-signal) child path, so logging is safe here. */
static void host_warn_table_full(DaemonLimitRegistry* registry, long long now) {
long long last = atomic_load_explicit(&registry->host_full_warn, memory_order_relaxed);
if (last != 0 && now - last < DAEMON_LIMITS_HOST_FULL_WARN_SEC)
return;
if (atomic_compare_exchange_strong_explicit(&registry->host_full_warn, &last, now,
memory_order_relaxed, memory_order_relaxed)) {
log_message(LOG_LEVEL_WARNING,
"daemon: per-source registry is full (%d slots) and no bucket can be reclaimed; "
"'max connections per host' and the auth lockout are temporarily not enforced for "
"new sources (the per-module cap and host ACLs still apply)",
registry->host_slots);
}
}
/* Find or insert the bucket for `peer_ip`. Insertion is a lock-free CAS so two
* forked children racing on the same source converge on one bucket.
*
* When the probe finds no empty bucket it reclaims, via a key CAS, the first
* bucket that is reclaimable (expired lockout or idle, and no active
* connection) and resets its counters. This bounds the table's lifetime so it
* cannot fill permanently and stay fail-open. Returns -1 only when the address
* is unparseable or the table is genuinely full of live/locked buckets
* (callers fail open: the global/module caps and ACLs still apply). */
static int host_intern(DaemonLimitRegistry* registry, const char* peer_ip) {
bool ok = false;
uint64_t key = daemon_limits_host_hash(peer_ip, &ok);
if (!ok)
return -1;
long long now = (long long)time(NULL);
size_t mask = (size_t)registry->host_slots - 1;
size_t start = (size_t)(key & mask);
/* A couple of passes bound the work: the first normally claims/seeds a bucket;
* a lost eviction CAS retries once against the freshly observed table. */
for (int pass = 0; pass < 2; pass++) {
int evict = -1;
uint64_t evict_key = 0;
for (size_t i = 0; i < (size_t)registry->host_slots; i++) {
size_t idx = (start + i) & mask;
uint64_t current = atomic_load_explicit(&registry->host_key[idx], memory_order_acquire);
if (current == key) {
atomic_store_explicit(&registry->host_last_use[idx], now, memory_order_relaxed);
return (int)idx;
}
if (current == 0) {
/* Stamp last_use *before* publishing the key so a reclaimer racing the
* claim can never observe a claimed bucket with last_use == 0 and
* evict it. A pre-stamp is harmless if the CAS loses: the bucket is
* either still empty (never inspected for reclaim) or has just been
* taken by another source that wants a fresh timestamp anyway. */
atomic_store_explicit(&registry->host_last_use[idx], now, memory_order_relaxed);
uint64_t expected = 0;
if (atomic_compare_exchange_strong_explicit(&registry->host_key[idx], &expected, key,
memory_order_acq_rel, memory_order_acquire)) {
return (int)idx;
}
if (atomic_load_explicit(&registry->host_key[idx], memory_order_acquire) == key) {
return (int)idx;
}
continue; /* another child won this empty bucket; keep probing */
}
if (evict < 0 && host_bucket_reclaimable(registry, idx, now)) {
evict = (int)idx;
evict_key = current;
}
}
if (evict >= 0) {
/* Refresh the timestamp before the key changes hands so the reused bucket
* is not seen as immediately idle by a racing reclaimer. */
atomic_store_explicit(&registry->host_last_use[evict], now, memory_order_relaxed);
uint64_t expected = evict_key;
if (atomic_compare_exchange_strong_explicit(&registry->host_key[evict], &expected, key,
memory_order_acq_rel, memory_order_acquire)) {
/* The bucket now belongs to the new source; clear the evicted source's
* stale lockout/failure state. */
atomic_store_explicit(&registry->host_active[evict], 0, memory_order_relaxed);
atomic_store_explicit(&registry->host_fail[evict], 0, memory_order_relaxed);
atomic_store_explicit(&registry->host_until[evict], 0, memory_order_relaxed);
/* Two children can race to intern the same brand-new key into different
* eviction targets, leaving the table with duplicate buckets for `key`.
* Re-scan for the first (canonical) bucket holding `key`; when it
* precedes `evict`, drop our duplicate's occupancy and hand back the
* canonical bucket so per-source counts are not orphaned on the
* duplicate. The duplicate keeps its key, so no tombstone hole is
* created and probe chains stay intact; it ages out normally. */
for (size_t i = 0; i < (size_t)registry->host_slots; i++) {
size_t candidate = (start + i) & mask;
uint64_t found =
atomic_load_explicit(&registry->host_key[candidate], memory_order_acquire);
if (found == key) {
if (candidate != (size_t)evict) {
atomic_store_explicit(&registry->host_active[evict], 0, memory_order_relaxed);
return (int)candidate;
}
break;
}
if (found == 0)
break; /* the key is present at `evict`, so this cannot happen first */
}
return evict;
}
continue; /* lost the race; re-probe with fresh observations */
}
break; /* no free and no reclaimable bucket: genuinely full */
}
host_warn_table_full(registry, now);
return -1;
}
DaemonLimitRegistry* daemon_limits_create(int max_slots, int module_count, int per_host_cap,
int lockout_threshold, int lockout_duration_sec) {
if (max_slots < DAEMON_LIMITS_MIN_SLOTS)
max_slots = DAEMON_LIMITS_MIN_SLOTS;
if (max_slots > DAEMON_LIMITS_MAX_SLOTS)
max_slots = DAEMON_LIMITS_MAX_SLOTS;
if (module_count < 1)
module_count = 1;
if (module_count > DAEMON_LIMITS_MAX_MODULES)
module_count = DAEMON_LIMITS_MAX_MODULES;
if (per_host_cap < 0)
per_host_cap = 0;
if (lockout_threshold < 0)
lockout_threshold = 0;
if (lockout_duration_sec < 0)
lockout_duration_sec = 0;
bool need_hosts = per_host_cap > 0 || (lockout_threshold > 0 && lockout_duration_sec > 0);
int host_slots = 1;
if (need_hosts) {
size_t want = (size_t)max_slots * 4;
if (want < 64)
want = 64;
if (want > DAEMON_LIMITS_MAX_HOST_SLOTS)
want = DAEMON_LIMITS_MAX_HOST_SLOTS;
host_slots = (int)next_pow2(want);
}
size_t header = round_up(sizeof(DaemonLimitRegistry), 16);
size_t slot_bytes =
round_up((size_t)max_slots * sizeof(_Atomic int), 16) * 4; /* state,pid,module,host */
size_t module_bytes = round_up((size_t)module_count * sizeof(_Atomic int), 16);
size_t host_key_bytes = round_up((size_t)host_slots * sizeof(_Atomic uint64_t), 16);
size_t host_int_bytes = round_up((size_t)host_slots * sizeof(_Atomic int), 16) * 2;
size_t host_until_bytes = round_up((size_t)host_slots * sizeof(_Atomic long long), 16) * 2;
size_t total =
header + slot_bytes + module_bytes + host_key_bytes + host_int_bytes + host_until_bytes + 16;
void* map = mmap(NULL, total, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0);
if (map == MAP_FAILED)
return NULL;
memset(map, 0, total);
DaemonLimitRegistry* registry = (DaemonLimitRegistry*)map;
registry->max_slots = max_slots;
registry->module_count = module_count;
registry->host_slots = host_slots;
registry->per_host_cap = per_host_cap;
registry->lockout_threshold = lockout_threshold;
registry->lockout_duration_sec = lockout_duration_sec;
registry->map_size = total;
unsigned char* cursor = (unsigned char*)map + header;
registry->slot_state = (atomic_int*)cursor;
cursor += (size_t)max_slots * sizeof(_Atomic int);
registry->slot_pid = (atomic_int*)cursor;
cursor += (size_t)max_slots * sizeof(_Atomic int);
registry->slot_module = (atomic_int*)cursor;
cursor += (size_t)max_slots * sizeof(_Atomic int);
registry->slot_host = (atomic_int*)cursor;
cursor += (size_t)max_slots * sizeof(_Atomic int);
registry->module_active = (atomic_int*)cursor;
cursor += (size_t)module_count * sizeof(_Atomic int);
cursor = (unsigned char*)round_up((size_t)(uintptr_t)cursor, 16);
registry->host_key = (_Atomic uint64_t*)cursor;
cursor += (size_t)host_slots * sizeof(_Atomic uint64_t);
registry->host_active = (atomic_int*)cursor;
cursor += (size_t)host_slots * sizeof(_Atomic int);
registry->host_fail = (atomic_int*)cursor;
cursor += (size_t)host_slots * sizeof(_Atomic int);
cursor = (unsigned char*)round_up((size_t)(uintptr_t)cursor, 16);
registry->host_until = (atomic_llong*)cursor;
cursor += (size_t)host_slots * sizeof(_Atomic long long);
registry->host_last_use = (atomic_llong*)cursor;
for (int i = 0; i < max_slots; i++) {
atomic_store(&registry->slot_module[i], -1);
atomic_store(&registry->slot_host[i], -1);
}
return registry;
}
void daemon_limits_destroy(DaemonLimitRegistry* registry) {
if (!registry)
return;
munmap(registry, registry->map_size);
}
int daemon_limits_claim_slot(DaemonLimitRegistry* registry) {
if (!registry)
return DAEMON_LIMITS_NO_SLOT;
for (int i = 0; i < registry->max_slots; i++) {
int expected = SLOT_FREE;
if (atomic_compare_exchange_strong(&registry->slot_state[i], &expected, SLOT_CLAIMED)) {
atomic_store(&registry->slot_pid[i], 0);
atomic_store(&registry->slot_module[i], -1);
atomic_store(&registry->slot_host[i], -1);
return i;
}
}
return DAEMON_LIMITS_NO_SLOT;
}
void daemon_limits_set_slot_pid(DaemonLimitRegistry* registry, int slot, long pid) {
if (!registry || slot < 0 || slot >= registry->max_slots)
return;
atomic_store(&registry->slot_pid[slot], (int)pid);
}
void daemon_limits_reclaim_slot(DaemonLimitRegistry* registry, int slot) {
if (!registry || slot < 0 || slot >= registry->max_slots)
return;
atomic_exchange_explicit(&registry->slot_state[slot], SLOT_FREE, memory_order_acq_rel);
atomic_store_explicit(&registry->slot_pid[slot], 0, memory_order_relaxed);
/* The module/host occupancy arrays are derived from the slot table; do not
* decrement here or a SIGKILL between a child's increment and its REGISTERED
* publish would leak a count. Callers that need the derived counts call
* daemon_limits_recompute. */
}
void daemon_limits_reclaim_pid(DaemonLimitRegistry* registry, long pid) {
if (!registry || pid <= 0)
return;
for (int i = 0; i < registry->max_slots; i++) {
if (atomic_load(&registry->slot_state[i]) == SLOT_FREE)
continue;
if (atomic_load(&registry->slot_pid[i]) == (int)pid) {
daemon_limits_reclaim_slot(registry, i);
return;
}
}
}
void daemon_limits_recompute(DaemonLimitRegistry* registry) {
if (!registry)
return;
/* Zero the derived arrays, then re-derive solely from the REGISTERED slots.
* A child that was SIGKILLed after incrementing a counter but before
* publishing REGISTERED is not counted, and its leaked increment is erased by
* the zeroing, so the leak cannot persist. */
for (int m = 0; m < registry->module_count; m++)
atomic_store_explicit(&registry->module_active[m], 0, memory_order_relaxed);
for (int h = 0; h < registry->host_slots; h++)
atomic_store_explicit(&registry->host_active[h], 0, memory_order_relaxed);
for (int i = 0; i < registry->max_slots; i++) {
if (atomic_load_explicit(&registry->slot_state[i], memory_order_acquire) != SLOT_REGISTERED)
continue;
int module = atomic_load_explicit(&registry->slot_module[i], memory_order_relaxed);
if (module >= 0 && module < registry->module_count)
atomic_fetch_add_explicit(&registry->module_active[module], 1, memory_order_relaxed);
int host = atomic_load_explicit(&registry->slot_host[i], memory_order_relaxed);
if (host >= 0 && host < registry->host_slots)
atomic_fetch_add_explicit(&registry->host_active[host], 1, memory_order_relaxed);
}
}
DaemonLimitResult daemon_limits_register(DaemonLimitRegistry* registry, int slot, int module_index,
const char* peer_ip, int module_cap) {
if (!registry || slot < 0 || slot >= registry->max_slots)
return DAEMON_LIMIT_UNAVAILABLE;
if (module_index < 0 || module_index >= registry->module_count)
return DAEMON_LIMIT_UNAVAILABLE;
if (atomic_load_explicit(&registry->slot_state[slot], memory_order_acquire) != SLOT_CLAIMED)
return DAEMON_LIMIT_UNAVAILABLE;
int host = -1;
if (registry_tracks_hosts(registry))
host = host_intern(registry, peer_ip);
int module_count = atomic_fetch_add(&registry->module_active[module_index], 1) + 1;
if (module_cap > 0 && module_count > module_cap) {
atomic_fetch_sub(&registry->module_active[module_index], 1);
return DAEMON_LIMIT_MODULE_FULL;
}
if (host >= 0) {
int host_count = atomic_fetch_add(&registry->host_active[host], 1) + 1;
if (registry->per_host_cap > 0 && host_count > registry->per_host_cap) {
atomic_fetch_sub(&registry->host_active[host], 1);
atomic_fetch_sub(&registry->module_active[module_index], 1);
return DAEMON_LIMIT_HOST_FULL;
}
}
atomic_store(&registry->slot_module[slot], module_index);
atomic_store(&registry->slot_host[slot], host);
atomic_store_explicit(&registry->slot_state[slot], SLOT_REGISTERED, memory_order_release);
return DAEMON_LIMIT_OK;
}
bool daemon_limits_auth_locked(DaemonLimitRegistry* registry, const char* peer_ip,
int* seconds_remaining) {
if (!registry || registry->lockout_threshold <= 0 || registry->lockout_duration_sec <= 0)
return false;
int bucket = host_lookup(registry, peer_ip);
if (bucket < 0)
return false;
long long until = atomic_load(&registry->host_until[bucket]);
long long now = (long long)time(NULL);
if (until > now) {
if (seconds_remaining)
*seconds_remaining = (int)(until - now);
return true;
}
if (until != 0) {
/* The previous lockout has expired: clear the stale counter so the source
* gets a fresh allowance. */
atomic_store(&registry->host_fail[bucket], 0);
atomic_store(&registry->host_until[bucket], 0);
}
return false;
}
void daemon_limits_auth_record_failure(DaemonLimitRegistry* registry, const char* peer_ip) {
if (!registry || registry->lockout_threshold <= 0 || registry->lockout_duration_sec <= 0)
return;
int bucket = host_intern(registry, peer_ip);
if (bucket < 0)
return;
int failures = atomic_fetch_add(&registry->host_fail[bucket], 1) + 1;
if (failures >= registry->lockout_threshold) {
long long now = (long long)time(NULL);
atomic_store(&registry->host_until[bucket], now + (long long)registry->lockout_duration_sec);
}
}
void daemon_limits_auth_record_success(DaemonLimitRegistry* registry, const char* peer_ip) {
if (!registry)
return;
int bucket = host_lookup(registry, peer_ip);
if (bucket < 0)
return;
atomic_store(&registry->host_fail[bucket], 0);
atomic_store(&registry->host_until[bucket], 0);
}
+147
View File
@@ -0,0 +1,147 @@
#ifndef DAEMON_LIMITS_H
#define DAEMON_LIMITS_H
#include <stdbool.h>
#include <stddef.h>
#include <stdint.h>
/* Cross-process daemon connection registry.
*
* The daemon listener forks ONE child per accepted connection, so any
* per-module / per-source accounting must live in state shared across the
* forked children. This module owns a fixed-size registry carved out of an
* anonymous shared mapping (mmap(MAP_SHARED | MAP_ANONYMOUS)) created by the
* accept-loop PARENT before it forks; every child inherits the mapping (and the
* pointer to it) across fork().
*
* Rules:
* - ONLY C11 atomics (atomic_*); never mtx_t/pthread locks, which can deadlock
* in a forked child if another thread held them at fork time.
* - No heap allocation after fork: the mapping is fixed-size and all access is
* atomic load/store/CAS over preallocated arrays.
*
* Slot lifecycle (the parent reclaims even when a child is SIGKILLed):
* FREE --(parent claim_slot)--> CLAIMED
* CLAIMED --(child register)--> REGISTERED
* any --(parent reclaim)--> FREE
* The child records its module index and per-source bucket into the slot before
* publishing REGISTERED; the parent's SIGCHLD handler matches the reaped pid to
* the slot and, when REGISTERED, decrements the module/per-source counters.
* A child killed before registering holds no counts, so reclaiming a CLAIMED
* slot only frees the slot.
*
* Per-source identity is the normalized numeric peer IP (IPv4-mapped IPv6 is
* already collapsed to IPv4 by utils_fd_peer_ip); it is interned into an
* open-addressed, linear-probing table keyed by a 64-bit hash. The same table
* also carries the cross-process auth-failure counter and lockout deadline.
*
* Per-source table lifetime: a bucket's key is never cleared back to empty (that
* would break every later probe chain that passed through it). Instead the
* table has a bounded-lifetime eviction policy: when no empty bucket exists, the
* first bucket that is reclaimable -- no active connection AND (its lockout
* deadline has passed OR it has been idle for
* DAEMON_LIMITS_HOST_EVICT_IDLE_SEC) -- is atomically repurposed for the new
* source via a CAS of its key, and its counters are reset. The table therefore
* cannot fill permanently, and a full table degrades to fail-open for the
* per-source cap/lockout of new sources (the per-module cap and host ACLs still
* apply) instead of staying fail-open forever. A rate-limited warning is logged
* on the fail-open path. The eviction race with a concurrent
* registration/reclaim on the same bucket is benign: it can at worst lose one
* source's counter (fail-open), never corrupt memory or the module caps.
*/
typedef struct DaemonLimitRegistry DaemonLimitRegistry;
/* Result of a per-connection admission check. */
typedef enum {
DAEMON_LIMIT_OK = 0, /* admitted; slot is now REGISTERED */
DAEMON_LIMIT_MODULE_FULL, /* module's `max connections` cap reached */
DAEMON_LIMIT_HOST_FULL, /* global `max connections per host` cap reached */
DAEMON_LIMIT_UNAVAILABLE, /* registry/slot unusable (caller fails open) */
} DaemonLimitResult;
/* Bounds for registry sizing. A slot is one concurrently live child. */
#define DAEMON_LIMITS_MIN_SLOTS 16
#define DAEMON_LIMITS_MAX_SLOTS 65536
#define DAEMON_LIMITS_MAX_HOST_SLOTS 65536
#define DAEMON_LIMITS_NO_SLOT (-1)
/* Upper bound on `module_count`, matching daemon_conf.h's DAEMON_CONF_MAX_MODULES
* (asserted in daemon_limits.c) so a caller can never size the per-module counter
* array larger than the config parser can produce. */
#define DAEMON_LIMITS_MAX_MODULES 256
/* Per-source table lifetime: a bucket with no active connection and no pending
* lockout is reclaimable once it has been idle this long, so a flood of distinct
* sources cannot pin the table full forever. A bucket whose lockout deadline
* has passed is reclaimable immediately (independent of this idle window). */
#define DAEMON_LIMITS_HOST_EVICT_IDLE_SEC 300
/* Minimum spacing between "per-source table is full" warnings, so a table-full
* attack cannot flood the log. */
#define DAEMON_LIMITS_HOST_FULL_WARN_SEC 60
/* Create the shared registry in the calling (parent) process. `max_slots` is
* the number of concurrently live children to track (clamped to
* [DAEMON_LIMITS_MIN_SLOTS, DAEMON_LIMITS_MAX_SLOTS]); `module_count` is the
* number of daemon modules (clamped to
* [1, DAEMON_LIMITS_MAX_MODULES]); `per_host_cap` and the lockout pair come
* from the daemon config (0 disables). Returns NULL on failure (e.g. mmap
* allocation); callers must degrade gracefully (global cap + ACLs still
* apply). */
DaemonLimitRegistry* daemon_limits_create(int max_slots, int module_count, int per_host_cap,
int lockout_threshold, int lockout_duration_sec);
/* Unmap the registry. Only the creating process may call this. */
void daemon_limits_destroy(DaemonLimitRegistry* registry);
/* Parent side: reserve a slot for the next fork. Returns the slot index or
* DAEMON_LIMITS_NO_SLOT when every slot is in use. */
int daemon_limits_claim_slot(DaemonLimitRegistry* registry);
/* Parent side: record the forked child's pid in a claimed slot. */
void daemon_limits_set_slot_pid(DaemonLimitRegistry* registry, int slot, long pid);
/* Parent side: release a slot. The slot becomes FREE; the module/per-source
* occupancy arrays are DERIVED state and are only refreshed by
* daemon_limits_recompute, which callers must invoke afterwards when they rely
* on the derived counts (the SIGCHLD handler batches one recompute for the whole
* reap). Idempotent. */
void daemon_limits_reclaim_slot(DaemonLimitRegistry* registry, int slot);
/* Parent SIGCHLD side: release the slot owned by `pid` (no-op when not found).
* Like reclaim_slot this does not touch the derived occupancy arrays; call
* daemon_limits_recompute after a batch of releases. */
void daemon_limits_reclaim_pid(DaemonLimitRegistry* registry, long pid);
/* Parent side (async-signal-safe; atomics only, no malloc/log): rebuild
* module_active[] / host_active[] from scratch by scanning the REGISTERED slots.
* The slot table is the single source of truth, so this self-heals any
* count leaked by a child that was SIGKILLed mid-registration (it zeroes the
* arrays and re-derives them). Bounded by max_slots + host_slots. A
* registration racing this call can be transiently undercounted until the next
* recompute, which can only relax a cap briefly -- never corrupt memory. */
void daemon_limits_recompute(DaemonLimitRegistry* registry);
/* Child side: admit the connection for `module_index` from `peer_ip`. Always
* tracks the module/per-source occupancy (so the parent's reclaim is
* symmetric); when `module_cap` > 0 it additionally enforces the per-module
* cap. A NULL/empty or non-numeric `peer_ip` skips the per-source track (the
* callers use that to exempt a trusted loopback peer from the per-host cap; the
* per-module cap still applies). Returns DAEMON_LIMIT_OK and publishes the
* slot, or a refusal reason. */
DaemonLimitResult daemon_limits_register(DaemonLimitRegistry* registry, int slot, int module_index,
const char* peer_ip, int module_cap);
/* Child side: true when `peer_ip` is currently locked out after too many failed
* authentications. `seconds_remaining` may be NULL. */
bool daemon_limits_auth_locked(DaemonLimitRegistry* registry, const char* peer_ip,
int* seconds_remaining);
/* Child side: count one failed authentication for `peer_ip`; once the threshold
* is reached the source is locked out for the configured duration. */
void daemon_limits_auth_record_failure(DaemonLimitRegistry* registry, const char* peer_ip);
/* Child side: clear the failure counter/lockout for a source that authenticated
* successfully (no-op when the source has no table entry). */
void daemon_limits_auth_record_success(DaemonLimitRegistry* registry, const char* peer_ip);
/* Pure helper: 64-bit FNV-1a hash of a numeric peer IP plus its family, used to
* index the per-source table. *ok is set false (and 0 returned) for a NULL or
* non-numeric address. Exposed for unit testing. */
uint64_t daemon_limits_host_hash(const char* peer_ip, bool* ok);
#endif
+8 -2
View File
@@ -23,6 +23,7 @@ Data* data_create_reserve(size_t size) {
d->data = NULL;
d->size = size;
d->protocol_charge = 0;
d->owner = NULL;
return d;
}
@@ -36,14 +37,19 @@ Data* data_create(void* data, size_t data_size) {
new_data->data = data;
new_data->size = data_size;
new_data->protocol_charge = 0;
new_data->owner = NULL;
return new_data;
}
void data_destroy(Data* data) {
if (data == NULL)
return;
if (data->protocol_charge != 0)
protocol_release_memory(data->protocol_charge);
if (data->protocol_charge != 0) {
if (data->owner != NULL)
protocol_release_memory_for_session(data->owner, data->protocol_charge);
else
protocol_release_memory(data->protocol_charge);
}
free(data->data);
free(data);
}
+18
View File
@@ -3,11 +3,25 @@
#include <stdlib.h>
/* Forward declaration for the connection budget a received Data is charged
* against; defined in protocol.h (which includes this header). */
typedef struct ProtocolSession ProtocolSession;
typedef struct {
void* data;
size_t size;
/* Non-zero only for a buffer charged to the protocol connection budget. */
size_t protocol_charge;
/* Session whose budget `protocol_charge` was reserved from. When non-NULL,
* the charge is returned to this session directly, regardless of which
* session (if any) is bound to the destroying thread. owner is not
* guaranteed to be set whenever protocol_charge is non-zero: it is NULL for
* uncharged Data and for Data that has no recorded owner, in which case any
* charge falls back to the session bound at destroy time.
*
* Lifetime contract: a Data with a non-NULL owner must not outlive that
* ProtocolSession -- data_destroy dereferences owner to return the charge. */
ProtocolSession* owner;
} Data;
Data* data_create_empty(size_t data_size);
@@ -15,5 +29,9 @@ Data* data_create_reserve(size_t size);
Data* data_create(void* data, size_t data_size);
void data_destroy(Data* data);
void protocol_release_memory(size_t charge);
/* Release `charge` against `session` directly instead of the thread-local bound
* session. Used by data_destroy to honor Data.owner; `session` must outlive
* the Data whose charge is being returned. A NULL session is a no-op. */
void protocol_release_memory_for_session(ProtocolSession* session, size_t charge);
#endif
+3 -2
View File
@@ -18,8 +18,9 @@ typedef struct {
/* Receiver-side --delay-updates staging registry. All successfully written
files land under a private staging directory inside the receive root and are
atomically renamed into their final destination only at the very end of the
transfer. A single PipelineContextReceiver has exactly one writer thread,
but the registry is still mutex-protected so the same object can be safely
transfer. A single receiver pipeline (see src/server/receiver_pipeline.h)
has exactly one writer thread, but the registry is still mutex-protected so
the same object can be safely
shared with the publish/cleanup phase that runs after the threads join. */
typedef struct DelayUpdatesContext {
char* root_directory; /* receive root the staging dir lives under */
+19 -5
View File
@@ -29,12 +29,21 @@ uint32_t delta_xxhash32(const void* data, uint32_t len) {
return XXH32(data, len, 0);
}
uint32_t delta_xxhash32_seeded(const void* data, uint32_t len, uint32_t seed) {
return XXH32(data, len, seed);
}
uint64_t delta_xxhash64(const void* data, size_t len) {
return XXH64(data, len, 0);
}
DeltaSignature* delta_signature_create(const void* old_file_data, uint64_t old_file_size,
uint32_t block_size) {
return delta_signature_create_seeded(old_file_data, old_file_size, block_size, 0);
}
DeltaSignature* delta_signature_create_seeded(const void* old_file_data, uint64_t old_file_size,
uint32_t block_size, uint32_t seed) {
if (old_file_data == NULL || old_file_size == 0 || block_size == 0)
return NULL;
@@ -67,7 +76,7 @@ DeltaSignature* delta_signature_create(const void* old_file_data, uint64_t old_f
uint32_t len =
(uint32_t)((old_file_size - offset < block_size) ? (old_file_size - offset) : block_size);
sig->blocks[i].adler32 = delta_adler32(data + offset, len);
sig->blocks[i].xxhash = delta_xxhash32(data + offset, len);
sig->blocks[i].xxhash = delta_xxhash32_seeded(data + offset, len, seed);
}
return sig;
@@ -297,7 +306,7 @@ static uint32_t* delta_build_index(const DeltaSignature* sig, uint32_t bucket_co
* O(1) per window); otherwise an exact linear scan is used. */
static uint32_t delta_find_match(const uint8_t* window, uint32_t window_len, uint32_t adler,
bool full_window, const DeltaSignature* sig, const uint32_t* heads,
const uint32_t* next, uint32_t mask) {
const uint32_t* next, uint32_t mask, uint32_t seed) {
if (!full_window || sig->block_count == 0)
return DELTA_NO_BLOCK;
@@ -309,7 +318,7 @@ static uint32_t delta_find_match(const uint8_t* window, uint32_t window_len, uin
if (sig->blocks[j].adler32 != adler)
continue;
if (!have_xxh) {
window_xxh = delta_xxhash32(window, window_len);
window_xxh = delta_xxhash32_seeded(window, window_len, seed);
have_xxh = true;
}
if (window_xxh == sig->blocks[j].xxhash)
@@ -321,7 +330,7 @@ static uint32_t delta_find_match(const uint8_t* window, uint32_t window_len, uin
/* Fallback used when the index could not be allocated. */
for (uint32_t j = 0; j < sig->block_count; j++) {
if (sig->blocks[j].adler32 == adler) {
uint32_t window_xxh = delta_xxhash32(window, window_len);
uint32_t window_xxh = delta_xxhash32_seeded(window, window_len, seed);
if (window_xxh == sig->blocks[j].xxhash)
return j;
}
@@ -331,6 +340,11 @@ static uint32_t delta_find_match(const uint8_t* window, uint32_t window_len, uin
Delta* delta_compute(const void* new_file_data, uint64_t new_file_size, const DeltaSignature* sig,
uint32_t block_size) {
return delta_compute_seeded(new_file_data, new_file_size, sig, block_size, 0);
}
Delta* delta_compute_seeded(const void* new_file_data, uint64_t new_file_size,
const DeltaSignature* sig, uint32_t block_size, uint32_t seed) {
if (!new_file_data || !sig || !sig->blocks || new_file_size == 0 || block_size == 0 ||
block_size > DELTA_BLOCK_SIZE_MAX || sig->block_size != block_size)
return NULL;
@@ -397,7 +411,7 @@ Delta* delta_compute(const void* new_file_data, uint64_t new_file_size, const De
bool matched = false;
uint32_t match_block = delta_find_match(new_data + i, window_len, adler, full_window, sig,
index, chain_next, mask);
index, chain_next, mask, seed);
if (match_block != DELTA_NO_BLOCK) {
if (has_literal) {
if (!flush_literal(&instrs, &capacity, &count, new_data, literal_start, i)) {
+11
View File
@@ -56,12 +56,22 @@ typedef struct {
DeltaSignature* delta_signature_create(const void* old_file_data, uint64_t old_file_size,
uint32_t block_size);
/* Seeded equivalent of delta_signature_create: the per-block strong (xxHash32)
* checksum uses `seed` (the low 32 bits of --checksum-seed). Passing seed 0 is
* identical to the unseeded function. */
DeltaSignature* delta_signature_create_seeded(const void* old_file_data, uint64_t old_file_size,
uint32_t block_size, uint32_t seed);
Data* delta_signature_serialize(const DeltaSignature* sig);
DeltaSignature* delta_signature_deserialize(const Data* data);
void delta_signature_destroy(DeltaSignature* sig);
Delta* delta_compute(const void* new_file_data, uint64_t new_file_size, const DeltaSignature* sig,
uint32_t block_size);
/* Seeded equivalent of delta_compute: the per-window strong (xxHash32) check
* uses `seed` (the low 32 bits of --checksum-seed). The receiver's signature
* must have been built with the same seed for matching. */
Delta* delta_compute_seeded(const void* new_file_data, uint64_t new_file_size,
const DeltaSignature* sig, uint32_t block_size, uint32_t seed);
Data* delta_serialize(const Delta* delta);
Delta* delta_deserialize(const Data* data);
void* delta_apply(const void* old_data, uint64_t old_size, const Delta* delta, uint32_t block_size);
@@ -72,6 +82,7 @@ bool delta_is_worthwhile(const Delta* delta, uint64_t new_file_size);
uint32_t delta_adler32(const void* data, uint32_t len);
uint32_t delta_xxhash32(const void* data, uint32_t len);
uint32_t delta_xxhash32_seeded(const void* data, uint32_t len, uint32_t seed);
uint64_t delta_xxhash64(const void* data, size_t len);
#endif
+609 -101
View File
@@ -1,3 +1,6 @@
#ifndef _GNU_SOURCE
#define _GNU_SOURCE /* statx + STATX_BTIME for --crtimes birth-time capture */
#endif
#include <errno.h>
#include <dirent.h>
#include <fcntl.h>
@@ -13,10 +16,13 @@
#include "data.h"
#include "delta.h"
#include "file.h"
#include "file_store.h"
#include "identity.h"
#include "log.h"
#include "metadata.h"
#include "utils.h"
#include "protocol.h"
#include "xattr.h"
static bool write_all(int fd, const void* data, unsigned long long size) {
const unsigned char* p = data;
@@ -32,6 +38,29 @@ static bool write_all(int fd, const void* data, unsigned long long size) {
return true;
}
/* Preallocate `size` bytes on `fd` before any data is written (--preallocate).
* posix_fallocate reserves real disk blocks, so an out-of-space condition
* (ENOSPC/EDQUOT) surfaces up front instead of partway through a transfer;
* unavoidable fragmentation of a streamed file is also reduced. Some
* filesystems (e.g. tmpfs, ZFS) do not support it and return EOPNOTSUPP/ENOSYS,
* where we fall back to ftruncate, which still extends the logical size so the
* fail-fast/contiguity intent degrades gracefully but never fails. Genuine
* allocation failures are propagated as the error code (caller fails the write).
* posix_fallocate leaves the fd's file offset unchanged, so the subsequent
* write_all at offset 0 is unaffected. Returns 0 on success (including the
* fallback) or a nonzero error code. */
static int preallocate_fd(int fd, unsigned long long size) {
if (size == 0)
return 0;
int rc = posix_fallocate(fd, 0, (off_t)size);
if (rc == EOPNOTSUPP || rc == ENOSYS) {
if (ftruncate(fd, (off_t)size) == 0)
return 0;
return errno;
}
return rc;
}
/* Process-wide counter for scratch temp names. A --temp-dir scratch directory
is flat: different destinations that share a basename must never race onto
the same temp name. Deriving the trailing number from a global atomic
@@ -43,17 +72,17 @@ static unsigned long long next_temp_sequence(void) {
return atomic_fetch_add_explicit(&sequence, 1, memory_order_relaxed);
}
bool file_checksum(File* file, uint64_t* checksum) {
if (!file || !checksum || !file->data)
bool file_checksum(File* file, ChecksumAlgo algo, uint64_t seed, uint8_t* out, size_t out_capacity,
size_t* out_len) {
if (!file || !out || !out_len || !file->data)
return false;
if (file->data->size == 0) {
*checksum = delta_xxhash64("", 0);
return true;
return checksum_digest(algo, seed, "", 0, out, out_capacity, out_len);
}
if (!file->data->data && !file_load_data(file))
return false;
*checksum = delta_xxhash64(file->data->data, file->data->size);
return true;
return checksum_digest(algo, seed, file->data->data, file->data->size, out, out_capacity,
out_len);
}
File* file_create(const char* path) {
@@ -84,7 +113,17 @@ File* file_create(const char* path) {
file->metadata = NULL;
file->skip = false;
file->is_dir = false;
file->dir_time_only = false;
file->basis_link = NULL;
file->link_group = 0;
file->link_first = false;
file->hardlink_target = NULL;
file->is_symlink = false;
file->symlink_target = NULL;
file->is_special = false;
file->rdev_major = 0;
file->rdev_minor = 0;
file->xattrs = NULL;
return file;
}
@@ -102,10 +141,17 @@ void file_destroy(void* item) {
file->send_path = NULL;
free(file->basis_link);
file->basis_link = NULL;
free(file->hardlink_target);
file->hardlink_target = NULL;
free(file->symlink_target);
file->symlink_target = NULL;
xattr_list_free(file->xattrs);
file->xattrs = NULL;
free(file);
}
FileMetadata* file_metadata_create(const struct stat* stats) {
FileMetadata* file_metadata_create(const char* path, const struct stat* stats, bool capture_atime,
bool capture_crtime) {
FileMetadata* m = protocol_alloc(sizeof(FileMetadata));
if (m == NULL) {
log_perror("ERROR: Could not allocate memory for file metadata");
@@ -120,6 +166,37 @@ FileMetadata* file_metadata_create(const struct stat* stats) {
#else
m->mtime_nsec = 0;
#endif
/* -U/--atimes: capture the access time from the same pre-read stat the
scanner already took, so the value is not clobbered by a later read for
transfer. The timestamp is populated (and atime_valid set) only on Linux,
where st_atim is populated; on other platforms the atime is left alone
rather than clobbered to the default 0/epoch by an unpopulated value. */
#ifdef __linux__
m->atime_valid = capture_atime;
m->atime_sec = stats->st_atim.tv_sec;
m->atime_nsec = stats->st_atim.tv_nsec;
#else
m->atime_valid = false;
m->atime_sec = 0;
m->atime_nsec = 0;
#endif
/* -N/--crtimes: birth time is not available via struct stat in general; on
Linux it needs statx STATX_BTIME. If unavailable it is captured as a
documented no-op (the flag stays accepted, crtime_valid stays false). */
m->crtime_valid = false;
m->crtime_sec = 0;
m->crtime_nsec = 0;
if (capture_crtime) {
#ifdef STATX_BTIME
struct statx stx;
if (path != NULL && statx(AT_FDCWD, path, AT_STATX_SYNC_AS_STAT, STATX_BTIME, &stx) == 0 &&
(stx.stx_mask & STATX_BTIME) != 0) {
m->crtime_valid = true;
m->crtime_sec = (time_t)stx.stx_btime.tv_sec;
m->crtime_nsec = (long)stx.stx_btime.tv_nsec;
}
#endif
}
return m;
}
@@ -127,6 +204,37 @@ void file_metadata_destroy(void* metadata) {
free(metadata);
}
/* --open-noatime: process-wide sender policy (client-only, never crosses the
* wire). When enabled, opening a source file for transfer uses O_NOATIME so
* the read does not bump the source's on-disk access time. It degrades safely
* to a normal open where O_NOATIME is unavailable (not defined) or refused
* (EPERM, because it needs CAP_FOWNER): the data path never silently changes,
* only the atime-bump is skipped. */
static bool file_open_noatime = false;
void file_set_open_noatime(bool enable) {
file_open_noatime = enable;
}
bool file_get_open_noatime(void) {
return file_open_noatime;
}
/* Open `path` read-only for transfer, honouring --open-noatime when set. */
int file_open_for_read(const char* path) {
int flags = O_RDONLY;
#ifdef O_NOATIME
if (file_get_open_noatime())
flags |= O_NOATIME;
#endif
int fd = open(path, flags);
#ifdef O_NOATIME
if (fd < 0 && (flags & O_NOATIME))
fd = open(path, O_RDONLY); /* degrade safely on EPERM / unsupported fs */
#endif
return fd;
}
bool file_load_data(File* file) {
if (file == NULL || !file->data)
return false;
@@ -153,8 +261,14 @@ bool file_load_data(File* file) {
size_t file_content_to_buffer(File* file) {
if (!file || !file->path || !file->data || (!file->data->data && file->data->size != 0))
return 0;
FILE* file_pointer = fopen(file->path, "rb");
int fd = file_open_for_read(file->path);
if (fd < 0) {
log_perror("Could not open the file!");
return 0;
}
FILE* file_pointer = fdopen(fd, "rb");
if (file_pointer == NULL) {
close(fd);
log_perror("Could not open the file!");
return 0;
}
@@ -170,28 +284,6 @@ size_t file_content_to_buffer(File* file) {
/* ---- Secure filesystem primitives ---- */
static int authorized_root_fd = -1;
static char* authorized_root_path;
static bool path_is_within_root(const char* root, const char* path) {
size_t root_len = strlen(root);
return strncmp(root, path, root_len) == 0 && (path[root_len] == '\0' || path[root_len] == '/');
}
bool file_set_authorized_root(int fd, const char* canonical_path) {
char* path_copy = canonical_path ? str_dup(canonical_path) : NULL;
if (canonical_path && !path_copy) {
authorized_root_fd = -1;
free(authorized_root_path);
authorized_root_path = NULL;
return false;
}
authorized_root_fd = fd;
free(authorized_root_path);
authorized_root_path = path_copy;
return true;
}
bool file_path_exists_secure(const char* path) {
if (!path)
return false;
@@ -236,6 +328,177 @@ bool file_destination_is_newer_secure(const char* path, const FileMetadata* meta
return file_stat_secure(path, &st) && stat_is_newer(&st, metadata);
}
/* --keep-dirlinks (-K) receiver process-wide policy: when set, a destination
* path component that is itself a symlink to an in-root directory is followed
* (used as that directory) instead of failing the O_NOFOLLOW walk. Only ever
* honoured when the resolved target is a directory that stays beneath the
* authorized root, so a malicious symlink can never redirect the write outside
* it. Client of record is the server's receiver. */
static bool file_keep_dirlinks = false;
void file_set_keep_dirlinks(bool enable) {
file_keep_dirlinks = enable;
}
/* --trust-sender (Phase 5) receiver process-wide policy: when set, the receiver
* trusts the sender's file list and skips its own redundant up-front re-
* validation (empty/".." path rejection, escaping-symlink-target containment).
* Kept OFF by default; the server's per-connection handler sets it once from the
* received config before any receiver/writer threads start (each connection is
* its own forked process, so this per-process value never bleeds across
* connections). */
static bool file_trust_sender = false;
void file_set_trust_sender(bool enable) {
file_trust_sender = enable;
}
bool file_get_trust_sender(void) {
return file_trust_sender;
}
/* True when `target` is a lexical symlink target that can never escape the
* receive root once created beneath it: relative (not absolute) and containing
* no ".." path component. Used by --munge-links' sender-side containment: an
* escaping target is never transmitted (the entry is skipped/contained). */
bool file_symlink_target_contained(const char* target) {
if (!target || target[0] == '\0' || target[0] == '/')
return false;
const char* p = target;
while (*p) {
const char* slash = strchr(p, '/');
size_t comp_len = slash ? (size_t)(slash - p) : strlen(p);
if (comp_len == 2 && p[0] == '.' && p[1] == '.')
return false;
if (!slash)
break;
p = slash + 1;
}
return true;
}
/* Remove a leading symlink munge marker (if present); returns true when the
* marker was stripped. `target` is a mutable NUL-terminated buffer. */
bool file_symlink_unmunge(char* target) {
if (!target)
return false;
static const char* const marker = SYMLINK_MUNGE_PREFIX;
size_t marker_len = strlen(marker);
if (strncmp(target, marker, marker_len) != 0)
return false;
size_t rest = strlen(target + marker_len) + 1;
memmove(target, target + marker_len, rest);
return true;
}
/* Owned copy of `target` prefixed with SYMLINK_MUNGE_PREFIX (the sender-side
* --munge-links rewriting). Returns NULL on allocation failure. */
char* file_symlink_munge(const char* target) {
if (!target)
return NULL;
static const char* const marker = SYMLINK_MUNGE_PREFIX;
size_t marker_len = strlen(marker);
size_t target_len = strlen(target);
char* out = malloc(marker_len + target_len + 1);
if (!out)
return NULL;
memcpy(out, marker, marker_len);
memcpy(out + marker_len, target, target_len + 1);
return out;
}
/* Create a symlink at `path` pointing to `target`, confined below the
* authorized root: the parent directory is opened with an O_NOFOLLOW fd walk
* and the link is created with symlinkat so neither the destination chain nor
* the target is ever followed. The final component is never dereferenced: an
* existing non-directory entry at `path` is unlinked by name before the link is
* placed; an existing directory there is left untouched (returns false, so a
* caller can treat it as a collision). As a receiver-side trust-boundary
* invariant, `target` must be file_symlink_target_contained() (relative and
* ".."-free): an absolute or escaping target is rejected outright (returns
* false) so a malicious sender can never materialize a symlink that points
* outside the receive root. */
bool file_symlink_at_secure(const char* path, const char* target) {
/* The link itself (`path`) is always kept below the authorized root. The
TARGET may point anywhere: normally only a contained (relative, ".."-free)
target is permitted so a malicious sender can never plant a symlink that
later dereferences outside the root. Under --trust-sender that target
containment check is relaxed (the receiver trusts the sender and copies the
link verbatim, matching rsync -l), but path/leaf confinement is never
disabled, so the link still cannot be placed outside the tree. */
if (!path || !target || has_path_traversal(path))
return false;
if (!file_trust_sender && !file_symlink_target_contained(target))
return false;
char* leaf = NULL;
int parent_fd = file_open_secure_parent(path, &leaf, true);
if (parent_fd < 0)
return false;
bool ok = false;
struct stat st;
bool exists = fstatat(parent_fd, leaf, &st, AT_SYMLINK_NOFOLLOW) == 0;
if (exists && S_ISDIR(st.st_mode)) {
/* A directory already at this path cannot be replaced atomically with a
symlink without --force semantics; leave it and report the collision. */
ok = false;
} else {
if (exists && unlinkat(parent_fd, leaf, 0) != 0 && errno != ENOENT)
goto out;
ok = symlinkat(target, parent_fd, leaf) == 0;
}
out:
close(parent_fd);
free(leaf);
return ok;
}
/* Open the directory named by canonical absolute `resolved`, which the caller
* has already verified lies beneath `root` (the canonical authorized root).
* Each component is opened relative to the authorized-root fd with O_NOFOLLOW,
* so a directory swapped for a symlink after the realpath() check cannot
* redirect the open outside the root -- the walk simply fails. This replaces
* re-opening the absolute resolved path (TOCTOU). Returns an O_DIRECTORY fd,
* or -1 (the root itself and any error are refused). */
static int open_dir_beneath_root(const char* resolved, const char* root) {
size_t root_len = strlen(root);
const char* rel = resolved + root_len;
while (*rel == '/')
rel++;
if (*rel == '\0')
return -1;
int root_fd = utils_get_authorized_root_fd();
if (root_fd < 0)
return -1;
int fd = dup(root_fd);
if (fd < 0)
return -1;
char* copy = str_dup(rel);
if (!copy) {
close(fd);
return -1;
}
char* save = NULL;
for (char* component = strtok_r(copy, "/", &save); component;
component = strtok_r(NULL, "/", &save)) {
if (strcmp(component, ".") == 0)
continue;
/* A canonical realpath() output never contains "." or ".."; refuse ".."
defensively rather than let it climb toward the root. */
int next = strcmp(component, "..") == 0
? -1
: openat(fd, component, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
if (next < 0) {
close(fd);
free(copy);
return -1;
}
close(fd);
fd = next;
}
free(copy);
return fd;
}
int file_open_secure_parent(const char* path, char** leaf_out, bool create_dirs) {
char* copy = str_dup(path);
if (!copy)
@@ -248,20 +511,21 @@ int file_open_secure_parent(const char* path, char** leaf_out, bool create_dirs)
return -1;
}
int fd;
if (authorized_root_fd >= 0) {
if (!authorized_root_path || path[0] != '/' ||
!path_is_within_root(authorized_root_path, path)) {
int root_fd = utils_get_authorized_root_fd();
const char* root_path = utils_get_authorized_root_path();
if (root_fd >= 0) {
if (!root_path || path[0] != '/' || !path_is_within_root(root_path, path)) {
free(copy);
free(leaf);
return -1;
}
fd = dup(authorized_root_fd);
fd = dup(root_fd);
if (fd < 0) {
free(copy);
free(leaf);
return -1;
}
size_t root_len = strlen(authorized_root_path);
size_t root_len = strlen(root_path);
char* relative = str_dup(path + root_len);
if (!relative) {
free(copy);
@@ -283,6 +547,7 @@ int file_open_secure_parent(const char* path, char** leaf_out, bool create_dirs)
}
char* save = NULL;
char* component = strtok_r(parent, "/", &save);
char rel_buf[PATH_MAX] = "";
while (component) {
if (strcmp(component, "..") == 0) {
close(fd);
@@ -292,9 +557,63 @@ int file_open_secure_parent(const char* path, char** leaf_out, bool create_dirs)
}
if (strcmp(component, ".") != 0) {
int next = openat(fd, component, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
if (create_dirs && next < 0 && errno == ENOENT) {
if (mkdirat(fd, component, 0755) == 0 || errno == EEXIST)
if (next < 0 && create_dirs && errno == ENOENT) {
bool created = mkdirat(fd, component, 0755) == 0;
if (created || errno == EEXIST) {
/* P7 Wave E: --copy-as owns EVERY entry, including the intermediate
directories this walk creates implicitly. Its target ids are a
global policy, so they are available here without per-entry source
metadata. Only a directory this walk actually created is chowned
(a pre-existing destination directory is left alone, matching
rsync's transferred-entry scope); the helper is a no-op unless an
identity policy is active. */
if (created && identity_copy_as_active() &&
!identity_apply_ownership_link(fd, component, 0, 0)) {
/* A REQUIRED --copy-as ownership that cannot be applied to a
directory this walk just created must fail the entry rather than
leave that implicit parent owned by the receiver. Preserve the
failing errno across the cleanup so the caller logs the real
reason. */
int saved_errno = errno;
close(fd);
free(copy);
free(leaf);
errno = saved_errno;
return -1;
}
next = openat(fd, component, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
}
}
/* --keep-dirlinks (-K): a path component that is an existing symlink to
an in-root directory is used as THAT directory rather than failing the
O_NOFOLLOW walk. Only honoured when the symlink resolves to a
directory that stays beneath the authorized root, so a malicious link
can never redirect the write outside it. */
if (next < 0 && file_keep_dirlinks && root_path != NULL &&
(errno == ELOOP || errno == ENOTDIR || errno == EACCES)) {
struct stat lst;
if (fstatat(fd, component, &lst, AT_SYMLINK_NOFOLLOW) == 0 && S_ISLNK(lst.st_mode)) {
char candidate[PATH_MAX];
char root[PATH_MAX];
if (realpath(root_path, root) && snprintf(candidate, sizeof(candidate), "%s%s/%s", root,
rel_buf, component) < (int)sizeof(candidate)) {
char resolved[PATH_MAX];
if (realpath(candidate, resolved) && strcmp(resolved, root) != 0 &&
strncmp(root, resolved, strlen(root)) == 0 &&
(resolved[strlen(root)] == '/' || resolved[strlen(root)] == '\0')) {
struct stat rst;
if (stat(resolved, &rst) == 0 && S_ISDIR(rst.st_mode)) {
/* Open the resolved directory through a relative no-follow walk
from the authorized-root fd instead of re-opening the
absolute `resolved` path: swapping an intermediate directory
for a symlink between realpath() and open() (TOCTOU) then
merely fails the walk rather than redirecting the fd outside
the root. */
next = open_dir_beneath_root(resolved, root);
}
}
}
}
}
if (next < 0) {
close(fd);
@@ -304,6 +623,20 @@ int file_open_secure_parent(const char* path, char** leaf_out, bool create_dirs)
}
close(fd);
fd = next;
/* Track the walked relative prefix so the -K candidate path can be
reconstructed. An overflow while building it means the whole path is
at the PATH_MAX edge, so fail hard rather than silently building a
wrong (truncated) candidate for a later -K follow. */
size_t need = strlen(rel_buf) + strlen(component) + 2;
if (need <= sizeof(rel_buf)) {
strcat(rel_buf, "/");
strcat(rel_buf, component);
} else if (file_keep_dirlinks) {
close(fd);
free(copy);
free(leaf);
return -1;
}
}
component = strtok_r(NULL, "/", &save);
}
@@ -338,8 +671,9 @@ bool file_ensure_directory_secure(const char* path) {
return false;
/* The authorized root is already an open directory, and the filesystem root
is always present: there is no final component left to create for them. */
const char* root_path = utils_get_authorized_root_path();
bool root_is_open =
authorized_root_fd >= 0 && authorized_root_path && strcmp(norm, authorized_root_path) == 0;
utils_get_authorized_root_fd() >= 0 && root_path && strcmp(norm, root_path) == 0;
if (root_is_open || strcmp(norm, "/") == 0) {
free(norm);
return true;
@@ -351,11 +685,23 @@ bool file_ensure_directory_secure(const char* path) {
return false;
int dir_fd = openat(parent_fd, leaf, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
bool created = false;
if (dir_fd < 0 && errno == ENOENT) {
if (mkdirat(parent_fd, leaf, 0755) == 0 || errno == EEXIST)
if (mkdirat(parent_fd, leaf, 0755) == 0) {
created = true;
dir_fd = openat(parent_fd, leaf, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
} else if (errno == EEXIST) {
dir_fd = openat(parent_fd, leaf, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
}
}
bool ok = dir_fd >= 0;
/* --copy-as owns a directory this call just created (the final component;
intermediate components were handled by file_open_secure_parent above). A
failed REQUIRED ownership fails the call rather than leaving the directory
owned by the receiver. */
if (ok && created && identity_copy_as_active() &&
!identity_apply_ownership_link(parent_fd, leaf, 0, 0))
ok = false;
if (dir_fd >= 0)
close(dir_fd);
close(parent_fd);
@@ -374,8 +720,9 @@ bool file_directory_exists_secure(const char* path) {
char* norm = normalize_directory_path(path);
if (!norm)
return false;
const char* root_path = utils_get_authorized_root_path();
bool root_is_open =
authorized_root_fd >= 0 && authorized_root_path && strcmp(norm, authorized_root_path) == 0;
utils_get_authorized_root_fd() >= 0 && root_path && strcmp(norm, root_path) == 0;
if (root_is_open || strcmp(norm, "/") == 0) {
free(norm);
return true;
@@ -508,11 +855,32 @@ int file_open_private_dir(const char* dir_path) {
return fd;
}
/* After the content and mode/times are restored on the just-written file, apply
* the per-file xattrs (-X/-A) and, for --fake-super, park the source's
* uid/gid/mode/mtime in the reserved xattr. All fd-relative (confined to the
* destination file) and best-effort: a per-attribute or privilege failure is
* logged and skipped, never fatal. */
static void restore_extra_fd(int fd, const FileMetadata* metadata, const FileXattrList* xattrs,
bool fake_super) {
xattr_apply_fd(fd, xattrs);
if (fake_super && metadata) {
fake_super_store_fd(fd, (uint32_t)metadata->uid, (uint32_t)metadata->gid,
(uint32_t)metadata->mode, metadata->mtime_sec, metadata->mtime_nsec);
/* Replay: re-apply the recorded uid/gid/mode/mtime fd-relative so a save
under --fake-super restores the attrs (when privileged) instead of only
recording them. Best-effort; fake_super_restore_fd silently skips a
non-root fchown EPERM/EACCES and never fatal. */
fake_super_restore_fd(fd);
}
}
static bool file_to_disk_secure_impl(const char* path, const void* data,
unsigned long long data_size, bool inplace, bool sparse,
const FileMetadata* metadata, bool preserve_executability,
bool update, bool no_replace, bool use_fsync,
const char* temp_dir) {
bool preallocate, const FileMetadata* metadata,
bool preserve_executability, bool update, bool no_replace,
bool use_fsync, const char* temp_dir,
const FileXattrList* xattrs, bool fake_super,
bool keep_partial) {
char* leaf = NULL;
int dirfd = file_open_secure_parent(path, &leaf, true);
if (dirfd < 0)
@@ -522,43 +890,90 @@ static bool file_to_disk_secure_impl(const char* path, const void* data,
if (inplace) {
/* --inplace writes directly into the destination; a scratch --temp-dir
does not apply and must never redirect these writes. */
fd = openat(dirfd, leaf, O_WRONLY | O_CREAT | O_CLOEXEC | O_NOFOLLOW, 0644);
/* Type gate BEFORE opening: an existing destination entry that is not a
regular file (FIFO, socket, char/block device, directory) must never be
opened for writing. Opening a FIFO would block the receive thread
forever and writing into a device would bypass the --write-devices /
super-mode gate (a client-controlled device write). fstatat with
AT_SYMLINK_NOFOLLOW does not follow a symlink and does not block. */
struct stat pre_stat;
if (fstatat(dirfd, leaf, &pre_stat, AT_SYMLINK_NOFOLLOW) == 0 && !S_ISREG(pre_stat.st_mode)) {
close(dirfd);
free(leaf);
return false;
}
/* O_NONBLOCK: a no-op for a regular file, but a raced-in FIFO cannot block
the open before the post-open S_ISREG re-check rejects it. */
fd = openat(dirfd, leaf, O_WRONLY | O_CREAT | O_CLOEXEC | O_NOFOLLOW | O_NONBLOCK, 0644);
if (fd >= 0) {
struct stat destination_stat;
/* Re-check the opened descriptor: a concurrent replacement between the
fstatat probe and the open (or a device/FIFO raced in) must never be
written through. */
if (fstat(fd, &destination_stat) != 0 || !S_ISREG(destination_stat.st_mode)) {
close(fd);
close(dirfd);
free(leaf);
return false;
}
bool newer = false;
if (update && metadata && fstat(fd, &destination_stat) == 0 &&
S_ISREG(destination_stat.st_mode)) {
newer = stat_is_newer(&destination_stat, metadata);
if (update && metadata && stat_is_newer(&destination_stat, metadata)) {
newer = true;
}
if (newer) {
ok = true;
} else {
/* In-place overwrites: pre-size sparse targets and always trim the
file to the new payload length afterwards so shorter payloads can
never leave stale trailing bytes from a previous version. */
if (sparse && data_size > 0)
ok = ftruncate(fd, (off_t)data_size) == 0;
if (ok || !sparse || data_size == 0)
ok = write_all(fd, data, data_size);
if (ok)
ok = ftruncate(fd, (off_t)data_size) == 0;
/* Normalize the mode: apply the metadata-derived safe mode when the
sender supplied metadata (setuid/setgid/sticky are never honored);
otherwise fall back to a safe default so dangerous bits on an
existing destination cannot survive an overwrite. */
if (ok) {
if (metadata)
ok = file_restore_metadata_fd(fd, metadata, preserve_executability);
else if (fchmod(fd, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH) != 0)
ok = false;
/* Preallocate the expected payload size before writing so an
out-of-space condition fails cleanly up front (--preallocate).
--sparse takes precedence: posix_fallocate would allocate every
block, defeating the holes the sparse writer would create, so the
two never combine here (the ftruncate presize below stays). */
int prealloc_rc = 0;
if (preallocate && !sparse && data_size > 0) {
prealloc_rc = preallocate_fd(fd, data_size);
if (prealloc_rc != 0) {
char* escaped_path = output_escape(path, log_get_8_bit_output());
log_message(LOG_LEVEL_ERROR, "preallocate failed for '%s' (%s); transfer aborted",
escaped_path ? escaped_path : "<allocation failed>", strerror(prealloc_rc));
free(escaped_path);
}
}
if (prealloc_rc == 0) {
/* posix_fallocate does not guarantee the fd's file offset is left
unchanged, so seek back to 0 before the data write. */
lseek(fd, 0, SEEK_SET);
if (sparse && data_size > 0)
ok = ftruncate(fd, (off_t)data_size) == 0;
if (ok || !sparse || data_size == 0)
ok = sparse && data_size > 0
? file_store_write_sparse(fd, (const unsigned char*)data, data_size)
: write_all(fd, data, data_size);
if (ok)
ok = ftruncate(fd, (off_t)data_size) == 0;
/* Normalize the mode: apply the metadata-derived safe mode when the
sender supplied metadata (setuid/setgid/sticky are never honored);
otherwise fall back to a safe default so dangerous bits on an
existing destination cannot survive an overwrite. */
if (ok) {
if (metadata)
ok = file_restore_metadata_fd(fd, metadata, preserve_executability);
else if (fchmod(fd, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH) != 0)
ok = false;
}
if (ok)
restore_extra_fd(fd, metadata, xattrs, fake_super);
if (ok && use_fsync)
ok = fsync(fd) == 0;
}
if (ok && use_fsync)
ok = fsync(fd) == 0;
}
}
} else {
/* The --update newer-destination check runs first so a skipped file never
creates an empty scratch directory behind it. */
/* True once the temp is being written: distinguishes a mid-write/metadata/
install failure (partial data may exist, --partial may retain it) from a
pre-write validation failure (nothing to retain). */
bool write_attempted = false;
if (update && metadata) {
/* This check protects the normal atomic path as far as possible. A
concurrent replacement can still occur before the final rename. */
@@ -623,14 +1038,36 @@ static bool file_to_disk_secure_impl(const char* path, const void* data,
O_WRONLY | O_CREAT | O_EXCL | O_CLOEXEC | O_NOFOLLOW, 0600);
if (fd < 0)
continue; /* EEXIST (or a transient open error): try a fresh name. */
if (sparse && data_size > 0)
ok = ftruncate(fd, (off_t)data_size) == 0;
if (ok || (!sparse || data_size == 0))
ok = write_all(fd, data, data_size);
if (ok && metadata)
ok = file_restore_metadata_fd(fd, metadata, preserve_executability);
if (ok && use_fsync)
ok = fsync(fd) == 0;
int prealloc_rc = 0;
if (preallocate && !sparse && data_size > 0) {
prealloc_rc = preallocate_fd(fd, data_size);
if (prealloc_rc != 0) {
char* escaped_path = output_escape(path, log_get_8_bit_output());
log_message(LOG_LEVEL_ERROR, "preallocate failed for '%s' (%s); transfer aborted",
escaped_path ? escaped_path : "<allocation failed>", strerror(prealloc_rc));
free(escaped_path);
}
}
if (prealloc_rc == 0) {
lseek(fd, 0, SEEK_SET);
if (sparse && data_size > 0)
ok = ftruncate(fd, (off_t)data_size) == 0;
/* A real write attempt begins here (the ftruncate presize succeeded or
no presize applies): a later mid-write / metadata / fsync / install
failure may leave partial data that --partial retention can rename. */
if (ok || (!sparse || data_size == 0)) {
write_attempted = true;
ok = sparse && data_size > 0
? file_store_write_sparse(fd, (const unsigned char*)data, data_size)
: write_all(fd, data, data_size);
}
if (ok && metadata)
ok = file_restore_metadata_fd(fd, metadata, preserve_executability);
if (ok)
restore_extra_fd(fd, metadata, xattrs, fake_super);
if (ok && use_fsync)
ok = fsync(fd) == 0;
}
if (close(fd) != 0)
ok = false;
fd = -1;
@@ -658,8 +1095,21 @@ static bool file_to_disk_secure_impl(const char* path, const void* data,
ok = false;
}
}
if (!ok)
unlinkat(scratch_dirfd >= 0 ? scratch_dirfd : dirfd, tmp, 0);
if (!ok) {
/* --partial retention (best-effort): on a failure that happened after
the temp held data (mid-write / metadata / fsync / install error),
keep the already-written temp at the final destination path instead
of unlinking it, so a later --append / --append-verify run can resume.
This only ever renames the already-written temp (never a corrupt
blend); the rename can fail (cross-device, permissions) and we then
fall through to the normal unlink cleanup. Never retains when
keep_partial is off, when nothing was actually written, or under
--ignore-existing/--existing (no_replace), where the destination is
not ours to overwrite. */
if (!keep_partial || !write_attempted || no_replace ||
renameat(scratch_dirfd >= 0 ? scratch_dirfd : dirfd, tmp, dirfd, leaf) != 0)
unlinkat(scratch_dirfd >= 0 ? scratch_dirfd : dirfd, tmp, 0);
}
/* Once the temp fd was created the outcome is permanent: a write,
metadata, fsync, close, linkat or renameat failure will not be fixed
by retrying under a fresh name, so stop here. Only the open-failure
@@ -678,33 +1128,55 @@ static bool file_to_disk_secure_impl(const char* path, const void* data,
}
bool file_to_disk_secure(const char* path, const void* data, unsigned long long data_size,
bool inplace, bool sparse, const FileMetadata* metadata,
bool inplace, bool sparse, bool preallocate, const FileMetadata* metadata,
bool preserve_executability, const char* temp_dir) {
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, metadata,
preserve_executability, false, false, false, temp_dir);
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
preserve_executability, false, false, false, temp_dir, NULL,
false, false);
}
bool file_to_disk_secure_update(const char* path, const void* data, unsigned long long data_size,
bool inplace, bool sparse, const FileMetadata* metadata,
bool preserve_executability, const char* temp_dir) {
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, metadata,
preserve_executability, true, false, false, temp_dir);
bool inplace, bool sparse, bool preallocate,
const FileMetadata* metadata, bool preserve_executability,
const char* temp_dir) {
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
preserve_executability, true, false, false, temp_dir, NULL, false,
false);
}
bool file_to_disk_secure_with_fsync(const char* path, const void* data,
unsigned long long data_size, bool inplace, bool sparse,
const FileMetadata* metadata, bool preserve_executability,
bool use_fsync, const char* temp_dir) {
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, metadata,
preserve_executability, false, false, use_fsync, temp_dir);
bool preallocate, const FileMetadata* metadata,
bool preserve_executability, bool use_fsync,
const char* temp_dir) {
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
preserve_executability, false, false, use_fsync, temp_dir, NULL,
false, false);
}
bool file_to_disk_secure_no_replace(const char* path, const void* data,
unsigned long long data_size, bool sparse,
unsigned long long data_size, bool sparse, bool preallocate,
const FileMetadata* metadata, bool preserve_executability,
const char* temp_dir) {
return file_to_disk_secure_impl(path, data, data_size, false, sparse, metadata,
preserve_executability, false, true, false, temp_dir);
return file_to_disk_secure_impl(path, data, data_size, false, sparse, preallocate, metadata,
preserve_executability, false, true, false, temp_dir, NULL, false,
false);
}
/* Receiver write-path variant that also applies the per-file xattrs (-X/-A)
* and, under --fake-super, parks the source stat in the reserved xattr, on the
* just-written file descriptor before the final rename. `no_replace` / `update`
* mirror the plain wrappers; `keep_partial` enables --partial retention of a
* failed write's temp. See file_to_disk_secure_impl for the semantics. */
bool file_to_disk_secure_attrs(const char* path, const void* data, unsigned long long data_size,
bool inplace, bool sparse, bool preallocate,
const FileMetadata* metadata, bool preserve_executability,
bool update, bool no_replace, bool use_fsync,
const FileXattrList* xattrs, bool fake_super, bool keep_partial,
const char* temp_dir) {
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, preallocate, metadata,
preserve_executability, update, no_replace, use_fsync, temp_dir,
xattrs, fake_super, keep_partial);
}
/* Atomic --link-dest install. The destination is replaced (via a temporary
@@ -716,9 +1188,18 @@ bool file_to_disk_secure_no_replace(const char* path, const void* data,
* fallback; a successful hard link keeps the basis inode's own attributes
* (applying metadata through the shared inode would mutate the basis file).
* Returns false only when both the link and the copy fallback fail. */
bool file_to_disk_secure_link(const char* path, const char* basis_path, const void* data,
unsigned long long data_size, const FileMetadata* metadata,
bool preserve_executability, bool use_fsync, const char* temp_dir) {
/* --link-dest / -H hardlink install with a byte-copy fallback. `metadata` is
* applied only on the copy fallback; a successful hard link keeps the basis
* inode's own attributes (applying through the shared inode would mutate the
* basis). Likewise `xattrs`/`fake_super` are applied only on the copy
* fallback, so a fallback copy preserves the per-file attributes instead of
* silently dropping them. */
static bool file_to_disk_secure_link_impl(const char* path, const char* basis_path,
const void* data, unsigned long long data_size,
bool preallocate, const FileMetadata* metadata,
bool preserve_executability, bool use_fsync,
const FileXattrList* xattrs, bool fake_super,
const char* temp_dir) {
if (!path || !basis_path)
return false;
char* leaf = NULL;
@@ -767,11 +1248,19 @@ bool file_to_disk_secure_link(const char* path, const char* basis_path, const vo
if (linked) {
int target_dirfd = scratch_dirfd >= 0 ? scratch_dirfd : dirfd;
if (use_fsync) {
int tfd = openat(target_dirfd, tmp, O_RDONLY | O_NOFOLLOW | O_CLOEXEC);
if (tfd < 0 || fsync(tfd) != 0) {
/* O_NONBLOCK: the freshly linked temp is normally the basis's regular
file, but a raced-in FIFO at the name must not block this reopen
forever. With O_NONBLOCK such an open fails with ENXIO instead of
blocking, which is treated as a benign fsync-skip (the link itself
is still installed); any other open/fsync failure falls back to the
byte-copy path as before. */
int tfd = openat(target_dirfd, tmp, O_RDONLY | O_NOFOLLOW | O_CLOEXEC | O_NONBLOCK);
if (tfd < 0) {
if (errno != ENXIO)
linked = false;
} else if (fsync(tfd) != 0) {
linked = false;
if (tfd >= 0)
close(tfd);
close(tfd);
} else {
close(tfd);
}
@@ -796,8 +1285,9 @@ bool file_to_disk_secure_link(const char* path, const char* basis_path, const vo
free(leaf);
/* The basis file could not be linked in (missing, cross-device, refused
by the filesystem). Write a byte-identical local copy instead. */
return file_to_disk_secure_with_fsync(path, data, data_size, false, false, metadata,
preserve_executability, use_fsync, temp_dir);
return file_to_disk_secure_attrs(path, data, data_size, false, false, preallocate, metadata,
preserve_executability, false, false, use_fsync, xattrs,
fake_super, false, temp_dir);
}
if (scratch_dirfd >= 0)
@@ -807,9 +1297,27 @@ bool file_to_disk_secure_link(const char* path, const char* basis_path, const vo
return true;
}
bool file_to_disk_secure_link(const char* path, const char* basis_path, const void* data,
unsigned long long data_size, bool preallocate,
const FileMetadata* metadata, bool preserve_executability,
bool use_fsync, const char* temp_dir) {
return file_to_disk_secure_link_impl(path, basis_path, data, data_size, preallocate, metadata,
preserve_executability, use_fsync, NULL, false, temp_dir);
}
bool file_to_disk_secure_link_attrs(const char* path, const char* basis_path, const void* data,
unsigned long long data_size, bool preallocate,
const FileMetadata* metadata, bool preserve_executability,
bool use_fsync, const FileXattrList* xattrs, bool fake_super,
const char* temp_dir) {
return file_to_disk_secure_link_impl(path, basis_path, data, data_size, preallocate, metadata,
preserve_executability, use_fsync, xattrs, fake_super,
temp_dir);
}
bool file_write_to_disk(const char* path, const void* data, unsigned long long data_size,
bool inplace, bool sparse) {
if (!path || (!data && data_size != 0) || has_path_traversal(path))
return false;
return file_to_disk_secure(path, data, data_size, inplace, sparse, NULL, false, NULL);
return file_to_disk_secure(path, data, data_size, inplace, sparse, false, NULL, false, NULL);
}
+75 -13
View File
@@ -4,6 +4,7 @@
#include "file_send.h"
#include "file_receive.h"
#include "file_types.h"
#include "checksum.h"
#include <stdbool.h>
#include <stdint.h>
#include <sys/stat.h>
@@ -14,15 +15,52 @@
File* file_create(const char* path);
void file_destroy(void* item);
bool file_load_data(File* file);
bool file_checksum(File* file, uint64_t* checksum);
/* Compute the whole-file content digest of `file` with the negotiated
* --checksum-choice algorithm and --checksum-seed. Writes the digest into
* `out` (capacity `out_capacity`) and its length into `*out_len`. Returns
* false on read/allocation failure or when the digest would not fit. */
bool file_checksum(File* file, ChecksumAlgo algo, uint64_t seed, uint8_t* out, size_t out_capacity,
size_t* out_len);
size_t file_content_to_buffer(File* file);
FileMetadata* file_metadata_create(const struct stat* stats);
FileMetadata* file_metadata_create(const char* path, const struct stat* stats, bool capture_atime,
bool capture_crtime);
void file_metadata_destroy(void* metadata);
/* --open-noatime process-wide sender policy; see file.c. */
void file_set_open_noatime(bool enable);
bool file_get_open_noatime(void);
/* Open `path` read-only for transfer, honouring --open-noatime when set. */
int file_open_for_read(const char* path);
bool file_write_to_disk(const char* path, const void* data, unsigned long long data_size,
bool inplace, bool sparse);
/* A configured fd without a canonical identity deliberately rejects paths. */
bool file_set_authorized_root(int fd, const char* canonical_path);
/* Symlink trust-boundary helpers (Phase 4, symlink wave). --munge-links
* sender-side marker: every transmitted symlink target is prefixed with this
* while the flag is on; the receiver strips it to restore the real target. */
#define SYMLINK_MUNGE_PREFIX "#SYMLINK/"
char* file_symlink_munge(const char* target);
/* True when a lexical target is relative and contains no ".." component, so it
* can never escape the receive root once created beneath it. */
bool file_symlink_target_contained(const char* target);
/* Strip a leading SYMLINK_MUNGE_PREFIX from `target` (mutable, in place);
* returns true when a marker was removed. */
bool file_symlink_unmunge(char* target);
/* Create a symlink at `path` -> `target`, confined below the authorized root
* (O_NOFOLLOW parent walk, symlinkat; the target is never followed). Returns
* false when a directory already occupies `path`. */
bool file_symlink_at_secure(const char* path, const char* target);
/* --keep-dirlinks (-K) receiver process-wide policy: allow an in-root existing
* symlink-to-directory to be followed as a directory. */
void file_set_keep_dirlinks(bool enable);
/* --trust-sender receiver process-wide policy (Phase 5). When set, the
* receiver trusts that the sender already produced a clean file list and skips
* its own redundant up-front re-validation of incoming paths (the empty/".."
* rejection and the escaping-symlink-target containment). The low-level
* fd-relative confinement primitives below are deliberately NOT disabled by
* this flag, so a hostile sender still cannot escape the authorized root. */
void file_set_trust_sender(bool enable);
bool file_get_trust_sender(void);
/* Secure path/filesystem primitives (symlink-safe, O_NOFOLLOW, root-confined). */
bool file_path_exists_secure(const char* path);
@@ -51,28 +89,52 @@ int file_open_private_dir(const char* dir_path);
silently copied into place. Pass NULL for the historical same-directory
behavior. --inplace writes never use temp_dir. */
bool file_to_disk_secure(const char* path, const void* data, unsigned long long data_size,
bool inplace, bool sparse, const FileMetadata* metadata,
bool inplace, bool sparse, bool preallocate, const FileMetadata* metadata,
bool preserve_executability, const char* temp_dir);
bool file_to_disk_secure_with_fsync(const char* path, const void* data,
unsigned long long data_size, bool inplace, bool sparse,
const FileMetadata* metadata, bool preserve_executability,
bool use_fsync, const char* temp_dir);
bool preallocate, const FileMetadata* metadata,
bool preserve_executability, bool use_fsync,
const char* temp_dir);
/* With update enabled, an existing newer destination is left untouched. The
check is descriptor-based for inplace writes; atomic replacement still has
an unavoidable final rename race without filesystem locking. */
bool file_to_disk_secure_update(const char* path, const void* data, unsigned long long data_size,
bool inplace, bool sparse, const FileMetadata* metadata,
bool preserve_executability, const char* temp_dir);
bool inplace, bool sparse, bool preallocate,
const FileMetadata* metadata, bool preserve_executability,
const char* temp_dir);
bool file_to_disk_secure_no_replace(const char* path, const void* data,
unsigned long long data_size, bool sparse,
unsigned long long data_size, bool sparse, bool preallocate,
const FileMetadata* metadata, bool preserve_executability,
const char* temp_dir);
/* Receiver write-path variant that also applies per-file xattrs (-X/-A) and the
* --fake-super stat xattr fd-relative before the final rename. `update` /
* `no_replace` / `use_fsync` mirror the plain wrappers above; `keep_partial`
* enables --partial best-effort retention of a failed write's temp. */
bool file_to_disk_secure_attrs(const char* path, const void* data, unsigned long long data_size,
bool inplace, bool sparse, bool preallocate,
const FileMetadata* metadata, bool preserve_executability,
bool update, bool no_replace, bool use_fsync,
const FileXattrList* xattrs, bool fake_super, bool keep_partial,
const char* temp_dir);
/* Atomic --link-dest install: replace `path` with a hard link to `basis_path`
(via a temp name + rename); fall back to a byte-identical local copy from
`data` when the link is impossible (EXDEV/EPERM/unsupported filesystem).
`metadata` is applied only on the copy fallback. */
`metadata` is applied only on the copy fallback. `preallocate` applies to
that copy fallback only (a hard-linked file shares the basis inode and is
never re-allocated). */
bool file_to_disk_secure_link(const char* path, const char* basis_path, const void* data,
unsigned long long data_size, const FileMetadata* metadata,
bool preserve_executability, bool use_fsync, const char* temp_dir);
unsigned long long data_size, bool preallocate,
const FileMetadata* metadata, bool preserve_executability,
bool use_fsync, const char* temp_dir);
/* Like file_to_disk_secure_link, but the byte-copy fallback also applies the
* per-file xattrs (-X/-A) and --fake-super stat xattr (fd-relative). On a
* successful hard link no attributes are applied (the shared inode already
* carries the basis's). */
bool file_to_disk_secure_link_attrs(const char* path, const char* basis_path, const void* data,
unsigned long long data_size, bool preallocate,
const FileMetadata* metadata, bool preserve_executability,
bool use_fsync, const FileXattrList* xattrs, bool fake_super,
const char* temp_dir);
#endif
+75 -24
View File
@@ -2,6 +2,7 @@
#include "log.h"
#include "utils.h"
#include <errno.h>
#include <limits.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
@@ -22,6 +23,8 @@ static void string_list_destroy(StringList* list) {
static bool string_list_add(StringList* list, const char* text) {
if (list->count == list->capacity) {
if (list->capacity > INT_MAX / 2)
return false;
int new_cap = list->capacity > 0 ? list->capacity * 2 : 16;
char** grown = realloc(list->items, (size_t)new_cap * sizeof(char*));
if (!grown)
@@ -51,11 +54,18 @@ static int normalize_entry(const char* raw, size_t len, bool strip_line_endings,
if (len == 0)
return 0;
if (raw[0] == '/') {
snprintf(err, err_size, "absolute path entries are not allowed: '%.*s'", (int)len, raw);
int print_len = len > (size_t)INT_MAX ? INT_MAX : (int)len;
snprintf(err, err_size, "absolute path entries are not allowed: '%.*s'", print_len, raw);
return -1;
}
/* Reject NUL bytes inside a token defensively. In NUL-delimited mode the
* delimiter itself is the final byte and is expected; in line mode any NUL is
* embedded garbage (strlen-based parsing would otherwise silently truncate). */
size_t scan_len = strip_line_endings ? len : len - 1;
if (memchr(raw, '\0', scan_len)) {
snprintf(err, err_size, "entry contains an embedded NUL byte");
return -1;
}
/* Reject NUL bytes inside a token defensively (NUL-delimited mode splits on
* them, so this only guards against embedded garbage). */
char* dup = malloc(len + 1);
if (!dup) {
snprintf(err, err_size, "memory allocation failed");
@@ -102,8 +112,27 @@ static int normalize_entry(const char* raw, size_t len, bool strip_line_endings,
return result;
}
/* Build the membership index over the exact entries only. `file_list_affects`
combines the exact/descendant lookups with a walk of the query's own ancestor
prefixes, so no ancestor prefix is ever materialized as a copy and the index
stays O(entry count) memory regardless of path depth. An empty entry (the
source root) sets whole_tree and short-circuits every query. */
static bool file_list_index_build(FileListSet* set, char* err, size_t err_size) {
if (!path_index_build(&set->index, (const char* const*)set->entries, (size_t)set->count)) {
snprintf(err, err_size, "memory allocation failed");
return false;
}
for (int i = 0; i < set->count; i++) {
if (set->entries[i][0] == '\0') {
set->whole_tree = true;
break;
}
}
return true;
}
static FileListSet* string_list_to_set(StringList* raw, char* err, size_t err_size) {
FileListSet* set = malloc(sizeof(FileListSet));
FileListSet* set = calloc(1, sizeof(FileListSet));
if (!set) {
snprintf(err, err_size, "memory allocation failed");
return NULL;
@@ -112,6 +141,10 @@ static FileListSet* string_list_to_set(StringList* raw, char* err, size_t err_si
set->entries = raw->items;
raw->items = NULL;
raw->count = 0;
if (!file_list_index_build(set, err, err_size)) {
file_list_destroy(set);
return NULL;
}
return set;
}
@@ -133,10 +166,20 @@ FileListSet* file_list_load(const char* path, bool null_separated, char* err, si
StringList raw = {0};
char* line = NULL;
size_t line_cap = 0;
ssize_t n;
bool ok = true;
char delim = null_separated ? '\0' : '\n';
while (ok && (n = getdelim(&line, &line_cap, delim, fp)) != -1) {
while (ok) {
ssize_t n = utils_getdelim_bounded(fp, &line, &line_cap, delim, UTILS_MAX_LINE_LEN);
if (n < 0) {
if (errno == EFBIG)
snprintf(err, err_size, "entry in file list exceeds %d bytes", (int)UTILS_MAX_LINE_LEN);
else
snprintf(err, err_size, "error reading file list: %s", strerror(errno));
ok = false;
break;
}
if (n == 0)
break;
int r = normalize_entry(line, (size_t)n, !null_separated, &raw, err, err_size);
if (r < 0) {
ok = false;
@@ -158,34 +201,42 @@ FileListSet* file_list_load(const char* path, bool null_separated, char* err, si
void file_list_destroy(FileListSet* set) {
if (!set)
return;
path_index_free(&set->index);
for (int i = 0; i < set->count; i++)
free(set->entries[i]);
free(set->entries);
free(set);
}
static bool path_has_prefix(const char* path, const char* prefix) {
size_t plen = strlen(prefix);
if (strncmp(path, prefix, plen) != 0)
return false;
return path[plen] == '/' || path[plen] == '\0';
}
bool file_list_affects(const FileListSet* set, const char* rel) {
if (!set)
return true;
if (!rel)
return false;
for (int i = 0; i < set->count; i++) {
const char* entry = set->entries[i];
if (entry[0] == '\0')
return true; /* whole tree listed */
if (strcmp(rel, entry) == 0)
return true; /* the entry itself is listed */
if (path_has_prefix(rel, entry))
return true; /* rel lives under a listed directory */
if (path_has_prefix(entry, rel))
return true; /* rel is an ancestor directory of a listed entry */
if (set->whole_tree)
return true; /* whole tree listed */
/* An exact entry match means `rel` itself is listed. */
if (path_index_contains(&set->index, rel))
return true;
/* Otherwise `rel` is affected when a listed entry is an ancestor directory of
it; walk rel's own directory prefixes (which preserve path-boundary
semantics) and test each for an exact entry. No prefixes are stored. */
size_t len = strlen(rel);
while (len > 0) {
const char* slash = NULL;
for (size_t i = len; i-- > 0;) {
if (rel[i] == '/') {
slash = rel + i;
break;
}
}
if (!slash)
break;
len = (size_t)(slash - rel);
if (path_index_contains_n(&set->index, rel, len))
return true;
}
return false;
/* Finally `rel` is affected when it is an ancestor directory of a listed
entry (binary search for the first entry at or after `rel` + '/'). */
return path_index_has_descendant(&set->index, rel);
}
+10 -2
View File
@@ -1,6 +1,7 @@
#ifndef FILE_LIST_H
#define FILE_LIST_H
#include "utils.h"
#include <stdbool.h>
#include <stddef.h>
@@ -12,11 +13,18 @@
* of "." means the whole tree, absolute entries and ".." traversal are
* rejected at parse time. The set is immutable and shared read-only across
* scanner worker threads.
*/
*
* Membership is answered from `index`, built once at load time over the exact
* entries only: `index.exact` matches a listed path, the sorted view detects an
* ancestor directory of a listed entry, and `rel`'s own directory prefixes are
* matched against the exact set while descending. No ancestor prefix is stored
* as a separate string, so the index is O(entry count) memory however deep the
* paths are, and each query is O(path length) comparisons. */
typedef struct {
char** entries; /* normalized rel paths; "" means the whole tree */
int count;
PathIndex index;
bool whole_tree; /* an entry of "" lists the source root */
} FileListSet;
/* Load and validate a --files-from file. When `null_separated` (-0/--from0)
+1744 -341
View File
File diff suppressed because it is too large. Load diff
+56 -1
View File
@@ -7,9 +7,64 @@
/* Server-side file receive/save path. */
/* Cumulative caps for the deferred directory-time accumulator. The sender may
* legitimately split a large tree across repeated STATUS_DIR_TIMES frames, so a
* per-frame bound is not enough: the receiver must bound the TOTAL it retains
* against a hostile sender. Mirror the delete-manifest limits
* (MAX_MANIFEST_ENTRIES / MAX_MANIFEST_BYTES): the entry count bounds the
* metadata array and the byte budget bounds the concatenated path strings. */
#define MAX_DIR_TIME_ENTRIES (1024 * 1024)
#define MAX_DIR_TIME_BYTES (16ULL * 1024 * 1024)
File* file_receive(const Config* config, int file_descriptor);
File* file_receive_directory(int file_descriptor);
File* file_receive_directory(int file_descriptor, const Config* config);
File* file_receive_dir_time(int file_descriptor, const Config* config);
File* file_receive_hardlink(int file_descriptor);
File* file_receive_symlink(int file_descriptor, const Config* config);
File* file_receive_special(int file_descriptor);
bool file_special_rdev_valid(int32_t major, int32_t minor, mode_t mode);
File* receive_incremental_check(int fd, const Config* config, bool* skipped);
/* Extended variant used by the receiver. `would_transfer` (may be NULL) is set
* true only on the server-contacting --dry-run path when the file is not up to
* date: the receiver has already sent STATUS_DRY_RUN_TRANSFER and returns NULL
* without storing anything. On that path `*skipped` is true for an up-to-date
* (STATUS_OK) file and both flags are false for a genuine error. */
File* receive_incremental_check_ex(int fd, const Config* config, bool* skipped,
bool* would_transfer);
/* P7 Wave D directory-time accumulator. The receiver collects the metadata of
* every directory it creates/receives (STATUS_MKDIR with metadata and/or the
* trailing STATUS_DIR_TIMES frame(s)) and applies the times only at the END of the
* transfer, after all children have been written and after the delete /
* --delay-updates phases have committed (writing or removing a child bumps the
* parent's mtime). -O/--omit-dir-times skips the application entirely. The
* list owns deep copies of the paths and metadata; freed on every path. */
typedef struct {
char** paths; /* owned, destination-relative wire paths */
FileMetadata* entries; /* owned, parallel to paths */
size_t count;
size_t capacity;
size_t bytes; /* cumulative strlen of every retained path */
} DirTimeList;
/* Capture gate shared by the sender-side and receiver-side sinks: directory
* metadata is accumulated only when --times/--metadata is in effect and
* -O/--omit-dir-times does not suppress it. Kept here, next to the accumulator
* it guards, so both call sites express the same condition. */
bool dir_times_should_capture(const Config* config);
void dir_time_list_init(DirTimeList* list);
void dir_time_list_free(DirTimeList* list);
/* Deep-copy one directory's path + metadata into the list. Returns false on
* allocation failure OR when the cumulative entry/byte caps would be exceeded
* (the caller fails the transfer). */
bool dir_time_list_add(DirTimeList* list, const char* wire_path, const FileMetadata* metadata);
/* Apply every accumulated directory's mtime (and atime when captured) beneath
* `root_directory`, confined fd-relative. Best-effort per entry: an absent
* directory (an empty/pruned source dir that was deliberately not created) or a
* non-directory at the path is skipped QUIETLY, an unreachable one with a
* warning, and never fatal. */
void dir_time_list_apply(const DirTimeList* list, const char* root_directory);
/* A received delete-manifest frame: the keep-set (`keeps`, destination-relative
paths the sender transferred/keeps) plus `protected`, destination-relative
+36 -9
View File
@@ -10,23 +10,44 @@
#include <time.h>
#include <unistd.h>
#include "charset.h"
#include "compression.h"
#include "data.h"
#include "file.h"
#include "log.h"
#include "metadata.h"
#include "protocol.h"
#include "xattr.h"
/* Transmit a device/special node (--devices / --specials) as a STATUS_SPECIAL
* frame: the destination path, the metadata frame (whose mode's S_IFMT bits
* carry the node kind) and the device rdev major/minor. The receiver validates
* the kind and rdev and recreates the node (privilege-gating the mknod). */
bool file_send_special(const File* file, int file_descriptor, bool use_metadata) {
if (!file || !file_wire_path(file))
return false;
if (!send_status(file_descriptor, STATUS_SPECIAL))
return false;
if (!send_wire_str(file_descriptor, file_wire_path(file)))
return false;
if (use_metadata && !metadata_send(file_descriptor, file->metadata))
return false;
int32_t major = file->rdev_major;
int32_t minor = file->rdev_minor;
return send_n_data(file_descriptor, &major, sizeof(major)) &&
send_n_data(file_descriptor, &minor, sizeof(minor));
}
bool file_send_single_calls(File* file, int file_descriptor, bool use_metadata,
int compression_level, bool send_path) {
return file_send_single_calls_with_skip(file, file_descriptor, use_metadata, compression_level,
send_path, NULL, -1, 0);
send_path, NULL, -1, 0, false);
}
bool file_send_single_calls_with_skip(File* file, int file_descriptor, bool use_metadata,
int compression_level, bool send_path,
char* const* skip_suffixes, int skip_count,
int compression_threads) {
int compression_threads, bool send_xattrs) {
if (!file || !file->path || !file->data || (file->data->size != 0 && !file->data->data))
return false;
const Data* data_to_send = file->data;
@@ -41,7 +62,7 @@ bool file_send_single_calls_with_skip(File* file, int file_descriptor, bool use_
}
data_to_send = compressed_data;
}
if (send_path && !send_str(file_descriptor, file_wire_path(file))) {
if (send_path && !send_wire_str(file_descriptor, file_wire_path(file))) {
data_destroy(compressed_data);
return false;
}
@@ -49,6 +70,10 @@ bool file_send_single_calls_with_skip(File* file, int file_descriptor, bool use_
data_destroy(compressed_data);
return false;
}
if (send_xattrs && !xattr_send(file_descriptor, file ? file->xattrs : NULL)) {
data_destroy(compressed_data);
return false;
}
if (!send_data(file_descriptor, data_to_send)) {
data_destroy(compressed_data);
return false;
@@ -60,25 +85,27 @@ bool file_send_single_calls_with_skip(File* file, int file_descriptor, bool use_
bool file_send_sendfile(File* file, int file_descriptor, bool use_metadata, int compression_level,
bool send_path) {
return file_send_sendfile_with_skip(file, file_descriptor, use_metadata, compression_level,
send_path, NULL, -1, 0);
send_path, NULL, -1, 0, false);
}
bool file_send_sendfile_with_skip(File* file, int file_descriptor, bool use_metadata,
int compression_level, bool send_path, char* const* skip_suffixes,
int skip_count, int compression_threads) {
int skip_count, int compression_threads, bool send_xattrs) {
if (!file || !file->path || !file->data)
return false;
if (compression_level > 0)
return file_send_single_calls_with_skip(file, file_descriptor, use_metadata, compression_level,
send_path, skip_suffixes, skip_count,
compression_threads);
compression_threads, send_xattrs);
if (send_path && !send_str(file_descriptor, file_wire_path(file)))
if (send_path && !send_wire_str(file_descriptor, file_wire_path(file)))
return false;
if (use_metadata && !metadata_send(file_descriptor, file->metadata))
return false;
if (send_xattrs && !xattr_send(file_descriptor, file ? file->xattrs : NULL))
return false;
int fd = open(file->path, O_RDONLY);
int fd = file_open_for_read(file->path);
if (fd == -1) {
log_perror("Could not open file for sendfile");
return false;
@@ -118,7 +145,7 @@ bool file_send_sendfile_with_skip(File* file, int file_descriptor, bool use_meta
off_t offset = 0;
struct timespec deadline;
clock_gettime(CLOCK_MONOTONIC, &deadline);
deadline.tv_sec += 60;
deadline.tv_sec += protocol_get_io_timeout_sec();
while ((unsigned long long)offset < file_size) {
struct timespec now;
clock_gettime(CLOCK_MONOTONIC, &now);
+3 -2
View File
@@ -6,16 +6,17 @@
/* Client-side file send path. */
bool file_send_special(const File* file, int file_descriptor, bool use_metadata);
bool file_send_single_calls(File* file, int file_descriptor, bool use_metadata,
int compression_level, bool send_path);
bool file_send_single_calls_with_skip(File* file, int file_descriptor, bool use_metadata,
int compression_level, bool send_path,
char* const* skip_suffixes, int skip_count,
int compression_threads);
int compression_threads, bool send_xattrs);
bool file_send_sendfile(File* file, int file_descriptor, bool use_metadata, int compression_level,
bool send_path);
bool file_send_sendfile_with_skip(File* file, int file_descriptor, bool use_metadata,
int compression_level, bool send_path, char* const* skip_suffixes,
int skip_count, int compression_threads);
int skip_count, int compression_threads, bool send_xattrs);
#endif
+34 -176
View File
@@ -1,129 +1,7 @@
#include <errno.h>
#include <fcntl.h>
#include <libgen.h>
#include <limits.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include "file_store.h"
#include "metadata.h"
#include "utils.h"
static int authorized_root_fd = -1;
static char* authorized_root_path;
static bool path_is_within_root(const char* root, const char* path) {
size_t root_length = strlen(root);
return strncmp(root, path, root_length) == 0 &&
(path[root_length] == '\0' || path[root_length] == '/');
}
bool file_store_set_authorized_root(int fd, const char* canonical_path) {
char* new_path = canonical_path ? str_dup(canonical_path) : NULL;
if (canonical_path && !new_path) {
authorized_root_fd = -1;
free(authorized_root_path);
authorized_root_path = NULL;
return false;
}
free(authorized_root_path);
authorized_root_path = new_path;
authorized_root_fd = fd;
return true;
}
int file_store_open_secure_parent(const char* path, char** leaf_out) {
char* copy = str_dup(path);
if (!copy)
return -1;
char* parent = dirname(copy);
const char* slash = strrchr(path, '/');
char* leaf = str_dup(slash ? slash + 1 : path);
if (!leaf) {
free(copy);
return -1;
}
int fd;
if (authorized_root_fd >= 0) {
if (!authorized_root_path || path[0] != '/' ||
!path_is_within_root(authorized_root_path, path)) {
free(copy);
free(leaf);
return -1;
}
fd = dup(authorized_root_fd);
if (fd < 0) {
free(copy);
free(leaf);
return -1;
}
size_t root_length = strlen(authorized_root_path);
char* relative = str_dup(path + root_length);
if (!relative) {
free(copy);
free(leaf);
close(fd);
return -1;
}
free(copy);
copy = relative;
parent = dirname(copy);
} else {
fd = (parent[0] == '/') ? open("/", O_RDONLY | O_DIRECTORY | O_CLOEXEC)
: open(".", O_RDONLY | O_DIRECTORY | O_CLOEXEC);
}
if (fd < 0) {
free(copy);
free(leaf);
return -1;
}
char* save = NULL;
char* component = strtok_r(parent, "/", &save);
while (component) {
if (strcmp(component, "..") == 0) {
close(fd);
free(copy);
free(leaf);
return -1;
}
if (strcmp(component, ".") != 0) {
int next = openat(fd, component, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
if (next < 0 && errno == ENOENT) {
if (mkdirat(fd, component, 0755) == 0 || errno == EEXIST)
next = openat(fd, component, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
}
if (next < 0) {
close(fd);
free(copy);
free(leaf);
return -1;
}
close(fd);
fd = next;
}
component = strtok_r(NULL, "/", &save);
}
free(copy);
*leaf_out = leaf;
return fd;
}
bool file_store_rename_secure(const char* old_path, const char* new_path) {
char *old_leaf = NULL, *new_leaf = NULL;
int old_parent = file_store_open_secure_parent(old_path, &old_leaf);
int new_parent = file_store_open_secure_parent(new_path, &new_leaf);
bool ok = old_parent >= 0 && new_parent >= 0 &&
renameat(old_parent, old_leaf, new_parent, new_leaf) == 0;
if (old_parent >= 0)
close(old_parent);
if (new_parent >= 0)
close(new_parent);
free(old_leaf);
free(new_leaf);
return ok;
}
static bool write_all(int fd, const void* data, unsigned long long size) {
const unsigned char* p = data;
@@ -139,60 +17,40 @@ static bool write_all(int fd, const void* data, unsigned long long size) {
return true;
}
bool file_store_write_secure(const char* path, const void* data, unsigned long long data_size,
bool inplace, bool sparse, const FileMetadata* metadata,
bool preserve_executability) {
char* leaf = NULL;
int dirfd = file_store_open_secure_parent(path, &leaf);
if (dirfd < 0)
return false;
int fd = -1;
bool ok = false;
if (inplace) {
fd = openat(dirfd, leaf, O_WRONLY | O_CREAT | O_TRUNC | O_CLOEXEC | O_NOFOLLOW, 0644);
if (fd >= 0) {
if (!sparse || data_size == 0 || ftruncate(fd, (off_t)data_size) == 0)
ok = write_all(fd, data, data_size);
if (ok && metadata)
ok = file_restore_metadata_fd(fd, metadata, preserve_executability);
/* A run of NUL bytes at least this long is emitted as a hole (lseek) rather
* than written, so the resulting file is genuinely sparse on the filesystem. */
#define SPARSE_HOLE_MIN 4096U
/* Sparse-aware writer (--sparse/-S). Walks `data`; any all-zero run of at
* least SPARSE_HOLE_MIN bytes is skipped with lseek(SEEK_CUR) so the block is
* never allocated (a real hole on the destination); every other byte is written
* normally. The file is pre-sized with ftruncate by the callers before this
* runs, so holes are guaranteed and the offset bookkeeping stays correct
* (each lseek advances the fd offset exactly as a write of that many bytes
* would). After the final run, ftruncate(size) guarantees the logical size is
* exactly `size` even when the tail was a hole. The full file image is in
* memory, so no wire change is needed. Returns false on I/O error. */
bool file_store_write_sparse(int fd, const unsigned char* data, unsigned long long size) {
unsigned long long i = 0;
while (i < size) {
if (data[i] == 0) {
unsigned long long run_start = i;
while (i < size && data[i] == 0)
i++;
unsigned long long run_len = i - run_start;
if (run_len >= SPARSE_HOLE_MIN) {
if (lseek(fd, (off_t)run_len, SEEK_CUR) < 0)
return false;
} else if (!write_all(fd, data + run_start, run_len)) {
return false;
}
} else {
unsigned long long run_start = i;
while (i < size && data[i] != 0)
i++;
if (!write_all(fd, data + run_start, i - run_start))
return false;
}
} else {
int tmp_size = snprintf(NULL, 0, ".%s.tmp.%ld.%u", leaf, (long)getpid(), 99U);
if (tmp_size < 0) {
close(dirfd);
free(leaf);
return false;
}
char* tmp = malloc((size_t)tmp_size + 1);
if (!tmp) {
close(dirfd);
free(leaf);
return false;
}
for (unsigned int i = 0; i < 100 && !ok; ++i) {
snprintf(tmp, (size_t)tmp_size + 1, ".%s.tmp.%ld.%u", leaf, (long)getpid(), i);
fd = openat(dirfd, tmp, O_WRONLY | O_CREAT | O_EXCL | O_CLOEXEC | O_NOFOLLOW, 0600);
if (fd < 0)
continue;
if (sparse && data_size > 0)
ok = ftruncate(fd, (off_t)data_size) == 0;
if (ok || (!sparse || data_size == 0))
ok = write_all(fd, data, data_size);
if (ok && metadata)
ok = file_restore_metadata_fd(fd, metadata, preserve_executability);
if (close(fd) != 0)
ok = false;
fd = -1;
if (ok && renameat(dirfd, tmp, dirfd, leaf) != 0)
ok = false;
if (!ok)
unlinkat(dirfd, tmp, 0);
}
free(tmp);
}
if (fd >= 0)
close(fd);
close(dirfd);
free(leaf);
return ok;
return ftruncate(fd, (off_t)size) == 0;
}
+7 -7
View File
@@ -1,14 +1,14 @@
#ifndef FILE_STORE_H
#define FILE_STORE_H
#include "file.h"
#include <stdbool.h>
bool file_store_set_authorized_root(int fd, const char* canonical_path);
int file_store_open_secure_parent(const char* path, char** leaf_out);
bool file_store_rename_secure(const char* old_path, const char* new_path);
bool file_store_write_secure(const char* path, const void* data, unsigned long long data_size,
bool inplace, bool sparse, const FileMetadata* metadata,
bool preserve_executability);
/* Sparse-aware write (--sparse/-S): every all-zero run of at least
* SPARSE_HOLE_MIN bytes is skipped with lseek(SEEK_CUR) so it becomes a real
* hole; every other byte is written. The caller pre-sizes the file with
* ftruncate; this function also ftruncate()s to `size` at the end so a trailing
* hole keeps the exact logical length. Shared by the file_store and file write
* paths. Returns false on write/lseek/ftruncate error. */
bool file_store_write_sparse(int fd, const unsigned char* data, unsigned long long size);
#endif
+51
View File
@@ -2,6 +2,7 @@
#define FILE_TYPES_H
#include "data.h"
#include "xattr.h"
#include <stdbool.h>
#include <sys/stat.h>
@@ -13,6 +14,18 @@ typedef struct {
gid_t gid;
time_t mtime_sec;
long mtime_nsec;
/* Optional access time (-U/--atimes) and creation/birth time (-N/--crtimes),
* appended for protocol 2.12.0. The SENDER sets the corresponding *_valid
* flag only when the preserve option is active (and, for crtime, only when
* the source platform exposed a birth time via statx STATX_BTIME). The wire
* always carries the fields and the flags; a false flag tells the receiver to
* ignore the value. */
bool atime_valid;
time_t atime_sec;
long atime_nsec;
bool crtime_valid;
time_t crtime_sec;
long crtime_nsec;
} FileMetadata;
typedef struct {
@@ -29,12 +42,50 @@ typedef struct {
/* True when this entry is an explicit directory entry (--dirs mode): the
* receiver creates the directory instead of writing a regular file. */
bool is_dir;
/* Receiver-only (P7 Wave D): this is a STATUS_DIR_TIMES entry. It carries a
* traversed source directory's metadata for DEFERRED application, but must
* NEVER create the directory: the scanner captures every traversed directory
* (including empty ones whose parents no child write created), so creation
* would resurrect the empty dirs that FastSync deliberately never transfers.
* file_save_to_disk_full short-circuits such an entry as FILE_SAVE_SKIPPED,
* and the sink still accumulates the metadata into its DirTimeList. */
bool dir_time_only;
/* Receiver-only, --link-dest: when set, install the destination entry as a
* hard link to this absolute (root-confined) path instead of writing
* `data`. The matching code has already verified the link target's content
* equals the incoming file, and `data` is kept as the cross-filesystem
* fallback (a local copy) if the hard link cannot be created. */
char* basis_link;
/* --hard-links (-H), sender + receiver wire state. link_group is a run-local
* id shared by every member of one source inode (0 = not part of a group).
* The FIRST member (link_first == true) carries its data on the wire and is
* written normally; every sibling (link_first == false) carries NO data and
* hardlink_target holds the first member's wire path so the receiver can link
* to (or copy from) the already-installed first member. */
int link_group;
bool link_first;
char* hardlink_target;
/* Symlink-type entry (-l/--links, or -k/--copy-dirlinks' keep-as-symlink
* branch). When true, `symlink_target` holds the (sender-munged, if
* --munge-links) target string that is carried on the wire; the receiver
* creates a symlink to (an unmunged) target instead of writing regular-file
* data. `data` is empty for a symlink entry. Sender + receiver state. */
bool is_symlink;
char* symlink_target;
/* Phase 4 special/devices: when `is_special` is true this entry is a device
* or special node to be RECREATED on the destination (mknod/mkfifo) rather
* than written from `data`. The concrete node kind is derived from the
* metadata mode's S_IFMT bits (receiver-validated), and rdev_major/minor
* carry the device major/minor numbers for char/block devices. CROSSES the
* wire (protocol 2.13.0). */
bool is_special;
int32_t rdev_major;
int32_t rdev_minor;
/* Phase-4 xattrs (-X/--xattrs, -A/--acls). Sender: captured from the source
* file when use_xattrs is set; transmitted in the per-file metadata frame.
* Receiver: parsed off the wire, attached here, and applied fd-relative on
* the written file. NULL/0 == the file carries no xattrs. */
FileXattrList* xattrs;
} File;
/* The path that should be sent on the wire and used for the receiver-side
+20 -4
View File
@@ -2,6 +2,7 @@
#include "log.h"
#include "utils.h"
#include <errno.h>
#include <limits.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
@@ -176,6 +177,8 @@ bool filter_rule_list_add(FilterRuleList* list, FilterRule* rule) {
if (!list || !rule)
return false;
if (list->count == list->capacity) {
if (list->capacity > INT_MAX / 2)
return false;
int new_cap = list->capacity > 0 ? list->capacity * 2 : 8;
FilterRule** grown = realloc(list->items, (size_t)new_cap * sizeof(FilterRule*));
if (!grown)
@@ -322,8 +325,10 @@ FilterRuleList* filter_file_read(const char* dir_path, const char* owner_rel, bo
if (!fp) {
if (errno == ENOENT || errno == ENOTDIR)
return filter_rule_list_create();
log_message(LOG_LEVEL_WARNING, "Could not read .rsync-filter in %s: %s", dir_path,
strerror(errno));
char* escaped_dir = output_escape(dir_path, log_get_8_bit_output());
log_message(LOG_LEVEL_WARNING, "Could not read .rsync-filter in %s: %s",
escaped_dir ? escaped_dir : "<allocation failed>", strerror(errno));
free(escaped_dir);
return filter_rule_list_create();
}
if (exists)
@@ -336,9 +341,20 @@ FilterRuleList* filter_file_read(const char* dir_path, const char* owner_rel, bo
}
char* line = NULL;
size_t line_cap = 0;
ssize_t n;
bool ok = true;
while ((n = getline(&line, &line_cap, fp)) != -1) {
while (true) {
ssize_t n = utils_getdelim_bounded(fp, &line, &line_cap, '\n', UTILS_MAX_LINE_LEN);
if (n < 0) {
if (errno == EFBIG) {
snprintf(err, err_size, "line in .rsync-filter exceeds %d bytes", (int)UTILS_MAX_LINE_LEN);
} else {
snprintf(err, err_size, "error reading .rsync-filter: %s", strerror(errno));
}
ok = false;
break;
}
if (n == 0)
break;
const char* p = line;
while (*p == ' ' || *p == '\t')
p++;
+127
View File
@@ -0,0 +1,127 @@
#include "hardlink.h"
#include <stdint.h>
#include <stdlib.h>
#include <string.h>
#include "log.h"
#include "utils.h"
/* ---- Sender-side detection table ---- */
HardLinkTable* hardlink_table_create(void) {
HardLinkTable* table = calloc(1, sizeof(HardLinkTable));
if (!table)
return NULL;
if (mtx_init(&table->mutex, mtx_plain) != thrd_success) {
free(table);
return NULL;
}
table->next_gid = 1;
return table;
}
static void hardlink_item_destroy(HardLinkItem* item) {
if (!item)
return;
free(item->first_path);
item->first_path = NULL;
}
void hardlink_table_destroy(HardLinkTable* table) {
if (!table)
return;
for (size_t i = 0; i < table->count; i++)
hardlink_item_destroy(&table->items[i]);
free(table->items);
table->items = NULL;
table->count = 0;
table->capacity = 0;
mtx_destroy(&table->mutex);
free(table);
}
static HardLinkItem* hardlink_table_find_locked(HardLinkTable* table, dev_t dev, ino_t ino) {
for (size_t i = 0; i < table->count; i++) {
if (table->items[i].dev == dev && table->items[i].ino == ino)
return &table->items[i];
}
return NULL;
}
static bool hardlink_table_add_locked(HardLinkTable* table, dev_t dev, ino_t ino, const char* path,
int gid, HardLinkItem** out) {
if (table->count == table->capacity) {
size_t new_capacity = table->capacity == 0 ? 8 : table->capacity * 2;
if (new_capacity < table->capacity)
return false;
HardLinkItem* grown = realloc(table->items, new_capacity * sizeof(HardLinkItem));
if (!grown)
return false;
table->items = grown;
table->capacity = new_capacity;
}
HardLinkItem* item = &table->items[table->count];
char* dup = str_dup(path);
if (!dup)
return false;
memset(item, 0, sizeof(*item));
item->dev = dev;
item->ino = ino;
item->gid = gid;
item->first_path = dup;
table->count++;
*out = item;
return true;
}
bool hardlink_table_assign(HardLinkTable* table, const char* wire_path, dev_t dev, ino_t ino,
int* gid, bool* is_first, char** first_path_out) {
if (!table || !wire_path || !gid || !is_first || !first_path_out)
return false;
if (mtx_lock(&table->mutex) != thrd_success)
return false;
bool ok = true;
const HardLinkItem* item = hardlink_table_find_locked(table, dev, ino);
int next_gid;
if (item) {
*is_first = false;
char* dup = str_dup(item->first_path);
if (!dup) {
ok = false;
} else {
*gid = item->gid;
*first_path_out = dup;
}
next_gid = -1;
} else {
if (table->next_gid <= 0) {
ok = false;
next_gid = -1;
} else {
next_gid = table->next_gid;
HardLinkItem* created = NULL;
if (!hardlink_table_add_locked(table, dev, ino, wire_path, next_gid, &created)) {
ok = false;
} else {
char* dup = str_dup(wire_path);
if (!dup) {
hardlink_item_destroy(created);
table->count--;
ok = false;
} else {
*is_first = true;
*gid = next_gid;
*first_path_out = dup;
}
}
}
}
if (ok && next_gid > 0)
table->next_gid++;
mtx_unlock(&table->mutex);
if (!ok) {
log_message(LOG_LEVEL_ERROR, "memory allocation failed while detecting hard links");
}
return ok;
}
+66
View File
@@ -0,0 +1,66 @@
#ifndef HARDLINK_H
#define HARDLINK_H
#include <stdbool.h>
#include <stddef.h>
#include <sys/types.h>
#include <threads.h>
/*
* --hard-links / -H support.
*
* Sender side: a HardLinkTable detects regular files on the source that share
* an (st_dev, st_ino) identity (a `cp -al`-style hard-linked tree) and assigns
* each distinct inode a stable, run-local link-group id. The first member
* encountered carries the file data; every later member is marked as a sibling
* (no data payload) that the receiver creates as a hard link to the first
* member's destination file. Grouping is scoped by st_dev so inode reuse
* across different filesystems is never conflated. The table is mutex-guarded
* so the parallel (multi-threaded) scanner COULD share one instance across its
* worker threads; the first-thread-to-call designates the data-carrying member,
* which is safe because a hard-link group's members are byte-identical. (In
* practice the sender forces the sequential scanner whenever -H is on; the
* mutex guards the shared table for any path that supplies one.)
*
* ORDERING (why there is no receiver-side handshake): the receiver stores every
* file - including a hard-link group's first member - through a SINGLE writer
* thread draining a single FIFO queue driven by a single receive thread, so
* wire order == write order and every sibling is processed AFTER its group's
* first member. The sender additionally forces the sequential scanner with -H
* so the first-member frame always precedes its siblings on the wire. Sibling
* install therefore needs no present/wait registry: it hard-links to the first
* member (or copies it) knowing that path is already installed - or that, if
* the first member was skipped (already up to date), its destination still
* exists. This guarantee is REQUIRED; do not introduce a concurrent
* multi-writer receiver for -H without re-adding an ordering mechanism.
*/
typedef struct HardLinkItem {
dev_t dev;
ino_t ino;
int gid;
char* first_path; /* wire path of the group's data-carrying first member */
} HardLinkItem;
typedef struct HardLinkTable {
mtx_t mutex;
HardLinkItem* items;
size_t count;
size_t capacity;
int next_gid;
} HardLinkTable;
HardLinkTable* hardlink_table_create(void);
void hardlink_table_destroy(HardLinkTable* table);
/* Assign a link-group id to the regular file at `wire_path` with (dev, ino).
* On the first encounter the file becomes the group's first (data-carrying)
* member (*is_first = true) and a fresh gid is allocated. On a later member
* *is_first = false and *first_path_out is set to a malloc'd copy of the first
* member's wire path (the caller stores it and owns it; on the first member
* path the returned *first_path_out is a malloc'd copy of its own wire path).
* Returns false on allocation failure (transfer should abort). */
bool hardlink_table_assign(HardLinkTable* table, const char* wire_path, dev_t dev, ino_t ino,
int* gid, bool* is_first, char** first_path_out);
#endif
+748
View File
@@ -0,0 +1,748 @@
#include "identity.h"
#include "log.h"
#include "utils.h"
#include <errno.h>
#include <fcntl.h>
#include <grp.h>
#include <limits.h>
#include <pwd.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <unistd.h>
/* The active identity snapshot lives in a per-process global. The TCP server
* forks one child process per connection, so a connection never shares this
* with another; within a connection the multithreaded receiver reads it without
* mutation. This is what lets the fd-relative metadata path consult the
* negotiated policy without threading a Config through every write helper. */
typedef struct {
bool numeric_ids;
bool chown_uid_set;
int32_t chown_uid;
bool chown_gid_set;
int32_t chown_gid;
IdentityMap* usermap;
int usermap_count;
IdentityMap* groupmap;
int groupmap_count;
/* --super / --no-super tri-state (SUPER_MODE_AUTO when unset). Snapshotted
* per connection so privilege_super_permitted() can gate super-user
* activities without a Config argument. */
SuperMode super_mode;
/* --copy-as=USER[:GROUP]: snapshotted so the ownership resolver can force the
* target ids without a Config argument. */
bool copy_as_set;
int32_t copy_as_uid;
int32_t copy_as_gid;
bool set;
} IdentityActive;
static IdentityActive g_identity;
static void identity_active_reset(void) {
free(g_identity.usermap);
free(g_identity.groupmap);
g_identity.usermap = NULL;
g_identity.groupmap = NULL;
g_identity.usermap_count = 0;
g_identity.groupmap_count = 0;
g_identity.numeric_ids = false;
g_identity.chown_uid_set = false;
g_identity.chown_uid = 0;
g_identity.chown_gid_set = false;
g_identity.chown_gid = 0;
g_identity.super_mode = SUPER_MODE_AUTO;
g_identity.copy_as_set = false;
g_identity.copy_as_uid = 0;
g_identity.copy_as_gid = 0;
g_identity.set = false;
}
void identity_clear_active(void) {
identity_active_reset();
}
bool identity_set_active(const Config* config) {
identity_active_reset();
if (!config)
return true;
g_identity.numeric_ids = config->numeric_ids;
g_identity.chown_uid_set = config->chown_uid_set;
g_identity.chown_uid = config->chown_uid;
g_identity.chown_gid_set = config->chown_gid_set;
g_identity.chown_gid = config->chown_gid;
g_identity.super_mode = config->super_mode;
g_identity.copy_as_set = config->copy_as_set;
g_identity.copy_as_uid = config->copy_as_uid;
g_identity.copy_as_gid = config->copy_as_gid;
if (config->usermap_count > 0) {
g_identity.usermap = calloc((size_t)config->usermap_count, sizeof(IdentityMap));
if (!g_identity.usermap)
goto alloc_failed;
memcpy(g_identity.usermap, config->usermap,
(size_t)config->usermap_count * sizeof(IdentityMap));
g_identity.usermap_count = config->usermap_count;
}
if (config->groupmap_count > 0) {
g_identity.groupmap = calloc((size_t)config->groupmap_count, sizeof(IdentityMap));
if (!g_identity.groupmap)
goto alloc_failed;
memcpy(g_identity.groupmap, config->groupmap,
(size_t)config->groupmap_count * sizeof(IdentityMap));
g_identity.groupmap_count = config->groupmap_count;
}
g_identity.set = true;
/* A root receiver would honor any client-supplied ownership request (a
--usermap/--groupmap/--chown/--copy-as, or raw ids under --numeric-ids).
Surface that prominently; a privileged daemon applying arbitrary client
ownership is a deliberate, opt-in choice the operator should be aware of. */
if (geteuid() == 0)
log_message(LOG_LEVEL_WARNING,
"identity mapping active and running as root: client-supplied "
"ownership (usermap/groupmap/chown/numeric-ids) will be honored; "
"run the daemon as an unprivileged user unless intended");
/* --super explicitly requests super-user activities, but FastSync never
elevates privileges: when the receiver is not already root the kernel will
refuse those confined attempts and each is skipped per entry. Warn exactly
once at activation time (never abort) so the operator knows the flag cannot
succeed on this host. */
if (g_identity.super_mode == SUPER_MODE_ON && geteuid() != 0)
log_message(LOG_LEVEL_WARNING,
"--super requested but the receiver is not privileged; super-user "
"activities (ownership, device nodes) will be attempted but refused "
"by the kernel and skipped per entry");
return true;
alloc_failed:
/* Never proceed with a partial (count-left-zero) map: that would silently
apply the WRONG ownership policy. Fail closed and let the caller refuse
the connection. */
log_message(LOG_LEVEL_ERROR, "memory allocation failed while activating identity policy");
identity_active_reset();
return false;
}
bool privilege_super_permitted(void) {
return privilege_super_mode_permitted(g_identity.super_mode);
}
bool privilege_super_mode_permitted(SuperMode mode) {
/* AUTO and ON both attempt the confined operation; OFF forbids it even for a
* root receiver. AUTO is the historical FastSync behavior (always attempt
* and let the kernel refuse an unprivileged call, which the caller skips), so
* it must stay permissive or a group-only chown that a non-root receiver is
* allowed to make would regress. */
return mode != SUPER_MODE_OFF;
}
bool identity_active_enabled(void) {
/* numeric_ids is included: this set only gates identity_apply_ownership,
which runs only when metadata is present (a -M/--preserve transfer). A
standalone --numeric-ids (no ownership-affecting flag) carries no
metadata, never reaches identity_apply_ownership, and therefore correctly
stays inert; combined with -M it activates raw-id application. --super /
--no-super does NOT enable ownership: it only permits or forbids the
already-requested super-user activities, so a --super with no explicit
identity flag must never silently apply client-chosen ownership. */
return g_identity.set &&
(g_identity.numeric_ids || g_identity.chown_uid_set || g_identity.chown_gid_set ||
g_identity.usermap_count > 0 || g_identity.groupmap_count > 0 || g_identity.copy_as_set);
}
bool identity_ownership_requested(const Config* config) {
if (!config)
return false;
/* Every value that makes the receiver act on a client-chosen owner, plus an
* explicit --super (super-user device-node activities). Pure config, so the
* daemon gate can evaluate it before identity_set_active(). */
return config->numeric_ids || config->chown_uid_set || config->chown_gid_set ||
config->usermap_count > 0 || config->groupmap_count > 0 || config->copy_as_set ||
config->fake_super || config->super_mode == SUPER_MODE_ON;
}
bool identity_copy_as_active(void) {
return g_identity.set && g_identity.copy_as_set;
}
bool identity_copy_as_refused(const Config* config) {
if (!config || !config->copy_as_set)
return false;
/* The safe-subset --copy-as needs a privileged (root) receiver, and an
* operator/--no-super veto forbids the ownership change even for root. This
* is deliberately a pure function of the config and the current effective uid
* (never the active snapshot) because the server evaluates it at the
* pre-STATUS_OK config gate, before identity_set_active() has run. */
return geteuid() != 0 || config->super_mode == SUPER_MODE_OFF;
}
bool identity_wire_valid(const Config* config) {
if (!config)
return false;
if (config->usermap_count < 0 || config->usermap_count > MAX_IDENTITY_MAP ||
config->groupmap_count < 0 || config->groupmap_count > MAX_IDENTITY_MAP)
return false;
if (config->chown_uid_set && config->chown_uid < IDENTITY_MATCH_ANY)
return false;
if (config->chown_gid_set && config->chown_gid < IDENTITY_MATCH_ANY)
return false;
for (int i = 0; i < config->usermap_count; i++) {
if (config->usermap[i].from < IDENTITY_MATCH_ANY || config->usermap[i].to < IDENTITY_CURRENT)
return false;
}
for (int i = 0; i < config->groupmap_count; i++) {
if (config->groupmap[i].from < IDENTITY_MATCH_ANY || config->groupmap[i].to < IDENTITY_CURRENT)
return false;
}
/* Defense-in-depth: a --copy-as block must never carry a negative (sentinel)
* id into the ownership path. receive_copy_as_options already rejects them,
* but identity_wire_valid is the shared validation used by both the receiver
* and unit tests, so re-assert it here. */
if (config->copy_as_set && (config->copy_as_uid < 0 || config->copy_as_gid < 0))
return false;
return true;
}
/* ---- CLI-time name/number resolution ---- */
/* Parse a single FROM/TO token into an int32 id. Returns 0 on success, -1 on a
* malformed or unresolvable token. When is_group, name lookups use the group
* database; otherwise the user database. A `*` token returns IDENTITY_MATCH_ANY
* / IDENTITY_CURRENT (the same -1 value, disambiguated by the caller's
* position). An `@`-prefixed or bare-decimal token is a numeric id. */
static int identity_resolve_token(const char* token, bool is_group, int32_t* out) {
if (!token || *token == '\0')
return -1;
if (strcmp(token, "*") == 0) {
*out = IDENTITY_MATCH_ANY;
return 0;
}
const char* num = (token[0] == '@') ? token + 1 : token;
if (*num != '\0') {
bool all_digits = true;
for (const char* p = num; *p; p++)
if (*p < '0' || *p > '9')
all_digits = false;
if (all_digits) {
char* endptr = NULL;
errno = 0;
long val = strtol(num, &endptr, 10);
if (errno == 0 && endptr && *endptr == '\0' && val >= 0 && val <= INT32_MAX) {
*out = (int32_t)val;
return 0;
}
return -1;
}
}
/* A name (or a name-like numeric that failed strict numeric parse). */
if (is_group) {
struct group* gr = getgrnam(token);
if (!gr)
return -1;
*out = (int32_t)gr->gr_gid;
return 0;
}
struct passwd* pw = getpwnam(token);
if (!pw)
return -1;
*out = (int32_t)pw->pw_uid;
return 0;
}
static int identity_append_rule(IdentityMap** map, int* count, int32_t from, int32_t to) {
if (*count >= MAX_IDENTITY_MAP)
return -1;
IdentityMap* grown = realloc(*map, (size_t)(*count + 1) * sizeof(IdentityMap));
if (!grown)
return -1;
*map = grown;
(*map)[*count].from = from;
(*map)[*count].to = to;
(*count)++;
return 0;
}
int identity_parse_map(Config* config, const char* value, bool is_group) {
if (!config || !value || *value == '\0') {
log_message(LOG_LEVEL_ERROR, "%smap requires a value", is_group ? "--group" : "--user");
return -1;
}
char* list = str_dup(value);
if (!list)
return -1;
const char* optname = is_group ? "--groupmap" : "--usermap";
char* saveptr = NULL;
for (char* rule = strtok_r(list, ",", &saveptr); rule; rule = strtok_r(NULL, ",", &saveptr)) {
char* colon = strchr(rule, ':');
if (!colon || colon == rule) {
/* Log before freeing: `rule` points into the str_dup'd list. */
log_message(LOG_LEVEL_ERROR, "%s rules must be FROM:TO (got '%s')", optname, rule);
free(list);
return -1;
}
*colon = '\0';
char* from_token = rule;
char* to_token = colon + 1;
if (*to_token == '\0') {
free(list);
log_message(LOG_LEVEL_ERROR, "%s rule 'FROM:' is missing the TO value (got '%s')", optname,
value);
return -1;
}
int32_t from_id, to_id;
if (identity_resolve_token(from_token, is_group, &from_id) != 0 ||
identity_resolve_token(to_token, is_group, &to_id) != 0) {
free(list);
log_message(LOG_LEVEL_ERROR,
"%s could not resolve '%s' (name must exist on the source; use "
"@N for a numeric id)",
optname, value);
return -1;
}
if (identity_append_rule(is_group ? &config->groupmap : &config->usermap,
is_group ? &config->groupmap_count : &config->usermap_count, from_id,
to_id) != 0) {
free(list);
log_message(LOG_LEVEL_ERROR, "%s has too many rules (max %d)", optname, MAX_IDENTITY_MAP);
return -1;
}
}
free(list);
return 0;
}
/* Split --chown=USER:GROUP on the first UNESCAPED colon, honoring backslash
* escapes (a `\:` is a literal colon inside a name; a lone backslash before any
* other character is kept verbatim). Both sides are returned as malloc'd
* strings (the absent side is NULL). */
static int identity_split_chown(const char* value, char** puser, char** pgroup) {
size_t len = strlen(value);
char* user = malloc(len + 1);
char* group = malloc(len + 1);
if (!user || !group) {
free(user);
free(group);
return -1;
}
const char* p = value;
size_t ui = 0;
bool split_seen = false;
size_t gi = 0;
while (*p) {
if (*p == '\\' && p[1] == ':') {
/* an escaped colon: a literal ':' in the current side's name */
if (split_seen)
group[gi++] = ':';
else
user[ui++] = ':';
p += 2;
continue;
}
if (*p == ':') {
split_seen = true;
p++;
continue;
}
if (split_seen)
group[gi++] = *p;
else
user[ui++] = *p;
p++;
}
user[ui] = '\0';
group[gi] = '\0';
char* u = str_dup(user);
char* g = str_dup(group);
free(user);
free(group);
if (!u || !g) {
free(u);
free(g);
return -1;
}
*puser = u;
*pgroup = g;
return 0;
}
int identity_parse_chown(Config* config, const char* value) {
if (!config || !value || *value == '\0') {
log_message(LOG_LEVEL_ERROR, "--chown requires a value (USER:GROUP, USER, or :GROUP)");
return -1;
}
/* Reject more than one UNESCAPED colon (a name or group may not contain an
* unescaped ':' in the spec). The scan is escape-aware: a `\:` is a literal
* colon inside a name, not a field separator. */
int colons = 0;
bool saw_colon = false;
const char* p = value;
while (*p) {
if (*p == '\\' && p[1] == ':') {
p += 2;
continue;
}
if (*p == ':') {
colons++;
saw_colon = true;
}
p++;
}
if (colons > 1) {
log_message(LOG_LEVEL_ERROR, "--chown must have at most one ':' (got '%s')", value);
return -1;
}
char *user = NULL, *group = NULL;
if (identity_split_chown(value, &user, &group) != 0) {
log_message(LOG_LEVEL_ERROR, "memory allocation failed for --chown");
return -1;
}
int ret = 0;
if (!saw_colon) {
/* --chown=USER: owner only. */
if (*user == '\0') {
log_message(LOG_LEVEL_ERROR, "--chown requires a user or group (got '%s')", value);
ret = -1;
} else if (identity_resolve_token(user, false, &config->chown_uid) != 0) {
log_message(LOG_LEVEL_ERROR,
"--chown could not resolve user '%s' (use a name that exists "
"on the source, '*', or @N)",
value);
ret = -1;
} else {
config->chown_uid_set = true;
}
} else {
/* --chown=USER:GROUP, --chown=:GROUP, --chown=USER: */
if (*user != '\0') {
if (identity_resolve_token(user, false, &config->chown_uid) != 0) {
log_message(LOG_LEVEL_ERROR, "--chown could not resolve user '%s'", value);
ret = -1;
goto done;
}
config->chown_uid_set = true;
}
if (*group != '\0') {
if (identity_resolve_token(group, true, &config->chown_gid) != 0) {
log_message(LOG_LEVEL_ERROR, "--chown could not resolve group '%s'", value);
ret = -1;
goto done;
}
config->chown_gid_set = true;
}
if (!*user && !*group) {
log_message(LOG_LEVEL_ERROR, "--chown must set a user, a group, or both (got '%s')", value);
ret = -1;
}
}
done:
free(user);
free(group);
return ret;
}
/* uid_t/gid_t are unsigned and may hold a value wider than the signed int32 the
* wire (and the identity policy) uses. Reject such an id instead of truncating
* it to an out-of-range (possibly negative sentinel) value. */
static bool identity_id_fits_int32(unsigned long id) {
return id <= (unsigned long)INT32_MAX;
}
/* Resolve one --copy-as id token. A '*' token means the caller's current
* effective uid (user) or gid (group). Returns 0 on success. On failure sets
* *overflow when a '*' id was wider than int32 so the caller can log the
* specific message; otherwise the token was simply unresolvable. */
static int identity_resolve_copy_as_id(const char* token, bool is_group, int32_t* out,
bool* overflow) {
*overflow = false;
if (strcmp(token, "*") == 0) {
unsigned long current = is_group ? (unsigned long)getegid() : (unsigned long)geteuid();
if (!identity_id_fits_int32(current)) {
*overflow = true;
return -1;
}
*out = (int32_t)current;
return 0;
}
return identity_resolve_token(token, is_group, out);
}
int identity_parse_copy_as(Config* config, const char* value) {
if (!config || !value || *value == '\0') {
log_message(LOG_LEVEL_ERROR, "--copy-as requires USER[:GROUP]");
return -1;
}
/* --copy-as=USER[:GROUP] is the whole grammar: at most one field separator.
* (Unlike --chown there is no escaped-colon form; a name containing ':' is
* simply not expressible, and the extra colon is a clear parse error.) */
int colons = 0;
for (const char* p = value; *p; p++)
if (*p == ':')
colons++;
if (colons > 1) {
char* escaped = output_escape(value, false);
log_message(LOG_LEVEL_ERROR, "--copy-as must be USER[:GROUP] (got '%s')",
escaped ? escaped : "<allocation failed>");
free(escaped);
return -1;
}
char* spec = str_dup(value);
if (!spec) {
log_message(LOG_LEVEL_ERROR, "memory allocation failed for --copy-as");
return -1;
}
const char* user_token = spec;
const char* group_token = NULL;
char* colon = strchr(spec, ':');
if (colon) {
*colon = '\0';
group_token = colon + 1;
}
/* The spec is untrusted user input echoed back in error paths: escape it once
* (8-bit-safe) so a control byte cannot forge a log line. */
char* escaped_spec = output_escape(value, false);
const char* shown = escaped_spec ? escaped_spec : "<allocation failed>";
int ret = -1;
if (*user_token == '\0') {
log_message(LOG_LEVEL_ERROR, "--copy-as is missing the user (got '%s')", shown);
goto done;
}
bool overflow = false;
int32_t uid;
if (identity_resolve_copy_as_id(user_token, false, &uid, &overflow) != 0) {
if (overflow)
log_message(LOG_LEVEL_ERROR, "--copy-as: current user id %lu exceeds INT32_MAX",
(unsigned long)geteuid());
else
log_message(LOG_LEVEL_ERROR,
"--copy-as could not resolve user (use a name that exists on the "
"source, '*', or @N): %s",
shown);
goto done;
}
int32_t gid;
if (group_token) {
if (*group_token == '\0') {
log_message(LOG_LEVEL_ERROR, "--copy-as group is empty (got '%s')", shown);
goto done;
}
if (identity_resolve_copy_as_id(group_token, true, &gid, &overflow) != 0) {
if (overflow)
log_message(LOG_LEVEL_ERROR, "--copy-as: current group id %lu exceeds INT32_MAX",
(unsigned long)getegid());
else
log_message(LOG_LEVEL_ERROR, "--copy-as could not resolve group (got '%s')", shown);
goto done;
}
} else {
/* Group omitted: use the user's primary gid. A numeric id with no local
* passwd entry has no primary gid to look up, so fall back to gid == uid
* (the rsync-style numeric convention; documented divergence). */
struct passwd* pw = getpwuid((uid_t)uid);
if (pw) {
if (!identity_id_fits_int32((unsigned long)pw->pw_gid)) {
log_message(LOG_LEVEL_ERROR,
"--copy-as: primary group id %lu for the requested user exceeds INT32_MAX",
(unsigned long)pw->pw_gid);
goto done;
}
gid = (int32_t)pw->pw_gid;
} else {
gid = uid;
}
}
/* The group-default and gid==uid fallbacks must never store a negative
* (sentinel) value; the explicit numeric path is already capped by
* identity_resolve_token. */
if (uid < 0 || gid < 0) {
log_message(LOG_LEVEL_ERROR, "--copy-as resolved id does not fit in int32 (got '%s')", shown);
goto done;
}
config->copy_as_set = true;
config->copy_as_uid = uid;
config->copy_as_gid = gid;
/* Ownership application needs the metadata path (the source uid/gid must be
* transmitted); imply it exactly like --chown/--usermap/--groupmap. */
config->use_metadata = true;
ret = 0;
done:
free(escaped_spec);
free(spec);
return ret;
}
/* ---- Receiver-side ownership application ---- */
static bool identity_map_lookup(const IdentityMap* map, int count, int32_t source_id,
int32_t* out_to) {
for (int i = 0; i < count; i++) {
if (map[i].from == IDENTITY_MATCH_ANY || map[i].from == source_id) {
*out_to = map[i].to;
return true;
}
}
return false;
}
/* Resolve the target ownership from the negotiated policy against the entry's
* current stat. Shared by the fd (regular file) and no-follow (symlink) apply
* paths. Returns false when no side is to be changed. */
static bool identity_resolve_targets(const struct stat* st, int32_t source_uid, int32_t source_gid,
uid_t* out_uid, gid_t* out_gid) {
bool set_uid = false;
bool set_gid = false;
uid_t uid = 0;
gid_t gid = 0;
/* --copy-as (P7 Wave E) has the highest priority: it forces BOTH the owner
* and group of every written entry to the requested ids, beating usermap /
* groupmap / --chown / --numeric-ids and the best-effort name lookup. Only
* skip when the entry already carries exactly those ids. */
if (g_identity.copy_as_set) {
uid = (uid_t)g_identity.copy_as_uid;
gid = (gid_t)g_identity.copy_as_gid;
if (st->st_uid == uid && st->st_gid == gid)
return false;
*out_uid = uid;
*out_gid = gid;
return true;
}
int32_t target;
if (identity_map_lookup(g_identity.usermap, g_identity.usermap_count, source_uid, &target)) {
uid = target == IDENTITY_CURRENT ? geteuid() : (uid_t)target;
set_uid = true;
} else if (g_identity.chown_uid_set) {
uid = g_identity.chown_uid == IDENTITY_CURRENT ? geteuid() : (uid_t)g_identity.chown_uid;
set_uid = true;
} else if (g_identity.numeric_ids) {
uid = (uid_t)source_uid;
set_uid = true;
} else {
/* Best-effort name mapping against the receiver's own database: if the
* transmitted (numeric) id resolves to a name present on this machine,
* re-resolve it. On a shared-account host this is the identity operation;
* when the id has no name here, the user side is left alone. */
struct passwd* pw = getpwuid((uid_t)source_uid);
if (pw) {
const struct passwd* mapped = getpwnam(pw->pw_name);
if (mapped) {
uid = mapped->pw_uid;
set_uid = true;
}
}
}
if (identity_map_lookup(g_identity.groupmap, g_identity.groupmap_count, source_gid, &target)) {
gid = target == IDENTITY_CURRENT ? getegid() : (gid_t)target;
set_gid = true;
} else if (g_identity.chown_gid_set) {
gid = g_identity.chown_gid == IDENTITY_CURRENT ? getegid() : (gid_t)g_identity.chown_gid;
set_gid = true;
} else if (g_identity.numeric_ids) {
gid = (gid_t)source_gid;
set_gid = true;
} else {
struct group* gr = getgrgid((gid_t)source_gid);
if (gr) {
const struct group* mapped = getgrnam(gr->gr_name);
if (mapped) {
gid = mapped->gr_gid;
set_gid = true;
}
}
}
if (!set_uid && !set_gid)
return false;
/* An unset side keeps the file's current id so the other side can change. */
if (!set_uid)
uid = st->st_uid;
if (!set_gid)
gid = st->st_gid;
/* Only change ownership when the target differs (avoid needless syscalls and
* any chance of clearing setuid/setgid on an already-correct entry). */
if (st->st_uid == uid && st->st_gid == gid)
return false;
*out_uid = uid;
*out_gid = gid;
return true;
}
static void identity_log_chown_failure(const char* what, uid_t uid, gid_t gid) {
/* EPERM/EACCES are expected when the receiver is not privileged (e.g. the CI
* `nobody` user): warn and continue, never abort the transfer. Any other
* error (EIO/EROFS/ENOSPC/...) is a real failure and must not be silently
* downgraded to a warning.
*
* --copy-as is different: the whole point of the flag is that the target
* ownership is REQUIRED (the pre-flight gate already refused an unprivileged
* receiver). If the chown still fails with EPERM/EACCES (a capability-
* restricted root, root-squash, or a read-only mount) the run would be
* silently producing the WRONG ownership, so surface it at ERROR. The
* caller (identity_apply_ownership*) then reports the ENTRY as failed rather
* than as written, which becomes a FILE_SAVE_ERROR and fails the transfer
* (fail-fast) instead of reporting overall success with the wrong owner. */
if (errno == EPERM || errno == EACCES) {
if (identity_copy_as_active())
log_message(LOG_LEVEL_ERROR,
"could not apply --copy-as ownership on %s (uid=%ld gid=%ld): %s; "
"entry was written with the wrong owner",
what, (long)uid, (long)gid, strerror(errno));
else
log_message(LOG_LEVEL_WARNING,
"could not apply ownership (uid=%ld gid=%ld): %s; leaving as-is", (long)uid,
(long)gid, strerror(errno));
} else {
log_message(LOG_LEVEL_ERROR, "failed to apply ownership on %s (uid=%ld gid=%ld): %s", what,
(long)uid, (long)gid, strerror(errno));
}
}
bool identity_apply_ownership(int fd, int32_t source_uid, int32_t source_gid) {
/* Ownership application is OFF unless the client requested an identity flag.
* This is the controlled gate: a default (or plain -M) transfer never changes
* ownership, byte-for-byte preserving FastSync's existing behavior. --no-super
* additionally forbids it even when the receiver is root. */
if (!identity_active_enabled() || !privilege_super_permitted() || fd < 0)
return true;
struct stat st;
if (fstat(fd, &st) != 0)
return !identity_copy_as_active();
uid_t uid;
gid_t gid;
if (!identity_resolve_targets(&st, source_uid, source_gid, &uid, &gid))
return true;
if (fchown(fd, uid, gid) != 0) {
identity_log_chown_failure("file", uid, gid);
/* A required --copy-as ownership that did not land is a per-entry failure;
* every other policy stays best-effort (rsync parity). */
return !identity_copy_as_active();
}
return true;
}
bool identity_apply_ownership_link(int parent_fd, const char* leaf, int32_t source_uid,
int32_t source_gid) {
if (!identity_active_enabled() || !privilege_super_permitted() || parent_fd < 0 || !leaf)
return true;
struct stat st;
if (fstatat(parent_fd, leaf, &st, AT_SYMLINK_NOFOLLOW) != 0)
return !identity_copy_as_active();
uid_t uid;
gid_t gid;
if (!identity_resolve_targets(&st, source_uid, source_gid, &uid, &gid))
return true;
if (fchownat(parent_fd, leaf, uid, gid, AT_SYMLINK_NOFOLLOW) != 0) {
identity_log_chown_failure("no-follow entry", uid, gid);
return !identity_copy_as_active();
}
return true;
}
+133
View File
@@ -0,0 +1,133 @@
#ifndef IDENTITY_H
#define IDENTITY_H
#include "config.h"
#include <stdbool.h>
#include <stdint.h>
#include <sys/types.h>
/*
* Identity mapping: --numeric-ids / --usermap / --groupmap / --chown / --copy-as.
*
* FastSync transmits uid/gid numerically (int32 on the wire) and, by design,
* NEVER applies client-supplied ownership unless a user explicitly opts in with
* an identity flag below. This module is the controlled, opt-in,
* privilege-gated path for applying ownership on the receiver: the wire config
* is snapshotted once per connection via identity_set_active() and applied
* through an fd-relative fchown() in the receiver's metadata-restore path.
*
* Because only numeric ids cross the wire, name-based values are resolved to
* numbers at CLI parse time using the CLIENT (sender) machine's databases. On
* a shared-account source/destination this reproduces rsync's semantics; a
* genuinely different destination database is a documented divergence (see
* RSYNC_COMPAT.md).
*/
/* Parse one --usermap= / --groupmap= value (comma-separated FROM:TO rules,
* first match wins) into config->usermap / config->groupmap. is_group selects
* the group tables and name databases. Returns 0 on success, -1 on a
* malformed spec or an unresolvable name (never a silent no-op). */
int identity_parse_map(Config* config, const char* value, bool is_group);
/* Parse --chown=USER:GROUP. Supports USER:GROUP, USER (owner only), :GROUP
* (group only), '*' (current/root as appropriate) and numeric ids. Returns 0
* on success, -1 on a malformed spec / unresolvable name. */
int identity_parse_chown(Config* config, const char* value);
/* Parse --copy-as=USER[:GROUP] (P7 Wave E). USER is resolved with the same
* user-database rules as --chown (a name, @N/bare N numeric id, or '*' meaning
* the client's current euid); when ':GROUP' is present the group is resolved
* with the group database ('*' meaning the client's egid). When the group is
* omitted, the user's primary gid is used (getpwuid(uid)->pw_gid); if the
* resolved user is a numeric id with no local passwd entry, gid falls back to
* uid. On success sets copy_as_set/copy_as_uid/copy_as_gid and forces
* metadata transmission (ownership application needs the metadata path).
* Returns 0 on success, -1 on a malformed / empty / unresolvable spec (never a
* silent no-op). */
int identity_parse_copy_as(Config* config, const char* value);
/* True when a --copy-as request is active but the receiver is not permitted to
* perform the privileged ownership application it needs. This is the up-front
* refusal predicate: the server rejects the whole transfer at the config
* handshake rather than silently ignoring the requested ownership. It is a
* pure function of the config mode and the current effective uid (it does NOT
* read the active snapshot, so it is valid at the pre-STATUS_OK gate, before
* identity_set_active() has run). `super_mode` is the EFFECTIVE mode after any
* server-side policy veto. */
bool identity_copy_as_refused(const Config* config);
/* True when the CURRENT per-connection snapshot has a --copy-as active (i.e.
* identity_set_active() has run against a config with copy_as_set). The
* --fake-super owner replay consults this so a copy-as run never lets the
* recorded source owner overwrite the forced target owner. Reads the active
* snapshot, so call identity_set_active() first (the receiver does, before any
* write). */
bool identity_copy_as_active(void);
/* Receiver-side snapshot of the negotiated identity config. The server calls
* identity_set_active() once per connection (before any file write) using the
* config received over the wire; the snapshot is a deep copy so the caller may
* free its Config immediately. identity_clear_active() releases it.
*
* Returns true on success. On an allocation failure while deep-copying a
* requested usermap/groupmap it logs a LOG_LEVEL_ERROR, leaves the snapshot
* cleared (never a partial/wrong policy) and returns false; the caller must
* refuse the connection. */
bool identity_set_active(const Config* config);
void identity_clear_active(void);
/* True when any ownership-affecting identity option is present in the active
* snapshot. Ownership stays OFF ("do not apply") for every transfer that
* requests none of them, preserving FastSync's existing behavior. --super /
* --no-super alone does NOT enable ownership; an explicit identity flag
* (--numeric-ids / --chown / --usermap / --groupmap / --copy-as) is required. */
bool identity_active_enabled(void);
/* Pure, config-only predicate: true when the client requested ANY
* client-chosen ownership or super-user activity (--numeric-ids, --chown,
* --usermap/--groupmap, --copy-as, --fake-super, or an explicit --super). Used
* by the daemon module gate to decide whether a module's per-module opt-in is
* required; it never reads the per-connection snapshot. */
bool identity_ownership_requested(const Config* config);
/* Apply the negotiated ownership to an already-written file descriptor.
* source_uid/source_gid are the transmitted numeric ids. Resolution order:
* --copy-as (highest priority, forces both ids), then a matching
* usermap/groupmap rule, then --chown, then --numeric-ids (raw), then a
* best-effort name lookup on the receiver's own databases (skipped when the
* transmitted id has no name on this system). Only calls fchown() when the
* result differs from the current value.
*
* Returns false ONLY when an active --copy-as ownership application failed: its
* forced ownership is REQUIRED, so the caller must treat the entry as failed
* rather than reporting success with the wrong owner. For every other identity
* policy an fchown EPERM/EACCES is logged and ignored and true is returned
* (rsync parity: the transfer must not abort). A no-op when no identity policy
* is active returns true. */
bool identity_apply_ownership(int fd, int32_t source_uid, int32_t source_gid);
/* P7 Wave D: the no-follow (symlink) counterpart. Resolves the same
* usermap/groupmap/chown/numeric-ids/copy-as policy but applies it with
* fchownat(..., AT_SYMLINK_NOFOLLOW) so a symlink's own ownership is changed
* without ever dereferencing it. A no-op unless an identity flag is active.
* The return value follows identity_apply_ownership(): false only when an
* active --copy-as application failed. */
bool identity_apply_ownership_link(int parent_fd, const char* leaf, int32_t source_uid,
int32_t source_gid);
/* Receiver-side wire validation of the resolved identity fields. */
bool identity_wire_valid(const Config* config);
/* P7 Wave E receiver-side permission gate for super-user activities (ownership
* application and char/block device-node creation). `privilege_super_permitted`
* consults the per-connection snapshot (call identity_set_active() first);
* `privilege_super_mode_permitted` is the pure mode predicate and is what
* callers holding a Config use (the config-frame gate, file_receive). Both
* return false only for SUPER_MODE_OFF; SUPER_MODE_ON and SUPER_MODE_AUTO (the
* default) permit a confined attempt, matching FastSync's historical
* best-effort behavior where an unprivileged attempt is refused by the kernel
* and skipped. Neither EVER elevates privileges. */
bool privilege_super_permitted(void);
bool privilege_super_mode_permitted(SuperMode mode);
#endif
+77 -27
View File
@@ -3,7 +3,9 @@
#include <stdbool.h>
#include <stdarg.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <threads.h>
#include <time.h>
static const char* log_level_strings[] = {"DEBUG", "INFO", "WARN", "ERROR"};
@@ -15,6 +17,18 @@ static FILE* log_fp = NULL;
static _Thread_local bool eight_bit_output;
static LogStderrMode stderr_mode = LOG_STDERR_ERRORS;
/* Serializes access to log_fp and makes each emitted line atomic: the
* timestamp prefix, formatted body, and trailing newline are written as one
* critical section so concurrent threads cannot interleave partial lines.
* Initialized lazily (matching the protocol.c bw_mutex idiom) because logging
* can happen before main() installs any synchronization. */
static mtx_t log_mutex;
static once_flag log_mutex_once = ONCE_FLAG_INIT;
static void log_mutex_init(void) {
mtx_init(&log_mutex, mtx_plain);
}
void set_log_level(LogLevel level) {
current_log_level = level;
}
@@ -27,6 +41,10 @@ uint32_t get_log_debug_flags(void) {
return current_debug_flags;
}
bool log_debug_enabled(LogDebugFlag flag) {
return current_log_level <= LOG_LEVEL_DEBUG && (current_debug_flags & flag) != 0;
}
void set_log_info_flags(uint32_t flags) {
info_flags = flags;
info_flags_explicit = true;
@@ -37,7 +55,10 @@ uint32_t get_log_info_flags(void) {
}
void log_set_file(FILE* fp) {
call_once(&log_mutex_once, log_mutex_init);
mtx_lock(&log_mutex);
log_fp = fp;
mtx_unlock(&log_mutex);
}
void log_set_8_bit_output(bool enabled) {
@@ -56,13 +77,48 @@ LogStderrMode log_get_stderr_mode(void) {
return stderr_mode;
}
static inline void write_message(FILE* dest_io, LogLevel log_level, struct tm t, const char* format,
va_list args) {
fprintf(dest_io, "%04d-%02d-%02d %02d:%02d:%02d [%s]: ", t.tm_year + 1900, t.tm_mon + 1,
t.tm_mday, t.tm_hour, t.tm_min, t.tm_sec, log_level_strings[log_level]);
/* Format one complete log line (timestamp prefix + body + newline) into a
* freshly allocated buffer. This is pure CPU/malloc work and must happen
* OUTSIDE the log mutex: the mutex only guards the log_fp pointer, so a
* stalled stderr/stdout pipe cannot block every logging thread. Returns NULL
* on allocation/formatting failure. */
static char* format_log_line(LogLevel log_level, const struct tm* t, const char* format,
va_list args) {
char prefix[64];
int prefix_len = snprintf(
prefix, sizeof(prefix), "%04d-%02d-%02d %02d:%02d:%02d [%s]: ", t->tm_year + 1900,
t->tm_mon + 1, t->tm_mday, t->tm_hour, t->tm_min, t->tm_sec, log_level_strings[log_level]);
if (prefix_len < 0 || prefix_len >= (int)sizeof(prefix))
return NULL;
va_list copy;
va_copy(copy, args);
int body_len = vsnprintf(NULL, 0, format, copy);
va_end(copy);
if (body_len < 0)
return NULL;
size_t total = (size_t)prefix_len + (size_t)body_len;
char* line = malloc(total + 2); /* body bytes + '\n' + NUL */
if (!line)
return NULL;
memcpy(line, prefix, (size_t)prefix_len);
vsnprintf(line + prefix_len, (size_t)body_len + 1, format, args);
line[total] = '\n';
line[total + 1] = '\0';
return line;
}
vfprintf(dest_io, format, args);
fprintf(dest_io, "\n");
/* Write an already-formatted line to the console and, if configured, the log
* file. Only the log_fp pointer is read under the mutex (so log_set_file /
* config_delete cannot free it while it is in use); the single console fputs
* runs unlocked but is internally atomic per stdio stream. */
static void emit_log_line(FILE* console, const char* line) {
fputs(line, console);
call_once(&log_mutex_once, log_mutex_init);
mtx_lock(&log_mutex);
FILE* file = log_fp;
if (file)
fputs(line, file);
mtx_unlock(&log_mutex);
}
void log_message(LogLevel log_level, const char* format, ...) {
@@ -82,14 +138,12 @@ void log_message(LogLevel log_level, const char* format, ...) {
va_list args;
va_start(args, format);
write_message(dest_io, log_level, t, format, args);
char* line = format_log_line(log_level, &t, format, args);
va_end(args);
if (log_fp) {
va_start(args, format);
write_message(log_fp, log_level, t, format, args);
va_end(args);
}
if (!line)
return;
emit_log_line(dest_io, line);
free(line);
}
void log_debug_message(LogDebugFlag flag, const char* format, ...) {
@@ -103,14 +157,12 @@ void log_debug_message(LogDebugFlag flag, const char* format, ...) {
va_list args;
va_start(args, format);
write_message(stdout, LOG_LEVEL_DEBUG, t, format, args);
char* line = format_log_line(LOG_LEVEL_DEBUG, &t, format, args);
va_end(args);
if (log_fp) {
va_start(args, format);
write_message(log_fp, LOG_LEVEL_DEBUG, t, format, args);
va_end(args);
}
if (!line)
return;
emit_log_line(stdout, line);
free(line);
}
void log_info_message(LogInfoFlag flag, const char* format, ...) {
@@ -125,14 +177,12 @@ void log_info_message(LogInfoFlag flag, const char* format, ...) {
va_list args;
va_start(args, format);
write_message(stdout, LOG_LEVEL_INFO, t, format, args);
char* line = format_log_line(LOG_LEVEL_INFO, &t, format, args);
va_end(args);
if (log_fp) {
va_start(args, format);
write_message(log_fp, LOG_LEVEL_INFO, t, format, args);
va_end(args);
}
if (!line)
return;
emit_log_line(stdout, line);
free(line);
}
void log_perror(const char* context) {
+4
View File
@@ -29,6 +29,10 @@ void log_perror(const char* context);
void set_log_level(LogLevel level);
void set_log_debug_flags(uint32_t flags);
uint32_t get_log_debug_flags(void);
/* True when a log_debug_message() call with the same flag would actually emit:
* the debug log level is enabled AND the flag is selected. Hot paths use this
* to skip expensive message formatting/escaping when the line is filtered. */
bool log_debug_enabled(LogDebugFlag flag);
void log_debug_message(LogDebugFlag flag, const char* message, ...);
void set_log_info_flags(uint32_t flags);
uint32_t get_log_info_flags(void);
+162 -79
View File
@@ -1,5 +1,6 @@
#include "metadata.h"
#include "file.h"
#include "identity.h"
#include "log.h"
#include "protocol.h"
#include "utils.h"
@@ -69,41 +70,87 @@ void metadata_to_buf(char** buf, const FileMetadata* m) {
int64_t mtime_nsec = (int64_t)m->mtime_nsec;
memcpy(*buf, &mtime_nsec, sizeof(mtime_nsec));
*buf += sizeof(mtime_nsec);
int32_t atime_valid = m->atime_valid ? 1 : 0;
memcpy(*buf, &atime_valid, sizeof(atime_valid));
*buf += sizeof(atime_valid);
int64_t atime_sec = (int64_t)m->atime_sec;
memcpy(*buf, &atime_sec, sizeof(atime_sec));
*buf += sizeof(atime_sec);
int64_t atime_nsec = (int64_t)m->atime_nsec;
memcpy(*buf, &atime_nsec, sizeof(atime_nsec));
*buf += sizeof(atime_nsec);
int32_t crtime_valid = m->crtime_valid ? 1 : 0;
memcpy(*buf, &crtime_valid, sizeof(crtime_valid));
*buf += sizeof(crtime_valid);
int64_t crtime_sec = (int64_t)m->crtime_sec;
memcpy(*buf, &crtime_sec, sizeof(crtime_sec));
*buf += sizeof(crtime_sec);
int64_t crtime_nsec = (int64_t)m->crtime_nsec;
memcpy(*buf, &crtime_nsec, sizeof(crtime_nsec));
*buf += sizeof(crtime_nsec);
}
FileMetadata* metadata_from_buf(char** buf) {
FileMetadata* metadata_from_buf(const uint8_t* buf, size_t len) {
if (buf == NULL || len < sizeof(int32_t))
return NULL;
int32_t present;
memcpy(&present, *buf, sizeof(present));
*buf += sizeof(present);
if (present != 0 && present != 1)
memcpy(&present, buf, sizeof(present));
if (present != 1)
return NULL;
if (!present)
if (len < sizeof(int32_t) + FILE_METADATA_WIRE_SIZE)
return NULL;
const uint8_t* cursor = buf + sizeof(int32_t);
FileMetadata* m = protocol_alloc(sizeof(FileMetadata));
if (m == NULL)
return NULL;
int32_t mode;
memcpy(&mode, *buf, sizeof(mode));
*buf += sizeof(mode);
memcpy(&mode, cursor, sizeof(mode));
cursor += sizeof(mode);
m->mode = (mode_t)mode;
int32_t uid;
memcpy(&uid, *buf, sizeof(uid));
*buf += sizeof(uid);
memcpy(&uid, cursor, sizeof(uid));
cursor += sizeof(uid);
m->uid = (uid_t)uid;
int32_t gid;
memcpy(&gid, *buf, sizeof(gid));
*buf += sizeof(gid);
memcpy(&gid, cursor, sizeof(gid));
cursor += sizeof(gid);
m->gid = (gid_t)gid;
int64_t mtime_sec;
memcpy(&mtime_sec, *buf, sizeof(mtime_sec));
*buf += sizeof(mtime_sec);
memcpy(&mtime_sec, cursor, sizeof(mtime_sec));
cursor += sizeof(mtime_sec);
m->mtime_sec = (time_t)mtime_sec;
int64_t mtime_nsec;
memcpy(&mtime_nsec, *buf, sizeof(mtime_nsec));
*buf += sizeof(mtime_nsec);
memcpy(&mtime_nsec, cursor, sizeof(mtime_nsec));
cursor += sizeof(mtime_nsec);
m->mtime_nsec = (long)mtime_nsec;
if (present != 1 || mtime_nsec < 0 || mtime_nsec >= 1000000000LL || mode < 0 || uid < 0 ||
gid < 0) {
int32_t atime_valid;
memcpy(&atime_valid, cursor, sizeof(atime_valid));
cursor += sizeof(atime_valid);
int64_t atime_sec;
memcpy(&atime_sec, cursor, sizeof(atime_sec));
cursor += sizeof(atime_sec);
int64_t atime_nsec;
memcpy(&atime_nsec, cursor, sizeof(atime_nsec));
cursor += sizeof(atime_nsec);
int32_t crtime_valid;
memcpy(&crtime_valid, cursor, sizeof(crtime_valid));
cursor += sizeof(crtime_valid);
int64_t crtime_sec;
memcpy(&crtime_sec, cursor, sizeof(crtime_sec));
cursor += sizeof(crtime_sec);
int64_t crtime_nsec;
memcpy(&crtime_nsec, cursor, sizeof(crtime_nsec));
cursor += sizeof(crtime_nsec);
m->atime_valid = atime_valid != 0;
m->atime_sec = (time_t)atime_sec;
m->atime_nsec = (long)atime_nsec;
m->crtime_valid = crtime_valid != 0;
m->crtime_sec = (time_t)crtime_sec;
m->crtime_nsec = (long)crtime_nsec;
if (mtime_nsec < 0 || mtime_nsec >= 1000000000LL || mode < 0 || uid < 0 || gid < 0 ||
atime_valid < 0 || atime_valid > 1 || crtime_valid < 0 || crtime_valid > 1 ||
(atime_valid && (atime_nsec < 0 || atime_nsec >= 1000000000LL)) ||
(crtime_valid && (crtime_nsec < 0 || crtime_nsec >= 1000000000LL))) {
free(m);
return NULL;
}
@@ -112,21 +159,17 @@ FileMetadata* metadata_from_buf(char** buf) {
bool metadata_send(int file_descriptor, const FileMetadata* m) {
if (m == NULL) {
int32_t zero = 0;
return send_n_data(file_descriptor, &zero, sizeof(zero));
int32_t absent = 0;
return send_n_data(file_descriptor, &absent, sizeof(absent));
}
int32_t present = 1;
int32_t mode = (int32_t)m->mode;
int32_t uid = (int32_t)m->uid;
int32_t gid = (int32_t)m->gid;
int64_t mtime_sec = (int64_t)m->mtime_sec;
int64_t mtime_nsec = (int64_t)m->mtime_nsec;
return send_n_data(file_descriptor, &present, sizeof(present)) &&
send_n_data(file_descriptor, &mode, sizeof(mode)) &&
send_n_data(file_descriptor, &uid, sizeof(uid)) &&
send_n_data(file_descriptor, &gid, sizeof(gid)) &&
send_n_data(file_descriptor, &mtime_sec, sizeof(mtime_sec)) &&
send_n_data(file_descriptor, &mtime_nsec, sizeof(mtime_nsec));
/* One packed frame (protocol 2.20.0): the int32 present flag followed by the
fixed FILE_METADATA_WIRE_SIZE-byte field record. metadata_to_buf() emits
exactly that layout (present + fields), so build it once and write the
whole record in a single call instead of one frame per field. */
char packed[sizeof(int32_t) + FILE_METADATA_WIRE_SIZE];
char* cursor = packed;
metadata_to_buf(&cursor, m);
return send_n_data(file_descriptor, packed, sizeof(packed));
}
FileMetadata* metadata_receive(int file_descriptor, int* ok) {
@@ -146,58 +189,21 @@ FileMetadata* metadata_receive(int file_descriptor, int* ok) {
*ok = 0;
return NULL;
}
FileMetadata* m = protocol_alloc(sizeof(FileMetadata));
/* Rebuild the packed record metadata_from_buf() expects: the present flag we
just read, followed by exactly FILE_METADATA_WIRE_SIZE field bytes. */
char packed[sizeof(int32_t) + FILE_METADATA_WIRE_SIZE];
memcpy(packed, &present, sizeof(present));
if (!receive_n_data(file_descriptor, packed + sizeof(present), FILE_METADATA_WIRE_SIZE)) {
if (ok)
*ok = 0;
return NULL;
}
FileMetadata* m = metadata_from_buf((const uint8_t*)packed, sizeof(packed));
if (m == NULL) {
if (ok)
*ok = 0;
return NULL;
}
int32_t mode;
if (!receive_n_data(file_descriptor, &mode, sizeof(mode))) {
free(m);
if (ok)
*ok = 0;
return NULL;
}
m->mode = (mode_t)mode;
int32_t uid;
if (!receive_n_data(file_descriptor, &uid, sizeof(uid))) {
free(m);
if (ok)
*ok = 0;
return NULL;
}
m->uid = (uid_t)uid;
int32_t gid;
if (!receive_n_data(file_descriptor, &gid, sizeof(gid))) {
free(m);
if (ok)
*ok = 0;
return NULL;
}
m->gid = (gid_t)gid;
int64_t mtime_sec;
if (!receive_n_data(file_descriptor, &mtime_sec, sizeof(mtime_sec))) {
free(m);
if (ok)
*ok = 0;
return NULL;
}
m->mtime_sec = (time_t)mtime_sec;
int64_t mtime_nsec;
if (!receive_n_data(file_descriptor, &mtime_nsec, sizeof(mtime_nsec))) {
free(m);
if (ok)
*ok = 0;
return NULL;
}
m->mtime_nsec = (long)mtime_nsec;
if (mtime_nsec < 0 || mtime_nsec >= 1000000000LL || mode < 0 || uid < 0 || gid < 0) {
free(m);
if (ok)
*ok = 0;
return NULL;
}
if (ok)
*ok = 1;
return m;
@@ -231,6 +237,16 @@ void file_restore_metadata(const char* path, const FileMetadata* metadata,
times[0].tv_nsec = UTIME_OMIT;
times[1].tv_sec = metadata->mtime_sec;
times[1].tv_nsec = metadata->mtime_nsec;
if (metadata->atime_valid) {
times[0].tv_sec = metadata->atime_sec;
times[0].tv_nsec = metadata->atime_nsec;
}
if (metadata->crtime_valid) {
log_message(LOG_LEVEL_DEBUG,
"crtime (birth time) %lld.%09ld transmitted for %s but not applied: no portable "
"setter exists",
(long long)metadata->crtime_sec, metadata->crtime_nsec, path);
}
if (utimensat(AT_FDCWD, path, times, 0) != 0) {
char* escaped_path = output_escape(path, log_get_8_bit_output());
log_message(LOG_LEVEL_WARNING, "Failed to set timestamps on %s: %s",
@@ -239,6 +255,47 @@ void file_restore_metadata(const char* path, const FileMetadata* metadata,
}
}
bool file_restore_symlink_metadata(const char* path, const FileMetadata* metadata,
bool omit_link_times) {
if (path == NULL || metadata == NULL)
return !identity_copy_as_active();
char* leaf = NULL;
int parent_fd = file_open_secure_parent(path, &leaf, false);
if (parent_fd < 0)
return !identity_copy_as_active();
/* Ownership (only when the identity policy is active) via lchown semantics:
fchownat with AT_SYMLINK_NOFOLLOW never dereferences the link. A failed
REQUIRED --copy-as ownership marks the entry failed; every other policy is
best-effort. */
bool owned = identity_apply_ownership_link(parent_fd, leaf, (int32_t)metadata->uid,
(int32_t)metadata->gid);
/* Symlink mode: not settable on Linux (fchmodat AT_SYMLINK_NOFOLLOW returns
EOPNOTSUPP/ENOTSUP); attempt it for platforms that support it and quietly
ignore the unsupported case so the transfer never fails over it. */
mode_t link_mode = metadata->mode & 0777;
if (fchmodat(parent_fd, leaf, link_mode, AT_SYMLINK_NOFOLLOW) != 0 && errno != EOPNOTSUPP &&
errno != ENOTSUP && errno != ENOSYS) {
log_message(LOG_LEVEL_DEBUG, "Could not set symlink mode on %s: %s", path, strerror(errno));
}
if (!omit_link_times) {
struct timespec times[2] = {{.tv_sec = 0, .tv_nsec = UTIME_OMIT},
{.tv_sec = metadata->mtime_sec, .tv_nsec = metadata->mtime_nsec}};
if (metadata->atime_valid) {
times[0].tv_sec = metadata->atime_sec;
times[0].tv_nsec = metadata->atime_nsec;
}
if (utimensat(parent_fd, leaf, times, AT_SYMLINK_NOFOLLOW) != 0) {
char* escaped_path = output_escape(path, log_get_8_bit_output());
log_message(LOG_LEVEL_WARNING, "Failed to set symlink timestamps on %s: %s",
escaped_path ? escaped_path : "<allocation failed>", strerror(errno));
free(escaped_path);
}
}
close(parent_fd);
free(leaf);
return owned;
}
bool file_restore_metadata_fd(int fd, const FileMetadata* metadata, bool preserve_executability) {
if (fd < 0 || metadata == NULL)
return metadata == NULL;
@@ -249,9 +306,35 @@ bool file_restore_metadata_fd(int fd, const FileMetadata* metadata, bool preserv
mode_t safe_mode = metadata_mode(metadata, current.st_mode, preserve_executability);
if (fchmod(fd, safe_mode) != 0)
ok = false;
/* Client uid/gid values are deliberately not authoritative. */
/* Client uid/gid values are deliberately not authoritative UNLESS the client
explicitly opted in with an identity flag (--numeric-ids / --usermap /
--groupmap / --chown). identity_apply_ownership is the controlled,
privilege-gated path: it consults the negotiated policy, resolves the
target ids, and applies them via an fd-relative fchown() that is confined
to the just-written file (EPERM/EACCES are logged, never fatal) -- EXCEPT
for an active --copy-as, whose forced ownership is REQUIRED: a failure
marks this entry as failed instead of reporting a wrong-owner write as
success. With no identity flag set it is a no-op, so a default or plain -M
transfer keeps FastSync's existing behavior of never applying client
ownership. */
if (!identity_apply_ownership(fd, (int32_t)metadata->uid, (int32_t)metadata->gid))
ok = false;
struct timespec times[2] = {{.tv_sec = 0, .tv_nsec = UTIME_OMIT},
{.tv_sec = metadata->mtime_sec, .tv_nsec = metadata->mtime_nsec}};
if (metadata->atime_valid) {
times[0].tv_sec = metadata->atime_sec;
times[0].tv_nsec = metadata->atime_nsec;
}
/* --crtimes captures and transmits the source birth time, but there is no
* portable way to set a birth time (utimensat can only set atime/mtime), so
* the receiver deliberately does NOT apply it. This is explicit, honest
* unsupported-attribute handling: log a debug note and continue — never fail
* the transfer and never pretend the crtime was applied. */
if (metadata->crtime_valid) {
log_message(LOG_LEVEL_DEBUG,
"crtime (birth time) %lld.%09ld transmitted but not applied: no portable setter",
(long long)metadata->crtime_sec, metadata->crtime_nsec);
}
if (futimens(fd, times) != 0)
ok = false;
return ok;
+32 -3
View File
@@ -3,6 +3,7 @@
#include "file.h"
#include <stdbool.h>
#include <stddef.h>
#include <stdint.h>
#include <sys/stat.h>
#include <time.h>
@@ -15,6 +16,12 @@
* int32_t gid (was gid_t, platform-dependent)
* int64_t mtime_sec (was time_t, platform-dependent)
* int64_t mtime_nsec (was long, platform-dependent)
* int32_t atime_valid (-U/--atimes; protocol 2.12.0)
* int64_t atime_sec
* int64_t atime_nsec
* int32_t crtime_valid (-N/--crtimes; protocol 2.12.0)
* int64_t crtime_sec
* int64_t crtime_nsec
*
* Prior to 2.0.0 the wire format used the raw platform-dependent types,
* which broke compatiblity across different systems. All fields are now
@@ -23,16 +30,38 @@
/* Size of metadata fields on wire, excluding the int32_t `present` field that
* is always sent first. The total wire size for present metadata is
* sizeof(int32_t) + FILE_METADATA_WIRE_SIZE (32 bytes on most platforms). */
#define FILE_METADATA_WIRE_SIZE (sizeof(int32_t) * 3 + sizeof(int64_t) * 2)
* sizeof(int32_t) + FILE_METADATA_WIRE_SIZE (68 bytes on most platforms).
*
* metadata_send()/metadata_receive() (protocol 2.20.0) frame the metadata as a
* single packed record: one int32 present flag (0 = absent) followed, when
* present, by exactly FILE_METADATA_WIRE_SIZE bytes of field data. This is the
* same present+fields byte layout metadata_to_buf()/metadata_from_buf() use, so
* the wire metadata is now one frame instead of one frame per field. */
#define FILE_METADATA_WIRE_SIZE (sizeof(int32_t) * 5 + sizeof(int64_t) * 6)
void metadata_to_buf(char** buf, const FileMetadata* m);
FileMetadata* metadata_from_buf(char** buf);
/* Decode one packed metadata record (an int32 present flag followed, when
* present, by FILE_METADATA_WIRE_SIZE field bytes) from `buf`, which has `len`
* readable bytes. Every read is bounds-checked against `len`, so the function
* can never over-read the caller's buffer: a too-short record, an absent
* (present == 0) record and a malformed record all return NULL. A successful
* decode returns a heap-allocated FileMetadata owned by the caller. */
FileMetadata* metadata_from_buf(const uint8_t* buf, size_t len);
bool metadata_send(int file_descriptor, const FileMetadata* m);
FileMetadata* metadata_receive(int file_descriptor, int* ok);
void file_restore_metadata(const char* path, const FileMetadata* metadata,
bool preserve_executability);
bool file_restore_metadata_fd(int fd, const FileMetadata* metadata, bool preserve_executability);
/* P7 Wave D: apply a SYMLINK's own metadata using no-follow primitives only
* (utimensat/lchown/fchmodat with AT_SYMLINK_NOFOLLOW), confined fd-relative
* under the authorized root. `omit_link_times` (-J/--omit-link-times)
* suppresses the timestamps; the link's mode/ownership are still attempted
* (ownership stays gated by the identity policy and by default is not applied).
* A null metadata or an unfollowable parent is a harmless no-op. Returns false
* only when a REQUIRED --copy-as ownership application failed, so the caller can
* report the entry as failed instead of claiming a wrong-owner success. */
bool file_restore_symlink_metadata(const char* path, const FileMetadata* metadata,
bool omit_link_times);
/* Compare timestamps using rsync's whole-second modification window. */
bool metadata_mtime_matches(time_t left_sec, long left_nsec, time_t right_sec, long right_nsec,
+76
View File
@@ -0,0 +1,76 @@
#include "motd.h"
#include "protocol.h"
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
char* motd_read_file(const char* path) {
if (!path || path[0] == '\0')
return NULL;
FILE* fp = fopen(path, "rb");
if (!fp)
return NULL;
char* buffer = malloc(MOTD_MAX_BYTES + 1);
if (!buffer) {
fclose(fp);
return NULL;
}
/* fread stops at the bound; a larger file is truncated rather than read
* unbounded. ferror distinguishes a truncated read from an I/O failure. */
size_t total = fread(buffer, 1, MOTD_MAX_BYTES, fp);
if (ferror(fp)) {
free(buffer);
fclose(fp);
return NULL;
}
fclose(fp);
buffer[total] = '\0';
return buffer;
}
char* motd_render(const char* motd, bool eight_bit_output) {
if (!motd)
return NULL;
size_t length = strlen(motd);
if (length > (SIZE_MAX - 1) / 5)
return NULL;
char* rendered = malloc(length * 5 + 1);
if (!rendered)
return NULL;
size_t out = 0;
for (size_t i = 0; i < length; i++) {
unsigned char byte = (unsigned char)motd[i];
if (byte == '\n' || byte == '\t') {
rendered[out++] = (char)byte;
} else if ((byte >= 32 && byte <= 126) || (eight_bit_output && byte >= 128)) {
rendered[out++] = (char)byte;
} else {
rendered[out++] = '\\';
rendered[out++] = '#';
rendered[out++] = (char)('0' + ((byte >> 6) & 7));
rendered[out++] = (char)('0' + ((byte >> 3) & 7));
rendered[out++] = (char)('0' + (byte & 7));
}
}
rendered[out] = '\0';
return rendered;
}
bool motd_send(int file_descriptor, const char* motd) {
return send_str(file_descriptor, motd ? motd : "");
}
char* motd_receive(int file_descriptor) {
char* motd = receive_str(file_descriptor);
if (!motd)
return NULL;
/* Guard against a hostile/oversized peer: receive_str already bounded the
* frame at MAX_STRING_SIZE and consumed it, so discarding an over-bound
* body here keeps the stream framed while refusing to display it. */
if (strlen(motd) > MOTD_MAX_BYTES) {
free(motd);
return NULL;
}
return motd;
}
+54
View File
@@ -0,0 +1,54 @@
#ifndef MOTD_H
#define MOTD_H
#include <stdbool.h>
/* Daemon Message-Of-The-Day (Wave C).
*
* The daemon listener (fastsync-server --daemon) may advertise a `motd file`
* configured in its globals. When a client connects with a host::module/path
* destination and the module gate accepts the connection, the server sends the
* MOTD as a single string frame BEFORE any transfer data (rsync sends its MOTD
* as the first thing from the server at the start of a daemon connection).
* The client reads that frame right after the config/status handshake and
* displays it on stdout unless --no-motd was given.
*
* The MOTD is ordinary display text, never a secret, so it uses the normal
* (non-redacted) string primitive. The exchange is strictly server->client
* and happens on the daemon listener path only; the --stdio SSH path has no
* MOTD.
*
* No PROTOCOL_VERSION bump is involved: the frame is sent and read
* symmetrically by every 2.15.0 daemon build (the strict same-version
* handshake rejects any other version before the frame), so it cannot
* desynchronize a peer. */
/* Upper bound on the MOTD bytes the server will read from disk and put on the
* wire. Kept far below MAX_STRING_SIZE (64 KB) so a huge/hostile motd file
* can never produce an unbounded frame or allocation. */
#define MOTD_MAX_BYTES 4096
/* Read a daemon MOTD file, bounded to MOTD_MAX_BYTES. Returns a malloc'd
* NUL-terminated copy of the file content (bytes beyond the bound are
* truncated) or NULL when path is NULL/empty, the file cannot be opened or
* read, or allocation fails. An absent or unreadable motd file is NOT an
* error: the caller simply sends an empty MOTD frame and continues. */
char* motd_read_file(const char* path);
/* Render MOTD text for terminal display. Newlines and tabs are preserved so
* a multi-line motd still reads naturally, while every other non-printable /
* control byte (ESC included) is escaped with FastSync's `\NNN` octal
* convention, so a hostile server cannot inject terminal escape sequences
* through the MOTD. eight_bit_output keeps bytes >= 0x80 verbatim (matching
* --8-bit-output). Returns a malloc'd string or NULL on allocation failure. */
char* motd_render(const char* motd, bool eight_bit_output);
/* Send/receive the MOTD string frame. These wrap the normal string
* primitive: the MOTD is not a credential, so no redaction is used. The
* receiver additionally rejects an over-bound frame (> MOTD_MAX_BYTES) as a
* hostile input guard; the frame itself is always fully consumed first, so the
* stream stays framed. */
bool motd_send(int file_descriptor, const char* motd);
char* motd_receive(int file_descriptor);
#endif
+104 -226
View File
@@ -1,15 +1,16 @@
#include "multiprocessing.h"
#include "receiver.h"
#include "array_list.h"
#include "chunk.h"
#include "config.h"
#include "data.h"
#include "file.h"
#include "file_receive.h"
#include "log.h"
#include "protocol.h"
#include "queue.h"
#include "utils.h"
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
@@ -25,12 +26,15 @@ PipelineContextSender* pipeline_context_sender_create(Config* config, Queue* que
context->queue_loader = queue_loader;
context->scanner_done = false;
context->loader_done = false;
context->queued_bytes = 0;
context->max_queue_bytes = 0;
context->manifest = NULL;
context->excluded_paths = NULL;
context->missing_args = NULL;
context->scan_had_io_error = false;
context->remove_source_files = NULL;
context->early_delete = false;
context->scan_stopped_early = false;
context->total_files = 0;
context->progress_bytes = 0;
context->total_bytes = 0;
@@ -38,7 +42,14 @@ PipelineContextSender* pipeline_context_sender_create(Config* config, Queue* que
atomic_init(&context->cancelled, false);
protocol_session_init(&context->allocation_session, -1, -1);
protocol_session_set_max_alloc(&context->allocation_session, config->max_alloc);
context->dir_entries = NULL;
context->dir_entries_mutex_init = false;
int init = 0;
if (config->use_metadata) {
context->dir_entries = array_list_create(file_destroy);
if (!context->dir_entries)
goto fail;
}
if (mtx_init(&context->mutex_scanner, mtx_plain) != thrd_success)
goto fail;
init++;
@@ -61,10 +72,17 @@ PipelineContextSender* pipeline_context_sender_create(Config* config, Queue* que
goto fail;
// cppcheck-suppress unreadVariable
init++;
if (mtx_init(&context->dir_entries_mutex, mtx_plain) != thrd_success)
goto fail;
context->dir_entries_mutex_init = true;
return context;
fail:
log_perror("Error initializing synchronization objects");
if (context->dir_entries_mutex_init)
mtx_destroy(&context->dir_entries_mutex);
if (context->dir_entries)
array_list_delete(context->dir_entries);
if (init >= 6)
cnd_destroy(&context->condition_not_empty_loader);
if (init >= 5)
@@ -81,7 +99,88 @@ fail:
return NULL;
}
void pipeline_context_sender_set_queue_byte_limit(PipelineContextSender* context,
size_t max_bytes) {
if (context == NULL)
return;
mtx_lock(&context->mutex_loader);
context->max_queue_bytes = max_bytes;
context->queued_bytes = 0;
cnd_broadcast(&context->condition_not_full_loader);
mtx_unlock(&context->mutex_loader);
}
size_t pipeline_context_sender_chunk_bytes(const Chunk* chunk) {
if (chunk == NULL || chunk->items == NULL)
return 0;
size_t total = 0;
for (int i = 0; i < chunk->element_count; i++) {
const File* file = chunk->items[i];
if (file == NULL || file->data == NULL || file->data->data == NULL)
continue;
if (file->data->size > SIZE_MAX - total)
return SIZE_MAX;
total += file->data->size;
}
return total;
}
void pipeline_context_sender_note_bytes_released(PipelineContextSender* context,
size_t released_bytes) {
if (context == NULL || context->max_queue_bytes == 0 || released_bytes == 0)
return;
mtx_lock(&context->mutex_loader);
if (released_bytes >= context->queued_bytes)
context->queued_bytes = 0;
else
context->queued_bytes -= released_bytes;
cnd_signal(&context->condition_not_full_loader);
mtx_unlock(&context->mutex_loader);
}
bool pipeline_context_sender_enqueue_chunk(PipelineContextSender* context, Chunk* chunk) {
if (context == NULL || chunk == NULL)
return false;
size_t chunk_bytes = pipeline_context_sender_chunk_bytes(chunk);
mtx_lock(&context->mutex_loader);
while (!atomic_load(&context->cancelled)) {
bool blocked_by_count = queue_is_full(context->queue_loader);
bool blocked_by_budget = false;
if (context->max_queue_bytes > 0) {
size_t budget = context->max_queue_bytes;
size_t used = context->queued_bytes;
if (used >= budget) {
blocked_by_budget = true;
} else if (chunk_bytes > budget - used) {
/* A single payload larger than the whole budget is only admitted to an
empty pipeline so the wait can never deadlock. */
blocked_by_budget = used != 0;
}
}
if (!blocked_by_count && !blocked_by_budget)
break;
cnd_wait(&context->condition_not_full_loader, &context->mutex_loader);
}
if (atomic_load(&context->cancelled)) {
mtx_unlock(&context->mutex_loader);
chunk_destroy(chunk);
return false;
}
if (!queue_enqueue(context->queue_loader, chunk)) {
mtx_unlock(&context->mutex_loader);
chunk_destroy(chunk);
return false;
}
context->queued_bytes += chunk_bytes;
cnd_signal(&context->condition_not_empty_loader);
mtx_unlock(&context->mutex_loader);
return true;
}
void pipeline_context_sender_destroy(PipelineContextSender* context) {
/* `config` is borrowed: the caller retains ownership and frees it after the
pipeline has been destroyed (the worker threads are already joined, so no
config access can outlive this call). */
if (context->manifest) {
array_list_delete(context->manifest);
}
@@ -91,7 +190,10 @@ void pipeline_context_sender_destroy(PipelineContextSender* context) {
array_list_delete(context->missing_args);
if (context->remove_source_files)
array_list_delete(context->remove_source_files);
config_delete(context->config);
if (context->dir_entries)
array_list_delete(context->dir_entries);
if (context->dir_entries_mutex_init)
mtx_destroy(&context->dir_entries_mutex);
queue_destroy(context->queue_scanner);
queue_destroy(context->queue_loader);
mtx_destroy(&context->mutex_scanner);
@@ -103,227 +205,3 @@ void pipeline_context_sender_destroy(PipelineContextSender* context) {
mtx_destroy(&context->mutex_progress);
free(context);
}
PipelineContextReceiver* pipeline_context_receiver_create(Config* config, Queue* queue,
int file_descriptor, SSL* ssl) {
PipelineContextReceiver* context = malloc(sizeof(PipelineContextReceiver));
if (context == NULL)
return NULL;
context->config = config;
context->queue = queue;
context->file_descriptor = file_descriptor;
context->ssl = ssl;
context->outcomes.entries = NULL;
context->outcomes.count = 0;
context->outcomes.capacity = 0;
protocol_session_init(&context->session, file_descriptor, file_descriptor);
protocol_session_set_ssl(&context->session, ssl);
context->receiver_done = false;
context->queued_bytes = 0;
context->max_queue_bytes = 0;
context->deferred_manifest = NULL;
atomic_init(&context->cancelled, false);
int init = 0;
if (mtx_init(&context->mutex, mtx_plain) != thrd_success)
goto fail;
init++;
if (cnd_init(&context->condition_not_full) != thrd_success)
goto fail;
init++;
if (cnd_init(&context->condition_not_empty) != thrd_success)
goto fail;
// cppcheck-suppress unreadVariable
init++;
return context;
fail:
log_perror("Error initializing synchronization objects");
if (init >= 3)
cnd_destroy(&context->condition_not_empty);
if (init >= 2)
cnd_destroy(&context->condition_not_full);
if (init >= 1)
mtx_destroy(&context->mutex);
free(context);
return NULL;
}
void pipeline_context_receiver_destroy(PipelineContextReceiver* context) {
config_delete(context->config);
if (context->deferred_manifest)
delete_manifest_free(context->deferred_manifest);
queue_destroy(context->queue);
receiver_outcomes_destroy(&context->outcomes);
mtx_destroy(&context->mutex);
cnd_destroy(&context->condition_not_full);
cnd_destroy(&context->condition_not_empty);
free(context);
}
void pipeline_context_receiver_set_queue_byte_limit(PipelineContextReceiver* context,
size_t max_bytes) {
if (context == NULL)
return;
mtx_lock(&context->mutex);
context->max_queue_bytes = max_bytes;
context->queued_bytes = 0;
cnd_broadcast(&context->condition_not_full);
mtx_unlock(&context->mutex);
}
void pipeline_context_receiver_note_bytes_released(PipelineContextReceiver* context,
size_t released_bytes) {
if (context == NULL || context->max_queue_bytes == 0 || released_bytes == 0)
return;
mtx_lock(&context->mutex);
if (released_bytes >= context->queued_bytes)
context->queued_bytes = 0;
else
context->queued_bytes -= released_bytes;
cnd_signal(&context->condition_not_full);
mtx_unlock(&context->mutex);
}
bool pipeline_context_receiver_enqueue_file(PipelineContextReceiver* context, File* file) {
if (context == NULL || file == NULL)
return false;
size_t file_bytes = file->data ? file->data->size : 0;
mtx_lock(&context->mutex);
while (!atomic_load(&context->cancelled)) {
bool blocked_by_count = queue_is_full(context->queue);
bool blocked_by_budget = false;
if (context->max_queue_bytes > 0) {
size_t budget = context->max_queue_bytes;
size_t used = context->queued_bytes;
if (used >= budget) {
blocked_by_budget = true;
} else if (file_bytes > budget - used) {
/* A single payload larger than the whole budget (not possible with
the per-file receive cap) is only admitted to an empty pipeline so
the wait can never deadlock. */
blocked_by_budget = used != 0;
}
}
if (!blocked_by_count && !blocked_by_budget)
break;
cnd_wait(&context->condition_not_full, &context->mutex);
}
if (atomic_load(&context->cancelled)) {
mtx_unlock(&context->mutex);
file_destroy(file);
return false;
}
if (!queue_enqueue(context->queue, file)) {
mtx_unlock(&context->mutex);
file_destroy(file);
return false;
}
context->queued_bytes += file_bytes;
cnd_signal(&context->condition_not_empty);
mtx_unlock(&context->mutex);
return true;
}
static bool receiver_enqueue_file(File* file, void* context_pointer) {
PipelineContextReceiver* context = (PipelineContextReceiver*)context_pointer;
return pipeline_context_receiver_enqueue_file(context, file);
}
static void receiver_thread_fail(PipelineContextReceiver* context) {
mtx_lock(&context->mutex);
atomic_store(&context->cancelled, true);
context->receiver_done = true;
cnd_broadcast(&context->condition_not_empty);
cnd_broadcast(&context->condition_not_full);
mtx_unlock(&context->mutex);
}
int receive_thread(void* pipeline_context) {
PipelineContextReceiver* context = (PipelineContextReceiver*)pipeline_context;
protocol_session_bind(&context->session);
mtx_lock(&context->mutex);
int file_descriptor = context->file_descriptor;
const Config* config = context->config;
mtx_unlock(&context->mutex);
ReceiverSink sink = {receiver_enqueue_file, context, false, false, NULL};
if (receiver_process_pending((Config*)config, file_descriptor, &sink,
&context->deferred_manifest) != 0) {
receiver_thread_fail(context);
protocol_session_unbind();
return thrd_error;
}
mtx_lock(&context->mutex);
context->receiver_done = true;
cnd_signal(&context->condition_not_empty);
mtx_unlock(&context->mutex);
protocol_session_unbind();
return thrd_success;
}
int write_thread(void* pipeline_context) {
PipelineContextReceiver* context = (PipelineContextReceiver*)pipeline_context;
protocol_session_bind(&context->session);
mtx_lock(&context->mutex);
bool save_to_disk = context->config->save_to_disk;
char* root_directory = str_dup(context->config->receive_root_directory);
mtx_unlock(&context->mutex);
if (save_to_disk && !root_directory) {
mtx_lock(&context->mutex);
atomic_store(&context->cancelled, true);
context->receiver_done = true;
cnd_broadcast(&context->condition_not_full);
cnd_broadcast(&context->condition_not_empty);
mtx_unlock(&context->mutex);
protocol_session_unbind();
return thrd_error;
}
while (true) {
File* file =
queue_dequeue_multithreaded(context->queue, &context->mutex, &context->condition_not_empty,
&context->condition_not_full, &context->receiver_done);
if (file == NULL) {
free(root_directory);
protocol_session_unbind();
return thrd_success;
}
size_t file_bytes = file->data ? file->data->size : 0;
FileSaveResult result = FILE_SAVE_SKIPPED;
if (save_to_disk) {
result = file_save_to_disk_full(root_directory, file, context->config);
if (result == FILE_SAVE_ERROR) {
file_destroy(file);
pipeline_context_receiver_note_bytes_released(context, file_bytes);
mtx_lock(&context->mutex);
atomic_store(&context->cancelled, true);
context->receiver_done = true;
cnd_broadcast(&context->condition_not_full);
cnd_broadcast(&context->condition_not_empty);
mtx_unlock(&context->mutex);
free(root_directory);
protocol_session_unbind();
return thrd_error;
}
}
/* Record the per-file outcome so a --remove-source-files sender learns
which sources were actually written versus skipped on the receiver.
Explicit directory entries have no source and are never acknowledged. */
if (context->config->remove_source_files && !file->is_dir && !file->skip &&
!receiver_outcomes_append(&context->outcomes, (unsigned char)result)) {
file_destroy(file);
pipeline_context_receiver_note_bytes_released(context, file_bytes);
mtx_lock(&context->mutex);
atomic_store(&context->cancelled, true);
context->receiver_done = true;
cnd_broadcast(&context->condition_not_full);
cnd_broadcast(&context->condition_not_empty);
mtx_unlock(&context->mutex);
free(root_directory);
protocol_session_unbind();
return thrd_error;
}
file_destroy(file);
pipeline_context_receiver_note_bytes_released(context, file_bytes);
}
}
+46 -48
View File
@@ -5,11 +5,12 @@
#include <stdatomic.h>
#include "array_list.h"
#include "chunk.h"
#include "config.h"
#include "file.h"
#include "protocol.h"
#include "queue.h"
#include "receiver.h"
#include "stop_condition.h"
#include <openssl/ssl.h>
typedef struct {
@@ -24,6 +25,15 @@ typedef struct {
cnd_t condition_not_full_loader;
cnd_t condition_not_empty_loader;
bool loader_done;
/* Aggregate loaded payload bytes queued on queue_loader but not yet released
by the sender. Guarded by `mutex_loader`. When `max_queue_bytes` is
non-zero the loader blocks before enqueueing a chunk that would push this
total over it, so the sender buffers a bounded number of bytes rather than
an unbounded count of chunks that may each be up to chunk_size (or a single
file) in size. Files streamed straight from disk by sendfile hold no
payload, so only in-memory (`data->data`) payloads are counted. */
size_t queued_bytes;
size_t max_queue_bytes;
ArrayList* manifest;
/* Protected prefixes (paths the source scan excluded by user rules) sent
with the keep-set manifest so --delete leaves them alone unless
@@ -56,56 +66,44 @@ typedef struct {
bool sender_done;
atomic_bool cancelled;
ProtocolSession allocation_session;
/* Phase 6: client-only sender stop deadline, computed once before the worker
* threads start and shared read-only by the scanner and the sender thread. */
StopCondition stop_condition;
/* Phase 6: set when the scanner/sender reached the stop deadline before the
* scan (and thus the keep-set manifest) completed naturally. When true the
* completion tail must NOT transmit the partial manifest, or the receiver
* would delete unscanned source mirrors. Written by the sender thread
* before it reads the manifest, so no additional synchronization is needed
* to suppress the manifest. */
bool scan_stopped_early;
/* P7 Wave D: captured source directory times, filled by the scanner thread
* (and its parallel workers, guarded by dir_entries_mutex) and drained by the
* sender thread in trailing STATUS_DIR_TIMES frame(s). Owned by the
* context; NULL for non-metadata transfers. */
ArrayList* dir_entries;
mtx_t dir_entries_mutex;
bool dir_entries_mutex_init;
} PipelineContextSender;
typedef struct PipelineContextReceiver {
Queue* queue;
Config* config;
int file_descriptor;
SSL* ssl;
ProtocolSession session;
ReceiverOutcomes outcomes;
mtx_t mutex;
cnd_t condition_not_full;
cnd_t condition_not_empty;
bool receiver_done;
atomic_bool cancelled;
/* Aggregate payload bytes that have been received but not yet released by
the disk writer (queued or in the writer's hand). Guarded by `mutex`.
When `max_queue_bytes` is non-zero the receiver blocks before enqueuing
once this total would exceed it, so decompressed/copied file payloads
buffered ahead of a slow disk writer respect the per-connection memory
budget instead of growing without bound. */
size_t queued_bytes;
size_t max_queue_bytes;
/* Keep-set manifest for the commit-style (late) deletion
(--delete/--delete-after/--delete-delay). receive_thread parses the whole
protocol stream but hands the manifest here instead of deleting while the
disk writer may still be draining; the caller (server.c) commits the
deletion after both threads have joined, so no extra is removed unless the
transfer truly succeeded. NULL in the early delete modes (which delete at
the manifest). */
DeleteManifest* deferred_manifest;
} PipelineContextReceiver;
/* `config` is borrowed and must outlive the context: destroy does NOT free it,
so the caller owns it and frees it with config_delete() afterwards. */
PipelineContextSender* pipeline_context_sender_create(Config* config, Queue* queue_scanner,
Queue* queue_loader);
void pipeline_context_sender_destroy(PipelineContextSender* context);
PipelineContextReceiver* pipeline_context_receiver_create(Config* config, Queue* queue_receiver,
int file_descriptor, SSL* ssl);
void pipeline_context_receiver_destroy(PipelineContextReceiver* context);
/* Bound the bytes buffered ahead of the disk writer (see max_queue_bytes). */
void pipeline_context_receiver_set_queue_byte_limit(PipelineContextReceiver* context,
size_t max_bytes);
/* Blocking enqueue used by the receive pipeline sink. Blocks while the queue
is full by element count or when adding `file` would push queued_bytes over
the configured byte limit; waits until the disk writer releases bytes.
Takes ownership of `file` on success and destroys it on failure/cancel. */
bool pipeline_context_receiver_enqueue_file(PipelineContextReceiver* context, File* file);
/* Account for `released_bytes` of payload memory that has been freed by the
disk writer, unblocking a receiver that is waiting on the byte limit. */
void pipeline_context_receiver_note_bytes_released(PipelineContextReceiver* context,
size_t released_bytes);
int receive_thread(void* pipeline_context);
int write_thread(void* pipeline_context);
/* Bound the loaded payload bytes the sender may buffer ahead of the network
writer (see max_queue_bytes). */
void pipeline_context_sender_set_queue_byte_limit(PipelineContextSender* context, size_t max_bytes);
/* Total payload bytes a chunk currently holds in memory (loaded file data
only; zero for entries with no payload or data streamed from disk). */
size_t pipeline_context_sender_chunk_bytes(const Chunk* chunk);
/* Blocking enqueue used by the sender's loader stage. Blocks while
queue_loader is full by element count or when adding `chunk` would push the
queued payload bytes over the configured byte limit; waits until the sender
releases bytes. Takes ownership of `chunk` on success and destroys it on
failure/cancel. */
bool pipeline_context_sender_enqueue_chunk(PipelineContextSender* context, Chunk* chunk);
/* Account for `released_bytes` of payload memory that the sender freed after
destroying a chunk, unblocking a loader waiting on the byte limit. */
void pipeline_context_sender_note_bytes_released(PipelineContextSender* context,
size_t released_bytes);
#endif
+410 -28
View File
@@ -22,6 +22,10 @@ static __thread ProtocolSession* bound_session;
static __thread ProtocolSession legacy_io_session = {
.read_fd = -1, .write_fd = -1, .max_alloc = DEFAULT_MAX_ALLOC};
/* Last STATUS_ERROR_DETAIL reason received on this thread (protocol 2.21.0).
* Empty when the last status read carried no detail. */
static __thread char io_error_detail[MAX_ERROR_DETAIL_BYTES + 1];
static unsigned long long io_bwlimit = 0;
static mtx_t bw_mutex;
static once_flag bw_mutex_once = ONCE_FLAG_INIT;
@@ -40,7 +44,9 @@ static bool protocol_reserve_memory(ProtocolSession* session, size_t charge) {
}
}
static void protocol_release_memory_for_session(ProtocolSession* session, size_t charge) {
void protocol_release_memory_for_session(ProtocolSession* session, size_t charge) {
if (!session)
return;
unsigned long long allocated = atomic_load(&session->total_allocated_bytes);
while (true) {
unsigned long long remaining = (unsigned long long)charge >= allocated ? 0 : allocated - charge;
@@ -57,6 +63,10 @@ void io_set_fds(int read_fd, int write_fd) {
bound_session = NULL;
io_read_fd = read_fd;
io_write_fd = write_fd;
/* A descriptor switch starts a new connection on this thread: a stale
rejection detail captured from the previous transport must not leak into
the new one. */
io_error_detail[0] = '\0';
/* A descriptor switch starts a new transport; never reuse a TLS object
belonging to a previous connection or test pipe. */
io_ssl = NULL;
@@ -76,10 +86,23 @@ void protocol_session_init(ProtocolSession* session, int read_fd, int write_fd)
session->read_fd = read_fd;
session->write_fd = write_fd;
session->max_alloc = DEFAULT_MAX_ALLOC;
session->io_timeout_sec = RECEIVE_TIMEOUT_SEC;
atomic_init(&session->total_allocated_bytes, 0);
protocol_session_set_bwlimit(session, global_bwlimit());
}
void protocol_session_set_io_timeout(ProtocolSession* session, int sec) {
if (!session)
return;
session->io_timeout_sec = sec;
}
int protocol_get_io_timeout_sec(void) {
const ProtocolSession* session = bound_session ? bound_session : &legacy_io_session;
int sec = session->io_timeout_sec;
return sec > 0 ? sec : RECEIVE_TIMEOUT_SEC;
}
void protocol_session_set_max_alloc(ProtocolSession* session, unsigned long long max_alloc) {
if (!session)
session = bound_session ? bound_session : &legacy_io_session;
@@ -257,10 +280,11 @@ bool protocol_send_n_data(ProtocolSession* session, const void* data, size_t dat
log_debug_message(LOG_DEBUG_IO, " Sending n Data: %zu", data_size);
if (!session)
return false;
int timeout_sec = session->io_timeout_sec > 0 ? session->io_timeout_sec : SEND_TIMEOUT_SEC;
int fd = session->write_fd;
struct timespec deadline;
clock_gettime(CLOCK_MONOTONIC, &deadline);
deadline.tv_sec += SEND_TIMEOUT_SEC;
deadline.tv_sec += timeout_sec;
short wait_events = POLLOUT;
ssize_t total_bytes_send = 0;
while ((size_t)total_bytes_send < data_size) {
@@ -278,10 +302,14 @@ bool protocol_send_n_data(ProtocolSession* session, const void* data, size_t dat
if (pfd.revents & (POLLERR | POLLNVAL))
return false;
ssize_t bytes_send;
if (session->ssl)
bytes_send = SSL_write(session->ssl, (const char*)data + total_bytes_send, chunk);
else
if (session->ssl) {
/* SSL_write takes an int length; clamp a >INT_MAX request into chunks so
* the size_t downcast can never truncate into a negative/partial write. */
size_t ssl_chunk = chunk > (size_t)INT_MAX ? (size_t)INT_MAX : chunk;
bytes_send = SSL_write(session->ssl, (const char*)data + total_bytes_send, (int)ssl_chunk);
} else {
bytes_send = write(fd, (const char*)data + total_bytes_send, chunk);
}
if (bytes_send <= 0) {
if (session->ssl) {
int ssl_err = SSL_get_error(session->ssl, (int)bytes_send);
@@ -289,6 +317,12 @@ bool protocol_send_n_data(ProtocolSession* session, const void* data, size_t dat
wait_events = ssl_err == SSL_ERROR_WANT_WRITE ? POLLOUT : POLLIN;
continue;
}
/* A signal (e.g. Ctrl-C) interrupts the blocking TLS write: retry so
the send loop can observe the abort flag at the next checkpoint. */
if (ssl_err == SSL_ERROR_SYSCALL && errno == EINTR)
continue;
} else if (errno == EINTR) {
continue;
}
log_message(LOG_LEVEL_ERROR, "Could not send data");
return false;
@@ -306,30 +340,31 @@ bool protocol_receive_n_data_timed(ProtocolSession* session, void* data, size_t
int timeout_sec);
bool protocol_receive_n_data(ProtocolSession* session, void* data, size_t data_size) {
return protocol_receive_n_data_timed(session, data, data_size, RECEIVE_TIMEOUT_SEC);
/* Honor the session's configured deadline; protocol_receive_n_data_timed
* re-applies the built-in 60 s default when the value is <= 0. */
int timeout_sec = session ? session->io_timeout_sec : 0;
return protocol_receive_n_data_timed(session, data, data_size, timeout_sec);
}
bool protocol_receive_n_data_timed(ProtocolSession* session, void* data, size_t data_size,
int timeout_sec) {
/* Read exactly `data_size` bytes from `session` before `deadline` elapses
* (CLOCK_MONOTONIC). Shared by the ordinary timed primitive and the error-detail
* body reader so the latter can clamp itself to whatever deadline its caller
* already established instead of always applying the session's 60 s window. */
static bool protocol_receive_n_data_until(ProtocolSession* session, void* data, size_t data_size,
const struct timespec* deadline) {
log_debug_message(LOG_DEBUG_IO, " Receiving n Data: %zu", data_size);
if (!session)
if (!session || !deadline)
return false;
int fd = session->read_fd;
if (timeout_sec <= 0)
timeout_sec = RECEIVE_TIMEOUT_SEC;
struct timespec deadline;
clock_gettime(CLOCK_MONOTONIC, &deadline);
deadline.tv_sec += timeout_sec;
size_t total_bytes_received = 0;
short wait_events = POLLIN;
while (total_bytes_received < data_size) {
if (!session->ssl || SSL_pending(session->ssl) == 0) {
struct pollfd pfd = {.fd = fd, .events = wait_events};
int poll_result = poll(&pfd, 1, deadline_remaining_ms(&deadline));
int poll_result = poll(&pfd, 1, deadline_remaining_ms(deadline));
if (poll_result == 0) {
log_message(LOG_LEVEL_ERROR, "Receive timeout after %ds", timeout_sec);
log_message(LOG_LEVEL_ERROR, "Receive timeout");
return false;
}
if (poll_result < 0) {
@@ -356,6 +391,13 @@ bool protocol_receive_n_data_timed(ProtocolSession* session, void* data, size_t
wait_events = ssl_err == SSL_ERROR_WANT_WRITE ? POLLOUT : POLLIN;
continue;
}
/* A signal interrupts the blocking TLS read: retry (mirrors the send
path and protocol_read_status_until) so the loop reaches its next
abort/deadline checkpoint instead of failing spuriously. */
if (ssl_err == SSL_ERROR_SYSCALL && errno == EINTR)
continue;
} else if (errno == EINTR) {
continue;
}
if (bytes_received == 0)
log_message(LOG_LEVEL_ERROR, "Connection closed while receiving data");
@@ -371,6 +413,18 @@ bool protocol_receive_n_data_timed(ProtocolSession* session, void* data, size_t
return true;
}
bool protocol_receive_n_data_timed(ProtocolSession* session, void* data, size_t data_size,
int timeout_sec) {
if (!session)
return false;
if (timeout_sec <= 0)
timeout_sec = RECEIVE_TIMEOUT_SEC;
struct timespec deadline;
clock_gettime(CLOCK_MONOTONIC, &deadline);
deadline.tv_sec += timeout_sec;
return protocol_receive_n_data_until(session, data, data_size, &deadline);
}
static const char* status_to_string(Status status) {
switch (status) {
case STATUS_OK:
@@ -397,12 +451,45 @@ static const char* status_to_string(Status status) {
return "CHECK_BATCH";
case STATUS_MKDIR:
return "MKDIR";
case STATUS_APPEND:
return "APPEND";
case STATUS_APPEND_SIG:
return "APPEND_SIG";
case STATUS_APPEND_OK:
return "APPEND_OK";
case STATUS_APPEND_DATA:
return "APPEND_DATA";
case STATUS_HARDLINK:
return "HARDLINK";
case STATUS_SYMLINK:
return "SYMLINK";
case STATUS_SPECIAL:
return "SPECIAL";
case STATUS_DIR_TIMES:
return "DIR_TIMES";
case STATUS_AUTH_CHALLENGE:
return "AUTH_CHALLENGE";
case STATUS_AUTH_RESPONSE:
return "AUTH_RESPONSE";
case STATUS_AUTH_OK:
return "AUTH_OK";
case STATUS_AUTH_FAILED:
return "AUTH_FAILED";
case STATUS_ERROR_DETAIL:
return "ERROR_DETAIL";
case STATUS_DRY_RUN_TRANSFER:
return "DRY_RUN_TRANSFER";
default:
return "UNKNOWN";
}
}
bool protocol_send_str(ProtocolSession* session, const char* data) {
/* Shared string send/receive implementation. `redact` selects whether the
* payload body is written to the LOG_DEBUG_PROTO debug log: daemon auth material
* (the username and the proof/signature fields) sets it so a --verbose log never
* captures a replayable credential, while every other string keeps its normal
* debug trace. */
static bool protocol_send_str_impl(ProtocolSession* session, const char* data, bool redact) {
if (data == NULL)
return false;
size_t size = strlen(data);
@@ -410,11 +497,18 @@ bool protocol_send_str(ProtocolSession* session, const char* data) {
return false;
if (!protocol_send_n_data(session, data, size))
return false;
log_debug_message(LOG_DEBUG_PROTO, "Send String: %s", data);
if (redact) {
log_debug_message(LOG_DEBUG_PROTO, "Send String: <redacted>");
} else if (log_debug_enabled(LOG_DEBUG_PROTO)) {
char* escaped_data = output_escape(data, log_get_8_bit_output());
log_debug_message(LOG_DEBUG_PROTO, "Send String: %s",
escaped_data ? escaped_data : "<allocation failed>");
free(escaped_data);
}
return true;
}
char* protocol_receive_str(ProtocolSession* session) {
static char* protocol_receive_str_impl(ProtocolSession* session, bool redact) {
size_t size;
if (!protocol_receive_n_data(session, &size, sizeof(size_t)))
return NULL;
@@ -436,10 +530,33 @@ char* protocol_receive_str(ProtocolSession* session) {
return NULL;
}
data[size] = '\0';
log_debug_message(LOG_DEBUG_PROTO, "Received String: %s", data);
if (redact) {
log_debug_message(LOG_DEBUG_PROTO, "Received String: <redacted>");
} else if (log_debug_enabled(LOG_DEBUG_PROTO)) {
char* escaped_data = output_escape(data, log_get_8_bit_output());
log_debug_message(LOG_DEBUG_PROTO, "Received String: %s",
escaped_data ? escaped_data : "<allocation failed>");
free(escaped_data);
}
return data;
}
bool protocol_send_str(ProtocolSession* session, const char* data) {
return protocol_send_str_impl(session, data, false);
}
bool protocol_send_str_redacted(ProtocolSession* session, const char* data) {
return protocol_send_str_impl(session, data, true);
}
char* protocol_receive_str(ProtocolSession* session) {
return protocol_receive_str_impl(session, false);
}
char* protocol_receive_str_redacted(ProtocolSession* session) {
return protocol_receive_str_impl(session, true);
}
bool protocol_send_data(ProtocolSession* session, const Data* data) {
if (!data || (!data->data && data->size != 0))
return false;
@@ -491,13 +608,10 @@ Data* protocol_receive_data_limited(ProtocolSession* session, unsigned long long
return NULL;
}
result->protocol_charge = allocation_size;
result->owner = session;
return result;
}
Data* protocol_receive_data(ProtocolSession* session) {
return protocol_receive_data_limited(session, MAX_DATA_PAYLOAD_SIZE);
}
bool protocol_send_int(ProtocolSession* session, int data) {
if (!protocol_send_n_data(session, &data, sizeof(int)))
return false;
@@ -519,8 +633,82 @@ bool protocol_send_status(ProtocolSession* session, Status status) {
return true;
}
/* Read the bounded, length-prefixed body of a STATUS_ERROR_DETAIL frame within
* `deadline` (CLOCK_MONOTONIC), polling `abort_check` (may be NULL) between
* drain chunks. The declared length is validated BEFORE any allocation:
*
* - `size > MAX_STRING_SIZE`: an absurd framing error. Reading/draining that
* many bytes could never finish, so it is fatal (the caller tears the
* connection down) rather than drained.
* - `MAX_ERROR_DETAIL_BYTES < size <= MAX_STRING_SIZE`: drain exactly `size`
* bytes through a small fixed scratch buffer so the stream stays in sync,
* leaving the captured detail empty. No allocation happens.
* - `size <= MAX_ERROR_DETAIL_BYTES`: read straight into the thread-local
* `io_error_detail` buffer (size+1 capacity, already reserved), so the
* session's --max-alloc / MAX_CONNECTION_MEMORY budgets are never touched.
*
* Returns false on a fatal framing problem or any I/O failure; the terminal
* detail is then empty. The body is consumed on every non-fatal path even when
* the caller ignores protocol_last_error(), so the stream never desyncs. */
static bool protocol_receive_error_detail_until(ProtocolSession* session,
const struct timespec* deadline,
ProtocolWaitAbort abort_check) {
io_error_detail[0] = '\0';
size_t size = 0;
if (!protocol_receive_n_data_until(session, &size, sizeof(size), deadline))
return false;
if (size > MAX_STRING_SIZE) {
log_message(LOG_LEVEL_ERROR, "Error detail length %zu exceeds maximum %llu", size,
(unsigned long long)MAX_STRING_SIZE);
return false;
}
if (size > MAX_ERROR_DETAIL_BYTES) {
char scratch[256];
size_t remaining = size;
while (remaining > 0) {
if (abort_check && abort_check())
return false;
size_t chunk = remaining < sizeof(scratch) ? remaining : sizeof(scratch);
if (!protocol_receive_n_data_until(session, scratch, chunk, deadline))
return false;
remaining -= chunk;
}
return true;
}
if (!protocol_receive_n_data_until(session, io_error_detail, size, deadline))
return false;
io_error_detail[size] = '\0';
return true;
}
/* Consume the optional detail body of a STATUS_ERROR_DETAIL frame and map the
* status back to STATUS_ERROR for existing callers. Invoked for EVERY status
* read so a stale detail from an earlier exchange is never reported for a later
* one -- except for STATUS_KEEPALIVE, which carries no body and whose drain
* (protocol_receive_status_keepalive) must NOT erase the terminal detail that
* arrived just before it. Returns false on a fatal framing error. */
static bool protocol_capture_error_detail(ProtocolSession* session, Status* status,
const struct timespec* deadline,
ProtocolWaitAbort abort_check) {
if (*status == STATUS_KEEPALIVE)
return true;
io_error_detail[0] = '\0';
if (*status != STATUS_ERROR_DETAIL)
return true;
*status = STATUS_ERROR;
return protocol_receive_error_detail_until(session, deadline, abort_check);
}
bool protocol_receive_status(ProtocolSession* session, Status* status) {
if (!protocol_receive_n_data(session, status, sizeof(Status)))
if (!session || !status)
return false;
int timeout_sec = session->io_timeout_sec > 0 ? session->io_timeout_sec : RECEIVE_TIMEOUT_SEC;
struct timespec deadline;
clock_gettime(CLOCK_MONOTONIC, &deadline);
deadline.tv_sec += timeout_sec;
if (!protocol_receive_n_data_until(session, status, sizeof(Status), &deadline))
return false;
if (!protocol_capture_error_detail(session, status, &deadline, NULL))
return false;
log_debug_message(LOG_DEBUG_PROTO, "Received Status: %s", status_to_string(*status));
return true;
@@ -529,10 +717,169 @@ bool protocol_receive_status(ProtocolSession* session, Status* status) {
/* protocol_receive_status with an explicit per-message deadline (seconds).
Used where a single reply may legitimately take far longer than the default
60 s receive window - e.g. the sender waiting for the early-delete ACK after
the receiver committed a large (up to MAX_SERVER_DELETE_COUNT) deletion. */
the receiver committed a large (up to MAX_SERVER_DELETE_COUNT) deletion. The
error-detail body shares the same deadline as the status header. */
bool protocol_receive_status_timed(ProtocolSession* session, Status* status, int timeout_sec) {
if (!protocol_receive_n_data_timed(session, status, sizeof(Status), timeout_sec))
if (!session || !status)
return false;
if (timeout_sec <= 0)
timeout_sec = RECEIVE_TIMEOUT_SEC;
struct timespec deadline;
clock_gettime(CLOCK_MONOTONIC, &deadline);
deadline.tv_sec += timeout_sec;
if (!protocol_receive_n_data_until(session, status, sizeof(Status), &deadline))
return false;
if (!protocol_capture_error_detail(session, status, &deadline, NULL))
return false;
log_debug_message(LOG_DEBUG_PROTO, "Received Status: %s", status_to_string(*status));
return true;
}
/* Read exactly one Status frame within `deadline` (CLOCK_MONOTONIC). Unlike
* protocol_receive_status_keepalive this never emits a keepalive: it is used
* to consume the first byte(s) of an already-signalled frame and to drain the
* peer's outstanding keepalive replies, where injecting a write could split a
* reply across a frame boundary. Returns false on timeout/EOF/error. */
static bool protocol_read_status_until(ProtocolSession* session, Status* status,
const struct timespec* deadline) {
Status received = STATUS_ERROR;
size_t got = 0;
short wait_events = POLLIN;
while (got < sizeof(Status)) {
if (!session->ssl || SSL_pending(session->ssl) == 0) {
int remaining_ms = deadline_remaining_ms(deadline);
if (remaining_ms <= 0) {
log_message(LOG_LEVEL_ERROR, "Receive timeout while reading status");
return false;
}
struct pollfd pfd = {.fd = session->read_fd, .events = wait_events};
int poll_result = poll(&pfd, 1, remaining_ms);
if (poll_result == 0) {
log_message(LOG_LEVEL_ERROR, "Receive timeout while reading status");
return false;
}
if (poll_result < 0) {
if (errno == EINTR)
continue;
return false;
}
if (pfd.revents & (POLLERR | POLLNVAL))
return false;
}
ssize_t bytes_received;
if (session->ssl)
bytes_received = SSL_read(session->ssl, (char*)&received + got, sizeof(Status) - got);
else
bytes_received = read(session->read_fd, (char*)&received + got, sizeof(Status) - got);
if (bytes_received <= 0) {
if (session->ssl) {
int ssl_err = SSL_get_error(session->ssl, (int)bytes_received);
if (ssl_err == SSL_ERROR_WANT_READ || ssl_err == SSL_ERROR_WANT_WRITE) {
wait_events = ssl_err == SSL_ERROR_WANT_WRITE ? POLLOUT : POLLIN;
continue;
}
}
if (bytes_received < 0 && errno == EINTR)
continue;
log_message(LOG_LEVEL_ERROR, "Connection closed while receiving status");
return false;
}
got += (size_t)bytes_received;
}
*status = received;
return true;
}
bool protocol_receive_status_keepalive(ProtocolSession* session, Status* status, int timeout_sec,
int keepalive_interval_sec, ProtocolWaitAbort abort_check) {
if (!session || !status)
return false;
if (timeout_sec <= 0)
timeout_sec = RECEIVE_TIMEOUT_SEC;
if (keepalive_interval_sec <= 0)
keepalive_interval_sec = timeout_sec;
struct timespec deadline;
clock_gettime(CLOCK_MONOTONIC, &deadline);
deadline.tv_sec += timeout_sec;
unsigned long keepalives_sent = 0;
unsigned long replies_seen = 0;
Status final = STATUS_ERROR;
while (true) {
if (abort_check && abort_check())
return false;
if (!session->ssl || SSL_pending(session->ssl) == 0) {
int remaining_ms = deadline_remaining_ms(&deadline);
if (remaining_ms <= 0) {
log_message(LOG_LEVEL_ERROR, "Receive timeout after %ds", timeout_sec);
return false;
}
/* Only interleave a keepalive while waiting for the FIRST byte of a
* frame; once part of a frame is buffered a write could race the peer's
* reply into the middle of it. */
long long interval_ms_ll = (long long)keepalive_interval_sec * 1000LL;
int interval_ms = interval_ms_ll > INT_MAX ? INT_MAX : (int)interval_ms_ll;
int wait_ms = interval_ms < remaining_ms ? interval_ms : remaining_ms;
struct pollfd pfd = {.fd = session->read_fd, .events = POLLIN};
int poll_result = poll(&pfd, 1, wait_ms);
if (poll_result == 0) {
if (abort_check && abort_check())
return false;
if (!protocol_send_status(session, STATUS_KEEPALIVE))
return false;
keepalives_sent++;
continue;
}
if (poll_result < 0) {
if (errno == EINTR)
continue;
return false;
}
if (pfd.revents & (POLLERR | POLLNVAL))
return false;
}
Status received;
if (!protocol_read_status_until(session, &received, &deadline))
return false;
if (!protocol_capture_error_detail(session, &received, &deadline, abort_check))
return false;
if (received == STATUS_KEEPALIVE) {
/* The receiver's answer to one of our keepalives. */
replies_seen++;
continue;
}
final = received;
break;
}
/* Drain the replies the receiver still owes for keepalives we sent while it
* was busy. It answers them only after the real status, so leaving them
* unread would put stale KEEPALIVE frames ahead of the next exchange and
* desynchronize the protocol. */
if (replies_seen < keepalives_sent) {
/* A short separate grace, not the (possibly exhausted) main deadline: the
terminal status already arrived, so a peer that never answers its owed
keepalives must not turn a successful ack into a reported failure. */
struct timespec drain_deadline;
clock_gettime(CLOCK_MONOTONIC, &drain_deadline);
drain_deadline.tv_sec += 1;
while (replies_seen < keepalives_sent) {
Status drained;
if (!protocol_read_status_until(session, &drained, &drain_deadline)) {
log_message(LOG_LEVEL_WARNING, "peer did not answer %lu keepalive(s); continuing",
keepalives_sent - replies_seen);
break;
}
if (!protocol_capture_error_detail(session, &drained, &drain_deadline, abort_check))
return false;
if (drained != STATUS_KEEPALIVE) {
log_message(LOG_LEVEL_ERROR, "Unexpected status while draining keepalive replies");
return false;
}
replies_seen++;
}
}
*status = final;
log_debug_message(LOG_DEBUG_PROTO, "Received Status: %s", status_to_string(*status));
return true;
}
@@ -543,6 +890,15 @@ bool send_str(int fd, const char* data) {
char* receive_str(int fd) {
return protocol_receive_str(legacy_session(fd, -1));
}
/* Redacted variants: identical framing, but the string body is never written to
the debug protocol log. Used for daemon auth material (username, proof,
signature). */
bool send_str_redacted(int fd, const char* data) {
return protocol_send_str_redacted(legacy_session(-1, fd), data);
}
char* receive_str_redacted(int fd) {
return protocol_receive_str_redacted(legacy_session(fd, -1));
}
bool send_data(int fd, const Data* data) {
return protocol_send_data(legacy_session(-1, fd), data);
}
@@ -567,3 +923,29 @@ bool receive_status(int fd, Status* status) {
bool receive_status_timed(int fd, Status* status, int timeout_sec) {
return protocol_receive_status_timed(legacy_session(fd, -1), status, timeout_sec);
}
bool receive_status_keepalive(int fd, Status* status, int timeout_sec, int keepalive_interval_sec,
ProtocolWaitAbort abort_check) {
return protocol_receive_status_keepalive(legacy_session(fd, -1), status, timeout_sec,
keepalive_interval_sec, abort_check);
}
bool send_error_detail(int fd, const char* message) {
if (!message)
message = "";
char bounded[MAX_ERROR_DETAIL_BYTES + 1];
size_t len = strlen(message);
if (len > MAX_ERROR_DETAIL_BYTES) {
memcpy(bounded, message, MAX_ERROR_DETAIL_BYTES);
bounded[MAX_ERROR_DETAIL_BYTES] = '\0';
message = bounded;
}
return send_status(fd, STATUS_ERROR_DETAIL) && send_str(fd, message);
}
const char* protocol_last_error(void) {
return io_error_detail;
}
void protocol_clear_last_error(void) {
io_error_detail[0] = '\0';
}
+138 -2
View File
@@ -9,6 +9,12 @@
/* Maximum allowed string size for receive_str (64 KB) */
#define MAX_STRING_SIZE (64 * 1024)
/* Hard cap on the optional server->client rejection detail carried by
* STATUS_ERROR_DETAIL (protocol 2.21.0). A longer message is sliced to this
* many bytes before it is sent, so a peer can never be made to retain more than
* this for a rejection and the detail frame stays a small, fixed bound. */
#define MAX_ERROR_DETAIL_BYTES 4096
/* Maximum uncompressed file payload accepted by the receiver's whole-file
* paths. A single whole file is charged against the per-connection memory
* reservation (MAX_CONNECTION_MEMORY) and against the server allocation
@@ -52,6 +58,12 @@ typedef struct ProtocolSession {
atomic_ullong total_allocated_bytes;
bool eight_bit_output;
unsigned long long max_alloc;
/* Per-session deadline (seconds) applied to every protocol send/receive by
* protocol_send_n_data / protocol_receive_n_data. Defaults to the built-in
* 60 s window; a value <= 0 falls back to that default. Set from the
* negotiated Config->timeout so --timeout is honored by the poll()-driven
* protocol I/O, not just the socket SO_RCVTIMEO/SO_SNDTIMEO. */
int io_timeout_sec;
} ProtocolSession;
typedef int Status;
@@ -70,7 +82,80 @@ enum NET_STATUS {
STATUS_CHECK_BATCH,
/* An explicit directory entry (--dirs): the sender transmits only the path;
* the receiver creates the directory below the receive root. */
STATUS_MKDIR
STATUS_MKDIR,
/* --append / --append-verify tail resume. STATUS_APPEND is sent by the
* receiver after a per-file STATUS_CHECK when the existing destination file
* is SHORTER than the source and an append mode is negotiated: its payload is
* the resume offset (the number of prefix bytes already present), after which
* the sender answers either directly with STATUS_APPEND_DATA (plain --append,
* prefix not verified) or, for --append-verify, first with STATUS_APPEND_SIG
* carrying the xxHash64 of the source prefix; the receiver then replies
* STATUS_APPEND_OK (prefix matched -> sender transmits the tail) or
* STATUS_NEXT (prefix mismatch -> sender falls back to a full transfer).
* STATUS_APPEND_DATA carries the tail bytes (compressed data frame). */
STATUS_APPEND,
STATUS_APPEND_SIG,
STATUS_APPEND_OK,
STATUS_APPEND_DATA,
/* --hard-links/-H: a sibling (later member) of a source hard-link group.
* The sender transmits only the path, the run-local link-group id, and the
* first (data-carrying) member's destination-relative wire path; the receiver
* creates this entry as a hard link to the first member's installed file
* (falling back to a byte-identical copy if link() fails). Protocol 2.12.0. */
STATUS_HARDLINK,
/* A symlink-type entry (-l/--links, -k/--copy-dirlinks' keep-as-symlink
* branch). The sender transmits the destination path, the (sender-munged,
* if --munge-links) symlink target, and optional metadata; the receiver
* creates a symlink to the unmunged target beneath the receive root (see
* file_receive_symlink). Protocol 2.13.0. */
STATUS_SYMLINK,
/* --devices / --specials (-D): a device or special node the sender wants
* recreated (not written from content). Payload: destination path, the
* metadata frame (whose mode's S_IFMT bits carry the node kind), and two
* int32 rdev major/minor fields. The receiver validates the kind and rdev,
* confines the node below the receive root, and recreates it (mknod/mkfifo),
* privilege-gating the mknod. Protocol 2.13.0. */
STATUS_SPECIAL,
/* Directory-time superstructure (P7 Wave D, protocol 2.17.0): one or more
* trailing frames sent after all file data (and after the optional delete
* manifest) carrying the source directories' captured metadata so the
* receiver can apply directory mtimes/atimes AFTER all of a directory's
* children have been written. Payload per frame: an int count, then count
* repetitions of (wire path string, metadata frame); an entry count larger
* than MAX_MANIFEST_ENTRIES is split across repeated frames. The receiver
* defers the actual utimensat until its own delete/publish phase has
* committed, then skips the whole set when -O/--omit-dir-times is set. */
STATUS_DIR_TIMES,
/* Daemon SCRAM-SHA-256 authentication (A7 remediation, protocol 2.19.0).
* STATUS_AUTH_CHALLENGE: the server requires auth and is about to send the
* iteration count, the base64 salt and the base64 server nonce.
* STATUS_AUTH_RESPONSE: the client's reply, followed by the base64 client
* nonce and the base64 ClientProof. STATUS_AUTH_OK: the client proof
* verified, followed by the base64 ServerSignature. STATUS_AUTH_FAILED:
* a single generic refusal (unknown user, off-list user, wrong proof,
* missing/malformed credentials) after which the server closes without
* writing any data. */
STATUS_AUTH_CHALLENGE,
STATUS_AUTH_RESPONSE,
STATUS_AUTH_OK,
STATUS_AUTH_FAILED,
/* Optional server->client rejection detail (protocol 2.21.0). When the
* server refuses a transfer for a concrete reason it may send
* STATUS_ERROR_DETAIL followed by a length-prefixed, bounded string instead
* of a bare STATUS_ERROR. receive_status() consumes the string and maps the
* status back to STATUS_ERROR, so every pre-2.21 call site keeps working;
* callers that want the human-readable reason consult protocol_last_error().
* Appended immediately after STATUS_AUTH_FAILED so the existing wire values
* never move. */
STATUS_ERROR_DETAIL,
/* Server-contacting --dry-run (protocol 2.21.0). Sent by the receiver in
* response to a per-file STATUS_CHECK when the wire config carries
* dry_run=true and the file is NOT already up to date: it tells the sender
* the file WOULD be transferred, and the sender must NOT transmit any data
* (the receiver reads none in dry-run). STATUS_OK keeps its meaning in this
* path ("already up to date / nothing to do"). Appended after
* STATUS_ERROR_DETAIL so no existing status is renumbered. */
STATUS_DRY_RUN_TRANSFER
};
void io_set_fds(int read_fd, int write_fd);
@@ -85,6 +170,15 @@ void protocol_session_unbind(void);
void protocol_session_set_ssl(ProtocolSession* session, SSL* ssl);
void protocol_session_set_bwlimit(ProtocolSession* session, unsigned long long bytes_per_sec);
void protocol_session_set_max_alloc(ProtocolSession* session, unsigned long long max_alloc);
/* Override the per-message send/receive deadline for this session.
* `sec` <= 0 restores the built-in 60 s default (used for --timeout=0/unset).
* An explicit long deadline (e.g. the delete-ack wait) is applied per-call by
* protocol_receive_status_timed and is unaffected by this setter. */
void protocol_session_set_io_timeout(ProtocolSession* session, int sec);
/* Effective per-message I/O deadline (seconds) for the currently-bound session,
* falling back to the built-in default. Used by the plaintext sendfile path
* which bypasses the protocol send primitive. */
int protocol_get_io_timeout_sec(void);
void* protocol_alloc(size_t size);
void* protocol_realloc(void* ptr, size_t size);
void protocol_session_set_8_bit_output(ProtocolSession* session, bool enabled);
@@ -93,8 +187,14 @@ bool protocol_send_n_data(ProtocolSession* session, const void* data, size_t dat
bool protocol_receive_n_data(ProtocolSession* session, void* data, size_t data_size);
bool protocol_send_str(ProtocolSession* session, const char* data);
char* protocol_receive_str(ProtocolSession* session);
/* Redacted string variants: identical wire framing to protocol_send_str /
* protocol_receive_str, but the payload body is replaced by `<redacted>` in the
* LOG_DEBUG_PROTO debug log. Used for daemon auth material (the username and
* the proof/signature fields) so a --verbose log can never capture a credential
* that could be replayed. */
bool protocol_send_str_redacted(ProtocolSession* session, const char* data);
char* protocol_receive_str_redacted(ProtocolSession* session);
bool protocol_send_data(ProtocolSession* session, const Data* data);
Data* protocol_receive_data(ProtocolSession* session);
Data* protocol_receive_data_limited(ProtocolSession* session, unsigned long long maximum_size);
bool protocol_send_int(ProtocolSession* session, int data);
bool protocol_receive_int(ProtocolSession* session, int* data);
@@ -105,6 +205,9 @@ bool receive_n_data(int file_descriptor, void* data, size_t data_size);
bool send_str(int file_descriptor, const char* data);
char* receive_str(int file_descriptor);
/* Redacted fd-level string variants (see protocol_send_str_redacted). */
bool send_str_redacted(int file_descriptor, const char* data);
char* receive_str_redacted(int file_descriptor);
bool send_data(int file_descriptor, const Data* data);
Data* receive_data(int file_descriptor);
Data* receive_data_limited(int file_descriptor, unsigned long long maximum_size);
@@ -112,10 +215,43 @@ bool send_int(int file_descriptor, int data);
bool receive_int(int file_descriptor, int* data);
bool send_status(int file_descriptor, Status status);
bool receive_status(int file_descriptor, Status* status);
/* Send STATUS_ERROR_DETAIL followed by a bounded (<= MAX_ERROR_DETAIL_BYTES)
* length-prefixed string. Over-long messages are sliced and NULL is treated
* as "". Returns false if the status or the string could not be sent. */
bool send_error_detail(int file_descriptor, const char* message);
/* Human-readable reason captured from the most recent STATUS_ERROR_DETAIL
* received on this thread, or "" when the last status was a bare STATUS_ERROR
* (or no detail was seen). Thread-local, and valid until the next non-keepalive
* status read on the same thread; a later STATUS_KEEPALIVE does NOT clear it.
* The detail body is bounded by MAX_ERROR_DETAIL_BYTES: an over-cap declared
* length is drained and yields "" (so the stream never desyncs), while an
* absurd length is a fatal framing error that fails the status read. */
const char* protocol_last_error(void);
/* Clear the thread-local last-error buffer. */
void protocol_clear_last_error(void);
/* receive_status with an explicit per-message deadline in seconds, instead of
the default RECEIVE_TIMEOUT_SEC. A reply that may legitimately take longer
(e.g. the early-delete ACK after a large receiver-side deletion) must use
this so the sender does not abort after the deletion already committed. */
bool receive_status_timed(int file_descriptor, Status* status, int timeout_sec);
/* Callback polled by protocol_receive_status_keepalive once per keepalive
interval. Return true to stop waiting (e.g. a SIGINT/SIGTERM abort flag was
set). Kept as a function pointer so the protocol layer does not depend on
client signal state. */
typedef bool (*ProtocolWaitAbort)(void);
/* Like receive_status_timed, but while the peer is silent it emits
STATUS_KEEPALIVE every keepalive_interval_sec (the receiver answers each with
STATUS_KEEPALIVE, which this function consumes and skips) so a long
server-side operation does not look like a dead connection. The total wait
is still bounded by timeout_sec; abort_check (may be NULL) is polled every
interval and, when it returns true, ends the wait immediately with false.
Runs entirely on the calling thread: the protocol send path is NOT safe for
concurrent writers, so this must not be paired with a helper thread. */
bool receive_status_keepalive(int file_descriptor, Status* status, int timeout_sec,
int keepalive_interval_sec, ProtocolWaitAbort abort_check);
bool protocol_receive_status_keepalive(ProtocolSession* session, Status* status, int timeout_sec,
int keepalive_interval_sec, ProtocolWaitAbort abort_check);
#endif
+157
View File
@@ -0,0 +1,157 @@
#include "stop_condition.h"
#include <errno.h>
#include <limits.h>
#include <stdlib.h>
#include <string.h>
/* Parse a strictly positive decimal integer: only ASCII digits, no leading
* whitespace, sign or trailing garbage. */
static bool parse_positive_minutes(const char* value, long* out) {
if (!value || *value == '\0')
return false;
if (*value < '0' || *value > '9')
return false;
long v = 0;
for (const char* p = value; *p != '\0'; p++) {
if (*p < '0' || *p > '9')
return false;
int digit = *p - '0';
if (v > (LONG_MAX - digit) / 10)
return false;
v = v * 10 + digit;
}
if (v <= 0 || v > INT_MAX)
return false;
*out = v;
return true;
}
bool stop_parse_after_minutes(const char* value, int* out_minutes) {
if (!out_minutes)
return false;
long minutes = 0;
if (!parse_positive_minutes(value, &minutes))
return false;
*out_minutes = (int)minutes;
return true;
}
/* Two consecutive ASCII digits -> 0..99. */
static bool parse_two_digits(const char* s, int* out) {
if (s[0] < '0' || s[0] > '9' || s[1] < '0' || s[1] > '9')
return false;
*out = (s[0] - '0') * 10 + (s[1] - '0');
return true;
}
bool stop_parse_at_time(const char* value, time_t now, time_t* out_deadline) {
if (!value || !out_deadline)
return false;
/* now+N[smhd]: N whole units from the current wall clock. */
if (strncmp(value, "now+", 4) == 0) {
const char* p = value + 4;
/* The count must be a bare non-negative digit run: reject leading
whitespace ('now+ 5s') and a leading sign ('now++5s'). */
if (*p < '0' || *p > '9')
return false;
errno = 0;
char* end = NULL;
long amount = strtol(p, &end, 10);
if (errno != 0 || end == p || amount < 0)
return false;
long unit_seconds;
switch (*end) {
case 's':
unit_seconds = 1;
break;
case 'm':
unit_seconds = 60;
break;
case 'h':
unit_seconds = 3600;
break;
case 'd':
unit_seconds = 86400;
break;
default:
return false;
}
if (end[1] != '\0')
return false;
if (amount > LONG_MAX / unit_seconds)
return false;
long long delta = (long long)amount * unit_seconds;
/* Guard against signed overflow of now + delta. */
if ((long long)now > 0 && delta > (long long)LLONG_MAX - (long long)now)
return false;
if ((long long)now < 0 && delta < (long long)LLONG_MIN - (long long)now)
return false;
*out_deadline = now + (time_t)delta;
return true;
}
/* HH:MM or HH:MM:SS on the current local day. */
size_t len = strlen(value);
if (len != 5 && len != 8)
return false;
if (value[2] != ':' || (len == 8 && value[5] != ':'))
return false;
int hh, mm, ss = 0;
if (!parse_two_digits(value, &hh) || !parse_two_digits(value + 3, &mm))
return false;
if (len == 8 && !parse_two_digits(value + 6, &ss))
return false;
if (hh > 23 || mm > 59 || ss > 59)
return false;
struct tm today;
if (!localtime_r(&now, &today))
return false;
today.tm_hour = hh;
today.tm_min = mm;
today.tm_sec = ss;
today.tm_isdst = -1;
time_t deadline = mktime(&today);
if (deadline == (time_t)-1)
return false;
*out_deadline = deadline;
return true;
}
StopCondition stop_condition_make(bool has_after, int after_minutes, bool has_at, time_t at_time,
struct timespec now_mono) {
StopCondition condition;
condition.has_monotonic = false;
condition.monotonic_deadline.tv_sec = 0;
condition.monotonic_deadline.tv_nsec = 0;
condition.has_wall = false;
condition.wall_deadline = 0;
if (has_after && after_minutes > 0) {
condition.has_monotonic = true;
condition.monotonic_deadline.tv_sec = now_mono.tv_sec + (time_t)after_minutes * 60;
condition.monotonic_deadline.tv_nsec = now_mono.tv_nsec;
}
if (has_at) {
condition.has_wall = true;
condition.wall_deadline = at_time;
}
return condition;
}
bool stop_condition_reached(const StopCondition* condition) {
if (!condition)
return false;
if (condition->has_wall && time(NULL) >= condition->wall_deadline)
return true;
if (condition->has_monotonic) {
struct timespec now;
if (clock_gettime(CLOCK_MONOTONIC, &now) != 0)
return false;
if (now.tv_sec > condition->monotonic_deadline.tv_sec ||
(now.tv_sec == condition->monotonic_deadline.tv_sec &&
now.tv_nsec >= condition->monotonic_deadline.tv_nsec))
return true;
}
return false;
}
+46
View File
@@ -0,0 +1,46 @@
#ifndef STOP_CONDITION_H
#define STOP_CONDITION_H
#include <stdbool.h>
#include <time.h>
/* Client-only transfer stop conditions (--stop-after=MINS / --stop-at=TIME).
* Both are local sender-side deadlines: they are never serialized into the
* config frame and never bump PROTOCOL_VERSION. A transfer checks the
* condition at natural chunk/file boundaries and, once reached, stops
* elegantly (everything already sent is finalized normally, exit 0).
*
* A condition combines an optional CLOCK_MONOTONIC instant (the relative
* --stop-after duration, immune to wall-clock changes) with an optional
* wall-clock instant (the absolute --stop-at form). Either one being reached
* ends the transfer. */
typedef struct StopCondition {
bool has_monotonic;
struct timespec monotonic_deadline;
bool has_wall;
time_t wall_deadline;
} StopCondition;
/* Parse --stop-after=MINS: a positive integer count of minutes. Zero,
* negative, empty and non-numeric values are rejected. Returns true when
* accepted and stores the value in *out_minutes. */
bool stop_parse_after_minutes(const char* value, int* out_minutes);
/* Parse --stop-at=TIME. Accepted forms are HH:MM, HH:MM:SS and
* now+N[smhd] (seconds/minutes/hours/days from now). The absolute forms are
* resolved against `now` (local wall clock) and written to *out_deadline; a
* time already in the past yields a deadline <= now ("stop immediately").
* Returns false on any malformed value. */
bool stop_parse_at_time(const char* value, time_t now, time_t* out_deadline);
/* Build the runtime condition at transfer start. after_minutes is the
* relative --stop-after duration (<= 0 disables it); at_time is the absolute
* --stop-at deadline (only consulted when has_at is true); now_mono is the
* CLOCK_MONOTONIC reading at start. */
StopCondition stop_condition_make(bool has_after, int after_minutes, bool has_at, time_t at_time,
struct timespec now_mono);
/* True once either deadline has passed (wall clock first, then monotonic). */
bool stop_condition_reached(const StopCondition* condition);
#endif
+214 -37
View File
@@ -7,6 +7,7 @@
#include <stdlib.h>
#include <string.h>
#include <sys/socket.h>
#include <sys/time.h>
#include <sys/wait.h>
#include <unistd.h>
@@ -16,7 +17,9 @@ typedef struct {
char* remote_path;
} RemoteDest;
static void ssh_child_setup_failed(int status_fd) {
/* Writes the exec-failure marker and exits the child. Marked noreturn so
* static analyzers prove the caller's error path never falls through. */
__attribute__((noreturn)) static void ssh_child_setup_failed(int status_fd) {
ssize_t wret = write(status_fd, "x", 1);
(void)wret;
_exit(1);
@@ -72,27 +75,39 @@ static int parse_remote_dest(const char* dest, RemoteDest* r) {
memcpy(r->host, dest, host_len);
r->host[host_len] = '\0';
}
/* The user@host token is handed to ssh in option position. Reject anything
* that ssh would consume as an option (a leading '-') or an empty host, so a
* crafted destination can never inject an ssh option such as
* -oProxyCommand=... . This mirrors config_parse_ssh_dest's validation and
* is defense-in-depth for callers that bypass it. */
if (r->host[0] == '\0' || r->host[0] == '-' || r->user[0] == '-') {
remote_dest_destroy(r);
return -1;
}
return 0;
}
char* ssh_build_remote_command(const char* server_path, bool old_args) {
char* ssh_build_remote_command(const char* server_path, bool old_args, char* const* remote_options,
int remote_option_count) {
const char* path = server_path ? server_path : "fastsync-server";
const char* suffix = " --stdio";
/* Each --remote-option=OPT is appended after " --stdio" as one shell word,
escaped with the SAME single-quote boundary used for the server path, so a
value containing shell metacharacters (; & | ` $ ()) can never break out of
the quoting to inject an unrelated remote command. Values are already
validated at CLI parse time (non-empty, no control characters); this layer
only adds the escaping boundary. */
size_t path_len = strlen(path);
size_t suffix_len = strlen(suffix);
if (old_args) {
if (path_len > SIZE_MAX - suffix_len - 1)
return NULL;
char* command = malloc(path_len + suffix_len + 1);
if (!command)
return NULL;
memcpy(command, path, path_len);
memcpy(command + path_len, suffix, suffix_len + 1);
return command;
}
/* Quote the executable as one remote-shell word. This is the default safety boundary. */
/* The base command: the server path is ALWAYS quoted as one single-quoted
shell word (remote options below reuse the same escaping), then
" --stdio". Quoting the path is the only injection-safe construction: an
unquoted path would carry shell metacharacters straight into the remote
shell command. --old-args is kept for CLI/ABI compatibility but no longer
disables that protection. */
(void)old_args;
size_t quote_count = 0;
for (const char* p = path; *p; p++)
if (*p == '\'')
@@ -101,7 +116,34 @@ char* ssh_build_remote_command(const char* server_path, bool old_args) {
quote_count > (SIZE_MAX - path_len - suffix_len - 4) / 4)
return NULL;
size_t command_len = path_len + quote_count * 4 + suffix_len + 4;
char* command = malloc(command_len + 1);
/* Add each remote option, escaped as one single-quoted word:
" '<body>'", i.e. 1 leading space + 1 open quote + body (len + 3 per
embedded single quote) + 1 close quote = len + q*3 + 3 bytes.
Defense-in-depth against a non-conforming caller: never forward an empty
or control-character value, independent of the CLI validation. */
for (int i = 0; i < remote_option_count; i++) {
const char* opt = remote_options[i];
if (!opt || opt[0] == '\0')
return NULL;
size_t len = 0, q = 0;
for (const char* p = opt; *p; p++) {
/* Defense-in-depth: never forward a control character (newline/CR/etc.)
that could break the single-quoted shell word regardless of the remote
shell, independent of the CLI validation. */
if ((unsigned char)*p < 0x20 || (unsigned char)*p == 0x7f)
return NULL;
if (*p == '\'')
q++;
len++;
}
if (q > (SIZE_MAX - len) / 3 || len + q * 3 + 3 > SIZE_MAX - command_len)
return NULL;
command_len += len + q * 3 + 3;
}
command_len += 1; /* NUL */
char* command = malloc(command_len);
if (!command)
return NULL;
char* out = command;
@@ -116,11 +158,146 @@ char* ssh_build_remote_command(const char* server_path, bool old_args) {
}
*out++ = '\'';
memcpy(out, suffix, suffix_len + 1);
out += suffix_len;
for (int i = 0; i < remote_option_count; i++) {
const char* opt = remote_options[i];
*out++ = ' ';
*out++ = '\'';
for (const char* p = opt; *p; p++) {
if (*p == '\'') {
memcpy(out, "'\\''", 4);
out += 4;
} else {
*out++ = *p;
}
}
*out++ = '\'';
}
*out = '\0';
return command;
}
/* A heap-owned, NULL-terminated argv whose every string is separately malloc'd
* (str_dup'd) so a caller can free arbitrary slots, including argv[0]. */
char** ssh_build_client_argv(const char* rsh_command, int port, const char* userhost,
const char* remote_command) {
const char* rsh = (rsh_command && *rsh_command) ? rsh_command : "ssh";
/* Whitespace-split the remote-shell command into the leading argv words so
* "-e 'ssh -p 2222'" (or "--rsh=ssh -p 2222") works like rsync's rsh. A
* blank command falls back to the default "ssh". */
char* copy = str_dup(rsh);
if (!copy)
return NULL;
char* save = NULL;
int nwords = 0;
char** words = NULL;
for (char* tok = strtok_r(copy, " \t", &save); tok; tok = strtok_r(NULL, " \t", &save)) {
char** grown = realloc(words, (size_t)(nwords + 1) * sizeof(char*));
if (!grown) {
for (int i = 0; i < nwords; i++)
free(words[i]);
free(words);
free(copy);
return NULL;
}
words = grown;
words[nwords] = str_dup(tok);
if (!words[nwords]) {
for (int i = 0; i < nwords; i++)
free(words[i]);
free(words);
free(copy);
return NULL;
}
nwords++;
}
free(copy);
if (nwords == 0) {
words = malloc(sizeof(char*));
if (!words)
return NULL;
words[0] = str_dup("ssh");
if (!words[0]) {
free(words);
return NULL;
}
nwords = 1;
}
/* Fixed tail: three -o pairs (6) + optional -p/value (2) + the "--" end of
* options marker + user@host + remote command + terminating NULL. */
int port_extra = (port > 0 && port != 22) ? 2 : 0;
size_t total = (size_t)nwords + 6 + (size_t)port_extra + 4;
char** argv = calloc(total, sizeof(char*));
if (!argv) {
for (int i = 0; i < nwords; i++)
free(words[i]);
free(words);
return NULL;
}
int ac = 0;
for (int i = 0; i < nwords; i++)
argv[ac++] = words[i];
free(words);
char* tail[] = {"-o", "Compression=no",
"-o", "ControlMaster=auto",
"-o", "ControlPath=~/.cache/fastsync-%r@%h:%p"};
for (size_t i = 0; i < sizeof(tail) / sizeof(tail[0]); i++) {
argv[ac] = str_dup(tail[i]);
if (!argv[ac])
goto fail_argv;
ac++;
}
if (port_extra) {
char port_str[16];
snprintf(port_str, sizeof(port_str), "%d", port);
argv[ac] = str_dup("-p");
if (!argv[ac])
goto fail_argv;
ac++;
argv[ac] = str_dup(port_str);
if (!argv[ac])
goto fail_argv;
ac++;
}
/* End of options: guarantees the user@host token that follows is treated as
* the destination and never re-interpreted as an ssh option, even if every
* caller-side validation were bypassed. */
argv[ac] = str_dup("--");
if (!argv[ac])
goto fail_argv;
ac++;
argv[ac] = str_dup(userhost);
if (!argv[ac])
goto fail_argv;
ac++;
argv[ac] = str_dup(remote_command);
if (!argv[ac])
goto fail_argv;
ac++;
argv[ac] = NULL;
return argv;
fail_argv:
for (int i = 0; i < ac; i++)
free(argv[i]);
free(argv);
return NULL;
}
void ssh_free_client_argv(char** argv) {
if (!argv)
return;
for (int i = 0; argv[i]; i++)
free(argv[i]);
free(argv);
}
Client* client_connect_ssh(const char* destination, int port, const char* server_path,
bool old_args) {
bool old_args, const char* rsh_command, bool blocking_io,
char* const* remote_options, int remote_option_count) {
RemoteDest r;
if (parse_remote_dest(destination, &r) != 0) {
char* escaped = output_escape(destination, false);
@@ -142,6 +319,17 @@ Client* client_connect_ssh(const char* destination, int port, const char* server
setsockopt(sv[1], SOL_SOCKET, SO_SNDBUF, &buf_size, sizeof(buf_size));
setsockopt(sv[1], SOL_SOCKET, SO_RCVBUF, &buf_size, sizeof(buf_size));
/* By default the SSH transport socket gets the same read/write timeout as
* the TCP transport so a wedged remote shell cannot hang forever. With
* --blocking-io the timeouts are skipped and the socket blocks naturally. */
if (!blocking_io) {
struct timeval tv;
tv.tv_sec = tcp_get_timeout_sec();
tv.tv_usec = 0;
setsockopt(sv[0], SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof(tv));
setsockopt(sv[0], SOL_SOCKET, SO_SNDTIMEO, &tv, sizeof(tv));
}
int exec_pipe[2];
if (pipe(exec_pipe) < 0) {
log_perror("pipe failed");
@@ -187,29 +375,18 @@ Client* client_connect_ssh(const char* destination, int port, const char* server
else
snprintf(ssh_user, ssh_user_len, "%s", r.host);
char* ssh_argv[16];
int ac = 0;
char port_str[16];
char* remote_command = ssh_build_remote_command(server_path, old_args);
char* remote_command =
ssh_build_remote_command(server_path, old_args, remote_options, remote_option_count);
if (!remote_command)
ssh_child_setup_failed(exec_pipe[1]);
ssh_argv[ac++] = "ssh";
ssh_argv[ac++] = "-o";
ssh_argv[ac++] = "Compression=no";
ssh_argv[ac++] = "-o";
ssh_argv[ac++] = "ControlMaster=auto";
ssh_argv[ac++] = "-o";
ssh_argv[ac++] = "ControlPath=~/.cache/fastsync-%r@%h:%p";
if (port > 0 && port != 22) {
ssh_argv[ac++] = "-p";
snprintf(port_str, sizeof(port_str), "%d", port);
ssh_argv[ac++] = port_str;
}
ssh_argv[ac++] = ssh_user;
ssh_argv[ac++] = remote_command;
ssh_argv[ac] = NULL;
execvp("ssh", ssh_argv);
log_perror("exec of ssh failed");
char** ssh_argv = ssh_build_client_argv(rsh_command, port, ssh_user, remote_command);
free(ssh_user);
free(remote_command);
if (!ssh_argv)
ssh_child_setup_failed(exec_pipe[1]);
execvp(ssh_argv[0], ssh_argv);
log_perror("exec of remote shell failed");
ssh_free_client_argv(ssh_argv);
ssh_child_setup_failed(exec_pipe[1]);
}
Loaded 100 of 181 files, more files were not shown because too many files have changed in this diff. Show more