178 Commits
Author SHA1 Message Date
TapTap a2ce0a8e41 docs: mark --checksum-choice and --checksum-seed implemented
CI / lint (pull_request) Failing after 35s
CI / build-and-test (pull_request) Skipped
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
Precise notes for both rows: supported algorithms (xxh64/xxhash, md5 via EVP),
rejection of unsupported names, seed semantics (full 64-bit for whole-file,
low 32 bits for the delta block hash, ignored by md5), the --cc alias, the
length-prefixed bounded handshake digest, the 2.9.0 -> 2.10.0 protocol bump,
default byte-for-byte preservation, and the divergence from rsync's randomized
seed (FastSync defaults to seed 0). Summary count line is intentionally untouched.
2026-09-07 17:43:00 +02:00
TapTap df250ec9c9 test: coverage for --checksum-choice/--cc and --checksum-seed
Unit: digest selection with known vectors (xxh64 seed 0/empty, md5 RFC vectors,
seed changes xxh64 but not md5, name mapping, bounded buffer rejection), CLI
parse (all spellings, = forms, --cc alias, unsupported rejected, seed strict
decimal), config wire round-trip and out-of-range algo rejection, and seeded
delta signature/compute symmetry (matching seed rebuilds, mismatched seed yields
no block matches). Integration: unchanged skip + same-size/mtime redetect for
xxh64 and md5 with and without -m, near-zero-data skip run, deterministic seed,
and a byte-exact seeded delta run.
2026-09-07 17:42:53 +02:00
TapTap 0afda6b094 feat(checksum): --checksum-choice/--cc and --checksum-seed for whole-file digest
Adds real algorithm selection (xxh64 default, plus md5 via OpenSSL EVP) and a
64-bit seed for the per-file whole-file digest used by the --incremental/
--checksum handshake and basis-dir content verification. The seed also feeds
the delta path's per-block xxHash32 strong checksum (low 32 bits) so an
explicit seed deterministically changes those digests too. Sender and receiver
hash identically: the algorithm id and seed cross the config wire frame and the
STATUS_CHECK handshake now carries a length-prefixed, bounded digest instead of
a fixed 64-bit value. Unsupported algorithm names are rejected at parse time
(never a silent no-op). PROTOCOL_VERSION bumped 2.9.0 -> 2.10.0; defaults
(xxh64, seed 0) preserve prior byte-for-byte behavior.
2026-09-07 17:42:47 +02:00
TapTap 6701c103cb docs: recount RSYNC_COMPAT summary after Phase-3 wave B
CI / lint (push) Successful in 38s
CI / sanitizers (address) (push) Successful in 51s
CI / sanitizers (undefined) (push) Successful in 49s
CI / fuzz-build (push) Successful in 19s
CI / coverage (push) Successful in 43s
CI / valgrind (push) Successful in 37s
CI / build-and-test (push) Successful in 12m53s
2026-09-07 00:06:41 +02:00
TapTap ebab335488 Merge feat/p3-fuzzy: --fuzzy/-y/--no-fuzzy similar-file delta basis 2026-09-06 23:52:24 +02:00
TapTap ef13a70a29 Merge feat/p3-delete-policy: delete policy (--delete-excluded/--max-delete/--ignore-errors/--force/--prune-empty-dirs) 2026-09-06 23:52:00 +02:00
TapTap 4f19f5bfe7 fix: satisfy cppcheck on the hard-bound walker test
CI / lint (pull_request) Successful in 31s
CI / sanitizers (address) (pull_request) Successful in 52s
CI / sanitizers (undefined) (pull_request) Successful in 51s
CI / fuzz-build (pull_request) Successful in 17s
CI / coverage (pull_request) Successful in 42s
CI / valgrind (pull_request) Successful in 36s
CI / build-and-test (pull_request) Successful in 12m49s
Drop the derived 'created = rootfd >= 0' that cppcheck flagged as always true
after the EXPECT_TRUE guard.
2026-09-06 23:33:21 +02:00
TapTap 87b58975de style(receiver): rework fuzzy DP suffix trim, silence cppcheck FP
CI / lint (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 42s
CI / sanitizers (address) (pull_request) Successful in 44s
CI / fuzz-build (pull_request) Successful in 17s
CI / coverage (pull_request) Successful in 34s
CI / valgrind (pull_request) Successful in 36s
CI / build-and-test (pull_request) Successful in 7m19s
The suffix-trim loop using computed end offsets tripped cppcheck's
knownConditionTrueFalse value-range analysis (it unsoundly concluded the trims
always consume the whole middle).  Rewrite it with explicit moving end indices
and add an inline suppression with a rationale for the residual false
positive; cppcheck --error-exitcode=1 is clean again.  The trimming logic is
unchanged and was verified against a full DP reference over 200k random name
pairs.
2026-09-06 23:11:01 +02:00
TapTap bb54113254 docs: all-or-nothing TOCTOU caveat and two-section manifest budget
CI / lint (pull_request) Failing after 32s
CI / build-and-test (pull_request) Skipped
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
The walker's all-or-nothing guarantee holds only while the destination is not
concurrently modified (rehearsal and delete are separate walks).  The
protected-prefix list shares the 16 MB MAX_MANIFEST_BYTES budget with the
keep-set and each section is capped at MAX_MANIFEST_ENTRIES; an over-budget
frame is rejected on the receiver with STATUS_ERROR rather than truncated.
2026-09-06 23:10:42 +02:00
TapTap 5fc49a8503 test: pin the review findings
- ignore-errors scan-error test now runs single-threaded and under -m (the
  exact scan_directory_multithreaded path that had the use-after-free);
- new integration test: --delete/--delete-before --ignore-errors with an
  unreadable SOURCE ROOT (sequential and -m) must fail and delete NOTHING;
- new integration test: a destination-only file that merely matches an exclude
  rule is deleted under plain --delete (protection is sender-derived), while a
  source-excluded mirror is protected;
- delete-protects/delete-excluded coverage extended to --delete-after and
  --delete-delay;
- new unit test: the 100000-entry server hard bound is all-or-nothing (more
  extras than the bound -> nothing removed);
- new integration test: --force is inert under --delay-updates (documented);
- fixed the ignore-errors assertion message that stated the opposite of what it
  asserted.
2026-09-06 23:10:37 +02:00
TapTap 8007aed5f6 fix: read scanner io_error before destroy (-m UAF); refuse an empty keep-set from an errored scan
scan_directory_multithreaded read directory_scanner_had_io_error() /
parallel_scanner_had_io_error() AFTER destroying the scanner object (a
heap-use-after-free on every successful -m run that recorded an io_error); the
flag is now captured before the destroy.  As a second line of defense against
an empty-keep-set wipe, all four manifest send sites (sequential and -m, early
pre-scan and commit data pass) now refuse to transmit a keep-set manifest when
the scan that built it recorded an io_error and produced no keep entries: a
source that merely LOOKS empty because part of it was unreadable must never
delete the whole destination.  A genuinely empty source (no io_error) still
sends its empty keep-set and prunes extras.
2026-09-06 23:10:32 +02:00
TapTap b031e73ab0 fix: treat an unreadable source root as fatal even under --ignore-errors
A sequential scanner records an opendir failure of its seed/root directory as a
skippable io_error and would complete an EMPTY scan, whose keep-set manifest
would then delete every destination entry.  The seed directory that maps to the
transfer root (relative path "") is now fatal regardless of --ignore-errors;
only subdirectories discovered during an otherwise-successful root scan are
skippable.  The -m path never had this hole (its root open failure aborts
scanner creation), so sequential and -m now agree.
2026-09-06 23:10:27 +02:00
TapTap 741d1f3b93 test: review follow-up coverage for --fuzzy
CI / lint (pull_request) Failing after 37s
CI / build-and-test (pull_request) Skipped
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
Unit (CLI): --fuzzy --no-incremental (either order) stays a valid plain-mode
config -- no forced handshake, no delta implication; --fuzzy --no-delta stays
covered.

Integration (TestFuzzy):
- worthless basis: a sibling that passes the name+size gates but shares no
  blocks makes the sender reply STATUS_NEXT; the whole file is consumed inside
  the delta handshake with byte-exact output (no protocol desync).
- non-displacement: an existing exact-path destination file INSIDE the delta
  size bounds (same size, different content, older mtime) is used as the delta
  basis instead of a byte-identical similar sibling (whole-file wire cost).
- asymmetric bases: basis larger than source (prefix reuse) and basis smaller
  than source (appended tail as literals) both reconstruct byte-exactly with a
  small delta.
- --no-fuzzy end-to-end equals the no-flag whole-file behavior.
CountingProxy closes its listener socket (fd hygiene).
2026-09-06 23:05:43 +02:00
TapTap c379e2dbdd docs: fuzzy oracle note and --no-incremental asymmetry
Review follow-ups on the --fuzzy row: note that the receiver's block
signature is derived from a sibling file it may not otherwise send, exposing
destination sibling files to the sender at block granularity (the same
known-plaintext information class as the ordinary delta path); and document
that --fuzzy honors an explicit --no-incremental (unlike the basis-dir
options, which force it), with the delta implication suppressed accordingly.
2026-09-06 23:05:39 +02:00
TapTap c1aabc68de feat(cli): --fuzzy honors an explicit --no-incremental
The --fuzzy implication previously forced use_incremental back on even when
the user passed --no-incremental, while --no-delta and -W were honored.
Track a --no-incremental latch (like the no_delta latch): with it set, do not
force the handshake on, and because delta needs the handshake, also suppress
the delta implication so no invalid '--delta requires --incremental' config
results.  A --fuzzy --no-incremental run is therefore a plain default-mode
transfer (fuzzy inert), consistent with -W/--no-delta.  Documented in the
usage text (this deliberately differs from the basis-dir options, which still
force incremental unconditionally).
2026-09-06 23:05:35 +02:00
TapTap be37a509a5 perf(receiver): bound the fuzzy edit-distance cost, harden the open
Review follow-ups on the --fuzzy candidate scan:
- Allocate the two DP rows once per directory scan instead of once per
  candidate (4096-entry directories no longer do thousands of malloc pairs).
- Pre-prune before the DP with two cheap lower bounds on the edit distance:
  the name length gap and the count of characters of one basename absent from
  the other; a candidate whose gate (distance*2 <= longer) already fails on
  the max of those bounds is skipped without running the DP.
- Trim the common prefix and non-overlapping common suffix before the DP so
  it only runs over the differing middles.
- Note that the 4096 readdir cap bounds iterations, not per-entry DP cost,
  and that the seen set is filesystem-order dependent (winner stays
  deterministic via the total comparator).
- Open the chosen candidate with O_NONBLOCK so a name raced to a FIFO cannot
  block the receive thread forever in open(2); the existing fstat S_ISREG gate
  still rejects non-regular files.  (The pre-existing basis_open_regular has
  the same latent FIFO pattern and is intentionally left unchanged.)
- Free old_data in receive_delta_file's defensive NULL guard.
2026-09-06 23:05:32 +02:00
TapTap 5f4c7ce638 fix: free walker-test root path after cleanup (ASan leak)
CI / lint (pull_request) Successful in 31s
CI / sanitizers (address) (pull_request) Successful in 40s
CI / sanitizers (undefined) (pull_request) Successful in 39s
CI / fuzz-build (pull_request) Successful in 18s
CI / coverage (pull_request) Successful in 36s
CI / valgrind (pull_request) Successful in 36s
CI / build-and-test (pull_request) Successful in 10m49s
make_walk_root() roots were removed from disk but never freed, leaking under
the address/valgrind sanitizer jobs.
2026-09-06 22:09:25 +02:00
TapTap 356b5592e0 feat: add confined symlink-safe directory-tree removal helper
CI / lint (pull_request) Successful in 32s
CI / sanitizers (address) (pull_request) Failing after 42s
CI / sanitizers (undefined) (pull_request) Successful in 43s
CI / fuzz-build (pull_request) Successful in 18s
CI / coverage (pull_request) Successful in 35s
CI / valgrind (pull_request) Failing after 36s
CI / build-and-test (pull_request) Successful in 10m49s
file_remove_tree_secure() opens the final component O_NOFOLLOW below the
authorized root and recursively wipes it with an fd-relative walk (symlinks are
removed by name, never followed); --force uses it to clear a destination
directory that blocks an incoming regular file.
2026-09-06 21:50:25 +02:00
TapTap 6e835fd9f7 docs: mark the delete-policy family implemented in RSYNC_COMPAT
--delete-excluded/--max-delete/--ignore-errors/--force/--prune-empty-dirs now
✅ with precise notes: the rsync-parity default (plain --delete protects
filter-excluded destination mirrors), the -m divergence (FastSync -m stays
multithreading, so --prune-empty-dirs is long-only), the all-or-nothing
max-delete/hard-bound error model, the 2.8.0 -> 2.9.0 protocol bump, and the
new two-section STATUS_MANIFEST frame.  Summary counts left for the orchestrator
to recount after the wave.
2026-09-06 21:50:16 +02:00
TapTap 1de2677bb1 test: delete-policy unit and integration coverage
Unit: walker all-or-nothing bounds (exceeded -> nothing removed + distinct
result; exact bound -> deletes), protected-prefix skipping, config wire
round-trip for force_delete/delete_excluded/prune_empty_dirs/max_delete, CLI
parse/validation for the new flags.  Integration (TestDeletePolicy): default
delete-excluded protection and --delete-excluded opt-out (single-thread, -m,
early --delete-before, excluded-dir subtrees), --max-delete all-or-nothing over
and at the limit, --force file-over-nonempty-dir replacement, --prune-empty-dirs
(--dirs mode + recursion-mode parity), and --ignore-errors keeping deletion
active across a genuine scan I/O error (run as an unprivileged user).
2026-09-06 21:50:12 +02:00
TapTap 0b25bacb18 feat: protect filter-excluded destination mirrors by default (--delete-excluded opt-in), --ignore-errors scan continuation, --prune-empty-dirs
The scanners now record every entry pruned by user-selection rules
(--filter/-C/per-dir, --exclude/--include, --max-size/--min-size) as a
destination-relative protected path on a caller-supplied sink (thread-safe in
the parallel scanner); --files-from subset pruning and -R relative wire paths
are never recorded.  The sender transmits these as manifest protected prefixes,
giving rsync's default --delete behavior (excluded mirrors survive) with
--delete-excluded opting back into deleting them.  --ignore-errors makes an
unreadable source directory a recorded, non-fatal scan error: the run continues,
the deletion still runs, and the exit code reports the ignored error.
--prune-empty-dirs omits an empty source directory's explicit --dirs entry.
Empty directories were never transferred by recursive scans (rsync -m parity).
2026-09-06 21:50:07 +02:00
TapTap c4e0de8f08 feat: split delete-manifest frame into keep-set + protected prefixes; enforce --max-delete and --force on the receiver
The STATUS_MANIFEST frame now carries two count-delimited sections: the kept
paths and a protected-prefix list (excluded-on-source paths the walker must not
delete unless --delete-excluded opted out).  The receiver's DeleteManifest is
passed through the commit/early paths unchanged.  manifest_delete_extras
honors a client --max-delete (all-or-nothing) and produces a distinct error for
it versus the 100000-entry server bound.  --force clears a non-empty directory
that blocks an incoming regular file (confined, symlink-safe) via a new
file_remove_tree_secure helper.
2026-09-06 21:50:02 +02:00
TapTap 1c9b660ac0 feat: all-or-nothing bounded delete walker
delete_extras_limited now rehearses a finite-capped deletion before unlinking
anything (an identical fd-relative walk that counts files and directories) and
returns DELETE_WALK_LIMIT_EXCEEDED with nothing removed when the run would
exceed the cap, so --max-delete is enforced per run instead of truncating the
deletion.  A directory that still holds entries the walker leaves in place
(protected excluded prefix, manifest-kept file, symlink) is left behind rather
than failing the whole deletion, matching rsync's leave-non-empty-dirs
behavior.  Rehearsal/delete each open an independent file description so a
prior pass cannot drain the directory stream.
2026-09-06 21:49:57 +02:00
TapTap 3631df0a3e feat: add delete-policy config fields and CLI flags
Adds ignore_errors (client-only) and force_delete (wire) booleans, makes
max_delete default -1 (no client limit), and parses --delete-excluded,
--max-delete=NUM, --ignore-errors, --force, --prune-empty-dirs.  Bumps
PROTOCOL_VERSION 2.8.0 -> 2.9.0 for the new on-the-wire force_delete field.
2026-09-06 21:49:53 +02:00
TapTap 2bc43d6084 docs: mark -y/--fuzzy/--no-fuzzy implemented in RSYNC_COMPAT
CI / lint (pull_request) Successful in 36s
CI / sanitizers (address) (pull_request) Successful in 42s
CI / sanitizers (undefined) (pull_request) Successful in 42s
CI / fuzz-build (pull_request) Successful in 17s
CI / coverage (pull_request) Successful in 34s
CI / valgrind (pull_request) Successful in 36s
CI / build-and-test (pull_request) Successful in 7m15s
Document the receiver-side decision location, the exact deterministic
similarity heuristic, the byte-exactness argument, when fuzzy applies (and
when it deliberately does not), the 2.8.0 -> 2.9.0 protocol bump, and the
divergences from rsync.  Update the basis-dir rows' protocol references to
the now-current 2.9.0.
2026-09-06 21:04:26 +02:00
TapTap 9c1ec6758c test: --fuzzy coverage (CLI, config wire, integration)
Unit: parse -y/--fuzzy and --no-fuzzy negation (order-independent), -W and
--no-delta leave fuzzy inert, --fuzzy implies --incremental/--delta, and
validate_config rejects fuzzy with -s (chunk serialization) and -f
(sendfile).  Config: fuzzy survives the socketpair wire round-trip.

Integration (TestFuzzy, byte counts via a new CountingProxy helper): the
rename case transfers a 2 MiB file in a few percent of its size with byte-
exact output under --fuzzy, while the no-fuzzy, no-candidate, dissimilar-
sibling and --whole-file runs send the whole file; -y alias; -m parity;
dest-holds-unsuitable-file falls back to the sibling; --delay-updates and
--remove-source-files keep their semantics on fuzzy transfers.
2026-09-06 21:04:23 +02:00
TapTap cebd23a239 feat(receiver): --fuzzy similar-file delta basis
When a file must be transferred and the destination holds no usable content
at the exact path (file absent, or outside the delta engine's size bounds),
the receiver now searches the target's own destination directory for an
existing regular file with a similar basename and uses it as the delta basis
via the existing receiver-driven STATUS_DELTA_SIGNATURE handshake.  The
sender never learns the basis was another file, so no wire change beyond the
new config flag was required.

Heuristic (deterministic, simpler than rsync's, documented): candidates are
sibling entries confined below the root and opened O_NOFOLLOW (symlinks are
never followed, nothing outside the destination root is read); dotfiles,
directories, the target's own name and stage/temp names are excluded; size
gate is delta_should_attempt; name gate is a Levenshtein distance <= half
the longer basename; the closest candidate (size tie-break, then lexical) is
loaded; the scan is capped at 4096 entries.

Byte-exactness is independent of the basis: block matches are verified by
Adler-32 + xxHash32, delta_apply validates every reference, and a basis that
shares nothing makes the sender reply with a whole-file transfer.  When no
candidate qualifies the normal whole-file transfer runs unchanged.
2026-09-06 21:04:19 +02:00
TapTap b753efcecc feat(cli): parse -y/--fuzzy and --no-fuzzy
Register --fuzzy (alias -y) as a boolean option and fuzzy as negatable so
--no-fuzzy works through the generic negation machinery.  FastSync's delta
machinery is off by default (unlike rsync, where --fuzzy implies nothing
because delta is the default), so --fuzzy implies --incremental and --delta
unless --whole-file or an explicit --no-delta switched delta off (leaving
fuzzy inert, matching rsync where -W makes fuzzy irrelevant).  Add help text.
2026-09-06 21:04:14 +02:00
TapTap f099a6e20f feat(config): add fuzzy flag and bump protocol to 2.9.0
-y/--fuzzy is receiver-side similar-file basis selection, so the receiver
must learn the flag: add a bool to Config, serialize it as a trailing field
on the config frame, and validate the received value.  Because the config
frame layout changed, PROTOCOL_VERSION moves to 2.9.0 (peers must match).
2026-09-06 21:04:11 +02:00
TapTap 3815b82306 docs: recount RSYNC_COMPAT summary after Phase-3 wave A
CI / lint (push) Successful in 32s
CI / sanitizers (undefined) (push) Successful in 41s
CI / sanitizers (address) (push) Successful in 42s
CI / fuzz-build (push) Successful in 16s
CI / coverage (push) Successful in 34s
CI / valgrind (push) Successful in 37s
CI / build-and-test (push) Successful in 7m13s
2026-09-06 20:11:39 +02:00
TapTap 01a5a93089 Merge feat/p3-basis-dest: alternate basis dirs (--compare-dest/--copy-dest/--link-dest) 2026-09-06 20:10:40 +02:00
TapTap 265b1e7669 Merge feat/p3-delete-timing: rsync delete timing (--delete-before/--delete-during/--del/--delete-after/--delete-delay) 2026-09-06 19:58:40 +02:00
TapTap 329fc60b14 test: make remove-source incremental-skip test deterministic
CI / lint (pull_request) Successful in 32s
CI / sanitizers (address) (pull_request) Successful in 42s
CI / sanitizers (undefined) (pull_request) Successful in 40s
CI / fuzz-build (pull_request) Successful in 17s
CI / coverage (pull_request) Successful in 35s
CI / valgrind (pull_request) Successful in 36s
CI / build-and-test (pull_request) Successful in 3m29s
The seed run did not preserve timestamps, so the destination copy's mtime was
the write time; the incremental --remove-source-files rerun only skipped the
file when both writes happened to land in the same whole second, making the
test flaky (observed intermittently in local full-suite runs and on CI).  Seed
with -M so the destination stores the source's exact mtime.
2026-09-06 19:53:22 +02:00
TapTap d6d502fbb4 docs: precise basis-dir caveats (shared-inode --inplace, attribute provenance, 256MiB limit)
CI / lint (pull_request) Successful in 32s
CI / sanitizers (address) (pull_request) Successful in 42s
CI / sanitizers (undefined) (pull_request) Successful in 40s
CI / fuzz-build (pull_request) Successful in 18s
CI / coverage (pull_request) Successful in 35s
CI / valgrind (pull_request) Successful in 36s
CI / build-and-test (pull_request) Failing after 3m31s
- --link-dest destination entries share the basis inode: a later --inplace run
  against such a path mutates the basis snapshot through the shared inode
  (recommend --copy-dest when the destination must stay independently writable).
- basis-hit files take mode/uid/gid and mtime from the basis file, not the
  sender's metadata (with --size-only the mtime can differ from the source).
- --remove-source-files sources satisfied by a basis dir are retained.
- basis runs refuse files above the 256 MiB whole-file limit up front (FastSync
  caps every whole-file payload path at 256 MiB; rsync supports arbitrary sizes);
  the config frame always carries a basis-count field (protocol 2.8.0).
2026-09-06 19:43:25 +02:00
TapTap e0e0ea6eac test: xxHash equal-size gate, basis priority, size-only/ignore-times, oversize
- unit: config basis-path normalization (trailing slash, a//b, ./x/./y collapse;
  degenerate inputs rejected).
- integration: same-size/same-mtime/different-content fixtures prove the xxHash
  gate -- the basis changed.txt now has the SAME byte size as the source so the
  size short-circuit can no longer mask the hash comparison, plus a dedicated
  parametrized same-size mismatch test asserting link/copy never use a
  content-mismatched basis and compare-dest transfers.
- basis-dir priority is first-match-wins: two link-dest dirs (inode of the
  first), and compare-dest before link-dest stays sparse while the reverse
  order hard-links.
- --size-only links a same-content basis file with a different mtime;
  --ignore-times never links even an exact match.
- --delay-updates + --delete removes extras inside a nested .fastsync-stage
  dir (regression guard) while keeping the real staging dir and basis tree.
- a basis run containing a file above the whole-file limit fails up front with
  a clear error and transfers nothing.
2026-09-06 19:43:21 +02:00
TapTap 4799d12e25 fix: refuse basis runs containing an over-limit file before any transfer
Basis dirs imply the per-file incremental check, which (like every whole-file
payload path in FastSync) is bounded by MAX_RECEIVE_WHOLE_FILE_SIZE.  A source
tree with a larger file used to abort the whole run mid-stream on the receiver
with no client-side diagnostic.  With basis dirs configured the client now
preflights the scan (respecting filters/size rules) and fails up front with a
clear error naming the offending file before connecting, matching the
documented 256 MiB whole-file limit instead of aborting silently.
2026-09-06 19:43:15 +02:00
TapTap 4bf4da37e5 fix: free basis path on copy-dest hits; keep --delete staging skip top-level-only
- copy-dest basis hits leaked the heap-allocated basis path: BASIS_DEST_COPY
  did not transfer it (only LINK does) and returned before the basis cleanup.
  basis_match_free is now called on every materialization return path (success
  and send-failure) after content/link ownership is transferred.
- the --delete walker regression: the delay-updates staging name must be
  protected only as a DIRECT child of the receive root, while basis dirs may
  be skipped at any depth.  delete_extras_limited now takes DeleteSkipEntry
  entries carrying a top_level_only flag instead of a flat prefix list, so a
  nested destination directory named .fastsync-stage is ordinary content again
  (its extras are deleted) and a basis tree is still never removed.
2026-09-06 19:43:11 +02:00
TapTap 08a5815ca7 refactor: unify basis-dir path validation and normalization
config_basis_path_valid and config_basis_append now share one normalizer
(basis_path_normalize): interior empty components (a//b) collapse, '.'
components and trailing slashes are dropped, and the stored form is exactly
the canonical relative path used by validation, the delete-walker prefix match
and the receiver's basis lookup.  Degenerate inputs (empty, absolute, '..',
'.' that normalizes to nothing) stay rejected.
2026-09-06 19:43:06 +02:00
TapTap 02679fe335 docs: clarify --del alias, early keep-set caps, ACK wait, --no-delete conflict
CI / lint (pull_request) Successful in 25s
CI / sanitizers (undefined) (pull_request) Successful in 41s
CI / sanitizers (address) (pull_request) Successful in 42s
CI / fuzz-build (pull_request) Successful in 16s
CI / coverage (pull_request) Successful in 33s
CI / valgrind (pull_request) Successful in 36s
CI / build-and-test (pull_request) Successful in 6m51s
Usage help now gives --delete-during a complete description with --del on its
own line, and notes that timing flags imply --delete while timing+--no-delete
is rejected regardless of argument order. RSYNC_COMPAT.md documents: the
receiver's MAX_MANIFEST_ENTRIES/MAX_MANIFEST_BYTES caps now abort an early-mode
run before any data (previously only the deletion step failed), the extended
early-delete ACK deadline, and the order-independent flag-conflict policy.
2026-09-06 19:35:28 +02:00
TapTap c0c315cf48 test: cover -m late-deletion failure, receiver leak exits, timed ACK read
- Unit (leak guards): drive receiver_process_pending() past a parked keep-set
  into STATUS_ABORT, EOF, and a second manifest frame; each must return -1 with
  no manifest handed out. Verified leak-free under ASan.
- Unit: receive_status_timed reads a status and fails cleanly on EOF.
- Integration: test_late_flags_commit_only_after_success now parametrizes the
  -m path, proving a failed -m late-timing run preserves every extra and that
  the deferred manifest is dropped (never applied) when the writer fails.
2026-09-06 19:35:25 +02:00
TapTap ebfaced5c2 fix: wait for the early-delete ACK with an extended deadline
The receiver performs the whole bounded deletion walk (up to
MAX_SERVER_DELETE_COUNT unlinks) before answering the delete-before/during
manifest, so its STATUS_OK reply can take far longer than the default 60 s
per-message receive window. Waiting with the default would make the sender
abort AFTER the deletion had already committed on the receiver. Add a timed
receive variant (receive_status_timed / protocol_receive_n_data_timed) and use
it for the early-manifest ACK with a 1 h explicit deadline; connection errors
and EOF still abort immediately.
2026-09-06 19:35:21 +02:00
TapTap bbf982dc0b fix: free the parked delete manifest on every receiver error exit
The late/commit path keeps the received keep-set in a local list until
STATUS_FINISHED. Error exits after it was parked (STATUS_ABORT, a failing
receive_status / non-FINISHED status, a second manifest frame, or a later
file/chunk/store failure) previously dropped the only reference and leaked up
to ~16 MB of path strings + pointer array per connection. Both failure labels
now discard the parked list exactly once; the successful FINISHED path still
hands ownership to *pending_manifest (the -m caller) without freeing it.
2026-09-06 19:35:17 +02:00
TapTap 36c2c04910 docs: mark rsync delete-timing family implemented
CI / lint (pull_request) Successful in 26s
CI / sanitizers (address) (pull_request) Successful in 39s
CI / fuzz-build (pull_request) Successful in 16s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / coverage (pull_request) Successful in 34s
CI / valgrind (pull_request) Successful in 37s
CI / build-and-test (pull_request) Successful in 6m22s
RSYNC_COMPAT.md: flip --delete-before, --del/--delete-during, --delete-delay
and --delete-after to Implemented with precise notes (default-under--delete,
safety model, 2.7.0 -> 2.8.0 protocol bump, exact divergences from rsync).
README option tables list the new flags and the delete-after default.
2026-09-06 18:53:28 +02:00
TapTap 7eacf7c180 test: cover rsync delete-timing flags, config wire, and semantics
- CLI: each timing flag (+ --del alias) accepted and implies --delete;
  conflicting timings and a timing with --no-delete are rejected.
- Config: delete_during/delete_delay survive config_send/config_receive;
  two simultaneous timings are rejected by the receiver-side validation;
  config_delete_timing_early() mapping is unit-tested.
- Integration (TCP, single- and multithreaded): every flag removes extras on
  a successful transfer; early modes (--delete-before/--delete-during/--del)
  delete before data is applied so a destination file blocking a nested
  write is removed and the transfer succeeds, while plain --delete /
  --delete-after / --delete-delay keep it and fail with every extra intact
  (commit-style). Early timing also completes (without deleting) when the
  server refuses deletion.
2026-09-06 18:53:24 +02:00
TapTap 4e725517f0 feat: implement rsync delete timing (--delete-before/--delete-during/--delete-delay/--delete-after)
Deletion timing is now real and selected by the four rsync flags plus the
plain --delete default. Wire protocol bumps to 2.8.0: two new config
booleans (delete_during, delete_delay) are serialized and validated, joining
the existing delete_before/delete_after.

- Early modes (--delete-before, --delete-during/--del): the sender pre-scans
  the whole tree (paths only), transmits the keep-set manifest BEFORE any
  file data, and the receiver removes extras and acks STATUS_OK; the sender
  only streams data after the deletion committed. Deletion is thus performed
  even if a later transfer phase fails (rsync delete-before/during are
  destructive by definition). FastSync streams in a single scan so it cannot
  interleave per-directory like rsync delete-during; --delete-during selects
  the same engine mode as --delete-before (documented divergence).
- Late/commit modes (plain --delete, --delete-after, --delete-delay): the
  manifest closes the data stream and deletion is committed only after
  STATUS_FINISHED proves the whole transfer succeeded, preserving FastSync's
  commit-style safety. --delete-delay converges with --delete-after because
  FastSync never snapshots the destination during data flow (documented).
- The STATUS_MANIFEST frame is now self-delimiting and position-independent.
  Single-threaded receivers delete before the success frame; the -m receiver
  hands the keep-set to server.c, which commits the deletion only after the
  disk writer thread has drained (fixes a delete-vs-in-flight-temp race).
- Every timing flag implies --delete; at most one timing flag is allowed.
- Each timing flag implies --delete, matching rsync; conflicts are rejected.
2026-09-06 18:53:20 +02:00
TapTap 196a27689f docs: mark --compare-dest/--copy-dest/--link-dest implemented
CI / lint (pull_request) Successful in 33s
CI / sanitizers (address) (pull_request) Successful in 45s
CI / sanitizers (undefined) (pull_request) Successful in 43s
CI / fuzz-build (pull_request) Successful in 18s
CI / coverage (pull_request) Successful in 35s
CI / valgrind (pull_request) Successful in 36s
CI / build-and-test (pull_request) Successful in 3m20s
Document receiver-side basis semantics, relative-to-destination-root
confinement, content-verified matching, hard-link vs copy vs compare-only
behavior, cross-filesystem copy fallback, repetition/priority, --delete
exclusion, the implied --incremental and -s incompatibility, and each exact
divergence from rsync (no dest deletion on compare-dest stale entries, no
attribute re-application on basis hits, no re-linking of already-up-to-date
dest files, sources matched from basis dirs kept under --remove-source-files).
2026-09-06 18:47:41 +02:00
TapTap 0d18b7fc87 test: integration coverage for compare/copy/link-dest basis directories
- compare-dest skips an exact basis match (leaving a sparse destination) and
  still transfers files the basis cannot satisfy; a content mismatch forces a
  normal transfer.
- copy-dest materializes the unchanged file as a real local copy (distinct
  inode) and transfers content mismatches.
- link-dest hard-links (asserted same inode/nlink to the DIR file) and falls
  back to a normal transfer on content mismatch.
- a missing basis dir is a clean full-transfer no-op for all three flags.
- link-dest works through -m multithreading and --delay-updates (staged and
  published as a real link, staging cleaned up).
- --delete removes genuine extras while leaving the basis dir untouched.
2026-09-06 18:47:37 +02:00
TapTap 1fc0cacc32 test: unit tests for basis-dir CLI parsing and config wire round-trip
- parse_args accepts each flag in both forms, keeps repetition order/types,
  implies --incremental + metadata, and rejects absolute/escaping/degenerate
  paths; validate_config rejects basis dirs combined with -s.
- config wire round-trips a mixed basis list and rejects escaping/absolute
  paths on the receiver side.
2026-09-06 18:47:33 +02:00
TapTap 8f846a43b8 feat: exclude basis directories from --delete
Generalize the delete walker's protected-root-child skip into a prefix list.
The receiver now passes both the --delay-updates staging directory and every
basis-dir path, so a --delete run can never treat a basis snapshot (which a
--link-dest run just linked from) as destination content to remove.
2026-09-06 18:47:29 +02:00
TapTap d38920c972 feat: receiver-side basis matching with link/copy materialization
The receiver's per-file incremental check now consults the ordered basis-dir
list whenever the destination is not already up to date.  An exact basis match
requires equal size, equal mtime (unless --size-only; --ignore-times disables
basis matching like rsync), and an equal content xxHash64 -- the sender sends
its xxHash for every file whenever basis dirs are configured (not only under
--checksum), so a hard link or local copy is only ever made from byte-identical
content.

On a match:
  - compare-dest: reply STATUS_OK and skip data only when the destination does
    not already hold the file (sparse, rsync parity).  A destination that holds
    a DIFFERENT version falls back to a normal transfer instead of rsync's
    delete, keeping the mirror complete.
  - copy-dest: reply STATUS_OK and hand a synthetic File (bytes read from the
    basis file, basis metadata) to the normal store sink, so the file is
    installed as a real local copy through the existing atomic temp+rename
    engine and honors --existing/--ignore-existing/--update/--backup/
    --delay-updates/--partial-dir unchanged.
  - link-dest: same, but File.basis_link records the basis path and the store
    engine calls the new file_to_disk_secure_link(): an atomic temp hard link +
    rename.  Cross-filesystem/refused links fall back to a byte-identical local
    copy (never a corrupt or partial file); the copy fallback applies metadata,
    while a successful link keeps the basis inode's own attributes so the basis
    file is never mutated.

Basis-materialized files carry File.skip so they are not acknowledged to a
--remove-source-files sender (the sender already saw STATUS_OK and keeps the
source).  The no-match path is byte-for-byte identical to the existing delta /
full-data transfer.
2026-09-06 18:47:26 +02:00
TapTap df890f76ea feat: basis-dir config/CLI for --compare-dest/--copy-dest/--link-dest
Replace the vestigial single compare_dest/copy_dest/link_dest Config fields with
an ordered BasisDest list (type + path per entry) that is serialized to the
receiver and interpreted relative to the destination root.  Paths must be
relative with no '.'/'..' components (confined like --backup-dir); trailing
slashes are normalized.  Wire layout changes, so PROTOCOL_VERSION -> 2.8.0.

CLI: each flag is parsed in both --flag=DIR and --flag DIR forms, is
repeatable, and keeps command-line order as basis priority.  Supplying any
basis dir implies --incremental (and therefore metadata) on the sender because
the unchanged decision is receiver-side; combining basis dirs with -s chunk
serialization is rejected in validate_config.  Usage text updated.
2026-09-06 18:47:20 +02:00
TapTap 4cf34026e7 test: drop racy queued_bytes==0 assert in byte-budget test
CI / lint (push) Successful in 25s
CI / sanitizers (address) (push) Successful in 40s
CI / sanitizers (undefined) (push) Successful in 40s
CI / fuzz-build (push) Successful in 17s
CI / coverage (push) Successful in 34s
CI / valgrind (push) Successful in 36s
CI / build-and-test (push) Successful in 3m6s
After the writer releases bytes, the blocked enqueuer may already have
admitted the next payload, so the transient queued_bytes==0 read is
scheduling-dependent (lost the race under ASan). The post-join state
(covers unblocking + budget re-limit) is deterministic.
2026-09-06 16:27:54 +02:00
TapTap b04ffa608b fix: free rel path on parallel root-scan entries without -R
CI / lint (push) Successful in 26s
CI / sanitizers (undefined) (push) Successful in 41s
CI / sanitizers (address) (push) Failing after 41s
CI / fuzz-build (push) Successful in 16s
CI / coverage (push) Successful in 35s
CI / valgrind (push) Successful in 37s
CI / build-and-test (push) Successful in 3m6s
scan_root_entry str_dup'd the entry name for every root-level file but only
consumed it on the -R + --files-from path, leaking 1 string per root file in
normal parallel scans (found by CI ASan/valgrind).
2026-09-06 16:21:23 +02:00
TapTap 4146814aee Merge feat/p2-relative-dirs-mkpath: -R/--no-implied-dirs/--dirs/--mkpath
CI / lint (push) Successful in 25s
CI / sanitizers (undefined) (push) Successful in 42s
CI / sanitizers (address) (push) Failing after 42s
CI / fuzz-build (push) Successful in 16s
CI / coverage (push) Successful in 34s
CI / valgrind (push) Failing after 37s
CI / build-and-test (push) Successful in 3m8s
Final Phase-2 row: -R+files-from dest layout (bare relative wire path),
--no-implied-dirs ancestor checks, --dirs/-d explicit directory entries
(STATUS_MKDIR + chunk dir marker), --mkpath server create-root (absent
root now rejected without it). Protocol 2.7.0. c-review REQUEST CHANGES
-> blocker (trailing-slash/root-equal dest) + warnings/nits fixed; PR #266.
2026-09-06 16:10:23 +02:00
TapTap 3275b6c978 fix: address c-review for -R/--dirs/--mkpath row
B1: destination-root existence/creation mishandled two legitimate forms.
file_directory_exists_secure/file_ensure_directory_secure now normalize a
trailing-slash destination (so the last component is never an empty leaf)
and treat a destination equal to the already-open authorized root as present
(no spurious <root>/<basename> nested dir with --mkpath). Confinement and
O_NOFOLLOW probing are unchanged. Regression integration tests: existing
dest with trailing slash works with and without --mkpath; dest == authorized
root works and creates no stray nested dir.

W1: chunk serialize/deserialize round-trip unit test for a directory entry
mixed with a regular file, with and without metadata; --dirs coverage under
-s and -s -m.
W2: --list-only and --dry-run now print file_wire_path() so all outputs show
the -R transformed relative name, matching -i/--out-format.
W3: RSYNC_COMPAT -d/--dirs note: dirs are created immediately under
--delay-updates (only regular files are staged).
W4: --dirs generator flushes chunks by element count too, so a long
--files-from list of empty directories cannot exceed the per-chunk file cap.
N1: STATUS_MKDIR added to status_to_string.
N2: removed dead TestMkpath._transfer.
N3: removed redundant --no-implied-dirs OPTION_TABLE row.
N4: integration tests: --dirs --delete keeps the just-created empty dir (and
deletes extras); a listed dir colliding with a regular file at the dest fails
cleanly.
2026-09-06 16:10:00 +02:00
TapTap c2cf231158 feat: implement -R/--relative, --no-implied-dirs, --dirs/-d, --mkpath
CI / lint (pull_request) Successful in 26s
CI / sanitizers (address) (pull_request) Failing after 41s
CI / sanitizers (undefined) (pull_request) Successful in 40s
CI / fuzz-build (pull_request) Successful in 18s
CI / coverage (pull_request) Successful in 35s
CI / valgrind (pull_request) Failing after 37s
CI / build-and-test (pull_request) Failing after 2m35s
RSYNC_COMPAT Phase-2 row M: path-list construction and destination
directory creation while preserving traversal safety.

- -R/--relative with --files-from: transmit each listed entry under its
  bare relative destination path (no source-root mirror). Files keep an
  absolute local read path plus a separate wire/dest path (File.send_path);
  manifest, incremental quick-check and change output follow the wire path,
  so --delete and --remove-source-files stay consistent. -R without
  --files-from is unchanged (full mirror).
- --no-implied-dirs: client-only, only meaningful with -R + --files-from.
  A listed file whose parent dir is not itself (or via an ancestor)
  explicitly listed cannot be placed; the run fails up front with a clear
  error. No effect otherwise.
- --dirs/-d + --old-dirs/--old-d aliases: -d <dir> transmits the source
  root as an explicit empty directory entry (STATUS_MKDIR frame); with
  --files-from listed dirs are created empty and listed files transferred,
  never descending. Works single-threaded, -m (sequential scanner in the
  -m scan thread) and chunk-serialization (per-file type marker).
  Directory entries appear in the delete manifest.
- --mkpath: new wire bool; server creates the destination root (and missing
  leading components under its authorized root) at connection start. A
  missing destination root is now rejected by default.
- Protocol bumped to 2.7.0 (mkpath wire field + STATUS_MKDIR + chunk type
  marker). All receive paths funnel through file_save_to_disk_full which
  creates directories via the secure confined mkdir engine; dir entries are
  excluded from --remove-source-files outcome acknowledgements on both ends.
- Unit coverage: CLI parse (relative/dirs aliases/mkpath/no-implied-dirs),
  config round-trip (relative + mkpath), scanner --dirs non-recursion and
  -R send_path (sequential + parallel), receiver dir-entry save.
- Integration coverage: TestRelativeFilesFrom, TestNoImpliedDirs, TestDirs,
  TestMkpath (single and -m).
- RSYNC_COMPAT: 4 rows move to Implemented (Summary 67/3/5/1/71 = 147).
2026-09-06 15:28:32 +02:00
TapTap a196522010 style: clang-format test_client_cli.c after merge resolution
CI / lint (push) Successful in 23s
CI / sanitizers (undefined) (push) Successful in 42s
CI / sanitizers (address) (push) Successful in 43s
CI / fuzz-build (push) Successful in 17s
CI / coverage (push) Successful in 34s
CI / valgrind (push) Successful in 36s
CI / build-and-test (push) Successful in 1m45s
2026-09-06 14:29:28 +02:00
TapTap 342dd152ef Merge feat/p2-files-from-filter: files-from/from0/filter/-F/-C
Client-side file-list selection and filter-rule layer (client-only; no wire
change). rsync-consistent inner-first per-dir filter precedence; listed-but-
missing files-from entries hard-error; root '/' scan regression fixed;
Summary recounted (63 implemented / 75 not).
c-review REQUEST CHANGES -> blockers fixed; PR #265.
2026-09-06 14:25:51 +02:00
TapTap a6c982cd86 Merge feat/p2-delay-updates: implement --delay-updates
Receiver stages writes and publishes only after the full transfer succeeds
(single and -m, before the outcomes frame). delete walker skips staging;
backup-dir name reserved; flock serializes concurrent delayed sessions;
protocol bump to 2.6.0. c-review REQUEST CHANGES -> blockers fixed; PR #264.
2026-09-06 14:19:38 +02:00
TapTap 4295fefaa3 fix: --delay-updates delete/backup collisions, publish-failure test, staging lock
Review fixes for --delay-updates:

- --delete no longer deletes the staged files: the delete walker gains a
  skip_root_child parameter and receive_manifest passes DELAY_UPDATES_STAGING_DIR
  when delay_updates is active, so deletion removes genuine extras while the
  staging dir (a direct child of the receive root) is left for publication in
  both single and -m modes.
- --backup-dir is rejected when it collides with the reserved internal staging
  name .fastsync-stage (trailing slash normalized), in client validation and in
  the received-config wire validation, preventing old backups from being
  silently installed as new files.
- Staging dir is now held under an exclusive advisory flock for the whole
  transfer (context lifetime): two simultaneous delayed transfers to one
  destination root no longer share/destroy each other's staged data - the
  second fails cleanly.  Cleanup only touches the staging dir when this context
  owns the lock, so a lock-contention failure cannot wipe a live session.
- Post-publish staging cleanup now returns/logs instead of discarding failures
  (warning when the staging dir cannot be fully removed).
- Reworked the publish-failure integration test to exercise real mid-publish
  semantics (top-level file published, nested rename fails, no rollback,
  sources retained under --remove-source-files) and added integration tests for
  --delete + --delay-updates ordering and reserved --backup-dir rejection.
- RSYNC_COMPAT note documents delete ordering, the reserved-name hazard, and
  the concurrency guard.
2026-09-06 14:19:18 +02:00
TapTap 7f2180ef90 fix: filter precedence, files-from errors, NUL/CRLF and filter-rule rejections
Address an independent c-review of the files-from/filter feature:

- .rsync-filter precedence now matches rsync: evaluate the innermost
  (current) directory's rules first, then ancestors, then the command-line
  base (--filter/-C), so a deeper file's '+' can re-include what a shallower
  '-' excluded (regression tests in both scan modes; single-thread and -m).
- --files-from: a listed entry missing on disk and an empty list are now hard
  errors surfaced pre-transfer in send_files, send_files_multithreaded,
  dry-run and --list-only; '.' (whole tree) and empty listed dirs stay valid.
- scanner_path_relative now handles a transfer root of / (previously the
  scanner aborted on children of /).
- Reject unsupported rsync filter syntax explicitly (no silent no-ops):
  +/- modifiers other than '/' (! C s r p x) and rules beginning with ':'/'.'
  /'!' (merge/dir-merge/list-clear shorthands). Docs updated.
- -0/--from0 NUL mode preserves entry bytes (no CR/LF trimming); only newline
  mode trims. Absolute-entry error message no longer includes the newline.
- --no-from0/--no-cvs-exclude registered as negatable booleans.
- RSYNC_COMPAT rows updated for the precedence, rejection list, NUL-mode
  detail and the documented O(entries x files) scalability bound of the
  allow-set (Summary unchanged: 62/3/5/1/76 = 147).
2026-09-06 14:16:12 +02:00
TapTap f4c15a7b78 feat: add --files-from/-0, --filter, -C and -F filter layer
CI / lint (pull_request) Successful in 23s
CI / sanitizers (address) (pull_request) Successful in 40s
CI / sanitizers (undefined) (pull_request) Successful in 39s
CI / fuzz-build (pull_request) Successful in 17s
CI / coverage (pull_request) Successful in 34s
CI / build-and-test (pull_request) Successful in 1m28s
CI / valgrind (pull_request) Successful in 36s
Implement the RSYNC_COMPAT Phase-2 filter/parser feature group:
- --files-from=FILE (repeatable) plus -0/--from0 NUL delimiters: parse the
  source file list relative to the source root into a shared read-only
  allow-set; the scanner transfers listed files and the whole subtree of
  listed directories and prunes everything else in single- and
  multithreaded mode. Absolute/'..' entries and missing files are hard
  CLI errors.
- --filter=RULE: rsync-style +/- rules (anchored '/', dir-only trailing '/',
  word include/exclude forms) evaluated first-match-wins with a default of
  include, as an independent layer from legacy --exclude/--include.
  Unsupported directives (merge/hide/... ) are rejected explicitly. -f stays
  sendfile.
- -C/--cvs-exclude: well-known rsync CVS default exclude set.
- -F: per-directory .rsync-filter files read during traversal and applied to
  the owning directory's subtree (single + parallel), never transferred.
- Delete manifest still derives from what was actually sent.

Client-only config fields; no wire/protocol change. Adds unit coverage
(CLI parse, allow-set and filter scanning single+parallel) and integration
tests (TestFilesFrom, TestFilters). RSYNC_COMPAT matrix rows updated:
5 rows move to Implemented (Summary 62/3/5/1/76 = 147).
2026-09-06 13:43:34 +02:00
TapTap a2a82dd856 feat: implement --delay-updates receiver staging and publication
CI / lint (pull_request) Failing after 22s
CI / build-and-test (pull_request) Skipped
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
Stage every successfully written file under a private 0700 .fastsync-stage
directory inside the receive root and atomically publish all staged files
only after the whole protocol stream (manifest/delete handling included)
has completed, immediately before the success/outcome frame.  On any
abort/error before publication nothing is installed and staging is removed;
a publish failure aborts the transfer with best-effort cleanup of the
remainder (already-published files are not rolled back).  Crash leftovers
are wiped when the next delayed transfer starts.

Wire: new delay_updates config flag (selection-options block), protocol
version bumped to 2.6.0, client/server validation rejects --inplace.
CLI/usage/validation updated.  Works in single-threaded and -m modes
(exactly one write_thread stages files; the staged-file registry is
mutex-protected; publication runs once after both threads join).
--existing/--ignore-existing/--update decide against the final destination
at stage time; --backup is deferred to publication.  remove_source_files
outcomes are only sent after publication so skipped/unpublished sources are
never deleted.  Default (no flag) behavior is unchanged.

Tests: config wire round-trip, CLI parse, --inplace rejection, new
test_delay_updates unit suite (27 suites total), and integration
TestDelayUpdates covering single/-m parity, incremental reruns, remove
source files, receiver-skip ordering, and a deterministic publish-failure
abort path.
2026-09-06 13:20:03 +02:00
TapTap 8577f95550 fix: cppcheck cleanups (printf sentinel type, scanner test style)
CI / lint (push) Successful in 22s
CI / sanitizers (address) (push) Successful in 40s
CI / sanitizers (undefined) (push) Successful in 39s
CI / fuzz-build (push) Successful in 16s
CI / coverage (push) Successful in 34s
CI / build-and-test (push) Successful in 1m23s
CI / valgrind (push) Successful in 36s
2026-09-06 12:56:54 +02:00
TapTap 2931bb97b4 docs: recount RSYNC_COMPAT summary after Phase-2 wave A
6 rows flipped to implemented across the three merges; total stays 147.
2026-09-06 12:53:50 +02:00
TapTap 88ab4f65cb Merge feat/p2-itemize-output: structured change output (-i, --list-only, --out-format, --log-file-format)
Client-only per-file event/format model (change_list.c). Reviewed
(c-review APPROVE WITH NITS, all fixed); PR #263.
2026-09-06 12:52:46 +02:00
TapTap 9fbb86ca68 Merge feat/p2-temp-dir: implement --temp-dir
Receiver writes temps into a confined scratch dir and atomically renames
into place; EXDEV aborts; inplace/partial bypass; thread-safe temp names.
Uses existing wire field; no protocol bump. Reviewed (c-review APPROVE
WITH NITS, all fixed); PR #262.
2026-09-06 12:52:11 +02:00
TapTap f91ca70eda Merge feat/p2-one-file-system: implement -x/--one-file-system
Sender-side scanner stays within the source filesystem (-x), single and
multithreaded; default unchanged; client-only, no wire change. Reviewed
(c-review APPROVE WITH NITS, all fixed); PR #261.
2026-09-06 12:51:12 +02:00
TapTap 1b67f5ffcf docs: clarify %b semantics, temper thread-safety claim, harden %M scan; add -m coverage
CI / lint (pull_request) Successful in 20s
CI / sanitizers (undefined) (pull_request) Successful in 41s
CI / sanitizers (address) (pull_request) Successful in 41s
CI / fuzz-build (pull_request) Successful in 16s
CI / coverage (pull_request) Successful in 35s
CI / build-and-test (pull_request) Successful in 1m22s
CI / valgrind (pull_request) Successful in 36s
Address c-review nits on the itemize/output feature:
- RSYNC_COMPAT.md: state that %b is the source length (always == %l) because
  no wire-byte counter exists; keep Summary equal to the matrix (recounted:
  54 implemented / 84 not-implemented, 147 rows total - four rows flipped).
- change_list.h/.c: document bytes_sent == size; note itemize/out-format lines
  never interleave with each other but may interleave with legacy log
  messages sharing the stream; mark the %M stat() path best-effort.
- change_render_format scan in format_uses_mtime now mirrors the tokenizer
  (skips '%%' and unknown '%X' pairs) so a literal '%%M' no longer triggers
  the stat() fallback.
- Integration tests: --list-only under -m; a changed file on a second
  --incremental run emits exactly one '>f' line while unchanged files print
  nothing; --log-file + --log-file-format under -m.
2026-09-06 12:49:49 +02:00
TapTap 2d841405e6 test: cover -x cross-device skip without root; harden OneFileSystem tests
CI / lint (pull_request) Failing after 20s
CI / build-and-test (pull_request) Skipped
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
Add a rootless unit test that reaches the actual st_dev skip branch in both
the sequential and parallel (-m) scanners: a symlink nested under the scan
root points at a directory on /dev/shm (a different device than the build
fs) and, under --copy-links semantics, -x must drop that subtree while a
plain scan includes it. Skips only when no cross-device target exists.
Integration OneFileSystem test now cleans both dest dirs up front and
reports a busy test mountpoint instead of ignoring the umount result.
RSYNC_COMPAT.md notes that cross-filesystem mount-point subdirectories are
dropped entirely (rsync parity).
2026-09-06 12:49:26 +02:00
TapTap 00e8df4bcd fix: address c-review nits for --temp-dir engine
CI / lint (pull_request) Failing after 21s
CI / build-and-test (pull_request) Skipped
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
2026-09-06 12:48:07 +02:00
TapTap 7f5547e900 feat: add structured per-file change output model (-i, --list-only, --out-format, --log-file-format)
Implement rsync-style itemized output backed by one shared change-event
engine (src/client/change_list.c):
- -i/--itemize-changes prints ">f+++++++++ <path>" for files actually sent
  (single-threaded and -m); unchanged files print nothing.
- --list-only prints an ls-style listing of files that would be transferred
  without contacting the server or writing anything.
- --out-format=FORMAT prints a printf-style template per changed file
  (tokens %%f %%n %%l %%b %%M %%%%; unknown escapes preserved).
- --log-file-format=FMT logs each transferred file when --log-file is set.
Events are emitted from the per-file sender path shared by both transfer
modes, so the single sender thread is the only reporter (no races).
2026-09-06 12:34:10 +02:00
TapTap 19e6fc2205 feat: implement rsync --temp-dir for atomic receiver installs 2026-09-06 12:21:36 +02:00
TapTap 2bcc20aa08 feat: add -x/--one-file-system to stay within the source filesystem
Capture the transfer root's device (st_dev) at scanner creation and skip
descending into any subdirectory on a different device (a mount point).
Implemented sender/client-side only: sequential BFS and parallel (-m) root
scan apply the same scanner_same_filesystem decision; no wire/protocol change
and default behavior is unchanged. Unit tests cover the pure decision, same
device scanning in both modes, and CLI parsing; integration tests prove -x
leaves a single-filesystem tree byte-identical and, when root can mount a
tmpfs, skips a genuine cross-device subtree.
2026-09-06 12:18:43 +02:00
TapTap 06e83f2402 Merge docs: document missing rsync flags and add phased implementation plan
CI / lint (push) Successful in 20s
CI / sanitizers (address) (push) Successful in 40s
CI / sanitizers (undefined) (push) Successful in 38s
CI / fuzz-build (push) Successful in 16s
CI / coverage (push) Successful in 34s
CI / build-and-test (push) Successful in 1m22s
CI / valgrind (push) Successful in 36s
Reconcile the security-fixes branch's RSYNC_COMPAT.md updates onto dev:
- Keep dev's implemented statuses as authoritative and port the previously
  undocumented rsync 3.4.1 rows (one-file-system, -F, temp-dir/-T, identity
  mapping options, remote-option/-M, max-alloc) with code-verified statuses.
- Add the Implementation Difficulty Plan (Phases 1-6 + delivery order) as the
  sequencing roadmap; note that it is a superset snapshot and the Summary
  matrix is authoritative for shipped features.
- Recomputed the Summary counts from the table (was stale on dev): 51
  implemented / 3 alt-arg / 5 partial / 1 no-op / 87 not implemented.
2026-09-06 11:42:37 +02:00
TapTap f7c6c91e13 fix: set *failed on delta oversize branch; add -m RSF skip test
CI / lint (push) Successful in 20s
CI / sanitizers (address) (push) Successful in 41s
CI / sanitizers (undefined) (push) Successful in 39s
CI / fuzz-build (push) Successful in 16s
CI / coverage (push) Successful in 34s
CI / build-and-test (push) Successful in 1m23s
CI / valgrind (push) Successful in 35s
Review nits from independent review of the four fix branches:
- receive_delta_file STATUS_NEXT oversize branch now sets *failed=true
- receive_incremental_check oversize branch returns NULL (receiver sends the
  single STATUS_ERROR) instead of double-sending
- add multithreaded -m --ignore-existing --remove-source-files integration
  coverage so the writer-thread outcome path is exercised
2026-09-05 13:16:42 +02:00
TapTap 39805e4eee Merge fix/quality-cleanup into dev
fixes #260 (dead code, CLI =form/missing-arg diagnostics)
2026-09-05 13:12:33 +02:00
TapTap 9ef78ef48b Merge fix/perf-delta into dev
fixes #259 (delta_compute hash index)
2026-09-05 13:12:33 +02:00
TapTap 68e7ed57d0 Merge fix/security-hardening into dev
fixes #254 (receiver queue byte-budget) #258 (inplace setuid/truncate)

# Conflicts:
#	src/shared/multiprocessing.c
2026-09-05 13:12:31 +02:00
TapTap 11c591c5aa Merge fix/receiver-correctness into dev
fixes #251 #252 #253 #255 #256 #257 (remove-source-files skips, backup NULL-empty,
partial-dir install, lazy STATUS_CHECK, delta *failed, >64MiB files)
2026-09-05 13:11:41 +02:00
TapTap 1d61a1426e fix: #260 accept --opt=value uniformly and report missing arguments
- Correct misleading doc comments on set_string_option /
  set_positive_int_option / set_nonneg_int_option (they return 0/-1,
  not true/false).
- find_table_option_with_equals() now matches every OPTION_TABLE value
  option (OPT_STRING/OPT_POS_INT/OPT_NONNEG_INT/OPT_ULL), so forms such
  as --max-size=2G, --min-size=1K, --suffix=.bak, --timeout=30,
  --max-depth=5, --backup-dir=X parse instead of dying as 'Unknown option'.
  --max-size/--min-size now accept rsync-style binary suffixes (0 remains a
  valid 'no limit' byte count). Existing special handling for
  --compress-choice, --compress-level, --modify-window=, --chmod=,
  --skip-compress=, --compress-threads= is preserved.
- Options that require a separate value (-p, --exclude, --include,
  --delta-block, --delta-max, --server-port, --bwlimit, --chunk-size,
  --log-file, --exclude-from, --include-from, -T, --skip-compress,
  --compress-threads) now emit an explicit 'missing argument' diagnostic
  instead of falling through to the generic 'Unknown option' branch when
  given as the final argv entry.
- Add unit tests covering the = forms (--max-size=2G, --min-size=1K,
  --suffix=.bak, --timeout=30, --max-depth=5, --backup-dir=X) and a clean
  'missing argument' (not 'Unknown option') diagnostic for trailing
  --exclude/--server-port/--skip-compress/-T.
2026-09-05 12:58:30 +02:00
TapTap beb681c2dc fix: #260 remove dead receive_files() and mkdir_r()
receive_files() in src/server/server.c was a non-static, unprototyped,
zero-caller duplicate of receiver_process()/receiver_receive_files() in
src/server/receiver.c. Delete it along with the includes it uniquely pulled
(chunk.h, metadata.h, sys/stat.h, duplicate quoted unistd.h); remaining code
still uses file.h/config.h/protocol.h (via multiprocessing.h) directly.

mkdir_r() in src/shared/utils.c had zero callers in src/ and tests/; remove
the function and its declaration in utils.h, plus the unused libgen.h include.
2026-09-05 12:58:25 +02:00
TapTap 3704a6adaa test: integration coverage for #251 #252 #253 #257
- remove-source-files keeps sources skipped by --existing/--ignore-existing/
  --update (rsync reference behavior)
- --backup keeps <file>~, --suffix .bak, and --backup-dir backups
- --partial --partial-dir installs completed files in the destination
- a 100 MB file transfers end to end (>64 MiB whole-file cap regression)
2026-09-05 12:46:47 +02:00
TapTap 14c064a093 fix: #251 #252 #253 #255 #256 #257 receiver & remove-source correctness
#251 --remove-source-files deletes sources that were skipped receiver-side
     (--existing/--ignore-existing/--update). The receiver now reports a
     per-file outcome for every processed data file when the sender requests
     removal; the client only unlinks sources the receiver actually wrote.
     add remove_source_files to the wire config and bump the protocol to 2.5.0.
#252 --backup/--suffix/--backup-dir broken by NULL-vs-empty wire loss. Receivers
     canonicalize the empty wire string back to NULL for backup_dir, temp_dir,
     partial_dir and suffix, and --suffix is received unconditionally.
#253 --partial --partial-dir never installed completed files. file_save_to_disk
     now renames a fully written partial-dir file into the real destination.
#255 STATUS_CHECK read the entire old file before the size/mtime quick check.
     Old contents are only read when a checksum compare or delta needs them.
#256 receive_delta_file failure paths did not set *failed, so the caller sent
     STATUS_NEXT and waited for a body that never came. Every NULL return now
     marks the transfer failed.
#257 files >64 MiB could not transfer. Whole-file receive caps raised to the
     256 MiB connection/allocation ceiling (chunk caps stay 64 MiB) and the
     client ignores SIGPIPE so a server-side close surfaces as a clean error.

Unit tests added: config NULL-vs-empty round trip, incremental quick-check
skip/NEXT paths, delta oversize failure, partial-dir install, save-result
skip reporting.
2026-09-05 12:46:44 +02:00
TapTap 2234879b2f fix: #258 normalize mode and truncate on --inplace overwrite
The --inplace branch opened the destination with O_WRONLY|O_CREAT (no
O_TRUNC) and only restored metadata when the sender supplied it.  Two
flaws resulted:
1. An existing destination file kept its original mode when no metadata
   was sent, so setuid/setgid/sticky bits survived an overwrite (a root
   sync could leave a root-owned setuid binary controlled by a client).
2. A shorter payload left stale trailing bytes from the previous version
   because the file was never truncated to the new length.

In the inplace branch of file_to_disk_secure_impl:
- Always trim the file to the new payload length (ftruncate after the
  write) so stale trailing bytes can never survive; sparse targets keep
  their pre-size ftruncate.
- Always normalize the mode after a successful overwrite: apply the
  metadata-derived safe mode when metadata is present (as before), else
  fchmod to a safe default 0644, so setuid/setgid/sticky are cleared in
  both cases.
- The --update newer-destination check still runs before any truncation
  or chmod, preserving the skip semantics.

Adds unit tests in test_file.c: (a) setuid/sticky bits on an existing
destination are cleared after an inplace write with and without metadata,
(b) a shorter inplace payload leaves no trailing stale bytes.
2026-09-05 12:29:46 +02:00
TapTap 98120fc722 fix: #254 bound receiver queue by aggregate payload bytes
The per-connection memory budget (MAX_CONNECTION_MEMORY, 256 MiB) only
charged wire buffers via receive_data_limited.  Decompression buffers and
per-file chunk copies were not accounted for, and the multithreaded
receiver could enqueue up to 100 files (each up to 64 MiB uncompressed)
ahead of a slow disk writer, retaining ~6.4 GiB per connection.  A client
sending highly compressible chunks with little bandwidth could OOM the
host while the reserve never tripped.

Bound the receive pipeline by aggregate payload bytes instead of item
count alone:
- Export MAX_CONNECTION_MEMORY from protocol.h.
- PipelineContextReceiver tracks queued_bytes (payload bytes received but
  not yet released by the disk writer, i.e. queued or in the writer's
  hand) under the existing mutex.
- receiver enqueue now blocks while the queue is full by count OR when
  adding the file would push queued_bytes over the configured byte limit,
  applying backpressure to the sender instead of failing the transfer.
- The disk writer releases the byte budget after each file is freed and
  signals the not-full condition.
- The server sets the byte ceiling to
  MAX_CONNECTION_MEMORY - 2*MAX_CHUNK_SIZE so that the queued payloads
  plus the transient wire/decompression buffers of the one in-flight
  chunk stay within the per-connection budget.

The single-threaded receive path is already bounded: it writes files to
disk before reading the next chunk, so its transient is at most one
chunk's wire + decompressed + copied payload (~3 * MAX_CHUNK_SIZE, below
the budget).  Wire buffers remain charged exactly once by
receive_data_limited; this change does not double charge them.

Adds a deterministic unit test in test_multiprocessing.c proving that an
enqueue which would exceed the byte budget blocks until the writer
releases bytes.
2026-09-05 12:29:41 +02:00
TapTap 5fed0888aa perf: #259 delta_compute hash index over signature blocks 2026-09-05 12:21:26 +02:00
TapTap 9f8b58893b Apply clang-format 18 to merge-conflict resolution code
CI / lint (push) Successful in 15s
CI / sanitizers (address) (push) Successful in 37s
CI / sanitizers (undefined) (push) Successful in 37s
CI / fuzz-build (push) Successful in 16s
CI / coverage (push) Successful in 33s
CI / build-and-test (push) Successful in 1m18s
CI / valgrind (push) Successful in 34s
The Phase 1 merge conflict resolutions introduced formatting that failed
the CI lint job (clang-format 18.1.3). Reformatted with the exact CI
version; no functional changes.
2026-09-05 10:57:41 +02:00
TapTap 48dfd5dfa0 Fix incremental skip regression from modify-window nsec comparison
CI / lint (push) Failing after 3s
CI / build-and-test (push) Skipped
CI / sanitizers (address) (push) Skipped
CI / sanitizers (undefined) (push) Skipped
CI / fuzz-build (push) Skipped
CI / coverage (push) Skipped
CI / valgrind (push) Skipped
metadata_mtime_matches() required exact nanosecond equality at the default
modify_window=0, but destination write-time nsecs never match source
creation-time nsecs unless -M preserves metadata. Compare whole seconds
(rsync's default quick-check) at window 0; keep the subsecond-refined
tolerance for explicit window values.
2026-09-05 10:43:58 +02:00
TapTap 8384a1997b Merge remote-tracking branch 'origin/feat/rsync-delete-during-alias' into dev 2026-09-05 02:53:59 +02:00
TapTap a1981a78ad Merge remote-tracking branch 'origin/feat/rsync-old-dirs-aliases' into dev 2026-09-05 02:48:48 +02:00
TapTap b5646a0b9e Merge remote-tracking branch 'origin/feat/rsync-checksum-choice-alias' into dev 2026-09-05 02:44:04 +02:00
TapTap c6a341bd1e Merge remote-tracking branch 'origin/feat/rsync-partial-progress' into dev 2026-09-05 02:39:03 +02:00
TapTap c1e9509f16 Merge remote-tracking branch 'origin/feat/rsync-old-args' into dev 2026-09-05 02:34:29 +02:00
TapTap 8689778233 Merge remote-tracking branch 'origin/feat/rsync-no-options' into dev 2026-09-05 02:30:50 +02:00
TapTap cf0d10fccb Merge remote-tracking branch 'origin/feat/rsync-info' into dev 2026-09-05 02:25:36 +02:00
TapTap 90112a7585 Merge remote-tracking branch 'origin/feat/max-alloc' into dev 2026-09-05 02:13:35 +02:00
TapTap 3304541337 Merge remote-tracking branch 'origin/feat/compress-threads' into dev 2026-09-05 02:01:43 +02:00
TapTap 6a95efbe41 Merge remote-tracking branch 'origin/feat/skip-compress' into dev 2026-09-04 18:35:19 +02:00
TapTap 3b013bf9d2 Merge remote-tracking branch 'origin/feat/rsync-compression-aliases' into dev 2026-09-04 18:24:21 +02:00
TapTap 4f21498ee6 Merge remote-tracking branch 'origin/feat/chmod' into dev 2026-09-04 18:20:08 +02:00
TapTap 315e572d18 Merge remote-tracking branch 'origin/feat/executability' into dev 2026-09-04 18:13:34 +02:00
TapTap 55172ff6de Merge remote-tracking branch 'origin/feat/ignore-existing' into dev 2026-09-04 17:59:50 +02:00
TapTap c650f9a3d0 Merge remote-tracking branch 'origin/feat/existing' into dev 2026-09-04 17:50:55 +02:00
TapTap 192aff8c46 Merge remote-tracking branch 'origin/feat/update' into dev
# Conflicts:
#	src/client/client_cli.c
#	src/client/usage.c
#	src/shared/file.c
#	src/shared/file.h
#	src/shared/file_receive.c
#	tests/test_client_cli.c
2026-09-04 17:44:07 +02:00
TapTap 6d10116d50 Merge remote-tracking branch 'origin/feat/modify-window' into dev
# Conflicts:
#	RSYNC_COMPAT.md
#	src/client/client_cli.c
#	src/client/usage.c
#	src/server/receiver.c
#	src/server/server.c
#	src/shared/config.c
#	src/shared/config.h
#	src/shared/file_receive.c
#	tests/integration/test_features.py
#	tests/test_client_cli.c
#	tests/test_config.c
2026-09-04 17:40:16 +02:00
TapTap e42df1bde3 Merge remote-tracking branch 'origin/feat/size-only' into dev
# Conflicts:
#	src/client/client_cli.c
#	src/client/usage.c
#	src/server/receiver.c
#	src/server/server.c
#	src/shared/config.c
#	src/shared/config.h
#	tests/integration/test_features.py
#	tests/test_config.c
2026-09-04 17:30:14 +02:00
TapTap 9fc1e72972 Merge remote-tracking branch 'origin/feat/ignore-times' into dev
# Conflicts:
#	src/shared/config.c
#	tests/integration/test_features.py
#	tests/test_client_cli.c
#	tests/test_config.c
2026-09-04 17:27:04 +02:00
TapTap 71f1529222 Merge remote-tracking branch 'origin/feat/whole-file' into dev
# Conflicts:
#	tests/test_client_cli.c
#	tests/test_config.c
2026-09-04 17:24:22 +02:00
TapTap 2adfa5a03b Merge remote-tracking branch 'origin/feat/rsync-secluded-args' into dev
# Conflicts:
#	RSYNC_COMPAT.md
#	tests/test_client_cli.c
2026-09-04 17:22:53 +02:00
TapTap 47d1cbdae8 Merge remote-tracking branch 'origin/feat/rsync-debug' into dev
# Conflicts:
#	RSYNC_COMPAT.md
#	src/client/client_cli.c
#	src/client/usage.c
#	src/shared/log.h
#	src/shared/protocol.c
2026-09-04 17:21:48 +02:00
TapTap b59139589d Merge remote-tracking branch 'origin/feat/rsync-stderr-mode' into dev
# Conflicts:
#	RSYNC_COMPAT.md
#	src/shared/log.c
#	src/shared/log.h
#	tests/test_client_cli.c
2026-09-04 17:20:36 +02:00
TapTap 49e4af275f Merge remote-tracking branch 'origin/feat/8-bit-output' into dev
# Conflicts:
#	src/client/client_cli.c
#	src/client/client_send.c
#	src/shared/config.c
#	tests/test_client_cli.c
#	tests/test_config.c
#	tests/test_shared_utils.c
2026-09-04 17:18:01 +02:00
TapTap 8e1bc9bb9c Merge remote-tracking branch 'origin/feat/human-readable' into dev
# Conflicts:
#	RSYNC_COMPAT.md
#	src/client/client_send.c
#	tests/test_client_cli.c
2026-09-04 17:16:24 +02:00
TapTap 421a94773f Merge remote-tracking branch 'origin/feat/quiet' into dev
# Conflicts:
#	src/client/client_send.c
2026-09-04 17:14:45 +02:00
TapTap ead4651d12 Merge remote-tracking branch 'origin/feat/remove-source-files' into dev 2026-09-04 17:14:23 +02:00
TapTap 638d953b1d fix: bump protocol version for max alloc wire format
CI / lint (pull_request) Successful in 12s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-04 02:29:06 +02:00
TapTap 04cea295b5 fix: enforce max alloc in delta deserialization
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-04 02:25:26 +02:00
TapTap 820afd334a fix: reject malformed max-alloc sizes
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 38s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-04 02:22:00 +02:00
TapTap 2f553a2a43 fix: honor explicit protocol allocation sessions
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 36s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 34s
2026-09-04 02:18:44 +02:00
TapTap f990e86309 fix: make protocol allocation accounting atomic
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 34s
2026-09-04 02:12:22 +02:00
TapTap 0d306940e1 fix: bind allocation sessions in worker threads
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 36s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-04 01:56:34 +02:00
TapTap 552c19d3fe fix: guard skip-compress cleanup
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 38s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m17s
CI / valgrind (pull_request) Successful in 34s
2026-09-04 01:49:38 +02:00
TapTap 14c5da98d8 fix: free old snapshot on full transfer fallback
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 36s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 22:51:42 +02:00
TapTap aab45873ac test: cover compression option equals forms
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 33s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 22:47:41 +02:00
TapTap 605f79a450 fix: make rsync info none override verbose
CI / lint (pull_request) Successful in 12s
CI / sanitizers (address) (pull_request) Successful in 36s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 16s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 22:44:44 +02:00
TapTap 066c7ed1af fix: address 8-bit output re-review findings
CI / lint (pull_request) Successful in 10s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 22:40:30 +02:00
TapTap 97a628c3fd docs: document delete-during aliases
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 34s
2026-09-03 22:34:52 +02:00
TapTap 9f23b23893 fix: clarify unsupported directory option diagnostics
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 36s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 22:32:09 +02:00
TapTap 2ace6416a7 test: cover configured fsync save path
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 22:29:40 +02:00
TapTap e491e811e3 fix: enforce max alloc across protocol workers
CI / lint (pull_request) Successful in 12s
CI / sanitizers (address) (pull_request) Successful in 36s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 22:26:29 +02:00
TapTap ec02ee6dde fix: bound compression worker threads
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 34s
2026-09-03 22:22:04 +02:00
TapTap 98cbf3495d fix: complete skip-compress protocol handling
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m17s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 22:19:00 +02:00
TapTap 245a34fa11 fix: guard source removal against file replacement
CI / lint (pull_request) Successful in 11s
CI / sanitizers (undefined) (pull_request) Successful in 38s
CI / sanitizers (address) (pull_request) Successful in 39s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 22:11:18 +02:00
TapTap 68aaf89da7 fix: address update option review findings
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 34s
2026-09-03 21:59:55 +02:00
TapTap e37d5b9438 fix: address modify-window review findings
CI / lint (pull_request) Successful in 13s
CI / sanitizers (undefined) (pull_request) Successful in 38s
CI / sanitizers (address) (pull_request) Successful in 39s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 21:54:46 +02:00
TapTap 5b8cdf457b fix: bump protocol version for size-only field
CI / lint (pull_request) Successful in 12s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 21:51:29 +02:00
TapTap 1ebe261e05 fix: complete ignore-times incremental handling
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 21:49:10 +02:00
TapTap 05313f00ea fix: prioritize whole-file validation
CI / lint (pull_request) Successful in 11s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 34s
2026-09-03 21:46:50 +02:00
TapTap 4152bf379e fix: honor compression choice aliases
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 36s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 21:44:21 +02:00
TapTap f8291d895d fix: classify partial progress as incomplete
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 38s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m14s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 21:40:07 +02:00
TapTap ba236d5b2e fix: clarify secluded args compatibility semantics
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 21:37:45 +02:00
TapTap ec6160a8b9 fix: propagate SSH child setup failures
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 21:34:32 +02:00
TapTap 302c3cb664 fix: preserve explicit rsync option negations
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 36s
CI / sanitizers (undefined) (pull_request) Successful in 35s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 21:30:55 +02:00
TapTap caabd2e314 fix: reject unsupported rsync debug categories
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 38s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 21:28:21 +02:00
TapTap 21e6b1c1bc fix: address rsync info review findings
CI / lint (pull_request) Successful in 12s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 21:25:56 +02:00
TapTap 5350290f52 fix: reject unsupported rsync client stderr mode
CI / lint (pull_request) Successful in 12s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 21:21:22 +02:00
TapTap 5b997d5d25 fix: complete 8-bit output negotiation
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 35s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 33s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 21:18:35 +02:00
TapTap fb96c21d93 test: complete human-readable progress coverage
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 36s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 16s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 21:14:49 +02:00
TapTap 17b5b31845 fix: complete quiet option handling
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 34s
2026-09-03 21:12:43 +02:00
TapTap 48a36b0038 feat: add rsync delete-during alias
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 38s
CI / fuzz-build (pull_request) Successful in 16s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 18:34:06 +02:00
TapTap ce64fcd4b1 feat: recognize rsync old-dirs aliases
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 16s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 18:31:09 +02:00
TapTap 4982e15969 feat: add rsync-compatible fsync
CI / lint (pull_request) Successful in 12s
CI / sanitizers (address) (pull_request) Successful in 40s
CI / sanitizers (undefined) (pull_request) Successful in 38s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 18:28:02 +02:00
TapTap d69f5db652 feat: add rsync-compatible max-alloc limit
CI / lint (pull_request) Successful in 11s
CI / sanitizers (undefined) (pull_request) Successful in 38s
CI / sanitizers (address) (pull_request) Successful in 39s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 18:23:45 +02:00
TapTap f443b2986d feat: add compression worker threads
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 38s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 18:15:14 +02:00
TapTap 2a08a7ba5c feat: add skip-compress option
CI / lint (pull_request) Successful in 12s
CI / sanitizers (address) (pull_request) Successful in 36s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 17:56:27 +02:00
TapTap 36a373ff5d Add executability preservation option
CI / lint (pull_request) Successful in 12s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 17:39:50 +02:00
TapTap 3b85658024 Add remove-source-files option
CI / lint (pull_request) Successful in 11s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 17:33:52 +02:00
TapTap 5e67a721a4 feat: add rsync-compatible update option
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 39s
CI / sanitizers (undefined) (pull_request) Successful in 39s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 17:07:02 +02:00
TapTap 0b5b1a8643 feat: add rsync modify-window option
CI / lint (pull_request) Successful in 14s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 38s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Failing after 1m15s
CI / valgrind (pull_request) Successful in 32s
2026-09-03 17:01:45 +02:00
TapTap 2afb1d9649 feat: add rsync-compatible size-only option
CI / lint (pull_request) Successful in 12s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 16:54:07 +02:00
TapTap 230401c629 feat: add ignore-times incremental option
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 16:50:07 +02:00
TapTap c91db3266a feat: add whole-file transfer mode
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 38s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 16:47:02 +02:00
TapTap 44ce154e37 feat: add checksum choice alias
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 36s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 16:42:29 +02:00
TapTap cd27886dde feat: add rsync compression aliases
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 34s
2026-09-03 16:38:12 +02:00
TapTap a0aff000ad feat: add rsync partial progress alias
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 32s
2026-09-03 16:34:18 +02:00
TapTap 50f9ebb55b feat: add secluded args compatibility option
CI / lint (pull_request) Successful in 11s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 16:31:37 +02:00
TapTap 393a440c78 feat: add rsync old-args compatibility mode
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 34s
2026-09-03 16:27:45 +02:00
TapTap 2ff2688ef1 feat: support rsync no-option handling
CI / lint (pull_request) Successful in 12s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 16:22:18 +02:00
TapTap 5c7d82b79c feat: add rsync debug flags
CI / lint (pull_request) Successful in 12s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 16:18:46 +02:00
TapTap 4872d425a3 feat: add rsync info flags
CI / lint (pull_request) Successful in 12s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 16:13:56 +02:00
TapTap 60eb7f11fe feat: add rsync stderr output modes
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 38s
CI / sanitizers (undefined) (pull_request) Successful in 38s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 16:07:51 +02:00
TapTap eb4e9fba1f feat: add 8-bit output option
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 16s
CI / coverage (pull_request) Successful in 33s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 16:03:19 +02:00
TapTap 8ab5026224 feat: add human-readable output flag
CI / lint (pull_request) Successful in 12s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 34s
2026-09-03 15:58:48 +02:00
TapTap dbacc4f0e5 feat: add quiet client option
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 36s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m16s
CI / valgrind (pull_request) Successful in 33s
2026-09-03 15:52:36 +02:00
TapTap 1517b3fc35 docs: add rsync feature implementation plan
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 37s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / fuzz-build (pull_request) Successful in 14s
CI / coverage (pull_request) Successful in 31s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-01 21:25:44 +02:00
TapTap 5c055960b8 docs: correct compatibility summary
CI / lint (pull_request) Successful in 11s
CI / sanitizers (address) (pull_request) Successful in 36s
CI / sanitizers (undefined) (pull_request) Successful in 35s
CI / fuzz-build (pull_request) Successful in 15s
CI / coverage (pull_request) Successful in 32s
CI / build-and-test (pull_request) Successful in 1m15s
CI / valgrind (pull_request) Successful in 33s
2026-09-01 21:18:23 +02:00
TapTap 405e5b3b00 docs: document missing rsync flags 2026-09-01 21:18:08 +02:00
81 changed files with 17479 additions and 915 deletions

No files matched your search

+1 -1
View File
@@ -82,7 +82,7 @@ set(TEST_INCLUDES tests src/shared src/server src/client)
# Monolithic test binary (backward compatible)
file(GLOB TEST_SRCS "tests/test_*.c" "tests/runner.c")
add_executable(tests ${TEST_SRCS} ${SHARED_SRCS} ${FILE_STORE_SRCS} ${SERVER_RECEIVER_SRCS} src/client/scanner.c src/client/client_cli.c src/client/client_validation.c src/client/usage.c)
add_executable(tests ${TEST_SRCS} ${SHARED_SRCS} ${FILE_STORE_SRCS} ${SERVER_RECEIVER_SRCS} src/client/scanner.c src/client/change_list.c src/client/client_cli.c src/client/client_validation.c src/client/usage.c)
target_include_directories(tests PRIVATE ${TEST_INCLUDES})
target_compile_definitions(tests PRIVATE FASTSYNC_TEST_BUILD)
target_link_libraries(tests PRIVATE ${TEST_LIBS})
+36 -8
View File
@@ -66,8 +66,12 @@ replacement for every rsync feature or protocol mode.
- Owner/group, ACL, xattr, hard-link, device, and special-file handling is
incomplete or unavailable.
- Sparse-file handling does not yet preserve all holes correctly.
- `--partial`, `--partial-dir`, `--append`, and `--append-verify` are not yet
full rsync-style resumable transfers.
- `--partial`, `--partial-dir`, `-P`, `--append`, and `--append-verify` are not
yet full rsync-style resumable transfers. Interrupted files are not retained
for resumption.
- `--dirs` is not implemented. Its compatibility aliases `--old-dirs` and
`--old-d` are recognized but rejected explicitly rather than silently using
FastSync's recursive directory behavior.
- Several rsync short options currently have FastSync-specific meanings. Do
not assume every short option is interchangeable yet.
@@ -89,19 +93,28 @@ partial, alternate, and planned behavior.
| `-a, --archive` | Archive mode: enables `-c -m -M` (no `-s`) |
| `-m` | Multithreading mode |
| `-s` | Chunk serialization (batch all files per chunk) |
| `--secluded-args` | Accepted as an rsync compatibility option with no effect; `-s` remains chunk serialization. |
| `-f, --sendfile` | Sendfile zero-copy. Incompatible with `-c` / `-s`. TCP only. |
| `-M, --preserve` | Preserve supported file metadata (mode and mtime; ownership and atime are unsupported) |
| `-n, --dry-run` | Scan and print what would be transferred |
| `-p <port>` | SSH port (default: 22) |
| `-v, --verbose` | Enable debug logging |
| `-q, --quiet` | Suppress non-error output |
| `--progress` | Show real-time transfer speed |
| `--delete` | Delete files on receiver not present in source |
| `-P` | Enables partial-transfer mode and progress output (partial retention is incomplete) |
| `--delete` | Delete files on receiver not present in source (default timing: delete-after, i.e. only after the whole transfer succeeded) |
| `--delete-before` | Delete extras before the transfer starts (implies `--delete`) |
| `--delete-during`, `--del` | Delete extras once the keep-set is known, before data is applied (implies `--delete`) |
| `--delete-delay` | Delete extras only after a successful transfer (implies `--delete`) |
| `--delete-after` | Explicit delete-after timing (implies `--delete`) |
| `--exclude <pattern>` | Exclude files matching glob pattern (repeatable) |
| `--exclude-from <file>` | Read exclude patterns from a file (one per line) |
| `--include <pattern>` | Only transfer files matching glob pattern (repeatable, whitelist) |
| `--max-size <n>` | Skip files larger than n bytes |
| `--min-size <n>` | Skip files smaller than n bytes |
| `--max-alloc <SIZE>` | Maximum single allocation (binary units: B, K, M, G, T, P, E; default 1G) |
| `--incremental` | Skip files unchanged since last transfer (size + mtime). Auto-enables `--preserve`. Incompatible with `-s`. |
| `--existing` | Skip files not already present at the destination; update existing files normally. |
| `--bwlimit <KB/s>` | Bandwidth limit in kilobytes per second |
| `--chunk-size <n>` | Chunk size in bytes (default: 10485760) |
| `--timeout <sec>` | I/O timeout in seconds (default: 30) |
@@ -109,6 +122,7 @@ partial, alternate, and planned behavior.
| `--backup` | Backup existing destination files before overwriting |
| `--backup-dir <dir>` | Target directory for backups (requires `--backup`) |
| `--stats` | Print transfer statistics at end (bytes, files, timing) |
| `-h, --human-readable` | Format transfer byte sizes with binary units |
| `--max-depth <n>` | Maximum directory depth to recurse (0 = unlimited, default: 0) |
| `--log-file <path>` | Write log messages to file instead of stderr |
| `--source-dir <path>` | Source directory (overrides `FASTSYNC_SOURCE_DIR`) |
@@ -339,6 +353,10 @@ features without changing the meaning of ordinary compatibility options.
| `-m` | Enable the multithreaded scanner/loader/sender pipeline. |
| `-c [level]`, `-z [level]` | Enable streaming zstd compression, levels 1-22. |
| `--compress-level <n>` | Set the zstd compression level. |
| `--zc <alg>` | Alias for `--compress-choice`. FastSync supports `zstd` and `none`. |
| `--zl <n>` | Alias for `--compress-level`. |
| `--skip-compress <list>` | Skip compression for comma-separated suffixes; incompatible with `-s`. |
| `--compress-threads <n>` | Use `n` zstd compression workers. Requires compression and a zstd build with threaded support; the setting affects sender CPU work only. |
| `--chunk-size <bytes>` | Set the transfer chunk size. |
| `-s` | Enable FastSync chunk serialization. |
| `-f`, `--sendfile` | Use TCP `sendfile()` zero-copy transfer. Incompatible with compression and chunk serialization. |
@@ -359,6 +377,11 @@ particular, FastSync currently uses `-p` for SSH port, `-s` for chunk
serialization, and `-S` for sparse handling. These meanings must be reconciled
before FastSync can claim full rsync CLI compatibility.
`--secluded-args` is accepted as a long-form compatibility no-op. It does not
change FastSync's transport or protocol behavior. The rsync short form `-s` is
intentionally not aliased because it remains FastSync's chunk-serialization
option.
## Client Options
### Selection and transfer
@@ -367,7 +390,11 @@ before FastSync can claim full rsync CLI compatibility.
|---|---|
| `-a`, `--archive` | Enable current archive preset. Full rsync archive semantics are planned. |
| `-n`, `--dry-run` | Scan and report without writing files. |
| `--delete` | Request removal of destination entries absent from the source. The server must allow deletion. |
| `--delete` | Request removal of destination entries absent from the source. The server must allow deletion. Default timing is delete-after: extras are removed only after the whole transfer succeeded. |
| `--delete-before` | Delete extras before the transfer starts (implies `--delete`). |
| `--delete-during`, `--del` | Delete extras once the keep-set manifest is known, before data is applied (implies `--delete`; early mode, same engine behaviour as `--delete-before`). |
| `--delete-delay` | Delete extras only after a successful transfer (implies `--delete`; commit mode, same behaviour as `--delete-after`). |
| `--delete-after` | Explicit delete-after timing: delete only after the transfer succeeded (implies `--delete`). |
| `--exclude <pattern>` | Exclude matching paths. Repeatable. |
| `--include <pattern>` | Include matching paths. Repeatable. |
| `--exclude-from <file>` | Read exclude patterns from a file. |
@@ -465,11 +492,12 @@ defaults to the current directory. |
## Protocol and Security
FastSync protocol version `2.2.0` is shared by the client and server. The
FastSync protocol version `2.5.0` is shared by the client and server. The
current protocol is sender-driven and includes configuration negotiation,
incremental checks, checksums, manifests, keep-alives, abort handling, and
FastSync-native delta messages. Client and server versions must currently
match exactly.
including the maximum allocation limit, incremental checks, checksums,
manifests, keep-alives, abort handling, per-file remove-source results, and
FastSync-native delta messages.
Client and server versions must currently match exactly.
TLS provides encrypted TCP transport. Supplying `--ca` enables certificate
verification; without it, traffic is encrypted but peer identity is not
+254 -60
View File
@@ -6,11 +6,12 @@ This document maps rsync's full feature set to FastSync's current implementation
| Status | Count | Description |
|--------|-------|-------------|
| ✅ Implemented | 34 | Feature works end-to-end |
| ✅ Implemented | 80 | Feature works end-to-end |
| 🔀 Alt Arg | 3 | Functionality exists but under different flag/semantics |
| ⚠️ Partial | 1 | Flag parsed/stored but behavior incomplete |
| ❌ Not Implemented | 98 | Flag not recognized or no behavior |
| **Total** | **136** | |
| ⚠️ Partial | 5 | Flag parsed/stored but behavior incomplete |
| 🔄 Compatibility No-op | 1 | Flag is accepted for CLI compatibility but has no effect |
| ❌ Not Implemented | 58 | Flag not recognized or no behavior |
| **Total** | **147** | |
---
@@ -20,31 +21,31 @@ This document maps rsync's full feature set to FastSync's current implementation
|------|-------------------|-----------------|-------|
| `-a`, `--archive` | Archive mode is -rlptgoD | 🔀 Alt Arg | Maps to -c -m -M (compression + multithread + metadata) |
| `-v`, `--verbose` | Increase verbosity | ✅ Implemented | Sets `log_level=DEBUG` |
| `-q`, `--quiet` | Suppress non-error messages | ❌ Not Implemented | Removed because it had no effect |
| `-q`, `--quiet` | Suppress non-error messages | ✅ Implemented | Suppresses client output while preserving errors |
| `--help` | Show help | ✅ Implemented | Prints usage and exits; `-h` is not accepted |
| `-V`, `--version` | Print version | ✅ Implemented | |
| `--info=FLAGS` | Fine-grained info verbosity | ❌ Not Implemented | Removed because it had no effect |
| `--debug=FLAGS` | Fine-grained debug verbosity | ❌ Not Implemented | Removed because it had no effect |
| `--stderr=MODE` | Change stderr output mode | ❌ Not Implemented | |
| `--info=FLAGS` | Fine-grained info verbosity | ✅ Implemented | Supports `copy`, `misc`, `skip`, `stats`, `all`, and `none`; explicit flags override `--verbose`, and `none` suppresses info output; unsupported names are rejected |
| `--debug=FLAGS` | Fine-grained debug verbosity | ✅ Implemented | `io`, `proto`, `pack`, and `util` are supported; `--debug=help` lists flags; other rsync categories are rejected |
| `--stderr=MODE` | Change stderr output mode | ⚠️ Partial | `errors` (default) and `all` are supported; `client` is rejected because FastSync has no rsync message channel |
| `--no-motd` | Suppress daemon MOTD | ❌ Not Implemented | |
| `--exclude=PATTERN` | Exclude files matching pattern | ✅ Implemented | Glob matching in scanner |
| `--include=PATTERN` | Include files matching pattern | ✅ Implemented | Glob matching in scanner |
| `-C`, `--cvs-exclude` | Auto-ignore CVS files | ❌ Not Implemented | Removed because it had no effect |
| `-C`, `--cvs-exclude` | Auto-ignore CVS files | ✅ Implemented | Applies the well-known rsync default exclude set as exclude rules during scanning (RCS SCCS CVS CVS.adm RCSLOG cvslog.* tags TAGS .make.state .nse_depinfo *~ #* .#* ,* _$* *$ *.old *.bak *.BAK *.orig *.rej .del-* *.a *.olb *.o *.obj *.so *.exe *.Z *.elc *.ln core .svn/ .git/ .hg/ .bzr/); `.git/`-style repo dirs are pruned without descending |
## 2. Modifying Output
| Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------|
| `--stats` | Give transfer stats | ✅ Implemented | Prints file/byte counts |
| `-h`, `--human-readable` | Human-readable numbers | ❌ Not Implemented | Removed because it had no effect |
| `-i`, `--itemize-changes` | Per-file change summary | ❌ Not Implemented | Removed because it had no effect |
| `-h`, `--human-readable` | Human-readable numbers | ✅ Implemented | Formats transfer byte sizes using binary units |
| `-i`, `--itemize-changes` | Per-file change summary | ✅ Implemented | Prints rsync-style `>f+++++++++` lines to stdout only for files actually sent (also under `-m`); unchanged files print nothing, matching single-`-i` behavior |
| `--progress` | Show progress | ✅ Implemented | Progress callback in sender |
| `-P` | Same as --partial --progress | ❌ Not Implemented | |
| `--out-format=FORMAT` | Custom output format | ❌ Not Implemented | Removed because it had no effect |
| `-P` | Same as --partial --progress | ⚠️ Partial | Parses and enables progress, but interrupted files are not retained for resumable transfers |
| `--out-format=FORMAT` | Custom output format | ✅ Implemented | Per-transfer template on stdout; tokens `%f` `%n` `%l` `%b` `%M` `%%` (`%b` is the source length, always `== %l`; post-compression/delta wire bytes are not counted); unknown escapes preserved |
| `--log-file=FILE` | Log to file | ✅ Implemented | `log_file` config field |
| `--log-file-format=FMT` | Log format | ❌ Not Implemented | |
| `--8-bit-output` | Leave high-bit chars unescaped | ❌ Not Implemented | |
| `--list-only` | List files instead of copying | ❌ Not Implemented | Removed because it had no effect |
| `--log-file-format=FMT` | Log format | ✅ Implemented | Requires `--log-file`; writes one template line per transferred file using the same token set as `--out-format` (including `%b` `==` source length) |
| `--8-bit-output`, `-8` | Leave high-bit chars unescaped | ✅ Implemented | Applies to displayed paths and protocol debug output |
| `--list-only` | List files instead of copying | ✅ Implemented | `ls -l`-style listing of files that would be transferred; scans the source only, contacts no server, writes nothing; also works with `-n` |
## 3. File Selection
@@ -52,27 +53,29 @@ This document maps rsync's full feature set to FastSync's current implementation
|------|-------------------|-----------------|-------|
| `--exclude-from=FILE` | Read exclude patterns from file | ✅ Implemented | Reads patterns from file |
| `--include-from=FILE` | Read include patterns from file | ✅ Implemented | Reads patterns from file |
| `--filter=RULE` | Add file-filtering rule | ❌ Not Implemented | Removed because it had no effect |
| `--files-from=FILE` | Read source file list from file | ❌ Not Implemented | Removed because it had no effect |
| `-0`, `--from0` | Delimit *-from files with NULs | ❌ Not Implemented | |
| `--filter=RULE` | Add file-filtering rule | ✅ Implemented | Long option only: rsync's short `-f` conflicts with FastSync sendfile (see FastSync-specific list), so `-f` is not reassigned. Supported subset: `+`/`-` include/exclude, implicit-exclude patterns, `include`/`exclude` word forms, a leading `/` anchor (to the transfer root, or to a `.rsync-filter` file's directory), and a trailing `/` for dir-only rules; first match wins with a default of include inside the filter layer. Filters are an independent layer from `--exclude`/`--include` (an entry must pass both). Rejected with a clear error (no silent no-ops): `merge`/`dir-merge`/`hide`/`show`/`protect`/`risk`/`clear` words, rules that begin with `:`/`.`/`!` (merge/dir-merge/list-clear shorthands), and include/exclude modifiers other than `/` (`! C s r p x`) |
| `--files-from=FILE` | Read source file list from file | ✅ Implemented | Entries are paths relative to the source root (leading `./` stripped, `..`/absolute entries rejected at parse time, blank lines ignored; NUL-delimited with `-0`). A listed regular file is transferred; a listed directory transfers its whole subtree (FastSync recursion is always on, unlike rsync's non-recursive default). Non-listed paths and their subtrees are pruned by the scanner. A listed entry that does not exist under the source (and an empty list) is a hard error reported before any transfer; listing `.` (whole tree) and empty listed directories are fine. Scalability note: `file_list_affects` is O(list size) per scanned entry, so a very large `--files-from` list against a huge tree is quadratic; lists are typically small enough that this is acceptable, but it is the documented bound. The delete manifest still derives from what was actually sent, so `--delete` stays consistent with the subset |
| `-0`, `--from0` | Delimit *-from files with NULs | ✅ Implemented | `--files-from` entries become NUL-delimited; the flag may appear before or after `--files-from` on the command line. NUL mode preserves entry bytes exactly (trailing CR/LF are part of the name; only newline mode trims them) |
| `--max-size=SIZE` | Skip files larger than SIZE | ✅ Implemented | `max_size` in scanner |
| `--min-size=SIZE` | Skip files smaller than SIZE | ✅ Implemented | `min_size` in scanner |
| `-I`, `--ignore-times` | Don't skip files matching size+time | ❌ Not Implemented | |
| `--size-only` | Skip based on size only | ❌ Not Implemented | |
| `-@`, `--modify-window=NUM` | Mod-time comparison accuracy | ❌ Not Implemented | |
| `--existing` | Skip creating new files on receiver | ❌ Not Implemented | |
| `--size-only` | Skip based on size only | ✅ Implemented | With `--incremental`, ignores mtime |
| `-@`, `--modify-window=NUM` | Mod-time comparison accuracy | ✅ Implemented | Whole-second tolerance with nanosecond-aware comparisons |
| `--existing` | Skip creating new files on receiver | ✅ Implemented | Existing destination files continue through normal update handling |
| `--ignore-existing` | Skip updating existing files | ❌ Not Implemented | |
| `--remove-source-files` | Sender removes synced files | ❌ Not Implemented | |
| `--remove-source-files` | Sender removes regular files after confirmed transfer | ✅ Implemented | |
| `-x`, `--one-file-system` | Do not cross filesystem boundaries | ✅ Implemented | Sender scanner captures the root device and skips descending into mount-point crossings (`st_dev` differs); cross-filesystem mount-point subdirectories are dropped entirely, matching rsync |
| `-F` | Add the default `.rsync-filter` rules | ✅ Implemented | Reads one filter rule per line from each directory's `.rsync-filter` file during traversal and applies it to that directory's subtree; the current directory's rules are evaluated before its ancestors', so deeper files override shallower ones and per-directory files override the command-line `--filter`/`-C` base by default (matching rsync's first-match-wins precedence); `.rsync-filter` files are never transferred. The rsync `-FF` behavior (also `.cvsignore`) is out of scope; unsupported rule types inside the file abort with a clear error |
## 4. Directory Options
| Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------|
| `-r`, `--recursive` | Recurse into directories | ✅ Implemented | Default behavior |
| `-R`, `--relative` | Use relative path names | ❌ Not Implemented | Removed because it had no effect |
| `--no-implied-dirs` | Don't send implied dirs with -R | ❌ Not Implemented | |
| `-d`, `--dirs` | Transfer dirs without recursing | ❌ Not Implemented | |
| `--mkpath` | Create missing path components | ❌ Not Implemented | |
| `-R`, `--relative` | Use relative path names | ✅ Implemented | Meaningful together with `--files-from` (FastSync's default full-tree scan always mirrors the full source argument path below the destination root, so -R does not change it). With `-R` + `--files-from` each listed entry is transmitted under its bare relative destination path: an entry `sub/x.txt` lands at `<dest>/sub/x.txt` (its leading components preserved) instead of under the `<dest>/<full source path>` mirror. Only the path sent on the wire changes; the client still reads the absolute source path, and the delete manifest derives from the sent (relative) paths so `--delete` and `--remove-source-files` stay consistent in both layouts. Works single-threaded and under `-m` (including chunk serialization) |
| `--no-implied-dirs` | Don't send implied dirs with -R | ✅ Implemented | Client-side, meaningful only with `-R` + `--files-from`. rsync would normally create the ancestor directories implied by a listed file so it can be written; with `--no-implied-dirs` a listed file whose parent directory is not itself (or via an ancestor) explicitly listed cannot be placed, and FastSync fails the whole run up front with a clear error (`--no-implied-dirs: cannot place file '...': parent directory '...' is not explicitly listed`). Listing the directory (or an ancestor of it, or the whole tree `.`) permits the file. In every other mode the option has no effect. FastSync has no per-entry skip channel, so the rsync "omit the file" case is surfaced as a hard pre-transfer error |
| `-d`, `--dirs`, `--old-dirs`, `--old-d` | Transfer dirs without recursing | ✅ Implemented | `-d <dir>` transmits an explicit directory entry for the source-root directory, so the destination mirror is created empty and nothing is descended into. With `--files-from` exactly the listed items are transferred: a listed directory is created empty (no descent) and a listed file is transferred with its content; the dest layout follows the same -R rules as plain files. A new wire frame (`STATUS_MKDIR`) carries each directory entry (path only); the receiver creates it with the same confined mkdir-parent semantics as regular writes, in single-threaded and `-m` receivers (chunk serialization carries a per-entry type marker). Directory entries appear in the delete manifest so `--delete` prunes correctly. FastSync divergences: directory mtimes/modes are not transmitted, filter/`--exclude` rules are not re-applied to the listed dirs mode (there is no descent during which they would apply), and `-d` never creates the intermediate directories between the destination root and a listed file beyond the usual on-demand parent creation. Under `--delay-updates` only regular files are staged: directory entries are created immediately, so a delayed run that fails part way can leave the already-created empty directories behind (matching rsync, which also creates directories as it processes the file list and only delays regular-file data) |
| `--mkpath` | Create missing path components | ✅ Implemented | Wire option (client → server). At connection start the server creates the client's destination root directory (and any missing leading components below its own authorized root) when `--mkpath` is set, failing the connection cleanly if it cannot. Without `--mkpath` a destination root that does not exist yet is rejected up front (rsync semantics), so the flag is the only way to transfer into a not-yet-created destination directory. Creation is confined by the same secure mkdir walk as file writes (`O_NOFOLLOW`, no `..`) |
## 5. Transfer Modifications
@@ -93,22 +96,98 @@ This document maps rsync's full feature set to FastSync's current implementation
| `-b`, `--backup` | Make backups of overwritten files | ✅ Implemented | Backup before overwrite |
| `--backup-dir=DIR` | Backup directory hierarchy | ✅ Implemented | `backup_dir` config field |
| `--suffix=SUFFIX` | Backup suffix (default ~) | ✅ Implemented | `suffix` config field |
| `--delay-updates` | Put updated files in place at end | ❌ Not Implemented | |
| `--delay-updates` | Put updated files in place at end | ✅ Implemented | Successfully received files are staged under a private 0700 `.fastsync-stage` dir inside the receive root and atomically renamed into their final destinations only after the whole transfer (manifest/delete handling included) succeeds, just before the success/outcome frame is sent. The delete walker deliberately skips the staging dir at the receive root, so `--delete` removes genuine extras but never the staged files (deletion runs before publication; rsync's delete-after ordering is not implemented). `--existing`/`--ignore-existing`/`--update` decide against the final destination path at stage time; `--backup` moves the old file aside at publication. Incompatible with `--inplace` and with `--backup-dir=.fastsync-stage` (the internal staging name is reserved; both are rejected). The staging dir name is fixed, so two simultaneous delayed transfers to the same destination root are serialized with an exclusive advisory lock held for the whole transfer: the second session fails cleanly instead of corrupting the first. Aborting or failing before publication installs nothing and removes the staging tree; a crash between stage and publish leaves staged leftovers that the next delayed run wipes at start (process death releases the lock). A stage→publish failure aborts the transfer (best-effort cleanup of the not-yet-published staged files; already-published files are not rolled back). Works in single-threaded and `-m` modes |
| `-T`, `--temp-dir=DIR` | Create temporary files in DIR | ✅ Implemented | `--temp-dir` only; `-T` stays FastSync's `--timeout` alias. Scratch dir is resolved under the receive root; temp copies use a unique name there and are atomically renamed into place. If the scratch dir and destination are on different filesystems the atomic rename fails with EXDEV and the file save fails, which aborts the whole transfer (FastSync has no per-file skip/resume on a save error; rsync's non-atomic copy fallback is deliberately not used). `--inplace` and `--partial-dir` writes bypass the scratch dir |
## 7. Deletion
| Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------|
| `--delete` | Delete extraneous files from dest | ✅ Implemented | `use_delete` config field |
| `--delete-before` | Delete before transfer | ❌ Not Implemented | Removed because it had no effect |
| `--delete-during` | Delete during transfer | ❌ Not Implemented | |
| `--delete-delay` | Find deletions during, delete after | ❌ Not Implemented | |
| `--delete-after` | Delete after transfer | ❌ Not Implemented | Removed because it had no effect |
| `--delete-excluded` | Also delete excluded files | ❌ Not Implemented | Removed because it had no effect |
| `--max-delete=NUM` | Max files to delete | ❌ Not Implemented | Removed because it had no effect |
| `--ignore-errors` | Delete even with I/O errors | ❌ Not Implemented | |
| `--force` | Force deletion of non-empty dirs | ❌ Not Implemented | |
| `--prune-empty-dirs` | Prune empty dir chains | ❌ Not Implemented | Removed because it had no effect |
| `--delete` | Delete extraneous files from dest | ✅ Implemented | `use_delete` config field. Deletion is always derived from the transmitted keep-set manifest of the paths the sender sent/keeps (never from unchecked input), runs through the symlink-safe walker bounded by `MAX_SERVER_DELETE_COUNT`, and skips the `.fastsync-stage` staging dir under `--delay-updates`. FastSync's default timing when no timing flag is given is **delete-after** (extras are removed only once the whole transfer succeeded) — intentionally NOT rsync's `--del`/delete-during default, to preserve FastSync's commit-style safety. By default the destination mirror of a path the source scan pruned (filter/exclude/size rules) is **protected** from deletion — matching rsync, which does not delete excluded files under `--delete`; `--delete-excluded` opts back into deleting them (see below). The bounded deletion is **all-or-nothing**: if the destination holds more extras than the effective bound (a client `--max-delete=NUM` or the 100000-entry server bound) nothing is deleted and the run fails with a distinct error instead of silently truncating |
| `--delete-before` | Delete before transfer | ✅ Implemented | Implies `--delete`. The sender runs a full source pre-scan (paths only) and transmits the keep-set manifest BEFORE any file data; the receiver validates it, removes every destination entry not listed (all-or-nothing bounded walk, staging-dir skip, protected prefixes honored), then acks `STATUS_OK`. The sender only starts streaming after the deletion committed, or aborts if the receiver reported a deletion error. By definition the deletions already happened when a later transfer phase fails — rsync's delete-before is destructive the same way; a subsequent failure does not restore the removed files. Divergence: the keep-set is the pre-scan snapshot, so a file that appears on the source between the pre-scan and the data pass is still transferred but was not protected from deletion |
| `--del`, `--delete-during` | Delete during transfer | ✅ Implemented | Both spellings accepted; imply `--delete`. FastSync streams the source in a single directory scan and has no per-directory generator pass, so deletions cannot be interleaved per-directory the way rsync's delete-during does. `--delete-during` therefore selects the same early engine mode as `--delete-before` (manifest transmitted before any data, extras removed and acknowledged before data is applied); observable success/failure behaviour equals `--delete-before`. That is the documented divergence from rsync, where `--del` is the default meaning of `--delete` |
| `--delete-delay` | Find deletions during, delete after | ✅ Implemented | Implies `--delete`. Commit-mode timing: extras are removed only after the whole transfer succeeded. rsync's delete-delay records the deletion list during its scan and applies it at the end; FastSync never snapshots the destination while data flows (the keep-set is the transmitted manifest and the destination is listed only at deletion time), so `--delete-delay` is implemented as the same end-of-transfer commit as `--delete-after` with identical safety. That is the documented divergence |
| `--delete-after` | Delete after transfer | ✅ Implemented | Implies `--delete`. The delete-after timing is also what plain `--delete` does: the keep-set manifest closes the data stream and the receiver commits the bounded deletion only after the terminal `STATUS_FINISHED` proves the whole transfer (every data frame received and stored) succeeded. A failed or aborted transfer removes nothing |
| `--delete-excluded` | Also delete excluded files | ✅ Implemented | `delete_excluded` config field. Under `--delete` FastSync now protects (rsync's default) the destination mirror of paths the sender's source scan pruned by user-selection rules — the `--filter`/`-F`/`-C` layer, the legacy `--exclude`/`--include` layer, and `--max-size`/`--min-size`. The sender transmits those concrete pruned paths as **protected prefixes** in the delete-manifest frame (see the Phase-3 notes below); the walker never descends into or removes them. `--delete-excluded` opts back in: the sender sends an empty protected list, so the excluded destination mirrors become ordinary extras and are removed. Divergences (documented): protection is derived only from what the source scan actually pruned — a stray destination-only file that happens to match an exclude rule is not protected (FastSync never re-applies rules to the destination, keeping deletion sender-derived), and `--files-from` subset pruning stays keep-set-only (an unlisted source path is treated as absent and its mirror is deletable, matching the `--files-from` delete note below). The two are orthogonal: `--delete-excluded` removes filter-excluded mirrors; it does not make `--files-from` prune things |
| `--max-delete=NUM` | Max files to delete | ✅ Implemented | `max_delete` config field (default -1 = no client limit; 0 = delete nothing). NUM bounds a `--delete` run with rsync's all-or-nothing semantics: the receiver rehearses the deletion first and, if the destination holds more than NUM extras, deletes NOTHING and fails the transfer with a distinct `--max-delete` error. A run at or below NUM deletes exactly the extras. NUM only applies together with `--delete` (it is inert otherwise, matching rsync). The hard server bound `MAX_SERVER_DELETE_COUNT` (100000) still caps the walk; a NUM above it never raises that cap, and exceeding the server bound is its own all-or-nothing error. Directories count toward the limit (each removed empty directory is one deletion), like rsync |
| `--ignore-errors` | Delete even with I/O errors | ✅ Implemented | Sender-side, client-only config field. rsync suppresses `--delete` when the transfer had I/O errors; FastSync's equivalent is a source-scan I/O error (an unreadable directory, e.g. EACCES): by default the scan aborts the run so no deletion happens. With `--ignore-errors` the scan continues past the unreadable directory, the readable tree is transferred and the deletion still runs (the mirror of the unreadable directory is treated as an extra). The run still exits non-zero (the error is reported, matching rsync's error status). Divergence: without the flag FastSync aborts the whole run on the scan error, whereas rsync transfers the rest of the tree and merely skips the deletion; both leave the deletion undone |
| `--force` | Force deletion of non-empty dirs | ✅ Implemented | `force_delete` receiver config field (crosses the wire). rsync's `--force` lets an incoming non-directory replace a destination directory; FastSync implements exactly that: when a regular file is written to a path that is currently a (possibly non-empty) destination directory, `--force` removes that directory tree first — confined to the receive root and symlink-safe (O_NOFOLLOW fd walk, symlinks removed by name, never followed) — so the atomic install can place the file. Without `--force` such a write fails and the run aborts. Divergence: `--force` acts on the immediate-install path only; under `--delay-updates` a blocking directory is not cleared (publication renames over regular files) |
| `--prune-empty-dirs` | Prune empty dir chains | ✅ Implemented | Long-only: FastSync's `-m` is already multithreading (recorded divergence — rsync's `-m` short form is not reassigned). FastSync's recursive transfer never emits directory entries, so empty directories are inherently never transferred (which is rsync's `-m` behavior) and truly-empty destination directory chains are removed by `--delete` regardless of this flag. The flag's additional real effect is on the `--dirs` explicit directory-entry generator: a plain `-d <empty-dir>` run omits the empty source directory's entry, so nothing is created at the destination (no `STATUS_MKDIR`, no `-i`/`--out-format` change line, and an existing empty mirror becomes an extra that `--delete` prunes). Explicitly `--files-from`-listed directories always pass through (documented `--files-from` behavior). A directory that still holds an excluded-but-protected file survives, matching the `--delete-excluded` default |
**Deletion-timing implementation notes (Phase 3):** the delete flags above are
real. Two new config booleans (`delete_during`, `delete_delay`) join the already
serialized `delete_before`/`delete_after`, so the on-the-wire config layout
changed and `PROTOCOL_VERSION` was bumped **2.7.0 → 2.8.0** (peers must match).
The `STATUS_MANIFEST` frame is count-delimited and position-independent: the
receiver commits the deletion either when the manifest arrives (early modes:
`--delete-before`/`--delete-during`, which additionally acknowledge with
`STATUS_OK` before data flows) or after the terminal `STATUS_FINISHED` proves
the whole transfer succeeded (commit modes: plain `--delete`/`--delete-after`/
`--delete-delay`). Timing is chosen purely from the config, so server policy
(`--allow-delete` off) still disables deletion without deadlocking the early
manifest ack. `--delete-delay` and `--delete-during` are each implemented as
the closest safe approximation their engine mode allows; the divergences are
noted in the rows above.
**Deletion-policy notes (Phase 3, delete-policy wave):** this wave made the
deletion family real — `--delete-excluded`, `--max-delete`, `--ignore-errors`,
`--force`, `--prune-empty-dirs` — and, to support them, the `STATUS_MANIFEST`
frame now carries **two sections**: the keep-set paths followed by a list of
**protected prefixes** (destination-relative paths the source scan pruned by
user-selection rules, which the walker must never delete unless
`--delete-excluded` opted out). Two config booleans were added for the wave:
`force_delete` (crosses the wire; the receiver clears a directory that blocks an
incoming file) and `ignore_errors` (client-only; the sender's scan continues
past an unreadable directory). `max_delete`'s default became -1 ("no client
limit"). These wire/layout changes bumped `PROTOCOL_VERSION` **2.8.0 → 2.9.0**
(peers must match). All four wire additions — `force_delete`,
`delete_excluded`, `prune_empty_dirs`, `max_delete` — round-trip unchanged and
are validated on receive.
The deletion walker is now **all-or-nothing**: before any unlink it rehearses
the deletion (an fd-relative walk identical to the delete pass, counting every
regular file it would unlink and every directory it would remove) and refuses to
start when the extras exceed the effective bound — a client `--max-delete=NUM`
below the hard bound, or the hard `MAX_SERVER_DELETE_COUNT` (100000) bound
itself. Previously the walker removed up to `MAX_SERVER_DELETE_COUNT` extras and
then reported an error (a truncated deletion); it now removes nothing and fails
with an error naming the bound. Directories count toward the bound. A directory
that still holds entries the walker leaves in place (a protected excluded file,
a kept manifest entry, a symlink) is left behind rather than failing the run —
matching rsync's "cannot delete non-empty directory" behaviour. The
all-or-nothing guarantee holds only while the destination is not concurrently
modified: rehearsal and delete are two separate walks, so a concurrent change
between them (another process adding or removing destination entries) can make
the actual deletion diverge from the counted set.
Manifest size: the sender's keep-set and protected-prefix collections (streaming
or early pre-scan) are unbounded, but the receiver rejects a manifest beyond
`MAX_MANIFEST_ENTRIES` (1 048 576 entries, applied to EACH section — a frame can
therefore total up to 2 097 152 entries) / `MAX_MANIFEST_BYTES` (16 MB of paths,
counted across BOTH sections) as a hard protocol error. A heavily filtered
source whose exclusion list grows large thus fails the run cleanly on the
receiver (STATUS_ERROR) instead of being silently truncated. In the commit
modes this only means the deletion is refused after the data already arrived; in
the early modes (`--delete-before`/`--delete-during`) the manifest is the first
frame, so an oversized keep-set or protected list aborts the whole transfer
BEFORE any data is sent. Keep the source tree small enough for the receiver's
manifest caps when using the early timing.
Early-delete ACK wait: after committing a large deletion (up to
`MAX_SERVER_DELETE_COUNT` removals) the receiver's `STATUS_OK`/`STATUS_ERROR`
reply can legitimately take much longer than a normal round trip, so the sender
waits for that single ACK with an extended explicit deadline (1 hour) instead
of the default 60 s per-message receive window. A receiver that is genuinely
gone still aborts the wait via connection close/error; the extended bound only
protects against aborting after the deletion already committed on the receiver.
Flag-conflict policy: unlike rsync's last-one-wins behaviour, every deletion
timing flag implies `--delete`, and combining a timing flag with `--no-delete`
(in either argument order) — or more than one timing flag — is rejected as a
configuration error rather than silently resolved. Note the check is
order-independent because it runs over the fully parsed config. The deletion
POLICY flags (`--delete-excluded`, `--max-delete`, `--ignore-errors`, `--force`)
do NOT imply `--delete`; without `--delete` they are inert (matching rsync).
## 8. Metadata Preservation
@@ -119,7 +198,7 @@ This document maps rsync's full feature set to FastSync's current implementation
| `-o`, `--owner` | Preserve owner | ✅ Implemented | Part of -M |
| `-g`, `--group` | Preserve group | ✅ Implemented | Part of -M |
| `-t`, `--times` | Preserve modification times | ✅ Implemented | Part of -M |
| `-E`, `--executability` | Preserve executability | ❌ Not Implemented | |
| `-E`, `--executability` | Preserve executability | ✅ Implemented | Preserves executable permission bits (implies metadata preservation) |
| `--chmod=CHMOD` | Affect file permissions | ✅ Implemented | Supports numeric and symbolic `ugo` `rwx` changes; retains receiver safety masking |
| `-A`, `--acls` | Preserve ACLs | ❌ Not Implemented | Removed because it had no effect |
| `-X`, `--xattrs` | Preserve extended attributes | ❌ Not Implemented | Removed because it had no effect |
@@ -135,6 +214,12 @@ This document maps rsync's full feature set to FastSync's current implementation
| `-J`, `--omit-link-times` | Omit symlinks from --times | ❌ Not Implemented | |
| `--super` | Receiver attempts super-user activities | ❌ Not Implemented | |
| `--fake-super` | Store/recover privileged attrs via xattrs | ❌ Not Implemented | |
| `--open-noatime` | Avoid changing access time when opening files | ❌ Not Implemented | |
| `--numeric-ids` | Do not map uid/gid by name | ❌ Not Implemented | |
| `--usermap=STRING` | Map usernames | ❌ Not Implemented | |
| `--groupmap=STRING` | Map group names | ❌ Not Implemented | |
| `--chown=USER:GROUP` | Map owner and group | ❌ Not Implemented | |
| `--copy-as=USER[:GROUP]` | Perform the copy as another user/group | ❌ Not Implemented | |
## 9. Symlink Handling
@@ -159,22 +244,22 @@ This document maps rsync's full feature set to FastSync's current implementation
| Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------|
| `--checksum` | Skip based on checksum | ✅ Implemented | With `--incremental`, compares xxHash64 content checksums; `-c` remains compression |
| `--checksum-choice=STR` | Choose checksum algorithm | ❌ Not Implemented | xxHash used internally |
| `--compare-dest=DIR` | Compare dest files relative to DIR | ❌ Not Implemented | Removed because it had no effect |
| `--copy-dest=DIR` | Include copies of unchanged files | ❌ Not Implemented | Removed because it had no effect |
| `--link-dest=DIR` | Hardlink to files when unchanged | ❌ Not Implemented | Removed because it had no effect |
| `--fuzzy`, `--no-fuzzy` | Find similar file for basis | ❌ Not Implemented | |
| `--checksum` | Skip based on checksum | ✅ Implemented | With `--incremental`, compares per-file whole-file content digests to skip unchanged files. The digest algorithm is `xxh64` with seed 0 by default and is selectable via `--checksum-choice`/`--cc` (xxh64/xxhash or md5) and `--checksum-seed=NUM` (see those rows); `-c` remains compression |
| `--checksum-choice=STR`, `--cc=STR` | Choose checksum algorithm | ✅ Implemented | Real algorithm selection for the per-file whole-file digest used by the `--incremental`/`--checksum` handshake and by the basis-dir content verification. FastSync genuinely supports `xxh64` (the default, exact xxHash64, seeded by `--checksum-seed`) and `md5` (via OpenSSL EVP); `xxhash` is accepted as rsync's spelling of xxHash64. Any other name (md4/sha1/sha256/crc32/none/…) is rejected with a clear error at parse time — never a silent no-op. `--cc` is the alias (`--cc=ALG` and space forms both parse). The algorithm id and seed cross the wire with the config frame, so the receiver hashes its on-disk old file with the SAME algorithm+seed the sender used and both agree on a match; the sender's digest and the receiver's comparison live in the per-file `STATUS_CHECK` handshake, which now carries a length-prefixed, bounded (1..16 byte) digest instead of a fixed 64-bit value. The digest length is validated on receive but not pinned to the negotiated algorithm id. Protocol/layout: `PROTOCOL_VERSION` bumped **2.9.0 → 2.10.0** (peers must match). Defaults preserve the pre-existing behavior byte-for-byte (xxh64, seed 0). Like rsync, the choice only takes effect where a whole-file digest is actually computed (`--checksum` on, or a basis-dir flag); it does not itself enable `--checksum`. Closely-related divergence: the delta BLOCK strong checksum (§11 delta) stays xxHash32 — `--checksum-choice` selects only the whole-file digest, matching rsync where the per-block checksum is independent of the whole-file checksum choice |
| `--compare-dest=DIR` | Compare dest files relative to DIR | ✅ Implemented | DIR is a receiver-side basis relative to the destination root (confined below it; absolute/`..`/`.` rejected, `//` collapsed and trailing `/` dropped). On the receiver's per-file check (implies `--incremental`) an exact match = same size + mtime (unless `--size-only`; `-I` disables matching) **and** equal xxHash64 of the sender's file; a match suppresses the data transfer. compare-dest never copies: it only skips a file the destination does **not** already hold (sparse destination, rsync parity), and is consulted before the normal delta/full paths. Repeatable; searched in command-line order, first match wins. Divergences: when the destination already holds a *different* version rsync deletes it but FastSync instead transfers the data (keeps the mirror complete; never deletes without `--delete`); attribute-only differences on a match are not re-applied (data is skipped so the sender never sends metadata); content is verified by xxHash64, stricter than rsync's default quick check. Sizing: FastSync's whole-file payload limit is 256 MiB on **every** transfer path (not basis-specific); rsync applies basis dirs to arbitrary sizes, so FastSync refuses a basis run whose source contains a larger file up front with a clear error before any transfer. Wire: a basis-count field is always present on the config frame (protocol 2.9.0, so clients and servers must both be 2.9.0) |
| `--copy-dest=DIR` | Include copies of unchanged files | ✅ Implemented | Same basis rules as `--compare-dest`, but an exact match materializes a **local copy** of the DIR file into the destination (via the normal atomic temp+rename store path, so `--existing`/`--ignore-existing`/`--update`/`--backup`/`--delay-updates` all still apply) instead of transferring data. Repeatable; command-line order = priority. Content is xxHash64-verified before the copy. Divergences: a basis-hit destination keeps the basis file's own mode/uid/gid and mtime (the sender sends no metadata on a skip), so with `--size-only` its mtime can differ from the source and attribute-only differences are copied with the basis attributes rather than rsync's "copy + fix attributes". Requires `--incremental` (implied); incompatible with `-s`. Wire: protocol 2.9.0 |
| `--link-dest=DIR` | Hardlink to files when unchanged | ✅ Implemented | Same basis rules as `--copy-dest`, but an exact match installs an atomic **hard link** to the DIR file (temp hard link + rename) so no data or disk space is used; where the link is impossible (basis on another filesystem, filesystem refuses links) it falls back cleanly to a byte-identical local copy, never a corrupt/partial file. `--delay-updates` stages the link and publishes by rename, so the final entry stays a real hard link. Repeatable (searched in command-line order, first match wins). Content is xxHash64-verified before linking. Divergences and caveats: an already up-to-date destination file is not re-linked to a basis file (only files that would otherwise be written are linked); a link keeps the basis inode's own mode/uid/gid and mtime — metadata is never written through the shared inode (that would mutate the basis file), so a later `--inplace` run that rewrites such a destination path **will mutate the basis snapshot** through the shared inode (use `--copy-dest` when the destination must stay independently writable); with `--size-only` the linked mtime can differ from the source; a `--remove-source-files` source satisfied by a basis dir is treated as skipped and therefore **retained** (never removed); basis dirs are excluded from `--delete`. Requires `--incremental` (implied); incompatible with `-s`. Wire: protocol 2.9.0 |
| `-y`, `--fuzzy`, `--no-fuzzy` | Find similar file for basis | ✅ Implemented | `-y/--fuzzy` is a pure bandwidth optimization on the existing receiver-driven delta path: when a file must be transferred and the destination holds no usable content at the exact path (file absent, or the destination file is outside the delta engine's size bounds), the receiver searches the SAME destination directory for an existing regular file whose basename is similar to the incoming name and uses it as the delta basis, so the sender transmits only the differences instead of the whole file. The output is always byte-exact regardless of which (or whether any) basis is chosen. Decision location: the receiver performs the candidate search inside `receive_incremental_check` and sends the normal `STATUS_DELTA_SIGNATURE`; the sender never learns the basis was a different file, so no new frame type or sender logic was needed — only the config frame grew a `fuzzy` boolean, so `PROTOCOL_VERSION` was bumped **2.8.0 → 2.9.0** (peers must match). Similarity heuristic (deterministic, simpler than rsync's deliberately-fuzzy matching, and documented precisely): candidates are the target's sibling entries in its destination directory, opened `O_NOFOLLOW`/`AT_SYMLINK_NOFOLLOW` under the confined root (symlinks never followed; nothing outside the destination root is ever read or hashed); dotfiles, directories, the target's own name, and the `.fastsync-stage`/temp scratch names are excluded; like the ordinary delta path, the block signature the receiver transmits is derived from on-disk content it may not otherwise send, so a negotiated `--fuzzy` run exposes the destination's sibling files (at block granularity) to the sender as a known-plaintext oracle — the same information class as the normal delta handshake over the file being replaced; the size gate is the delta engine's own bounds (both files ≥ 16 KiB, ≤ `--delta-max`, ratio ≤ 10×) rather than rsync's ~1.5× size window; the name gate is a Levenshtein edit distance between the basenames accepted only when ≤ half the length of the longer basename; the single best candidate (smallest distance, tie-break size closest to the incoming file then lexicographically smaller basename) is read; the directory scan is capped at 4096 entries so a pathological directory cannot stall a transfer. When fuzzy applies: only to files the receiver would otherwise send whole — the destination's own file is always preferred as the delta basis when it exists and fits the delta size bounds, so fuzzy does NOT replace an existing-but-different destination basis; FastSync's 10× delta size-ratio bound means an existing destination file that is too far away in size still lets the fuzzy search run. When no similar candidate exists the transfer falls back to the normal whole-file transfer. rsync-divergence note: rsync's own matching uses a fuzzy name/size rule set; FastSync implements the closest safe deterministic approximation above. Because FastSync's delta machinery is off by default (rsync's is on), `--fuzzy` implies `--incremental` + `--delta` (unless `--whole-file`/`-W` or an explicit `--no-delta` switched delta off, in which case fuzzy is inert — matching rsync where `--whole-file` makes fuzzy irrelevant). Unlike the basis-dir options, `--fuzzy` honors an explicit `--no-incremental` (it does not force the handshake back on); an explicit `--no-incremental` also suppresses the delta implication so no invalid `--delta requires --incremental` config results. `--no-fuzzy` negates it. All surrounding semantics are untouched: a fuzzy-reconstructed file is stored as a normal file, so `--remove-source-files`, itemize/`-i`, `--stats`, `--backup`, `--delay-updates`, `--existing`/`--ignore-existing`/`--update` behave exactly as for a whole-file transfer (the fuzzy delta does not skip the file) |
## 12. Compression
| Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------|
| `-z`, `--compress` | Compress file data | 🔀 Alt Arg | Always uses zstd (rsync supports multiple algorithms) |
| `--compress-choice=STR` | Choose compression algorithm | ❌ Not Implemented | Removed because it had no effect; FastSync always uses zstd |
| `--compress-level=NUM` | Set compression level | ✅ Implemented | 1-22, default 5 |
| `--compress-choice=STR`, `--zc=STR` | Choose compression algorithm | ✅ Implemented | FastSync supports `zstd` and `none` |
| `--compress-level=NUM`, `--zl=NUM` | Set compression level | ✅ Implemented | 1-22, default 5 |
| `--compress-threads=NUM` | Set compression threads | ❌ Not Implemented | |
| `--skip-compress=LIST` | Skip compress for suffixes | ❌ Not Implemented | Internal skip for hardcoded types; not user-configurable |
| `--skip-compress=LIST` | Skip compress for suffixes | ✅ Implemented | Comma-separated, case-insensitive suffix list; empty list skips none; incompatible with FastSync chunk serialization (`-s`) |
## 13. Connectivity
@@ -189,6 +274,7 @@ This document maps rsync's full feature set to FastSync's current implementation
| `--address=ADDRESS` | Bind address for outgoing socket | ❌ Not Implemented | Removed because it had no effect |
| `-4`, `--ipv4` | Prefer IPv4 | ❌ Not Implemented | Removed because it had no effect |
| `-6`, `--ipv6` | Prefer IPv6 | ❌ Not Implemented | Removed because it had no effect |
| `--remote-option=OPT`, `-M` | Send an option only to the remote side | ❌ Not Implemented | `-M` is FastSync's metadata-preservation flag |
## 14. Daemon Mode
@@ -210,8 +296,9 @@ This document maps rsync's full feature set to FastSync's current implementation
| Protocol version check | Verify compatible versions | ✅ Implemented | `config_receive()` |
| Max data/string/chunk sizes | Prevent OOM attacks | ✅ Implemented | Per-message limits |
| Per-connection memory limit | 1GB per connection | ✅ Implemented | `MAX_CONNECTION_MEMORY` |
| `--max-alloc=SIZE` | Limit a single memory allocation | ✅ Implemented | Caps the largest single allocation; binary units, default 1G |
| `--trust-sender` | Trust remote sender's file list | ❌ Not Implemented | |
| `--old-args` | Disable modern arg protection | ❌ Not Implemented | |
| `--old-args` | Disable modern arg protection | ✅ Implemented | SSH-only legacy mode; restores raw remote command construction and permits shell interpretation of the configured server path |
| `--ignore-missing-args` | Ignore missing source args | ❌ Not Implemented | |
| `--delete-missing-args` | Delete missing source args | ❌ Not Implemented | |
@@ -229,12 +316,118 @@ This document maps rsync's full feature set to FastSync's current implementation
|------|-------------------|-----------------|-------|
| `--stop-after=MINS` | Stop after N minutes | ❌ Not Implemented | |
| `--stop-at=TIME` | Stop at specified time | ❌ Not Implemented | |
| `--fsync` | Fsync every written file | ❌ Not Implemented | |
| `--fsync` | Fsync every written file before publication | ✅ Implemented | |
| `--protocol=NUM` | Force older protocol version | ❌ Not Implemented | |
| `--iconv=CONVERT_SPEC` | Charset conversion | ❌ Not Implemented | |
| `--checksum-seed=NUM` | Set checksum seed | ❌ Not Implemented | |
| `-s`, `--secluded-args` | Use protocol to send args | ❌ Not Implemented | |
| `--no-OPTION` | Turn off implied option | ❌ Not Implemented | |
| `--checksum-seed=NUM` | Set checksum seed | ✅ Implemented | Sets the seed for FastSync's whole-file xxHash64 digest (full 64-bit seed) and for the delta path's per-block xxHash32 strong checksum (low 32 bits of the seed). An explicit seed deterministically changes every computed digest on BOTH endpoints (sender and receiver share the seed via the config frame, protocol 2.10.0), so identical runs with the same seed skip the same files and a changed seed changes the digests — the explicit-seed path that makes xxHash comparisons deterministic. `--checksum-choice=md5` has no seed and ignores it (documented). The value is a strict decimal 0..2⁶⁴-1 (blank, signed, or non-numeric values are rejected). Like rsync, a seed only matters where a digest is actually computed (`--checksum` or a basis-dir run, or a delta transfer); it does not by itself enable `--checksum`/`--delta`. Divergence from rsync: the default is seed 0, and FastSync never randomizes the seed (rsync uses a random per-transfer seed when `--checksum-seed` is unset); FastSync's unset default therefore reproduces its historical byte-for-byte behavior |
| `--secluded-args` | Use protocol to send args | 🔄 Compatibility No-op | Accepted for CLI compatibility; it does not change FastSync transport or protocol behavior. `-s` remains chunk serialization. |
| `--no-OPTION` | Turn off implied option | ✅ Supported | Supported boolean FastSync options and archive-implied options; unsafe or value-taking options are rejected. |
---
## Implementation Difficulty Plan
The estimates below cover the currently unimplemented features in this document. They assume one engineer familiar with the codebase, include implementation and focused tests, and exclude production rollout time. A feature should not be marked implemented until its behavior is tested in both local and SSH/TCP paths where applicable.
> **Note:** This plan is a superset snapshot written while several of the listed features were still outstanding. The Summary matrix above is the authoritative record of what is already shipped (for example quiet/info/debug output, `--existing`, `--remove-source-files`, `-h`, and `--size-only` are now implemented on `dev`). Treat the phases as sequencing guidance for the work that remains unimplemented.
| Effort | Typical duration | Meaning |
|--------|------------------|---------|
| XS | 0.5-1 day | CLI alias or a local formatting/validation change |
| S | 1-3 days | Isolated behavior with little or no protocol change |
| M | 3-7 days | Cross-cutting client, server, or scanner behavior |
| L | 1-3 weeks | Protocol, filesystem, privilege, or compatibility work |
| XL | 3+ weeks | New transfer mode, daemon subsystem, or broad interoperability effort |
### Phase 1: Low-Risk CLI and Local Behavior
These are the best first changes because they require limited wire-format work and can be tested with existing transfer fixtures.
| Features | Effort | Implementation plan |
|----------|--------|--------------------|
| `--quiet`, `-q`; `--human-readable`, `-h`; `--8-bit-output`, `-8`; `--stderr=MODE`; `--info=FLAGS`; `--debug=FLAGS` | S | Extend logging and output formatting without changing transferred data. |
| `--no-OPTION`; `--old-args`; `--secluded-args`, `-s` | M | Add option implication/negation and safely serialize or protect remote arguments. `-s` currently has FastSync-specific semantics and needs a compatibility decision. |
| `-P`; `--del`; `--old-dirs`, `--old-d`; `--cc`; `--zc`; `--zl` | XS | Add aliases and composed behaviors after the underlying options exist. |
| `--whole-file`, `-W`; `--ignore-times`, `-I`; `--size-only`; `--modify-window`, `-@`; `--update`, `-u` | S | Extend the existing incremental comparison decision. |
| `--existing`; `--ignore-existing`; `--remove-source-files` | S | Add scanner/receiver eligibility checks and remove successfully synchronized source files. |
| `--executability`, `-E`; `--chmod=CHMOD` | M | Apply permission transformations safely while preserving current metadata behavior. |
| `--skip-compress=LIST`; `--compress-threads=NUM` | S | Make compression selection configurable and validate the thread setting against zstd behavior. |
| `--max-alloc=SIZE`; `--fsync` | S | Reuse existing allocation limits and add an explicit durability step after file writes. |
### Phase 2: Filesystem Selection and Update Semantics
These features are moderate because they affect traversal, temporary files, manifests, or the receiver's update policy.
| Features | Effort | Implementation plan |
|----------|--------|--------------------|
| `--one-file-system`, `-x` | M | Track the source device during scanner traversal and skip mount-point crossings. |
| `--relative`, `-R`; `--no-implied-dirs`; `--dirs`, `-d`; `--mkpath` | M | Extend path-list construction and destination directory creation while preserving traversal safety. |
| `--temp-dir`, `-T` | M | Separate temporary-file placement from FastSync's timeout alias and define collision, permissions, and cleanup rules. |
| `--delay-updates` | L | Stage all successful updates and publish them at completion, including crash and cancellation cleanup. |
| `--files-from=FILE`; `--from0`, `-0`; `--filter=RULE`, `-f`; `-F`; `--cvs-exclude`, `-C` | L | Build a complete filter/parser layer and integrate it with scanner pruning, manifests, and delete behavior. `-f` conflicts with FastSync sendfile mode. |
| `--list-only`; `--itemize-changes`, `-i`; `--out-format=FORMAT`; `--log-file-format=FMT` | M | Add a structured change-event model so output modes share one source of truth. |
### Phase 3: Deletion, Comparison, and Delta Compatibility
These features require careful interaction with manifests, incremental checks, backups, and the existing delta protocol.
| Features | Effort | Implementation plan |
|----------|--------|--------------------|
| `--delete-during`; `--delete-before`; `--delete-after`; `--delete-delay`; `--del` | L | Add deletion timing to the transfer state machine and ensure failures cannot remove files unexpectedly. |
| `--delete-excluded`; `--max-delete=NUM`; `--ignore-errors`; `--force`; `--prune-empty-dirs`, `-m` | M | Extend delete walks with policy limits, error handling, empty-directory pruning, and the `-m` short-flag conflict. |
| `--ignore-missing-args`; `--delete-missing-args` | M | Distinguish missing source arguments from traversal errors and apply explicit deletion policy. |
| `--compare-dest=DIR`; `--copy-dest=DIR`; `--link-dest=DIR` | L | Add alternate basis roots and hard-link handling, including metadata and cross-filesystem failures. |
| `--fuzzy`, `-y`; `--no-fuzzy` | L | Index candidate files and select a safe similar basis without making transfer time unbounded. |
| `--append`; `--append-verify` | M | Negotiate file length and verify the retained prefix before resuming. |
| `--checksum-choice=STR`, `--cc`; `--checksum-seed=NUM` | M | Negotiate checksum algorithms/seeds and preserve compatibility with existing xxHash checks. |
### Phase 4: Metadata, Links, and Devices
These features are platform-sensitive and need Linux permission, ACL, xattr, and special-file integration tests.
| Features | Effort | Implementation plan |
|----------|--------|--------------------|
| `--numeric-ids`; `--usermap=STRING`; `--groupmap=STRING`; `--chown=USER:GROUP` | L | Define identity mapping, privilege failures, and wire representation before applying ownership. |
| `--open-noatime`; `--atimes`, `-U`; `--crtimes`, `-N`; `--omit-dir-times`, `-O`; `--omit-link-times`, `-J` | L | Extend metadata capture/apply with platform capability checks and explicit unsupported-attribute handling. |
| `--acls`, `-A`; `--xattrs`, `-X`; `--fake-super` | XL | Add portable serialization, size limits, privilege behavior, and security tests for ACL/xattr data. |
| `--hard-links`, `-H` | L | Preserve inode relationships across the file list and coordinate hard-link creation order. |
| `--munge-links`; `--copy-dirlinks`, `-k`; `--keep-dirlinks`, `-K` | L | Define symlink trust boundaries and receiver-side directory/link collision behavior. |
| `--devices`; `--specials`; `-D`; `--copy-devices`; `--write-devices` | XL | Add privileged special-file handling with strict type, path, and authorization checks. |
| `--super`; `--copy-as=USER[:GROUP]` | XL | Requires a deliberate privilege model, identity switching, and refusal paths; do not implement by blindly elevating the process. |
| `--preallocate` | S | Use platform allocation APIs before writes and fall back cleanly when unsupported. |
### Phase 5: Connectivity and Daemon Compatibility
These options affect process startup, authentication, sockets, and remote execution. They should follow the filesystem and protocol work rather than being added as parser-only flags.
| Features | Effort | Implementation plan |
|----------|--------|--------------------|
| `--rsh=COMMAND`, `-e`; `--rsync-path=PROGRAM`; `--blocking-io`; `--outbuf=N\|L\|B` | M | Generalize SSH command construction and subprocess I/O while retaining argument escaping and timeout guarantees. |
| `--address=ADDRESS`; `--ipv4`, `-4`; `--ipv6`, `-6`; `--sockopts=OPTIONS`; `--port=PORT` daemon semantics | M | Add explicit socket-family/bind configuration and validate it independently for TCP client and daemon modes. |
| `--remote-option=OPT`, `-M`; `--trust-sender` | L | Add authenticated remote-option/config negotiation and reject unsafe sender-controlled values. `-M` conflicts with FastSync metadata mode. |
| `--daemon`; `--config=FILE`; `--dparam=OVERRIDE`; `--no-detach`; `--password-file=FILE`; `--early-input=FILE`; `--no-motd` | XL | Implement a real daemon lifecycle, module configuration, authentication, privilege separation, and process management. |
### Phase 6: Batch, Encoding, and Protocol Interoperability
These are the hardest compatibility items because they require durable formats or behavior that must interoperate with rsync itself.
| Features | Effort | Implementation plan |
|----------|--------|--------------------|
| `--write-batch=FILE`; `--only-write-batch=FILE`; `--read-batch=FILE` | XL | Specify a versioned batch format, persist all required metadata, and test replay, corruption, and partial application. |
| `--protocol=NUM` | XL | Add protocol-version negotiation and compatibility branches without weakening current validation. |
| `--iconv=CONVERT_SPEC` | L | Convert filenames at the protocol boundary with invalid-sequence and normalization tests. |
| `--stop-after=MINS`; `--stop-at=TIME` | M | Add deadline propagation, interruptible I/O, and safe checkpoint/cleanup behavior. |
| `--early-input=FILE`; `--password-file=FILE` | M | Securely read startup credentials/input with permission checks and no secret disclosure in logs. |
### Recommended Delivery Order
1. Resolve short-option conflicts (`-m`, `-M`, `-T`, `-f`, `-s`) and define the compatibility contract.
2. Implement Phase 1 comparison, update, output, and alias features with unit and integration coverage.
3. Implement Phase 2 traversal/filtering and Phase 3 deletion semantics.
4. Implement metadata and link features that are safe on the supported platforms.
5. Treat daemon mode, special files, batch mode, and protocol-version compatibility as separate projects.
The existing priority list below is a feature shortlist, not an implementation schedule; this plan supersedes it for effort and sequencing.
---
@@ -247,13 +440,14 @@ Ranked by user demand, implementation complexity, and interoperability impact:
| 1 | `--whole-file` / `-W` | Low | High — users expect opt-out of delta |
| 2 | `--ignore-times` / `-I` | Low | Medium — useful for forcing re-transfer |
| 3 | `--size-only` | Low | Medium — common migration scenario |
| 4 | `--existing` / `--ignore-existing` | Low | Medium — common sync patterns |
| 5 | `--remove-source-files` | Low | High — common for moves/backup |
| 6 | `--delete-during` | Medium | High — performance improvement |
| 7 | `--delay-updates` | Medium | High — atomic updates |
| 8 | `--chmod` | Low | Medium — permission flexibility |
| 9 | `--executability` / `-E` | Low | Low — simple flag |
| 10 | `--skip-compress` | Low | Medium — performance tuning |
| 4 | `--ignore-existing` | Low | Medium — common sync patterns |
| 5 | `--existing` | Low | Medium — common sync patterns |
| 6 | `--remove-source-files` | Low | High — common for moves/backup |
| 7 | `--delete-during` | Medium | High — performance improvement |
| 8 | `--delay-updates` | Medium | High — atomic updates |
| 9 | `--chmod` | Low | Medium — permission flexibility |
| 10 | `--executability` / `-E` | Low | Low — simple flag |
| 11 | `--skip-compress` | Low | Medium — performance tuning |
---
+329
View File
@@ -0,0 +1,329 @@
#include "change_list.h"
#include "utils.h"
#include <limits.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <time.h>
/* Itemize code emitted for a transferred regular file.
*
* Layout (rsync-compatible 11-char item): `>f` marks a regular file that was
* transferred to the remote host; the trailing nine markers are, in order,
* c(hecksum) s(ize) t(ime) p(erms) o(wner) g(roup) u(ser/acl) a(ttrs) x(attrs).
* Every marker is `+` (FastSync does not compare each attribute on the
* receiving side, so a sent file is reported as fully updated). Files that
* are already up to date print no line at all, matching rsync's single -i
* which only itemizes changes.
*
* Because the scanner only yields regular-file transfer candidates, `>d`
* (directory) lines are never produced; directories are not transferred as
* items by FastSync. */
#define ITEMIZE_SENT_FILE ">f+++++++++"
typedef struct {
char* data;
size_t length;
size_t capacity;
} StrBuf;
static void strbuf_free(StrBuf* buf) {
if (buf == NULL)
return;
free(buf->data);
buf->data = NULL;
buf->length = 0;
buf->capacity = 0;
}
static bool strbuf_reserve(StrBuf* buf, size_t extra) {
if (buf->length > SIZE_MAX - extra - 1)
return false;
size_t need = buf->length + extra + 1;
if (need <= buf->capacity)
return true;
size_t capacity = buf->capacity > 0 ? buf->capacity : 32;
while (capacity < need) {
if (capacity > SIZE_MAX / 2) {
capacity = need;
break;
}
capacity *= 2;
}
char* grown = realloc(buf->data, capacity);
if (!grown)
return false;
buf->data = grown;
buf->capacity = capacity;
return true;
}
static bool strbuf_append_char(StrBuf* buf, char c) {
if (!strbuf_reserve(buf, 1))
return false;
buf->data[buf->length++] = c;
buf->data[buf->length] = '\0';
return true;
}
static bool strbuf_append(StrBuf* buf, const char* text) {
if (text == NULL)
return true;
size_t length = strlen(text);
if (!strbuf_reserve(buf, length))
return false;
memcpy(buf->data + buf->length, text, length);
buf->length += length;
buf->data[buf->length] = '\0';
return true;
}
static bool strbuf_append_ull(StrBuf* buf, unsigned long long value) {
char digits[32];
int written = snprintf(digits, sizeof(digits), "%llu", value);
if (written < 0 || (size_t)written >= sizeof(digits))
return false;
return strbuf_append(buf, digits);
}
static bool strbuf_append_longlong(StrBuf* buf, long long value) {
char digits[32];
int written = snprintf(digits, sizeof(digits), "%lld", value);
if (written < 0 || (size_t)written >= sizeof(digits))
return false;
return strbuf_append(buf, digits);
}
bool change_list_enabled(const Config* config) {
return config != NULL && (config->itemize_changes || config->out_format != NULL ||
(config->log_file != NULL && config->log_file_format != NULL));
}
char* change_render_itemize(const ChangeEvent* event) {
if (event == NULL || event->decision != CHANGE_SENT)
return str_dup("");
const char* code = event->is_directory ? ">d+++++++++" : ITEMIZE_SENT_FILE;
StrBuf line = {0};
bool ok = strbuf_append(&line, code) && strbuf_append(&line, " ") &&
strbuf_append(&line, event->path != NULL ? event->path : "");
if (!ok) {
strbuf_free(&line);
return NULL;
}
return line.data;
}
static const char* leaf_name(const char* path) {
if (path == NULL)
return "";
const char* slash = strrchr(path, '/');
return slash != NULL && slash[1] != '\0' ? slash + 1 : path;
}
char* change_render_format(const char* format, const ChangeEvent* event) {
if (format == NULL)
return NULL;
StrBuf line = {0};
bool ok = true;
for (const char* p = format; *p != '\0' && ok;) {
if (*p != '%') {
ok = strbuf_append_char(&line, *p);
p++;
continue;
}
char token = p[1];
if (token == '\0') {
ok = strbuf_append_char(&line, '%');
break;
}
switch (token) {
case '%':
ok = strbuf_append_char(&line, '%');
break;
case 'f':
ok = strbuf_append(&line, event->path != NULL ? event->path : "");
break;
case 'n':
ok = strbuf_append(&line, leaf_name(event->path));
break;
case 'l':
ok = strbuf_append_ull(&line, event->size);
break;
case 'b':
ok = strbuf_append_ull(&line, event->bytes_sent);
break;
case 'M':
ok = strbuf_append_longlong(&line, (long long)event->mtime_sec);
break;
default:
/* Unknown escape sequences are preserved verbatim. */
ok = strbuf_append_char(&line, '%') && strbuf_append_char(&line, token);
break;
}
p += 2;
}
if (!ok) {
strbuf_free(&line);
return NULL;
}
if (line.data == NULL) {
line.data = str_dup("");
if (!line.data)
return NULL;
}
return line.data;
}
/* Format a mode as an `ls -l` permission string, e.g. `-rw-r--r--`. */
static void mode_to_ls_string(mode_t mode, char out[11]) {
out[0] = S_ISDIR(mode) ? 'd'
: S_ISLNK(mode) ? 'l'
: S_ISCHR(mode) ? 'c'
: S_ISBLK(mode) ? 'b'
: S_ISFIFO(mode) ? 'p'
: S_ISSOCK(mode) ? 's'
: '-';
mode_t bits = mode & 07777;
out[1] = (bits & S_IRUSR) ? 'r' : '-';
out[2] = (bits & S_IWUSR) ? 'w' : '-';
out[3] = (bits & S_IXUSR) ? (bits & S_ISUID ? 's' : 'x') : (bits & S_ISUID ? 'S' : '-');
out[4] = (bits & S_IRGRP) ? 'r' : '-';
out[5] = (bits & S_IWGRP) ? 'w' : '-';
out[6] = (bits & S_IXGRP) ? (bits & S_ISGID ? 's' : 'x') : (bits & S_ISGID ? 'S' : '-');
out[7] = (bits & S_IROTH) ? 'r' : '-';
out[8] = (bits & S_IWOTH) ? 'w' : '-';
out[9] = (bits & S_IXOTH) ? (bits & S_ISVTX ? 't' : 'x') : (bits & S_ISVTX ? 'T' : '-');
out[10] = '\0';
}
char* change_render_list_line(mode_t mode, unsigned long long size, time_t mtime,
const char* path) {
char permission[11];
mode_to_ls_string(mode, permission);
char date[32];
struct tm broken_down;
if (localtime_r(&mtime, &broken_down) != NULL) {
if (strftime(date, sizeof(date), "%Y/%m/%d %H:%M:%S", &broken_down) == 0)
snprintf(date, sizeof(date), "?");
} else {
snprintf(date, sizeof(date), "?");
}
StrBuf line = {0};
char size_field[32];
int written = snprintf(size_field, sizeof(size_field), "%llu", size);
if (written < 0 || (size_t)written >= sizeof(size_field)) {
strbuf_free(&line);
return NULL;
}
bool ok = strbuf_append(&line, permission) && strbuf_append_char(&line, ' ') &&
strbuf_append(&line, size_field) && strbuf_append_char(&line, ' ') &&
strbuf_append(&line, date) && strbuf_append_char(&line, ' ') &&
strbuf_append(&line, path != NULL ? path : "");
if (!ok) {
strbuf_free(&line);
return NULL;
}
return line.data;
}
static void print_escaped_line(FILE* stream, const char* line, bool eight_bit_output) {
char* escaped = output_escape(line, eight_bit_output);
if (escaped != NULL) {
fprintf(stream, "%s\n", escaped);
free(escaped);
} else {
fprintf(stream, "%s\n", line);
}
fflush(stream);
}
void change_emit(const Config* config, const ChangeEvent* event) {
if (event == NULL || !change_list_enabled(config))
return;
if (event->decision == CHANGE_UP_TO_DATE)
return;
bool to_stdout = config->itemize_changes || config->out_format != NULL;
bool to_log = config->log_file != NULL && config->log_file_format != NULL;
if (to_stdout) {
char* line = config->out_format != NULL ? change_render_format(config->out_format, event)
: change_render_itemize(event);
if (line != NULL) {
print_escaped_line(stdout, line, config->eight_bit_output);
free(line);
}
}
if (to_log) {
char* line = change_render_format(config->log_file_format, event);
if (line != NULL) {
print_escaped_line(config->log_file, line, config->eight_bit_output);
free(line);
}
}
}
static bool format_uses_mtime(const char* format) {
if (format == NULL)
return false;
/* Mirror change_render_format's tokenizer: "%%" is a literal percent (so
* "%%M" does NOT expand %M) and unknown "%X" escapes consume both chars.
* This keeps the optional stat() fallback below in step with the renderer. */
for (const char* p = format; *p != '\0';) {
if (*p != '%') {
p++;
continue;
}
char token = p[1];
if (token == '\0')
break;
if (token == 'M')
return true;
p += 2;
}
return false;
}
void change_emit_file_sent(const Config* config, const File* file) {
if (file == NULL || !change_list_enabled(config))
return;
ChangeEvent event;
memset(&event, 0, sizeof(event));
/* The displayed path is the one transmitted (with -R + --files-from this is
the bare relative destination path); the metadata fallback below still
stats the local absolute path. */
event.path = file_wire_path(file);
event.decision = CHANGE_SENT;
event.is_directory = false;
event.size = file->data != NULL ? file->data->size : 0;
/* FastSync has no wire-byte counter yet, so %b reports the source length
* that had to be delivered (always equal to %l); the actual bytes written
* to the socket (compressed/delta) are not measured. */
event.bytes_sent = event.size;
if (file->metadata != NULL) {
event.mtime_sec = file->metadata->mtime_sec;
} else if (format_uses_mtime(config->out_format) || format_uses_mtime(config->log_file_format)) {
/* Best-effort fallback for %M when no metadata was captured (no -M): the
* path is stat()ed just to fill the field, and any failure leaves 0. */
struct stat st;
if (file->path != NULL && stat(file->path, &st) == 0)
event.mtime_sec = st.st_mtime;
}
change_emit(config, &event);
}
/* Build and emit a CHANGE_SENT event for an explicit directory entry (-d). */
void change_emit_dir_sent(const Config* config, const File* file) {
if (file == NULL || !change_list_enabled(config))
return;
ChangeEvent event;
memset(&event, 0, sizeof(event));
event.path = file_wire_path(file);
event.decision = CHANGE_SENT;
event.is_directory = true;
event.size = 0;
event.bytes_sent = 0;
if (file->metadata != NULL)
event.mtime_sec = file->metadata->mtime_sec;
change_emit(config, &event);
}
+78
View File
@@ -0,0 +1,78 @@
#ifndef CHANGE_LIST_H
#define CHANGE_LIST_H
#include "config.h"
#include "file_types.h"
#include <stdbool.h>
#include <sys/stat.h>
#include <time.h>
/*
* Shared per-file change-event / output model (rsync --itemize-changes,
* --out-format, --log-file-format, and --list-only all render from here).
*
* FastSync is a push-style tool: the client sends files from the source tree
* to a server that writes them under the destination root. Events are
* emitted by whichever code path decides a file's fate (the single-threaded
* send loop and the `-m` sender thread both call the same per-file sender), so
* all change events are emitted by exactly one thread and itemize/out-format
* lines never interleave with each other. They may still interleave with
* legacy log messages (log.c) that share the same stdout/log-file stream.
*/
typedef enum {
CHANGE_SENT, /* file data (full or delta) was transmitted */
CHANGE_UP_TO_DATE, /* receiver already had an identical file; skipped */
} ChangeDecision;
typedef struct {
const char* path; /* full source path */
ChangeDecision decision;
bool is_directory;
unsigned long long size; /* source file length in bytes */
/* The number of bytes reported for a sent file. FastSync has no wire-byte
* counter, so this is always the source length (== size / %l); actual
* post-compression/delta bytes on the wire are not counted. */
unsigned long long bytes_sent;
time_t mtime_sec; /* 0 when unknown */
} ChangeEvent;
/* True when any output mode is active and per-file events matter. */
bool change_list_enabled(const Config* config);
/* Render the rsync-style itemize line for a transferred file:
* `>f+++++++++ <path>`
* The 11-char code is `>f` (regular file transferred to the remote host)
* followed by c/s/t/p/o/g/u/a/x markers that are all `+` (value will be set
* / differs) because FastSync does not separately compare checksums, size,
* mtime, perms, owner, group, uid, acl, or xattr on the receiving side, so a
* sent file is reported as fully updated. Up-to-date files print no line
* (rsync single `-i` only shows changes). Caller frees the result. */
char* change_render_itemize(const ChangeEvent* event);
/* Expand an --out-format/--log-file-format template. Tokens:
* %f full source path %b "bytes sent" == the source length (%l);
* %n leaf (base) name actual post-compression/delta wire bytes
* %l file length in bytes are not counted
* %M mtime in whole seconds %% a literal percent sign
* Unknown %X sequences are preserved verbatim. Caller frees the result. */
char* change_render_format(const char* format, const ChangeEvent* event);
/* Render one --list-only long-listing entry:
* `-rw-r--r-- 12 2026/09/06 10:00:00 <path>`
* (ls -l style columns; mtime in the local time zone). Caller frees it. */
char* change_render_list_line(mode_t mode, unsigned long long size, time_t mtime, const char* path);
/* Emit an event to every active destination:
* stdout: --itemize-changes line, or the --out-format expansion when set;
* log file: the --log-file-format expansion (requires --log-file).
* CHANGE_UP_TO_DATE events produce no output. */
void change_emit(const Config* config, const ChangeEvent* event);
/* Build and emit a CHANGE_SENT event for a file the client just sent. */
void change_emit_file_sent(const Config* config, const File* file);
/* Build and emit a CHANGE_SENT event for an explicit directory entry (-d). */
void change_emit_dir_sent(const Config* config, const File* file);
#endif
+825 -60
View File
File diff suppressed because it is too large. Load diff
+1016 -111
View File
File diff suppressed because it is too large. Load diff
+39 -3
View File
@@ -1,4 +1,5 @@
#include "client_validation.h"
#include "delay_updates.h"
#include "log.h"
#include "usage.h"
#include <stdio.h>
@@ -10,11 +11,21 @@ bool validate_config(const Config* config) {
print_usage();
return false;
}
if (config_has_basis(config) && config->use_chunk_serialization) {
log_message(LOG_LEVEL_ERROR,
"--compare-dest/--copy-dest/--link-dest require per-file incremental checks and "
"cannot be combined with -s (chunk serialization)");
return false;
}
if (config->use_sendfile && (config->use_chunk_serialization || config->use_compression)) {
log_message(LOG_LEVEL_ERROR, "-f/--sendfile cannot be combined with -c (compression) or -s "
"(chunk serialization)");
return false;
}
if (config->compression_threads > 0 && !config->use_compression) {
log_message(LOG_LEVEL_ERROR, "--compress-threads requires compression (-c or -z)");
return false;
}
if (config->transport == TRANSPORT_SSH && config->use_sendfile) {
log_message(LOG_LEVEL_ERROR, "-f/--sendfile is not supported with SSH transport");
return false;
@@ -23,18 +34,27 @@ bool validate_config(const Config* config) {
log_message(LOG_LEVEL_ERROR, "--incremental is not supported with -s (chunk serialization)");
return false;
}
if (config->use_delta && !config->use_incremental) {
if (config->skip_compress_set && config->use_chunk_serialization) {
log_message(LOG_LEVEL_ERROR,
"--skip-compress cannot be combined with -s (chunk serialization)");
return false;
}
if (config->use_delta && !config->whole_file && !config->use_incremental) {
log_message(LOG_LEVEL_ERROR, "--delta requires --incremental");
return false;
}
if (config->use_delta && config->use_chunk_serialization) {
if (config->use_delta && !config->whole_file && config->use_chunk_serialization) {
log_message(LOG_LEVEL_ERROR, "--delta cannot be combined with -s (chunk serialization)");
return false;
}
if (config->use_delta && config->use_sendfile) {
if (config->use_delta && !config->whole_file && config->use_sendfile) {
log_message(LOG_LEVEL_ERROR, "--delta cannot be combined with -f (sendfile)");
return false;
}
if (config->log_file_format && !config->log_file) {
log_message(LOG_LEVEL_ERROR, "--log-file-format requires --log-file");
return false;
}
if (config->append || config->append_verify) {
fprintf(
stderr,
@@ -47,5 +67,21 @@ bool validate_config(const Config* config) {
return false;
}
}
if (config->delay_updates && config->inplace) {
log_message(LOG_LEVEL_ERROR, "--delay-updates does not work with --inplace");
return false;
}
if (config->delay_updates && delay_updates_staging_name_conflict(config->backup_dir)) {
log_message(LOG_LEVEL_ERROR,
"--backup-dir is reserved when --delay-updates is active (used for the internal "
"staging directory)");
return false;
}
if (!config_has_valid_delete_timing(config)) {
log_message(LOG_LEVEL_ERROR,
"--delete-before/--delete-during/--delete-delay/--delete-after select the delete "
"timing; at most one may be given and each implies --delete");
return false;
}
return true;
}
+763 -50
View File
File diff suppressed because it is too large. Load diff
+92 -2
View File
@@ -2,11 +2,15 @@
#define SCANNER_H
#include "chunk.h"
#include "file_list.h"
#include "filter.h"
#include "protocol.h"
#include "queue.h"
#include <dirent.h>
#include <stdbool.h>
#include <threads.h>
#include <stdatomic.h>
#include <sys/types.h>
#include <threads.h>
typedef struct {
bool use_metadata;
@@ -24,8 +28,43 @@ typedef struct {
bool safe_links;
bool copy_unsafe_links;
bool checksum;
bool one_file_system;
/* Phase 2 (files-from / filter layer). All pointers are shared read-only
* across scanner instances and worker threads; ownership stays with the
* caller (client_send). */
const FileListSet* file_list; /* --files-from allow-set, or NULL */
const FilterRuleList* base_filters; /* command-line + -C rules, or NULL */
bool per_dir_filters; /* -F: read .rsync-filter per directory */
bool dirs; /* -d/--dirs: transfer dir entries, no recursion */
bool relative; /* -R/--relative (dest rel paths, with --files-from) */
/* --prune-empty-dirs (long only): in --dirs mode an empty source directory's
explicit entry is omitted from the transfer file list (so nothing is
created at the destination and it can be pruned by --delete); explicitly
--files-from-listed directories always pass through. Recursive transfers
never emit empty directories, so the flag has no additional effect there. */
bool prune_empty_dirs;
/* Delete-excluded protection sink (optional): when non-NULL the scanner
* appends the destination-relative path of every entry it prunes because a
* USER SELECTION rule excluded it (--filter/-C/per-dir rules, the legacy
* --exclude/--include layer, and --max-size/--min-size). The sender turns
* this list into the manifest's protected prefixes so `--delete` leaves the
* destination mirror of excluded source paths alone (rsync's default), and
* empties it when --delete-excluded opts back into deleting them. NOT
* recorded for --files-from subset pruning (whose delete semantics stay
* keep-set-only) or for -R/--files-from relative wire paths. When
* `excluded_mutex` is non-NULL it is taken around every append (the parallel
* scanner shares one list across its worker threads). */
ArrayList* excluded_paths;
mtx_t* excluded_mutex;
/* --ignore-errors: an unreadable directory during the scan is recorded as an
* I/O error and skipped instead of aborting the scan. Client-only. */
bool ignore_io_errors;
} ScannerOptions;
/* Internal per-scanner filter state. FilterNode chains represent the ordered
* per-directory .rsync-filter rules that apply below a directory. */
typedef struct FilterNode FilterNode;
typedef struct {
Queue* directories;
DIR* current_dir;
@@ -45,7 +84,38 @@ typedef struct {
bool safe_links;
bool copy_unsafe_links;
bool checksum;
bool one_file_system;
dev_t root_dev;
bool failed;
/* Phase 2 (files-from / filter layer). */
char* root_path; /* transfer root (fs path) for rel computation */
char* current_rel; /* rel path of the open directory ("" == root) */
bool at_seed_dir; /* next open is the seed directory */
FilterNode* seed_node; /* inherited context of the seed dir, or NULL */
FilterNode* current_node; /* filter context of the open directory */
ArrayList* filter_nodes; /* owned FilterNode arena (may be NULL) */
const FileListSet* file_list;
const FilterRuleList* base_filters;
bool per_dir_filters;
/* --dirs / -R state for the directory-entry generator (dirs_mode replaces
the recursive scan). */
bool dirs_mode;
bool relative_mode; /* file_list && relative: send bare relative wire paths */
bool prune_empty_dirs;
bool dirs_root_emitted;
int list_index;
ArrayList* dirs_batch; /* owned when non-NULL */
unsigned long long dirs_batch_size;
/* Excluded-path sink (see ScannerOptions). `excluded_mutex` is shared across
parallel worker threads. */
ArrayList* excluded_paths;
mtx_t* excluded_mutex;
/* --ignore-errors: continue past unreadable directories (records io_error). */
bool ignore_io_errors;
/* A directory could not be opened (I/O error, e.g. EACCES). With
--ignore-errors the scan continues past it and the caller decides what to
do; `failed` is reserved for fatal errors that always abort the scan. */
bool io_error;
} DirectoryScanner;
typedef struct {
@@ -59,9 +129,13 @@ typedef struct {
thrd_t* threads;
bool done;
bool failed;
/* A worker skipped an unreadable directory under --ignore-errors (non-fatal). */
bool io_error;
atomic_bool cancelled;
int completed;
Chunk* initial_chunk;
ProtocolSession* allocation_session;
FilterNode* root_filter_node; /* root .rsync-filter context (owned by ps) */
} ParallelScanner;
DirectoryScanner* directory_scanner_create(const char* root_directory, bool use_metadata,
@@ -77,10 +151,26 @@ Chunk* directory_scanner_next(DirectoryScanner* scanner);
bool directory_scanner_failed(const DirectoryScanner* scanner);
void directory_scanner_destroy(DirectoryScanner* scanner);
/* --one-file-system (-x) decision: a directory entry may be descended into
* only when the option is disabled or the entry lives on the same device as
* the transfer root. Exposed so tests can exercise the rule directly. */
bool scanner_same_filesystem(bool one_file_system, dev_t root_device, dev_t entry_device);
/* Relative path of an on-disk path below `root` ("" == the root itself, NULL
* when `fs_path` is not under `root`). Handles trailing slashes and a root of
* "/". Exposed so tests can exercise the mapping directly. */
char* scanner_path_relative(const char* root, const char* fs_path);
ParallelScanner* parallel_scanner_create_with_options(const char* root_directory,
const ScannerOptions* options);
const ScannerOptions* options,
ProtocolSession* allocation_session);
Chunk* parallel_scanner_next(ParallelScanner* scanner);
bool parallel_scanner_failed(const ParallelScanner* scanner);
bool parallel_scanner_had_io_error(const ParallelScanner* scanner);
void parallel_scanner_destroy(ParallelScanner* scanner);
/* True when a directory could not be opened during the scan (an I/O error,
recorded even when --ignore-errors keeps the scan going past it). */
bool directory_scanner_had_io_error(const DirectoryScanner* scanner);
#endif
+105
View File
@@ -18,26 +18,110 @@ void print_usage(void) {
printf(" -z [level] Alias for -c\n");
printf(" -a, --archive Archive mode (-c -m -M)\n");
printf(" -n, --dry-run Show what would be transferred\n");
printf(" --remove-source-files Remove regular source files after successful transfer\n");
printf(" -p <port> SSH port (default: 22)\n");
printf(" --progress Show transfer progress\n");
printf(" -P Partial mode with progress (retention incomplete)\n");
printf(" -8, --8-bit-output Leave high-bit characters unescaped in output\n");
printf(" --delete Delete files on receiver not in source\n");
printf(" (default timing: delete only after the whole\n");
printf(" transfer has succeeded)\n");
printf(" --delete-before Delete extras before the transfer starts\n");
printf(" (implies --delete)\n");
printf(" --delete-during Delete extras once the keep-set manifest is known,\n");
printf(" before the data is applied (implies --delete)\n");
printf(" --del Alias for --delete-during\n");
printf(" --delete-delay Delete extras only after a successful transfer\n");
printf(" (implies --delete)\n");
printf(" --delete-after Delete only after the whole transfer succeeded\n");
printf(" (the default --delete timing; implies --delete)\n");
printf(" --delete-excluded Also delete destination files that were excluded on\n");
printf(" the source (default protects them, matching rsync)\n");
printf(" --max-delete=NUM Never delete more than NUM destination entries per run;\n");
printf(" if the extras would exceed NUM, nothing is deleted and\n");
printf(" the run fails with a clear error (implies --delete only\n");
printf(" when used with it)\n");
printf(" --ignore-errors Continue (and still delete) when a source directory is\n");
printf(" unreadable during the scan, instead of aborting with no\n");
printf(" deletion\n");
printf(" --force A file may replace a destination directory by removing\n");
printf(" that (non-empty) directory first\n");
printf(" --prune-empty-dirs Do not transfer empty directory entries (--dirs mode);\n");
printf(" recursive transfers never send empty dirs. rsync's -m\n");
printf(" short form stays FastSync multithreading\n");
printf(" Note: each timing flag implies --delete. Combining a timing flag with\n");
printf(" --no-delete (in either order) is rejected as a config error.\n");
printf(" --ignore-existing Skip files that already exist on receiver\n");
printf(" --delay-updates Put updated files into place only at the end of transfer\n");
printf(" --dirs, -d, --old-dirs, --old-d Transfer the named directory entries without\n");
printf(" recursing into their contents (-d <dir> mirrors the source\n");
printf(" directory empty; with --files-from listed dirs are created\n");
printf(" empty and listed files are transferred)\n");
printf(" -R, --relative With --files-from, preserve each listed entry's relative path\n");
printf(" below the destination root instead of mirroring the full\n");
printf(" source path (no effect without --files-from)\n");
printf(" --no-implied-dirs With -R --files-from, refuse to place a listed file whose\n");
printf(" parent directory is not itself listed\n");
printf(" --mkpath Create the destination root directory on the server when it\n");
printf(" does not exist yet\n");
printf(" --exclude <pattern> Exclude files matching pattern\n");
printf(" --include <pattern> Only include files matching pattern\n");
printf(" --exclude-from <file> Read exclude patterns from file\n");
printf(" --include-from <file> Read include patterns from file\n");
printf(" --files-from <file> Read the source file list from FILE (paths relative to the "
"source root)\n");
printf(" -0, --from0 Entries in --files-from are NUL-delimited\n");
printf(" --filter=RULE rsync-style filter rule (+/- include/exclude; repeatable; the\n");
printf(" rsync -f short form conflicts with FastSync sendfile -f)\n");
printf(" -C, --cvs-exclude Auto-ignore common CVS/SCM files (.git/, .svn/, *.o, *~, ...)\n");
printf(" -F Apply per-directory .rsync-filter files during the scan\n");
printf(" --max-size <n> Skip files larger than n bytes\n");
printf(" --min-size <n> Skip files smaller than n bytes\n");
printf(" --max-alloc <SIZE> Maximum single allocation (default: 1G)\n");
printf(" --incremental Skip files unchanged since last transfer\n");
printf(" --size-only Skip incremental files matching in size, ignoring mtime\n");
printf(" -I, --ignore-times Transfer files even when size and mtime match\n");
printf(" -@, --modify-window <sec> Modification time tolerance\n");
printf(" -u, --update Skip files newer than the source on receiver\n");
printf(" --existing Skip files not already present at destination\n");
printf(" --compare-dest <dir> Treat DIR (relative to destination root) as an extra\n");
printf(" comparison basis: unchanged files are not transferred\n");
printf(" (requires --incremental, which is implied)\n");
printf(" --copy-dest <dir> Like --compare-dest, but copies the unchanged file from DIR\n");
printf(" into the destination instead of transferring its data\n");
printf(" --link-dest <dir> Like --copy-dest, but hard-links the unchanged file from DIR\n");
printf(" into the destination (repeatable; earlier DIRs win)\n");
printf(" --checksum-choice, --cc <alg> Whole-file checksum algorithm for --incremental/\n");
printf(" --checksum compares (xxh64/xxhash or md5; default xxh64 with\n");
printf(" seed 0). The seed comes from --checksum-seed\n");
printf(" --checksum-seed <num> Seed for the whole-file xxHash64 digest (and the delta\n");
printf(" block strong hash, low 32 bits); md5 ignores the seed. The\n");
printf(" digest algorithm and seed must match on sender and receiver\n");
printf(" --delta Delta transfer for changed files (requires --incremental)\n");
printf(" -W, --whole-file Transfer changed files without delta processing\n");
printf(" -y, --fuzzy Use a similar-named file already in the destination\n");
printf(" directory as the delta basis when the destination has no\n");
printf(" usable file at the exact path (saves bandwidth; implies\n");
printf(" --incremental and --delta; inert with --whole-file,\n");
printf(" --no-delta, or --no-incremental)\n");
printf(" --no-fuzzy Disable --fuzzy\n");
printf(" --delta-block <n> Delta block size in bytes (default: %d)\n",
DELTA_BLOCK_SIZE_DEFAULT);
printf(" --delta-max <n> Max file size for delta transfer (default: %llu)\n",
DELTA_MAX_FILE_SIZE);
printf(" -m Enable multithreading\n");
printf(" -s Enable chunk serialization\n");
printf(" --secluded-args Accept rsync compatibility option (no effect)\n");
printf(" -f Enable sendfile (TCP only, not with -c or -s)\n");
printf(" --compress-choice <alg> Compression algorithm (default: zstd)\n");
printf(" --zc <alg> Alias for --compress-choice\n");
printf(" -v, --verbose Enable debug logging\n");
printf(" -q, --quiet Suppress non-error output\n");
printf(" --debug=FLAGS Fine-grained debug logging (use --debug=help for flags)\n");
printf(" --info=FLAGS Fine-grained info: copy,misc,skip,stats,all,none\n");
printf(" none suppresses info even with --verbose\n");
printf(" -M, --preserve Preserve file metadata\n");
printf(" -E, --executability Preserve executable permission bits\n");
printf(" --chmod <changes> Modify transferred permissions (rsync syntax)\n");
printf(" --chunk-size <n> Chunk size in bytes (default: %d)\n", DEFAULT_CHUNK_SIZE);
printf(" --source-dir <path> Source directory\n");
@@ -57,19 +141,40 @@ void print_usage(void) {
printf(" --backup-dir <dir> Directory for backups (requires --backup)\n");
printf(" --suffix <str> Backup suffix (default: ~)\n");
printf(" --stats Print transfer statistics at end\n");
printf(" -i, --itemize-changes Print an rsync-style per-file change line\n");
printf(" --out-format=FORMAT Output format for changed files (%%f %%n %%l %%b %%M %%%%)\n");
printf(" --list-only List source files instead of transferring\n");
printf(" --log-file-format=FORMAT Per-file log line format (needs --log-file)\n");
printf(" -h, --human-readable Print byte sizes in human-readable form\n");
printf(" --max-depth <n> Maximum directory depth (0=unlimited)\n");
printf(" -x, --one-file-system Do not cross filesystem boundaries\n");
printf(" --log-file <path> Write log messages to file\n");
printf(" --stderr=MODE Route logging to stderr: errors or all\n");
printf(" --partial Keep partial files on interrupted transfer\n");
printf(" --partial-dir <dir> Directory for partial files\n");
printf(" --temp-dir <dir> Scratch dir for temp files before atomic install\n");
printf(" --fastsync-server-path <path>\n");
printf(" Path to fastsync-server on remote (default: fastsync-server)\n");
printf(
" --old-args Disable safe SSH command argument quoting (legacy compatibility)\n");
printf(" -l, --links Copy symlinks as symlinks\n");
printf(" --copy-links Transform symlinks into referent files\n");
printf(" --safe-links Skip symlinks that point outside transfer tree\n");
printf(" --copy-unsafe-links Only transform unsafe symlinks into referent files\n");
printf(" -S, --sparse Handle sparse files efficiently\n");
printf(" --inplace Update files in-place (no temp+rename)\n");
printf(" --fsync Fsync every written file before publication\n");
printf(" --compress-level <n> Compression level (default: 5)\n");
printf(" --zl <n> Alias for --compress-level\n");
printf(" --skip-compress=LIST Skip compression for comma-separated suffixes\n");
printf(" --compress-threads <n> Compression worker threads (requires zstd threaded support)\n");
printf(" --no-OPTION Disable a supported boolean option\n");
printf(" --help Show this help\n");
printf(" -V, --version Show version\n");
}
void print_debug_usage(void) {
printf("Supported debug flags: IO,PROTO,PACK,UTIL,ALL,NONE\n");
printf("Flags may be comma-separated, for example: --debug=io,proto\n");
printf("Other rsync debug flags are unsupported and rejected.\n");
}
+1
View File
@@ -2,5 +2,6 @@
#define USAGE_H
void print_usage(void);
void print_debug_usage(void);
#endif
+217 -22
View File
@@ -1,12 +1,58 @@
#include "receiver.h"
#include "chunk.h"
#include "config.h"
#include "delay_updates.h"
#include "file_receive.h"
#include "log.h"
#include "metadata.h"
#include "protocol.h"
#include "utils.h"
#include <stdlib.h>
#include <sys/stat.h>
bool receiver_outcomes_append(ReceiverOutcomes* outcomes, unsigned char code) {
if (!outcomes)
return false;
if (outcomes->count == outcomes->capacity) {
size_t new_capacity = outcomes->capacity == 0 ? 64 : outcomes->capacity * 2;
if (new_capacity < outcomes->capacity)
return false;
unsigned char* grown = realloc(outcomes->entries, new_capacity);
if (!grown)
return false;
outcomes->entries = grown;
outcomes->capacity = new_capacity;
}
outcomes->entries[outcomes->count++] = code;
return true;
}
void receiver_outcomes_destroy(ReceiverOutcomes* outcomes) {
if (!outcomes)
return;
free(outcomes->entries);
outcomes->entries = NULL;
outcomes->count = 0;
outcomes->capacity = 0;
}
/* End-of-transfer success frame. When --remove-source-files was negotiated
each processed data file is acknowledged first (STATUS_NEXT = written,
STATUS_OK = skipped) so the sender never removes a source the receiver did
not actually store. The frame always ends with a plain STATUS_OK. */
bool receiver_send_final_success(int fd, const Config* config, const ReceiverOutcomes* outcomes) {
if (!config->remove_source_files)
return send_status(fd, STATUS_OK);
size_t count = outcomes ? outcomes->count : 0;
for (size_t i = 0; i < count; i++) {
Status per_file = outcomes->entries[i] == FILE_SAVE_WRITTEN ? STATUS_NEXT : STATUS_OK;
if (!send_status(fd, per_file))
return false;
}
return send_status(fd, STATUS_OK);
}
static bool receiver_process_chunk(Chunk* chunk, const ReceiverSink* sink) {
if (!chunk || !sink || !sink->store_file)
return false;
@@ -37,9 +83,13 @@ static bool receiver_process_batch(Config* config, int file_descriptor) {
return false;
unsigned long long check_size;
long long check_mtime;
long long check_mtime_nsec;
if (!receive_n_data(file_descriptor, &check_size, sizeof(check_size)) ||
!receive_n_data(file_descriptor, &check_mtime, sizeof(check_mtime))) {
!receive_n_data(file_descriptor, &check_mtime, sizeof(check_mtime)) ||
!receive_n_data(file_descriptor, &check_mtime_nsec, sizeof(check_mtime_nsec)) ||
check_mtime_nsec < 0 || check_mtime_nsec >= 1000000000LL) {
free(check_path);
send_status(file_descriptor, STATUS_ERROR);
return false;
}
if (!utils_valid_batch_path(check_path)) {
@@ -47,7 +97,7 @@ static bool receiver_process_batch(Config* config, int file_descriptor) {
send_status(file_descriptor, STATUS_ERROR);
return false;
}
if (check_size > MAX_RECEIVE_FILE_SIZE) {
if (check_size > MAX_RECEIVE_WHOLE_FILE_SIZE) {
free(check_path);
send_status(file_descriptor, STATUS_ERROR);
return false;
@@ -60,8 +110,15 @@ static bool receiver_process_batch(Config* config, int file_descriptor) {
}
struct stat st;
bool has_old = file_stat_secure(full_path, &st);
bool match = has_old && (unsigned long long)st.st_size == check_size &&
(long long)st.st_mtime == check_mtime;
long long old_mtime_nsec = 0;
if (has_old) {
#ifdef __linux__
old_mtime_nsec = st.st_mtim.tv_nsec;
#endif
}
bool match = !config->ignore_times && has_old && (unsigned long long)st.st_size == check_size &&
metadata_mtime_matches(st.st_mtime, old_mtime_nsec, (time_t)check_mtime,
(long)check_mtime_nsec, config->modify_window);
bool sent = send_status(file_descriptor, match ? STATUS_OK : STATUS_NEXT);
free(full_path);
free(check_path);
@@ -72,19 +129,40 @@ static bool receiver_process_batch(Config* config, int file_descriptor) {
}
int receiver_process(Config* config, int file_descriptor, const ReceiverSink* sink) {
return receiver_process_pending(config, file_descriptor, sink, NULL);
}
/* Runs the whole receive loop. The delete manifest may legitimately arrive
either FIRST (--delete-before / --delete-during: the sender transmits the
validated keep-set before any file data) or LAST (plain --delete /
--delete-after / --delete-delay: the manifest closes the data stream). In
the early modes the receiver deletes as soon as the manifest has been read
and acknowledges with STATUS_OK so the sender only starts streaming once the
deletion has committed (or failed); in the late modes the manifest is held
and the deletion is committed only after the terminal STATUS_FINISHED proves
the whole transfer succeeded. See receiver_process_pending() for how the -m
receiver defers that commit until its disk writer has drained. */
int receiver_process_pending(Config* config, int file_descriptor, const ReceiverSink* sink,
DeleteManifest** pending_manifest) {
Status status;
if (!receive_status(file_descriptor, &status))
return -1;
bool early_delete = config_delete_timing_early(config);
/* Parked keep-set for the late/commit timing. Every exit path below frees it
exactly once; the only exception is the successful FINISHED handoff, which
transfers ownership to *pending_manifest (used by the -m receiver). */
DeleteManifest* deferred_manifest = NULL;
while (status == STATUS_NEXT || status == STATUS_CHUNK || status == STATUS_CHECK ||
status == STATUS_KEEPALIVE || status == STATUS_ABORT || status == STATUS_CHECK_BATCH) {
status == STATUS_KEEPALIVE || status == STATUS_ABORT || status == STATUS_CHECK_BATCH ||
status == STATUS_MKDIR || status == STATUS_MANIFEST) {
if (status == STATUS_KEEPALIVE) {
if (!send_status(file_descriptor, STATUS_KEEPALIVE))
return -1;
goto next;
goto fail;
goto next_status;
}
if (status == STATUS_ABORT) {
log_message(LOG_LEVEL_INFO, "Received abort from client, cleaning up");
return -1;
goto fail;
}
if (status == STATUS_CHECK) {
bool skipped;
@@ -97,8 +175,46 @@ int receiver_process(Config* config, int file_descriptor, const ReceiverSink* si
goto receive_error;
} else if (status == STATUS_CHECK_BATCH) {
if (!receiver_process_batch(config, file_descriptor))
return -1;
goto next;
goto fail;
goto next_status;
} else if (status == STATUS_MKDIR) {
File* dir = file_receive_directory(file_descriptor);
if (!dir || !sink->store_file(dir, sink->context))
goto receive_error;
} else if (status == STATUS_MANIFEST) {
DeleteManifest* manifest = receive_manifest_entries(file_descriptor);
if (!manifest)
goto fail; /* receive_manifest_entries already sent STATUS_ERROR */
if (early_delete) {
/* --delete-before / --delete-during: the manifest is authoritative the
moment it arrives, before any file data. Delete now and acknowledge
so the sender only starts streaming once the deletion committed (or
failed). This is the rsync delete-before/delete-during window: a
later transfer failure does not restore these deletions. */
bool deletion_ok = config->use_delete ? manifest_delete_extras(config, manifest) : true;
delete_manifest_free(manifest);
if (!deletion_ok) {
send_status(file_descriptor, STATUS_ERROR);
goto fail;
}
if (!send_status(file_descriptor, STATUS_OK))
goto fail;
} else if (config->use_delete) {
/* Plain --delete / --delete-after / --delete-delay: hold the keep-set
and commit the deletion only after STATUS_FINISHED. */
if (deferred_manifest) {
log_message(LOG_LEVEL_ERROR, "Received a second delete manifest");
delete_manifest_free(deferred_manifest);
deferred_manifest = NULL;
delete_manifest_free(manifest);
send_status(file_descriptor, STATUS_ERROR);
goto fail;
}
deferred_manifest = manifest;
} else {
delete_manifest_free(manifest);
}
goto next_status;
} else {
File* file = file_receive(config, file_descriptor);
if (!file) {
@@ -108,35 +224,114 @@ int receiver_process(Config* config, int file_descriptor, const ReceiverSink* si
if (!sink->store_file(file, sink->context))
goto receive_error;
}
next:
next_status:
if (!receive_status(file_descriptor, &status))
goto receive_error;
}
if (status == STATUS_MANIFEST && receive_manifest(file_descriptor, config, &status) != 0)
return -1;
if (status != STATUS_FINISHED) {
log_message(LOG_LEVEL_ERROR, "Did not receive FINISHED Status");
goto receive_error;
}
if (sink->send_success && !send_status(file_descriptor, STATUS_OK))
return -1;
/* Commit-style (late) deletion: every data frame has been received and the
sender proved the whole tree with STATUS_FINISHED. The single-threaded
receiver stores files synchronously, so everything is on disk here and the
deletion can be committed before the --delay-updates publication in
send_success (the walker skips the staging dir, so staged files are never
treated as extras). The -m receiver passes `pending_manifest` because its
disk writer may still be draining; the caller commits after the writer has
joined so no extra file is removed unless the transfer is known to have
succeeded. */
if (deferred_manifest) {
if (pending_manifest) {
*pending_manifest = deferred_manifest;
deferred_manifest = NULL;
} else {
bool deletion_ok = manifest_delete_extras(config, deferred_manifest);
delete_manifest_free(deferred_manifest);
deferred_manifest = NULL;
if (!deletion_ok) {
send_status(file_descriptor, STATUS_ERROR);
goto fail;
}
}
}
if (sink->send_success) {
if (sink->send_success_frame) {
if (!sink->send_success_frame(file_descriptor, sink->context))
goto fail;
} else if (!send_status(file_descriptor, STATUS_OK)) {
goto fail;
}
}
return 0;
fail:
/* Failure exits that must not (or already did) report a STATUS_ERROR. The
parked keep-set is dropped: never commit a deletion for a failed stream. */
if (deferred_manifest) {
delete_manifest_free(deferred_manifest);
deferred_manifest = NULL;
}
return -1;
receive_error:
if (deferred_manifest) {
delete_manifest_free(deferred_manifest);
deferred_manifest = NULL;
}
if (sink->send_error)
send_status(file_descriptor, STATUS_ERROR);
return -1;
}
static bool receiver_save_file(File* file, void* context) {
Config* config = context;
bool success =
!config->save_to_disk || file_save_to_disk(config->receive_root_directory, file, config);
/* ---- Single-threaded sink (used by receiver_receive_files) ---- */
typedef struct {
Config* config;
ReceiverOutcomes outcomes;
} ReceiverSaveContext;
static bool receiver_save_file(File* file, void* context_pointer) {
ReceiverSaveContext* context = context_pointer;
FileSaveResult result = FILE_SAVE_ERROR;
if (!context->config->save_to_disk) {
/* Nothing is stored; report the file as not-written so a
--remove-source-files sender keeps its source. */
result = FILE_SAVE_SKIPPED;
} else {
result = file_save_to_disk_full(context->config->receive_root_directory, file, context->config);
}
if (result != FILE_SAVE_ERROR && context->config->remove_source_files && !file->is_dir &&
!file->skip && !receiver_outcomes_append(&context->outcomes, (unsigned char)result)) {
file_destroy(file);
return false;
}
file_destroy(file);
return success;
return result != FILE_SAVE_ERROR;
}
static bool receiver_send_success_frame(int fd, void* context_pointer) {
ReceiverSaveContext* context = context_pointer;
/* --delay-updates: the whole protocol stream (including manifest/delete
handling, which ran inside receiver_process) has succeeded and every
staged file was fully written. Publish them atomically now, before the
success/outcome frame tells a --remove-source-files sender it may delete
its sources. */
if (context->config->delay_updates && context->config->delay_context) {
if (!delay_updates_publish(context->config->delay_context, context->config)) {
send_status(fd, STATUS_ERROR);
return false;
}
}
return receiver_send_final_success(fd, context->config, &context->outcomes);
}
int receiver_receive_files(Config* config, int file_descriptor) {
ReceiverSink sink = {receiver_save_file, config, true, true};
return receiver_process(config, file_descriptor, &sink);
ReceiverSaveContext context = {.config = config, .outcomes = {0}};
ReceiverSink sink = {receiver_save_file, &context, true, true, receiver_send_success_frame};
int ret = receiver_process(config, file_descriptor, &sink);
if (ret != 0 && config->delay_updates && config->delay_context)
delay_updates_cleanup(config->delay_context);
receiver_outcomes_destroy(&context.outcomes);
return ret;
}
+29
View File
@@ -3,17 +3,46 @@
#include "config.h"
#include "file.h"
#include "file_receive.h"
typedef bool (*ReceiverFileSink)(File* file, void* context);
/* Ordered per-file save outcomes for one connection. One entry is appended
for every data-bearing file the receiver processes (in the order the files
were sent) so the sender of a --remove-source-files transfer can be told
which sources were actually written versus skipped on the receiver. */
typedef struct {
unsigned char* entries; /* FILE_SAVE_WRITTEN or FILE_SAVE_SKIPPED */
size_t count;
size_t capacity;
} ReceiverOutcomes;
typedef bool (*ReceiverSuccessFrame)(int fd, void* context);
typedef struct {
ReceiverFileSink store_file;
void* context;
bool send_error;
bool send_success;
/* Emits the end-of-transfer success frame. When the sender requested
--remove-source-files this includes one per-file status per processed
data file followed by the final STATUS_OK; otherwise just STATUS_OK. */
ReceiverSuccessFrame send_success_frame;
} ReceiverSink;
bool receiver_outcomes_append(ReceiverOutcomes* outcomes, unsigned char code);
void receiver_outcomes_destroy(ReceiverOutcomes* outcomes);
bool receiver_send_final_success(int fd, const Config* config, const ReceiverOutcomes* outcomes);
int receiver_process(Config* config, int file_descriptor, const ReceiverSink* sink);
/* receiver_process with an escape hatch for the commit-style (late) deletion:
when `pending_manifest` is non-NULL the receiver does NOT delete at
STATUS_FINISHED itself; instead it stores the owned keep-set manifest there
(leaving *pending_manifest untouched on early modes/errors) so the caller can
commit the deletion only after its disk writer has fully drained. Pass NULL
to keep the default behaviour (delete before the success frame). */
int receiver_process_pending(Config* config, int file_descriptor, const ReceiverSink* sink,
DeleteManifest** pending_manifest);
int receiver_receive_files(Config* config, int file_descriptor);
#endif
+103 -129
View File
@@ -1,13 +1,13 @@
#include "config.h"
#include "chunk.h"
#include "delay_updates.h"
#include "file.h"
#include "log.h"
#include "multiprocessing.h"
#include "protocol.h"
#include "queue.h"
#include "receiver.h"
#include "transport_tcp.h"
#include "transport_tls.h"
#include "unistd.h"
#include "utils.h"
#include <fcntl.h>
#include <limits.h>
@@ -15,7 +15,6 @@
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <unistd.h>
#include <openssl/x509.h>
@@ -25,6 +24,13 @@ static bool allow_delete;
static bool allow_unauthenticated;
static const char* required_client_cn;
/* Aggregate payload bytes the multithreaded receiver may buffer ahead of the
slow disk writer. Receiving one more chunk adds up to ~2 * MAX_CHUNK_SIZE
of transient wire/decompression buffers on top of the queued payloads, so
this ceiling keeps total per-connection receive memory (decompressed and
per-file copied chunk buffers included) within MAX_CONNECTION_MEMORY. */
#define RECEIVER_QUEUE_MAX_BYTES (MAX_CONNECTION_MEMORY - 2 * MAX_CHUNK_SIZE)
static bool tls_client_identity_allowed(SSL* ssl) {
if (!ssl || !required_client_cn)
return false;
@@ -57,6 +63,21 @@ static bool path_is_within(const char* root, const char* path) {
return strncmp(root, path, n) == 0 && (path[n] == '\0' || path[n] == '/');
}
/* --mkpath contract: when the client's destination root directory does not
exist yet on the server side, --mkpath tells the server to create it (and
any missing leading components) below the authorized root at connection
start. Without --mkpath the destination root must already exist: a missing
root is rejected up front instead of being silently invented by a later
write. Both paths are confined to the authorized root by the secure file
helpers. */
static bool ensure_receive_root(const Config* config) {
if (!config || !config->receive_root_directory)
return false;
if (config->mkpath)
return file_ensure_directory_secure(config->receive_root_directory);
return file_directory_exists_secure(config->receive_root_directory);
}
static bool __attribute__((unused)) configure_authorization(const char* root) {
char resolved[PATH_MAX];
if (!root) {
@@ -100,125 +121,6 @@ static bool __attribute__((unused)) configure_authorization(const char* root) {
return true;
}
int receive_files(Config* config, int fd) {
Status status;
if (!receive_status(fd, &status))
return -1;
while (status == STATUS_NEXT || status == STATUS_CHUNK || status == STATUS_CHECK ||
status == STATUS_KEEPALIVE || status == STATUS_ABORT || status == STATUS_CHECK_BATCH) {
if (status == STATUS_KEEPALIVE) {
send_status(fd, STATUS_KEEPALIVE);
goto next;
}
if (status == STATUS_ABORT) {
log_message(LOG_LEVEL_INFO, "Received abort from client, cleaning up");
return -1;
}
if (status == STATUS_CHECK) {
bool skipped;
File* file = receive_incremental_check(fd, config, &skipped);
if (skipped)
goto next;
if (file == NULL && !skipped)
return -1;
if (config->save_to_disk &&
!file_save_to_disk(config->receive_root_directory, file, config)) {
file_destroy(file);
send_status(fd, STATUS_ERROR);
return -1;
}
file_destroy(file);
} else if (status == STATUS_CHUNK) {
Chunk* chunk = receive_chunk_data(fd, config);
if (chunk == NULL) {
send_status(fd, STATUS_ERROR);
return -1;
}
for (int i = 0; i < chunk->element_count; i++) {
if (config->save_to_disk &&
!file_save_to_disk(config->receive_root_directory, chunk->items[i], config)) {
chunk_destroy(chunk);
send_status(fd, STATUS_ERROR);
return -1;
}
}
chunk_destroy(chunk);
} else if (status == STATUS_CHECK_BATCH) {
int count;
/* Batch framing has no checksum field yet; never silently downgrade a
checksum-enabled transfer into mtime-only matching. */
if (config->checksum || !receive_int(fd, &count) || count < 0 || count > MAX_MANIFEST_ENTRIES)
return -1;
for (int i = 0; i < count; i++) {
char* check_path = receive_str(fd);
if (!check_path)
return -1;
unsigned long long check_size;
long long check_mtime;
if (!receive_n_data(fd, &check_size, sizeof(check_size)) ||
!receive_n_data(fd, &check_mtime, sizeof(check_mtime))) {
free(check_path);
return -1;
}
if (!utils_valid_batch_path(check_path)) {
free(check_path);
send_status(fd, STATUS_ERROR);
return -1;
}
struct stat st;
char* full_path = path_cat(config->receive_root_directory, check_path);
if (!full_path) {
free(check_path);
send_status(fd, STATUS_ERROR);
return -1;
}
bool has_old = full_path && file_stat_secure(full_path, &st);
bool match = has_old && (unsigned long long)st.st_size == check_size &&
(long long)st.st_mtime == check_mtime;
bool sent = send_status(fd, match ? STATUS_OK : STATUS_NEXT);
free(full_path);
free(check_path);
if (!sent)
return -1;
}
goto next;
} else {
File* file = file_receive(config, fd);
if (file == NULL) {
log_message(LOG_LEVEL_ERROR, "Failed to receive file");
send_status(fd, STATUS_ERROR);
return -1;
}
if (config->save_to_disk &&
!file_save_to_disk(config->receive_root_directory, file, config)) {
file_destroy(file);
send_status(fd, STATUS_ERROR);
return -1;
}
file_destroy(file);
}
next:
if (!receive_status(fd, &status)) {
send_status(fd, STATUS_ERROR);
return -1;
}
}
if (status == STATUS_MANIFEST) {
if (receive_manifest(fd, config, &status) != 0) {
return -1;
}
}
if (status != STATUS_FINISHED) {
log_message(LOG_LEVEL_ERROR, "Did not receive FINISHED Status");
send_status(fd, STATUS_ERROR);
return -1;
}
send_status(fd, STATUS_OK);
return 0;
}
void handler(int file_descriptor) {
SSL* ssl = io_get_ssl();
ProtocolSession session;
@@ -232,6 +134,7 @@ void handler(int file_descriptor) {
protocol_session_unbind();
return;
}
protocol_set_8_bit_output(config->eight_bit_output);
if (!authorized_root) {
log_message(LOG_LEVEL_ERROR, "No server-side destination root configured");
config_delete(config);
@@ -277,6 +180,31 @@ void handler(int file_descriptor) {
return;
}
config->use_delete = config->use_delete && allow_delete;
/* --mkpath: create the destination root (and its missing leading components)
before anything else; without it the root must pre-exist. A failure here
aborts the connection cleanly before any file data is exchanged. */
if (!ensure_receive_root(config)) {
log_message(LOG_LEVEL_ERROR, "destination root is not available: %s",
config->receive_root_directory);
config_delete(config);
close(file_descriptor);
protocol_session_unbind();
return;
}
/* A --delay-updates transfer stages under a private 0700 directory inside
the receive root. Create it up front (wiping leftovers of any previously
interrupted delayed transfer) so a fully-skipped run also starts clean. */
if (config->delay_updates) {
config->delay_context = delay_updates_context_create(config->receive_root_directory);
if (!config->delay_context || !delay_updates_prepare(config->delay_context)) {
log_message(LOG_LEVEL_ERROR, "Failed to initialize --delay-updates staging area");
delay_updates_cleanup(config->delay_context);
config_delete(config);
close(file_descriptor);
protocol_session_unbind();
return;
}
}
if (config->use_multithreading) {
Queue* q = queue_create(100, file_destroy);
if (q == NULL) {
@@ -294,7 +222,10 @@ void handler(int file_descriptor) {
protocol_session_unbind();
return;
}
context->session.total_allocated_bytes = session.total_allocated_bytes;
protocol_session_set_max_alloc(&context->session, config->max_alloc);
atomic_store(&context->session.total_allocated_bytes,
atomic_load(&session.total_allocated_bytes));
pipeline_context_receiver_set_queue_byte_limit(context, RECEIVER_QUEUE_MAX_BYTES);
thrd_t receiver, writer;
bool receiver_created = thrd_create(&receiver, receive_thread, context) == thrd_success;
bool writer_created = false;
@@ -323,9 +254,43 @@ void handler(int file_descriptor) {
int writer_result;
thrd_join(receiver, &receiver_result);
thrd_join(writer, &writer_result);
send_status(file_descriptor, receiver_result == thrd_success && writer_result == thrd_success
? STATUS_OK
: STATUS_ERROR);
bool transfer_ok = receiver_result == thrd_success && writer_result == thrd_success;
if (transfer_ok) {
/* Commit-style (late) deletion: receive_thread handed the keep-set
manifest here instead of deleting while write_thread might still be
draining, so by now every file is on disk and the whole transfer is
known to have succeeded. Remove the extras before publishing a
--delay-updates run; the walker skips the staging directory. */
if (context->deferred_manifest) {
if (!manifest_delete_extras(config, context->deferred_manifest)) {
transfer_ok = false;
}
delete_manifest_free(context->deferred_manifest);
context->deferred_manifest = NULL;
}
}
if (transfer_ok) {
/* --delay-updates: receive_thread has finished the whole protocol stream
(including manifest/delete handling) and write_thread has drained its
queue, so every staged file is complete. Publish atomically before the
success/outcome frame so a --remove-source-files sender only learns of
files that were actually installed. */
if (config->delay_updates && config->delay_context &&
!delay_updates_publish(config->delay_context, config)) {
transfer_ok = false;
}
}
if (transfer_ok) {
if (!receiver_send_final_success(file_descriptor, config, &context->outcomes))
transfer_ok = false;
} else {
send_status(file_descriptor, STATUS_ERROR);
}
if (!transfer_ok) {
log_message(LOG_LEVEL_ERROR, "Transfer failed");
if (config->delay_updates && config->delay_context)
delay_updates_cleanup(config->delay_context);
}
pipeline_context_receiver_destroy(context);
} else {
if (receiver_receive_files(config, file_descriptor) != 0)
@@ -380,6 +345,7 @@ int main(int argc, char* argv[]) {
stdio_mode = true;
} else if (strcmp(argv[i], "-v") == 0 || strcmp(argv[i], "--verbose") == 0) {
set_log_level(LOG_LEVEL_DEBUG);
set_log_debug_flags(LOG_DEBUG_ALL);
} else if (strcmp(argv[i], "--tls") == 0) {
use_tls = true;
} else if (strcmp(argv[i], "--cert") == 0 && i + 1 < argc) {
@@ -400,12 +366,17 @@ int main(int argc, char* argv[]) {
char* end;
long p = strtol(argv[++i], &end, 10);
if (*end || p <= 0 || p > 65535) {
fprintf(stderr, "Error: invalid port '%s' (must be 1-65535)\n", argv[i]);
char* escaped = output_escape(argv[i], false);
fprintf(stderr, "Error: invalid port '%s' (must be 1-65535)\n",
escaped ? escaped : "<allocation failed>");
free(escaped);
return 1;
}
port = (int)p;
} else if (argv[i][0] == '-') {
fprintf(stderr, "Unknown option: %s\n", argv[i]);
char* escaped = output_escape(argv[i], false);
fprintf(stderr, "Unknown option: %s\n", escaped ? escaped : "<allocation failed>");
free(escaped);
print_server_usage();
return 1;
}
@@ -415,7 +386,10 @@ int main(int argc, char* argv[]) {
signal(SIGINT, cleanup);
signal(SIGTERM, cleanup);
if (!configure_authorization(destination_root)) {
fprintf(stderr, "Error: invalid destination root '%s'\n", destination_root);
char* escaped = output_escape(destination_root, false);
fprintf(stderr, "Error: invalid destination root '%s'\n",
escaped ? escaped : "<allocation failed>");
free(escaped);
return 1;
}
if (stdio_mode) {
+5 -4
View File
@@ -1,17 +1,18 @@
#include "log.h"
#include "array_list.h"
#include "protocol.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
ArrayList* array_list_create(void (*item_destroyer)(void* item)) {
ArrayList* list = (ArrayList*)malloc(sizeof(ArrayList));
ArrayList* list = (ArrayList*)protocol_alloc(sizeof(ArrayList));
if (list == NULL) {
log_perror("ERROR: Could not allocate memory for array list struct");
return NULL;
}
list->items = malloc(INITIAL_ARRAY_SIZE * sizeof(void*));
list->items = protocol_alloc(INITIAL_ARRAY_SIZE * sizeof(void*));
if (list->items == NULL) {
free(list);
return NULL;
@@ -41,7 +42,7 @@ static bool array_list_extend(ArrayList* array_list) {
int new_capacity = array_list->capacity * 2;
if (new_capacity == 0)
new_capacity = INITIAL_ARRAY_SIZE;
void* new_items = realloc(array_list->items, new_capacity * sizeof(void*));
void* new_items = protocol_realloc(array_list->items, new_capacity * sizeof(void*));
if (new_items == NULL) {
log_perror("ERROR: Could not reallocate memory for array list items");
return false;
@@ -67,7 +68,7 @@ void** array_list_to_array(const ArrayList* array_list) {
if (array_list == NULL) {
return NULL;
}
void** array = malloc(array_list->size * sizeof(void*));
void** array = protocol_alloc(array_list->size * sizeof(void*));
if (array == NULL) {
log_perror("Could not malloc space for array from array list!");
return NULL;
+78
View File
@@ -0,0 +1,78 @@
#include "checksum.h"
#include <openssl/evp.h>
#include <string.h>
#include <strings.h>
/* delta.c owns the single XXH_IMPLEMENTATION that provides the xxHash symbols
* for the whole binary; this TU only needs the declarations. */
#include <xxhash.h>
bool checksum_digest(ChecksumAlgo algo, uint64_t seed, const void* data, size_t size, uint8_t* out,
size_t out_capacity, size_t* out_len) {
if (!out || !out_len || out_capacity < CHECKSUM_MAX_DIGEST_LEN)
return false;
if (data == NULL && size != 0)
return false;
if (algo == CHECKSUM_ALGO_XXH64) {
uint64_t digest = XXH64(data, size, seed);
uint8_t buf[CHECKSUM_MAX_DIGEST_LEN];
memcpy(buf, &digest, sizeof(digest));
memcpy(out, buf, sizeof(digest));
*out_len = sizeof(digest);
return true;
}
if (algo == CHECKSUM_ALGO_MD5) {
/* md5 takes no seed; the caller's seed is deliberately ignored (documented
* in RSYNC_COMPAT.md). OpenSSL's one-shot EVP_Digest needs a non-NULL
* buffer even for an empty input, so map a NULL data + size==0 to an empty
* buffer. */
static const uint8_t empty = 0;
const void* input = data ? data : &empty;
unsigned int digest_len = 0;
if (EVP_Digest(input, size, out, &digest_len, EVP_md5(), NULL) != 1)
return false;
if (digest_len > out_capacity)
return false;
*out_len = digest_len;
return true;
}
return false;
}
int checksum_algo_from_name(const char* name) {
if (!name)
return -1;
if (strcasecmp(name, "xxh64") == 0 || strcasecmp(name, "xxhash") == 0 ||
strcasecmp(name, "xxh3") == 0)
return (int)CHECKSUM_ALGO_XXH64;
if (strcasecmp(name, "md5") == 0)
return (int)CHECKSUM_ALGO_MD5;
return -1;
}
const char* checksum_algo_name(ChecksumAlgo algo) {
switch (algo) {
case CHECKSUM_ALGO_XXH64:
return "xxh64";
case CHECKSUM_ALGO_MD5:
return "md5";
}
return "<unknown>";
}
bool checksum_algo_valid(int algo) {
return algo == (int)CHECKSUM_ALGO_XXH64 || algo == (int)CHECKSUM_ALGO_MD5;
}
uint8_t checksum_digest_len(ChecksumAlgo algo) {
switch (algo) {
case CHECKSUM_ALGO_XXH64:
return 8;
case CHECKSUM_ALGO_MD5:
return 16;
}
return 0;
}
+45
View File
@@ -0,0 +1,45 @@
#ifndef CHECKSUM_H
#define CHECKSUM_H
#include <stdbool.h>
#include <stddef.h>
#include <stdint.h>
/* Whole-file content-digest algorithms selectable with --checksum-choice and
* seeded with --checksum-seed. The ids are the values actually placed on the
* wire (config frame), so they must be kept stable and validated on receive.
* CHECKSUM_ALGO_XXH64 == 0 is the default and is byte-for-byte what FastSync
* computed before these options existed (xxHash64 with seed 0). */
typedef enum { CHECKSUM_ALGO_XXH64 = 0, CHECKSUM_ALGO_MD5 = 1 } ChecksumAlgo;
/* md5 digest is 16 bytes, the longest supported. */
#define CHECKSUM_MAX_DIGEST_LEN 16
/* Compute the whole-file digest of the first `size` bytes of `data`.
*
* - CHECKSUM_ALGO_XXH64: xxHash64(data, size, seed) (full 64-bit seed).
* - CHECKSUM_ALGO_MD5: md5(data, size) via OpenSSL EVP.
* md5 has no seed, so `seed` is ignored (documented).
* - `size == 0` hashes the empty input (plus its seed), not a NULL input.
*
* Writes up to `out_capacity` bytes into `out`, storing the digest length in
* *out_len. Returns false on NULL out* or when the digest would not fit.
* Never writes more than CHECKSUM_MAX_DIGEST_LEN bytes. */
bool checksum_digest(ChecksumAlgo algo, uint64_t seed, const void* data, size_t size, uint8_t* out,
size_t out_capacity, size_t* out_len);
/* Resolve a --checksum-choice string (case-insensitive) to an algorithm id.
* Accepts "xxh64" and "xxhash" (both map to CHECKSUM_ALGO_XXH64, rsync's
* xxhash spelling) and "md5". Returns -1 for any unsupported name. */
int checksum_algo_from_name(const char* name);
/* Canonical name of an algorithm (used in CLI error messages). */
const char* checksum_algo_name(ChecksumAlgo algo);
/* True when `algo` is a supported id (used by config receive validation). */
bool checksum_algo_valid(int algo);
/* Digest length in bytes for an algorithm (xxx64 = 8, md5 = 16). */
uint8_t checksum_digest_len(ChecksumAlgo algo);
#endif /* CHECKSUM_H */
+45 -11
View File
@@ -22,7 +22,7 @@
Chunk* chunk_create(File** items, int element_count) {
if (element_count < 0 || (element_count > 0 && items == NULL))
return NULL;
Chunk* chunk = (Chunk*)malloc(sizeof(Chunk));
Chunk* chunk = (Chunk*)protocol_alloc(sizeof(Chunk));
if (chunk == NULL) {
log_perror("ERROR: Could not allocate memory for chunk structure");
return NULL;
@@ -35,7 +35,7 @@ Chunk* chunk_create(File** items, int element_count) {
free(chunk);
return NULL;
}
chunk->items = (File**)malloc((size_t)element_count * sizeof(File*));
chunk->items = (File**)protocol_alloc((size_t)element_count * sizeof(File*));
if (chunk->items == NULL) {
free(chunk);
return NULL;
@@ -65,7 +65,7 @@ void chunk_destroy(void* item) {
static unsigned long long per_file_serialize_size(File* file, bool use_metadata) {
unsigned long long size = sizeof(size_t);
size_t path_len = strlen(file->path);
size_t path_len = strlen(file_wire_path(file));
unsigned long long metadata_size =
use_metadata ? sizeof(int) + (file->metadata ? FILE_METADATA_WIRE_SIZE : 0) : 0;
if ((unsigned long long)path_len > ULLONG_MAX - size)
@@ -74,6 +74,10 @@ static unsigned long long per_file_serialize_size(File* file, bool use_metadata)
if (metadata_size > ULLONG_MAX - size)
return 0;
size += metadata_size;
/* Entry type marker: 0 = regular file, 1 = explicit directory entry. */
if (sizeof(int) > ULLONG_MAX - size)
return 0;
size += sizeof(int);
if (sizeof(size_t) > ULLONG_MAX - size)
return 0;
size += sizeof(size_t);
@@ -89,7 +93,8 @@ Data* chunk_serialize(Chunk* chunk, bool use_metadata) {
for (int i = 0; i < chunk->element_count; i++) {
if (!chunk->items[i] || !chunk->items[i]->path || !chunk->items[i]->data ||
(chunk->items[i]->data->size > 0 && !chunk->items[i]->data->data) ||
chunk->items[i]->path[0] == '\0' || has_path_traversal(chunk->items[i]->path))
chunk->items[i]->path[0] == '\0' || has_path_traversal(chunk->items[i]->path) ||
(file_wire_path(chunk->items[i]))[0] == '\0')
return NULL;
unsigned long long file_size = per_file_serialize_size(chunk->items[i], use_metadata);
if (file_size == 0 || file_size > ULLONG_MAX - data_size || data_size + file_size > SIZE_MAX)
@@ -104,19 +109,25 @@ Data* chunk_serialize(Chunk* chunk, bool use_metadata) {
char* data_pointer = data->data;
for (int i = 0; i < chunk->element_count; i++) {
File* file = chunk->items[i];
size_t path_len = strlen(file->path);
const char* wire_path = file_wire_path(file);
size_t path_len = strlen(wire_path);
memcpy(data_pointer, &path_len, sizeof(size_t));
data_pointer += sizeof(size_t);
memcpy(data_pointer, file->path, path_len);
memcpy(data_pointer, wire_path, path_len);
data_pointer += path_len;
int entry_type = file->is_dir ? 1 : 0;
memcpy(data_pointer, &entry_type, sizeof(int));
data_pointer += sizeof(int);
if (use_metadata)
metadata_to_buf(&data_pointer, file->metadata);
size_t file_data_size = file->data->size;
memcpy(data_pointer, &file_data_size, sizeof(size_t));
data_pointer += sizeof(size_t);
memcpy(data_pointer, file->data->data, file_data_size);
if (file_data_size > 0)
memcpy(data_pointer, file->data->data, file_data_size);
data_pointer += file_data_size;
}
return data;
@@ -158,7 +169,7 @@ Chunk* chunk_deserialize(Data* data, bool use_metadata) {
array_list_delete(files);
return NULL;
}
char* path = malloc(path_len + 1);
char* path = protocol_alloc(path_len + 1);
if (path == NULL) {
log_perror("Could not allocate memory for file path");
array_list_delete(files);
@@ -187,6 +198,24 @@ Chunk* chunk_deserialize(Data* data, bool use_metadata) {
return NULL;
}
if (remaining_size < sizeof(int)) {
log_message(LOG_LEVEL_ERROR, "Invalid chunk format: not enough data for entry type");
file_destroy(file);
array_list_delete(files);
return NULL;
}
int entry_type;
memcpy(&entry_type, data_pointer, sizeof(int));
if (entry_type != 0 && entry_type != 1) {
log_message(LOG_LEVEL_ERROR, "Invalid chunk format: bad entry type");
file_destroy(file);
array_list_delete(files);
return NULL;
}
file->is_dir = entry_type == 1;
data_pointer += sizeof(int);
remaining_size -= sizeof(int);
if (use_metadata) {
if (remaining_size < sizeof(int)) {
log_message(LOG_LEVEL_ERROR, "Invalid chunk format: not enough data for metadata");
@@ -245,7 +274,7 @@ Chunk* chunk_deserialize(Data* data, bool use_metadata) {
}
size_t allocation_size = file_data_size > 0 ? file_data_size : 1;
void* file_data = malloc(allocation_size);
void* file_data = protocol_alloc(allocation_size);
if (file_data == NULL) {
log_perror("Could not allocate memory for file data");
file_destroy(file);
@@ -290,15 +319,20 @@ Chunk* chunk_deserialize(Data* data, bool use_metadata) {
}
Data* chunk_compress(Chunk* chunk, int compression_level, bool use_metadata) {
return chunk_compress_with_threads(chunk, compression_level, use_metadata, 0);
}
Data* chunk_compress_with_threads(Chunk* chunk, int compression_level, bool use_metadata,
int compression_threads) {
log_message(LOG_LEVEL_DEBUG, "Starting to compress chunk");
Data* serialized = chunk_serialize(chunk, use_metadata);
if (serialized == NULL)
return NULL;
Data* compressed = data_compress(serialized, compression_level);
Data* compressed = data_compress_with_threads(serialized, compression_level, compression_threads);
data_destroy(serialized);
if (compressed == NULL)
return NULL;
log_message(LOG_LEVEL_DEBUG, "Chunk successfully compressed");
log_debug_message(LOG_DEBUG_PACK, "Chunk successfully compressed");
return compressed;
}
+2
View File
@@ -19,6 +19,8 @@ void chunk_destroy(void* chunk);
Data* chunk_serialize(Chunk* chunk, bool use_metadata);
Chunk* chunk_deserialize(Data* data, bool use_metadata);
Data* chunk_compress(Chunk* chunk, int compression_level, bool use_metadata);
Data* chunk_compress_with_threads(Chunk* chunk, int compression_level, bool use_metadata,
int compression_threads);
Chunk* receive_chunk_data(int fd, const Config* config);
#endif
+53 -9
View File
@@ -1,33 +1,53 @@
#include "compression.h"
#include "data.h"
#include "log.h"
#include "protocol.h"
#include <stdlib.h>
#include <limits.h>
#include <stdint.h>
#include <string.h>
#include <strings.h>
#include <unistd.h>
#include <zstd.h>
#define INITIAL_DECOMPRESS_BUF_SIZE (1024 * 1024)
#define MAX_DECOMPRESSED_SIZE (100ULL * 1024 * 1024) /* 100 MB hard ceiling */
static const char* SKIP_COMPRESSION_EXTENSIONS[] = {".jpg", ".jpeg", ".png", ".gif", ".mp4", ".mkv",
".zip", ".gz", ".xz", ".zst", NULL};
static char* SKIP_COMPRESSION_EXTENSIONS[] = {".jpg", ".jpeg", ".png", ".gif", ".mp4", ".mkv",
".zip", ".gz", ".xz", ".zst", NULL};
bool compression_should_skip(const char* path) {
return compression_should_skip_with_suffixes(path, NULL, -1);
}
bool compression_should_skip_with_suffixes(const char* path, char* const* suffixes, int count) {
if (!path)
return false;
const char* dot = strrchr(path, '.');
if (!dot)
return false;
for (int i = 0; SKIP_COMPRESSION_EXTENSIONS[i]; i++) {
if (strcasecmp(dot, SKIP_COMPRESSION_EXTENSIONS[i]) == 0)
if (count < 0) {
suffixes = SKIP_COMPRESSION_EXTENSIONS;
count = 0;
while (SKIP_COMPRESSION_EXTENSIONS[count])
count++;
}
for (int i = 0; i < count; i++) {
if (strcasecmp(dot, suffixes[i]) == 0)
return true;
}
return false;
}
Data* data_compress(Data* data_to_compress, int compression_level) {
return data_compress_with_threads(data_to_compress, compression_level, 0);
}
Data* data_compress_with_threads(Data* data_to_compress, int compression_level,
int compression_threads) {
if (!data_to_compress || (!data_to_compress->data && data_to_compress->size != 0) ||
compression_threads < 0 || compression_threads > COMPRESSION_MAX_THREADS)
return NULL;
log_message(LOG_LEVEL_DEBUG, "Starting to compress data");
size_t dst_size = ZSTD_compressBound(data_to_compress->size);
Data* compressed_data = data_create_empty(dst_size);
@@ -49,6 +69,30 @@ Data* data_compress(Data* data_to_compress, int compression_level) {
return NULL;
}
if (compression_threads > 0) {
long online_cpus = sysconf(_SC_NPROCESSORS_ONLN);
int available_threads = online_cpus > 0 && online_cpus < compression_threads
? (int)online_cpus
: compression_threads;
zret = ZSTD_CCtx_setParameter(cctx, ZSTD_c_nbWorkers, available_threads);
if (ZSTD_isError(zret)) {
log_message(LOG_LEVEL_ERROR, "Failed to set compression threads: %s",
ZSTD_getErrorName(zret));
ZSTD_freeCCtx(cctx);
data_destroy(compressed_data);
return NULL;
}
/* Streaming compression needs the source size before threaded mode can end a frame. */
zret = ZSTD_CCtx_setPledgedSrcSize(cctx, data_to_compress->size);
if (ZSTD_isError(zret)) {
log_message(LOG_LEVEL_ERROR, "Failed to set compression source size: %s",
ZSTD_getErrorName(zret));
ZSTD_freeCCtx(cctx);
data_destroy(compressed_data);
return NULL;
}
}
ZSTD_inBuffer input = {data_to_compress->data, data_to_compress->size, 0};
ZSTD_outBuffer output = {compressed_data->data, dst_size, 0};
@@ -66,8 +110,8 @@ Data* data_compress(Data* data_to_compress, int compression_level) {
compressed_data->size = output.pos;
ZSTD_freeCCtx(cctx);
log_message(LOG_LEVEL_DEBUG, "Data succesfully compressed from %zu to %zu",
data_to_compress->size, compressed_data->size);
log_debug_message(LOG_DEBUG_UTIL, "Data succesfully compressed from %zu to %zu",
data_to_compress->size, compressed_data->size);
return compressed_data;
}
@@ -75,7 +119,7 @@ Data* data_decompress_limited(Data* compressed_data, size_t maximum_size) {
if (!compressed_data || (!compressed_data->data && compressed_data->size != 0) ||
maximum_size == 0)
return NULL;
log_message(LOG_LEVEL_DEBUG, "Start to decompress data");
log_debug_message(LOG_DEBUG_UTIL, "Start to decompress data");
unsigned long long dst_size =
ZSTD_getFrameContentSize(compressed_data->data, compressed_data->size);
if (ZSTD_isError(dst_size)) {
@@ -139,7 +183,7 @@ Data* data_decompress_limited(Data* compressed_data, size_t maximum_size) {
buf_size *= 2;
if (buf_size > hard_limit)
buf_size = (size_t)hard_limit;
void* new_data = realloc(uncompressed_data->data, buf_size);
void* new_data = protocol_realloc(uncompressed_data->data, buf_size);
if (!new_data) {
log_message(LOG_LEVEL_ERROR, "Failed to grow decompression buffer");
ZSTD_freeDCtx(dctx);
@@ -155,7 +199,7 @@ Data* data_decompress_limited(Data* compressed_data, size_t maximum_size) {
uncompressed_data->size = output.pos;
ZSTD_freeDCtx(dctx);
log_message(LOG_LEVEL_DEBUG, "Decompressed data successfully");
log_debug_message(LOG_DEBUG_UTIL, "Decompressed data successfully");
return uncompressed_data;
}
+5
View File
@@ -4,9 +4,14 @@
#include "data.h"
#include <stdbool.h>
#define COMPRESSION_MAX_THREADS 64
Data* data_compress(Data* data_to_compress, int compression_level);
Data* data_compress_with_threads(Data* data_to_compress, int compression_level,
int compression_threads);
Data* data_decompress(Data* compressed_data);
Data* data_decompress_limited(Data* compressed_data, size_t maximum_size);
bool compression_should_skip(const char* path);
bool compression_should_skip_with_suffixes(const char* path, char* const* suffixes, int count);
#endif
+375 -50
View File
@@ -1,6 +1,8 @@
#include "config.h"
#include "chmod.h"
#include "delay_updates.h"
#include "delta.h"
#include "file_list.h"
#include "log.h"
#include "protocol.h"
#include "utils.h"
@@ -18,10 +20,14 @@ static void config_set_defaults(Config* config) {
config->use_chunk_serialization = false;
config->use_compression = false;
config->use_metadata = false;
config->use_executability = false;
config->metadata_explicitly_disabled = false;
config->show_progress = false;
config->dry_run = false;
config->remove_source_files = false;
config->use_delete = false;
config->compression_level = 5;
config->compression_threads = 0;
config->use_sendfile = false;
config->chunk_size = DEFAULT_CHUNK_SIZE;
config->ssh_port = 22;
@@ -34,8 +40,14 @@ static void config_set_defaults(Config* config) {
config->include_count = 0;
config->max_size = 0;
config->min_size = 0;
config->max_alloc = DEFAULT_MAX_ALLOC;
config->use_incremental = false;
config->ignore_times = false;
config->size_only = false;
config->use_delta = false;
config->whole_file = false;
config->fuzzy = false;
config->modify_window = 0;
config->delta_block_size = DELTA_BLOCK_SIZE_DEFAULT;
config->delta_max_file_size = DELTA_MAX_FILE_SIZE;
config->use_tls = false;
@@ -65,31 +77,48 @@ static void config_set_defaults(Config* config) {
config->preserve_sparse = false;
config->itemize_changes = false;
config->out_format = NULL;
config->log_file_format = NULL;
config->info_level = 0;
config->debug_level = 0;
config->list_only = false;
config->human_readable = false;
config->eight_bit_output = false;
config->existing = false;
config->ignore_existing = false;
config->update = false;
config->inplace = false;
config->delay_updates = false;
config->use_fsync = false;
config->append = false;
config->append_verify = false;
config->delete_excluded = false;
config->delete_after = false;
config->max_delete = 0;
config->max_delete = -1;
config->ignore_errors = false;
config->force_delete = false;
config->filters = NULL;
config->files_from = NULL;
config->files_from_set = NULL;
config->from0 = false;
config->cvs_exclude = false;
config->per_dir_filter = false;
config->prune_empty_dirs = false;
config->one_file_system = false;
config->relative = false;
config->no_implied_dirs = false;
config->dirs = false;
config->mkpath = false;
config->rsh_command = NULL;
config->rsync_path = NULL;
config->old_args = false;
config->temp_dir = NULL;
config->compare_dest = NULL;
config->copy_dest = NULL;
config->link_dest = NULL;
config->basis_dirs = NULL;
config->basis_count = 0;
config->partial_dir = NULL;
config->suffix = NULL;
config->delete_before = false;
config->delete_during = false;
config->delete_delay = false;
config->address = NULL;
config->bind_address = NULL;
config->ipv6 = false;
@@ -98,8 +127,14 @@ static void config_set_defaults(Config* config) {
config->daemon_config = NULL;
config->server_mode = false;
config->checksum = false;
config->checksum_algo = CHECKSUM_ALGO_XXH64;
config->checksum_seed = 0;
config->compress_choice = NULL;
config->chmod_spec = NULL;
config->skip_compress_suffixes = NULL;
config->skip_compress_count = 0;
config->skip_compress_set = false;
config->delay_context = NULL;
}
static bool valid_wire_bool(int value) {
@@ -118,25 +153,37 @@ static bool validate_received_config(const Config* config) {
return valid_wire_bool(config->save_to_disk) && valid_wire_bool(config->use_multithreading) &&
valid_wire_bool(config->use_chunk_serialization) &&
valid_wire_bool(config->use_compression) && valid_wire_bool(config->use_metadata) &&
valid_wire_bool(config->use_sendfile) && valid_wire_bool(config->use_delete) &&
valid_wire_bool(config->use_incremental) && valid_wire_bool(config->use_delta) &&
valid_wire_bool(config->backup) && valid_wire_bool(config->follow_symlinks) &&
valid_wire_bool(config->copy_links) && valid_wire_bool(config->safe_links) &&
valid_wire_bool(config->copy_unsafe_links) &&
valid_wire_bool(config->use_executability) && valid_wire_bool(config->use_sendfile) &&
valid_wire_bool(config->use_delete) && valid_wire_bool(config->use_incremental) &&
valid_wire_bool(config->size_only) && valid_wire_bool(config->ignore_times) &&
valid_wire_bool(config->use_delta) && valid_wire_bool(config->backup) &&
valid_wire_bool(config->fuzzy) && valid_wire_bool(config->remove_source_files) &&
valid_wire_bool(config->follow_symlinks) && valid_wire_bool(config->copy_links) &&
valid_wire_bool(config->safe_links) && valid_wire_bool(config->copy_unsafe_links) &&
valid_wire_bool(config->preserve_hard_links) && valid_wire_bool(config->preserve_acls) &&
valid_wire_bool(config->preserve_xattrs) && valid_wire_bool(config->preserve_devices) &&
valid_wire_bool(config->preserve_sparse) && valid_wire_bool(config->update) &&
valid_wire_bool(config->preserve_sparse) && valid_wire_bool(config->ignore_existing) &&
valid_wire_bool(config->existing) && valid_wire_bool(config->update) &&
valid_wire_bool(config->inplace) && valid_wire_bool(config->append) &&
valid_wire_bool(config->append_verify) && valid_wire_bool(config->delete_excluded) &&
valid_wire_bool(config->delete_after) && valid_wire_bool(config->relative) &&
valid_wire_bool(config->prune_empty_dirs) && valid_wire_bool(config->partial) &&
valid_wire_bool(config->delete_before) && valid_wire_bool(config->checksum) &&
valid_wire_bool(config->use_fsync) && valid_wire_bool(config->append_verify) &&
valid_wire_bool(config->delete_excluded) && valid_wire_bool(config->force_delete) &&
valid_wire_bool(config->delete_after) && valid_wire_bool(config->delete_delay) &&
valid_wire_bool(config->delete_during) && valid_wire_bool(config->relative) &&
valid_wire_bool(config->prune_empty_dirs) && valid_wire_bool(config->delay_updates) &&
valid_wire_bool(config->mkpath) && !(config->delay_updates && config->inplace) &&
!(config->delay_updates && delay_updates_staging_name_conflict(config->backup_dir)) &&
valid_wire_bool(config->partial) && valid_wire_bool(config->delete_before) &&
valid_wire_bool(config->checksum) && valid_wire_bool(config->eight_bit_output) &&
checksum_algo_valid(config->checksum_algo) && config_has_valid_delete_timing(config) &&
!(config->skip_compress_set && config->use_chunk_serialization) &&
(!config->use_compression ||
(config->compression_level >= 1 && config->compression_level <= 22)) &&
config->chunk_size > 0 && config->chunk_size <= MAX_CHUNK_SIZE &&
config->delta_block_size >= DELTA_BLOCK_SIZE_MIN &&
config->delta_block_size <= DELTA_BLOCK_SIZE_MAX &&
config->delta_max_file_size <= DELTA_MAX_FILE_SIZE && config->max_delete >= 0 &&
config->delta_max_file_size <= DELTA_MAX_FILE_SIZE && config->modify_window >= 0 &&
config->max_delete >= -1 && config->skip_compress_count >= 0 &&
config->skip_compress_count <= 10000 && config->max_alloc > 0 &&
(!config->chmod_spec || !*config->chmod_spec ||
chmod_apply(0, config->chmod_spec, &(mode_t){0}));
}
@@ -149,6 +196,107 @@ Config* config_create(void) {
return config;
}
bool config_delete_timing_early(const Config* config) {
if (!config)
return false;
return config->delete_before || config->delete_during;
}
/* A delete-timing flag is only meaningful together with --delete. At most one
of the four flags may be set; several simultaneous timings are a client bug
and are rejected on both ends. */
bool config_has_valid_delete_timing(const Config* config) {
if (!config)
return false;
if (!config->use_delete)
return !config->delete_before && !config->delete_during && !config->delete_delay &&
!config->delete_after;
int timing_count = (config->delete_before ? 1 : 0) + (config->delete_during ? 1 : 0) +
(config->delete_delay ? 1 : 0) + (config->delete_after ? 1 : 0);
return timing_count <= 1;
}
bool config_has_basis(const Config* config) {
return config && config->basis_count > 0;
}
/* A basis-dir path travels from the client to the receiver and is resolved
* below the destination root, so it must be a non-empty relative path with no
* "." or ".." component and no traversal: an absolute or escaping path would
* make the receiver read or link files outside its authorized root.
*
* Returns a malloc'd CANONICAL copy of an accepted path, or NULL when the path
* is rejected. Canonicalization collapses interior empty components ("a//b" ->
* "a/b"), drops "." components and trailing "/"s, so validation, the delete
* walker prefix match and the receiver's basis lookup all agree on one form.
* The normalizer is the single source of truth for both config_basis_path_valid
* and config_basis_append. */
static char* basis_path_normalize(const char* path) {
if (!path || path[0] == '\0' || path[0] == '/' || has_path_traversal(path))
return NULL;
if (strcmp(path, ".") == 0)
return NULL;
char* dup = str_dup(path);
if (!dup)
return NULL;
size_t out_len = 0;
char* out = malloc(strlen(path) + 1);
if (!out) {
free(dup);
return NULL;
}
char* saveptr = NULL;
bool ok = true;
for (char* part = strtok_r(dup, "/", &saveptr); part; part = strtok_r(NULL, "/", &saveptr)) {
if (strcmp(part, "..") == 0) {
ok = false;
break;
}
if (strcmp(part, ".") == 0)
continue;
if (out_len > 0)
out[out_len++] = '/';
size_t len = strlen(part);
memcpy(out + out_len, part, len);
out_len += len;
}
free(dup);
if (!ok || out_len == 0) {
free(out);
return NULL;
}
out[out_len] = '\0';
return out;
}
bool config_basis_path_valid(const char* path) {
char* normalized = basis_path_normalize(path);
if (!normalized)
return false;
free(normalized);
return true;
}
int config_basis_append(Config* config, BasisDestType type, const char* path) {
if (!config ||
(type != BASIS_DEST_COMPARE && type != BASIS_DEST_COPY && type != BASIS_DEST_LINK) ||
config->basis_count >= MAX_BASIS_DIRS)
return -1;
char* normalized = basis_path_normalize(path);
if (!normalized)
return -1;
BasisDest* grown = realloc(config->basis_dirs, (config->basis_count + 1) * sizeof(BasisDest));
if (!grown) {
free(normalized);
return -1;
}
config->basis_dirs = grown;
config->basis_dirs[config->basis_count].type = type;
config->basis_dirs[config->basis_count].path = normalized;
config->basis_count++;
return 0;
}
bool config_is_remote_dest(const char* s) {
if (s == NULL)
return false;
@@ -199,13 +347,19 @@ void config_delete(Config* config) {
free(config->backup_dir);
free(config->server_host);
free(config->out_format);
free(config->log_file_format);
free(config->files_from);
file_list_destroy((FileListSet*)config->files_from_set);
free(config->rsh_command);
free(config->rsync_path);
free(config->temp_dir);
free(config->compare_dest);
free(config->copy_dest);
free(config->link_dest);
for (int i = 0; i < config->basis_count; i++) {
free(config->basis_dirs[i].path);
config->basis_dirs[i].path = NULL;
}
free(config->basis_dirs);
config->basis_dirs = NULL;
config->basis_count = 0;
free(config->partial_dir);
free(config->suffix);
free(config->address);
@@ -213,9 +367,20 @@ void config_delete(Config* config) {
free(config->daemon_config);
free(config->compress_choice);
free(config->chmod_spec);
if (config->skip_compress_suffixes) {
for (int i = 0; i < config->skip_compress_count; i++)
free(config->skip_compress_suffixes[i]);
free(config->skip_compress_suffixes);
}
if (config->filters) {
array_list_delete(config->filters);
}
/* A --delay-updates staging tree is transient receiver state: remove any
leftovers on every exit path (success already emptied it). */
if (config->delay_context)
delay_updates_cleanup(config->delay_context);
delay_updates_context_destroy(config->delay_context);
config->delay_context = NULL;
free(config);
}
@@ -223,54 +388,109 @@ void config_delete(Config* config) {
* helper call order in config_send and config_receive unchanged when adding
* fields. */
static bool send_core_fields(int fd, const Config* c) {
return send_str(fd, c->version) && send_str(fd, c->send_directory) &&
send_str(fd, c->receive_root_directory) && send_int(fd, c->save_to_disk) &&
send_int(fd, c->use_multithreading) && send_int(fd, c->use_chunk_serialization) &&
send_int(fd, c->use_compression) && send_int(fd, c->use_metadata) &&
if (!send_str(fd, c->version) || !send_int(fd, c->eight_bit_output))
return false;
protocol_set_8_bit_output(c->eight_bit_output);
if (!send_n_data(fd, &c->max_alloc, sizeof(c->max_alloc)))
return false;
return send_str(fd, c->send_directory) && send_str(fd, c->receive_root_directory) &&
send_int(fd, c->save_to_disk) && send_int(fd, c->use_multithreading) &&
send_int(fd, c->use_chunk_serialization) && send_int(fd, c->use_compression) &&
send_int(fd, c->use_metadata) && send_int(fd, c->use_executability) &&
send_int(fd, c->compression_level) &&
send_n_data(fd, &c->chunk_size, sizeof(c->chunk_size)) && send_int(fd, c->use_sendfile);
}
static bool send_delta_fields(int fd, const Config* c) {
return send_int(fd, c->use_delete) && send_int(fd, c->use_incremental) &&
send_int(fd, c->use_delta) &&
send_int(fd, c->size_only) && send_int(fd, c->ignore_times) &&
send_int(fd, c->use_delta && !c->whole_file) &&
send_n_data(fd, &c->delta_block_size, sizeof(c->delta_block_size)) &&
send_n_data(fd, &c->delta_max_file_size, sizeof(unsigned long long));
}
static bool send_file_options(int fd, const Config* c) {
return send_int(fd, c->backup) && send_str(fd, c->backup_dir ? c->backup_dir : "") &&
send_int(fd, c->follow_symlinks) && send_int(fd, c->copy_links) &&
send_int(fd, c->safe_links) && send_int(fd, c->copy_unsafe_links) &&
send_int(fd, c->preserve_hard_links) && send_int(fd, c->preserve_acls) &&
send_int(fd, c->preserve_xattrs) && send_int(fd, c->preserve_devices) &&
send_int(fd, c->preserve_sparse);
send_int(fd, c->remove_source_files) && send_int(fd, c->follow_symlinks) &&
send_int(fd, c->copy_links) && send_int(fd, c->safe_links) &&
send_int(fd, c->copy_unsafe_links) && send_int(fd, c->preserve_hard_links) &&
send_int(fd, c->preserve_acls) && send_int(fd, c->preserve_xattrs) &&
send_int(fd, c->preserve_devices) && send_int(fd, c->preserve_sparse);
}
static bool send_selection_options(int fd, const Config* c) {
return send_int(fd, c->update) && send_int(fd, c->inplace) && send_int(fd, c->append) &&
send_int(fd, c->append_verify) && send_int(fd, c->delete_excluded) &&
return send_int(fd, c->ignore_existing) && send_int(fd, c->existing) && send_int(fd, c->update) &&
send_int(fd, c->inplace) && send_int(fd, c->delay_updates) && send_int(fd, c->append) &&
send_int(fd, c->use_fsync) && send_int(fd, c->append_verify) &&
send_int(fd, c->delete_excluded) && send_int(fd, c->force_delete) &&
send_int(fd, c->delete_after) && send_n_data(fd, &c->max_delete, sizeof(c->max_delete)) &&
send_int(fd, c->relative) && send_int(fd, c->prune_empty_dirs);
send_int(fd, c->relative) && send_int(fd, c->prune_empty_dirs) &&
send_int(fd, c->mkpath) && send_int(fd, c->delete_during) && send_int(fd, c->delete_delay);
}
static bool send_skip_compress_options(int fd, const Config* c) {
if (!send_int(fd, c->skip_compress_set) || !send_int(fd, c->skip_compress_count))
return false;
for (int i = 0; i < c->skip_compress_count; i++) {
if (!send_str(fd, c->skip_compress_suffixes[i]))
return false;
}
return true;
}
static bool send_resume_options(int fd, const Config* c) {
return send_str(fd, c->temp_dir ? c->temp_dir : "") && send_int(fd, c->partial) &&
send_str(fd, c->partial_dir ? c->partial_dir : "") &&
send_str(fd, c->suffix ? c->suffix : "") && send_int(fd, c->delete_before) &&
send_int(fd, c->checksum) && send_str(fd, c->compress_choice ? c->compress_choice : "") &&
send_str(fd, c->chmod_spec ? c->chmod_spec : "");
send_int(fd, c->checksum) && send_int(fd, c->modify_window) &&
send_str(fd, c->compress_choice ? c->compress_choice : "") &&
send_str(fd, c->chmod_spec ? c->chmod_spec : "") && send_skip_compress_options(fd, c);
}
static bool send_basis_options(int fd, const Config* c) {
if (!send_int(fd, c->basis_count))
return false;
for (int i = 0; i < c->basis_count; i++) {
if (!send_int(fd, (int)c->basis_dirs[i].type) ||
!send_str(fd, c->basis_dirs[i].path ? c->basis_dirs[i].path : ""))
return false;
}
return true;
}
/* -y/--fuzzy (receiver-side similar-file basis selection). Trailing field on
* the config frame; protocol 2.9.0. */
static bool send_fuzzy_option(int fd, const Config* c) {
return send_int(fd, c->fuzzy);
}
/* --checksum-choice/--cc + --checksum-seed. The algorithm id and seed travel
* with the config so the receiver hashes the on-disk old file with the same
* parameters the sender used for its digest (see checksum.h). Trailing fields
* on the config frame; protocol 2.10.0. */
static bool send_checksum_options(int fd, const Config* c) {
return send_int(fd, c->checksum_algo) &&
send_n_data(fd, &c->checksum_seed, sizeof(c->checksum_seed));
}
static bool receive_core_fields(int fd, Config* c) {
int value;
if (!receive_wire_bool(fd, &c->eight_bit_output))
return false;
protocol_set_8_bit_output(c->eight_bit_output);
if (!receive_n_data(fd, &c->max_alloc, sizeof(c->max_alloc)) || c->max_alloc == 0)
return false;
if (c->max_alloc > MAX_SERVER_ALLOC)
c->max_alloc = MAX_SERVER_ALLOC;
protocol_session_set_max_alloc(NULL, c->max_alloc);
c->send_directory = receive_str(fd);
c->receive_root_directory = receive_str(fd);
if (!c->send_directory || !c->receive_root_directory)
return false;
if (!receive_wire_bool(fd, &c->save_to_disk) || !receive_wire_bool(fd, &c->use_multithreading) ||
!receive_wire_bool(fd, &c->use_chunk_serialization) ||
!receive_wire_bool(fd, &c->use_compression) || !receive_wire_bool(fd, &c->use_metadata))
!receive_wire_bool(fd, &c->use_compression) || !receive_wire_bool(fd, &c->use_metadata) ||
!receive_wire_bool(fd, &c->use_executability))
return false;
if (!receive_int(fd, &value))
return false;
@@ -287,6 +507,10 @@ static bool receive_delta_fields(int fd, Config* c) {
return false;
if (!receive_wire_bool(fd, &c->use_incremental))
return false;
if (!receive_wire_bool(fd, &c->size_only))
return false;
if (!receive_wire_bool(fd, &c->ignore_times))
return false;
if (!receive_wire_bool(fd, &c->use_delta))
return false;
return receive_n_data(fd, &c->delta_block_size, sizeof(c->delta_block_size)) &&
@@ -296,8 +520,17 @@ static bool receive_delta_fields(int fd, Config* c) {
static bool receive_file_options(int fd, Config* c) {
if (!receive_wire_bool(fd, &c->backup))
return false;
c->backup_dir = receive_str(fd);
if (!c->backup_dir)
char* backup_dir = receive_str(fd);
if (!backup_dir)
return false;
if (*backup_dir != '\0') {
c->backup_dir = backup_dir;
} else {
/* The sender serializes an unset (NULL) string as "", so canonicalize the
empty wire value back to NULL to preserve NULL-vs-empty semantics. */
free(backup_dir);
}
if (!receive_wire_bool(fd, &c->remove_source_files))
return false;
bool* flags[] = {&c->follow_symlinks, &c->copy_links, &c->safe_links,
&c->copy_unsafe_links, &c->preserve_hard_links, &c->preserve_acls,
@@ -310,8 +543,9 @@ static bool receive_file_options(int fd, Config* c) {
}
static bool receive_selection_options(int fd, Config* c) {
bool* flags[] = {&c->update, &c->inplace, &c->append,
&c->append_verify, &c->delete_excluded, &c->delete_after};
bool* flags[] = {&c->ignore_existing, &c->existing, &c->update, &c->inplace,
&c->delay_updates, &c->append, &c->use_fsync, &c->append_verify,
&c->delete_excluded, &c->force_delete, &c->delete_after};
for (size_t i = 0; i < sizeof(flags) / sizeof(flags[0]); i++) {
if (!receive_wire_bool(fd, flags[i]))
return false;
@@ -322,31 +556,114 @@ static bool receive_selection_options(int fd, Config* c) {
return false;
if (!receive_wire_bool(fd, &c->prune_empty_dirs))
return false;
return true;
if (!receive_wire_bool(fd, &c->mkpath))
return false;
if (!receive_wire_bool(fd, &c->delete_during))
return false;
return receive_wire_bool(fd, &c->delete_delay);
}
static bool receive_resume_options(int fd, Config* c) {
c->temp_dir = receive_str(fd);
if (!c->temp_dir || !receive_wire_bool(fd, &c->partial))
char* temp_dir = receive_str(fd);
if (!temp_dir)
return false;
c->partial_dir = receive_str(fd);
c->suffix = c->partial_dir ? receive_str(fd) : NULL;
if (!c->partial_dir || !c->suffix || !receive_wire_bool(fd, &c->delete_before))
if (*temp_dir != '\0') {
c->temp_dir = temp_dir;
} else {
free(temp_dir);
}
if (!receive_wire_bool(fd, &c->partial))
return false;
/* These options have NULL client defaults, so the sender transmits an empty
string for "unset". Canonicalize the empty wire value back to NULL so
receivers observe exactly what the client configured (plain --backup, for
example, must not look like --backup-dir ""). */
char* partial_dir = receive_str(fd);
if (!partial_dir)
return false;
if (*partial_dir != '\0') {
c->partial_dir = partial_dir;
} else {
free(partial_dir);
}
char* suffix = receive_str(fd);
if (!suffix)
return false;
if (*suffix != '\0') {
c->suffix = suffix;
} else {
free(suffix);
}
if (!receive_wire_bool(fd, &c->delete_before))
return false;
if (!receive_wire_bool(fd, &c->checksum))
return false;
if (!receive_n_data(fd, &c->modify_window, sizeof(c->modify_window)))
return false;
c->compress_choice = receive_str(fd);
if (!c->compress_choice)
return false;
c->chmod_spec = receive_str(fd);
return c->chmod_spec != NULL;
if (!c->chmod_spec || !receive_wire_bool(fd, &c->skip_compress_set) ||
!receive_int(fd, &c->skip_compress_count) || c->skip_compress_count < 0 ||
c->skip_compress_count > 10000)
return false;
if (c->skip_compress_count > 0) {
c->skip_compress_suffixes = calloc((size_t)c->skip_compress_count, sizeof(char*));
if (!c->skip_compress_suffixes)
return false;
for (int i = 0; i < c->skip_compress_count; i++) {
c->skip_compress_suffixes[i] = receive_str(fd);
if (!c->skip_compress_suffixes[i])
return false;
}
}
return true;
}
static bool receive_basis_options(int fd, Config* c) {
int count;
if (!receive_int(fd, &count))
return false;
if (count < 0 || count > MAX_BASIS_DIRS)
return false;
for (int i = 0; i < count; i++) {
int type;
if (!receive_int(fd, &type) || type <= BASIS_DEST_NONE || type > BASIS_DEST_LINK)
return false;
char* path = receive_str(fd);
if (!path)
return false;
/* config_basis_append validates and canonicalizes the path; a rejected
path (absolute / traversal / empty) drops the whole connection. */
bool ok = config_basis_append(c, (BasisDestType)type, path) == 0;
free(path);
if (!ok)
return false;
}
return true;
}
static bool receive_fuzzy_option(int fd, Config* c) {
return receive_wire_bool(fd, &c->fuzzy);
}
static bool receive_checksum_options(int fd, Config* c) {
int algo;
if (!receive_int(fd, &algo) || !checksum_algo_valid(algo))
return false;
c->checksum_algo = algo;
return receive_n_data(fd, &c->checksum_seed, sizeof(c->checksum_seed));
}
bool config_send(int file_descriptor, const Config* config) {
protocol_session_set_max_alloc(NULL, config->max_alloc);
if (!send_core_fields(file_descriptor, config) || !send_delta_fields(file_descriptor, config) ||
!send_file_options(file_descriptor, config) ||
!send_selection_options(file_descriptor, config) ||
!send_resume_options(file_descriptor, config))
!send_resume_options(file_descriptor, config) ||
!send_basis_options(file_descriptor, config) || !send_fuzzy_option(file_descriptor, config) ||
!send_checksum_options(file_descriptor, config))
return false;
Status status;
if (!receive_status(file_descriptor, &status))
@@ -367,8 +684,10 @@ Config* config_receive(int file_descriptor) {
if (!config->version)
goto error;
if (strcmp(config->version, PROTOCOL_VERSION) != 0) {
fprintf(stderr, "Protocol version mismatch: client=%s, server=%s\n", config->version,
PROTOCOL_VERSION);
char* escaped_version = output_escape(config->version, false);
fprintf(stderr, "Protocol version mismatch: client=%s, server=%s\n",
escaped_version ? escaped_version : "<allocation failed>", PROTOCOL_VERSION);
free(escaped_version);
send_status(file_descriptor, STATUS_ERROR);
goto error;
}
@@ -376,11 +695,17 @@ Config* config_receive(int file_descriptor) {
!receive_delta_fields(file_descriptor, config) ||
!receive_file_options(file_descriptor, config) ||
!receive_selection_options(file_descriptor, config) ||
!receive_resume_options(file_descriptor, config))
!receive_resume_options(file_descriptor, config) ||
!receive_basis_options(file_descriptor, config) ||
!receive_fuzzy_option(file_descriptor, config) ||
!receive_checksum_options(file_descriptor, config))
goto error;
if (config->compress_choice[0] != '\0' && strcmp(config->compress_choice, "zstd") != 0 &&
strcmp(config->compress_choice, "none") != 0) {
fprintf(stderr, "Unsupported compression choice: %s\n", config->compress_choice);
char* escaped_choice = output_escape(config->compress_choice, config->eight_bit_output);
fprintf(stderr, "Unsupported compression choice: %s\n",
escaped_choice ? escaped_choice : "<allocation failed>");
free(escaped_choice);
send_status(file_descriptor, STATUS_ERROR);
goto error;
}
+143 -8
View File
@@ -2,12 +2,33 @@
#define CONFIG_H
#include "array_list.h"
#include "checksum.h"
#include <stdbool.h>
#include <stdint.h>
#include <stdio.h>
typedef enum { TRANSPORT_TCP, TRANSPORT_SSH } TransportType;
/* Receiver-side staging state for --delay-updates. Forward-declared here so
Config can carry it; the concrete type lives in delay_updates.h. */
typedef struct DelayUpdatesContext DelayUpdatesContext;
/* Alternate basis-directory modes (--compare-dest / --copy-dest /
* --link-dest). Each flag adds one entry to the ordered Config->basis_dirs
* list; the receiver consults entries in command-line order and stops at the
* first exact match, mirroring rsync's basis-dir priority rules. */
typedef enum {
BASIS_DEST_NONE = 0,
BASIS_DEST_COMPARE, /* compare only: never copies, never materializes */
BASIS_DEST_COPY, /* local copy of the matched basis file */
BASIS_DEST_LINK /* hard link to the matched basis file */
} BasisDestType;
typedef struct BasisDest {
BasisDestType type;
char* path; /* relative to the destination root (receiver-confined) */
} BasisDest;
typedef struct Config {
char* version;
char* send_directory;
@@ -18,10 +39,14 @@ typedef struct Config {
bool use_compression;
bool use_sendfile;
bool use_metadata;
bool use_executability;
bool metadata_explicitly_disabled;
bool show_progress;
bool dry_run;
bool remove_source_files;
bool use_delete;
int compression_level;
int compression_threads;
unsigned long long chunk_size;
int ssh_port;
TransportType transport;
@@ -33,8 +58,21 @@ typedef struct Config {
int include_count;
unsigned long long max_size;
unsigned long long min_size;
unsigned long long max_alloc;
bool use_incremental;
bool ignore_times;
bool size_only;
bool use_delta;
bool whole_file;
/* -y/--fuzzy: when a file must be transferred and the destination holds no
* usable file at the exact path, the receiver may reuse a SIMILAR-named
* existing regular file in the same destination directory as the delta
* basis so the sender transmits only the differences. Crosses the wire
* (the receiver performs the candidate search); the CLI implies
* --incremental + --delta because the similar-basis only matters on the
* receiver-driven delta path. Off by default. */
bool fuzzy;
int modify_window;
uint32_t delta_block_size;
unsigned long long delta_max_file_size;
bool use_tls;
@@ -70,36 +108,80 @@ typedef struct Config {
// Issue #122: Output/logging options
bool itemize_changes;
char* out_format;
char* log_file_format;
int info_level;
int debug_level;
bool list_only;
bool human_readable;
bool eight_bit_output;
// Issue #127: Transfer modes
bool existing;
bool ignore_existing;
bool update;
bool inplace;
bool delay_updates;
bool use_fsync;
bool append;
bool append_verify;
// Issue #128: Extended delete options
/* --delete-excluded: also delete destination entries that were excluded on
* the source. Default (off) matches rsync: excluded paths are protected from
* deletion. Crosses the wire (the sender encodes the choice by whether it
* transmits a protected-prefix list with the keep-set manifest). */
bool delete_excluded;
bool delete_after;
/* --max-delete=NUM: the receiver refuses to delete more than NUM entries per
* run (all-or-nothing: when the extras would exceed NUM nothing is removed and
* the transfer fails with a distinct error). -1 == no client limit (the
* server hard bound MAX_SERVER_DELETE_COUNT still applies). */
int max_delete;
/* --ignore-errors (client-only, never serialized): a sender-side source I/O
* error (an unreadable directory during the scan) normally aborts the run so
* no deletion happens; with --ignore-errors the scan continues and the
* (partial) keep-set is still transmitted so the deletion runs. */
bool ignore_errors;
/* --force (receiver-side): a regular file may replace a destination
* directory by removing that (possibly non-empty, symlink-safe) directory
* tree first, instead of failing the write. Crosses the wire. */
bool force_delete;
// Issue #129: Advanced file selection
ArrayList* filters;
char* files_from;
bool cvs_exclude;
// Issue #129: Advanced file selection. These fields are CLIENT-ONLY: they are
// never serialized to the wire (the receiver must not learn them).
ArrayList* filters; /* --filter=RULE rule strings, in order */
char* files_from; /* --files-from path (may be NULL) */
void* files_from_set; /* parsed FileListSet* allow-set, or NULL */
bool from0; /* -0/--from0: NUL-delimited *-from files */
bool cvs_exclude; /* -C/--cvs-exclude: standard CVS ignore set */
bool per_dir_filter; /* -F: apply per-directory .rsync-filter files */
bool prune_empty_dirs;
bool one_file_system; /* -x/--one-file-system: do not cross filesystem boundaries */
/* -R/--relative: crosses the wire; with --files-from listed entries keep
* their bare relative destination path (no source-root mirror prefix). */
bool relative;
/* --no-implied-dirs: client-only. With -R + --files-from, refuse to place a
* listed file whose ancestor directory is not itself explicitly listed. */
bool no_implied_dirs;
/* -d/--dirs: client-only. Transfer the directory entries named by the
* source argument / --files-from list without recursing into contents. */
bool dirs;
/* --mkpath: crosses the wire. Tells the server to create the destination
* root directory (and missing leading components below its authorized root)
* at connection start instead of requiring it to already exist. */
bool mkpath;
// Issue #130: Remote shell/connection options
char* rsh_command;
char* rsync_path;
bool old_args;
char* temp_dir;
char* compare_dest;
char* copy_dest;
char* link_dest;
/* Alternate basis directories, ordered by command-line appearance. Each
* entry's type selects compare/copy/link behavior on an exact match. These
* cross the wire so the receiver can consult them; they are interpreted
* relative to the destination root and confined there. */
BasisDest* basis_dirs;
int basis_count;
// PR #174: Partial transfer resumption
char* partial_dir;
@@ -110,6 +192,19 @@ typedef struct Config {
// PR #179: Delete policies
bool delete_before;
/* rsync deletion-timing family (real from Phase 3). At most one of
delete_before / delete_during / delete_delay / delete_after may be set, and
only together with use_delete (the CLI implies --delete for each of them).
delete_before and delete_during select the EARLY engine mode: the keep-set
manifest is transmitted before any file data and extras are removed then,
acknowledged, before the first data byte. delete_delay and delete_after
select the LATE commit mode: extras are removed only after the whole
transfer has succeeded (plain --delete keeps this mode). The exact
semantics and the divergences from rsync are documented in RSYNC_COMPAT.md
and in config_delete_timing_early() below. */
bool delete_during;
bool delete_delay;
// PR #181: IPv6 and bind address
char* address;
char* bind_address;
@@ -127,10 +222,32 @@ typedef struct Config {
// PR #184: Compression algorithm negotiation
char* compress_choice;
char* chmod_spec;
/* --checksum-choice / --cc and --checksum-seed. checksum_algo is the id of
* the whole-file content-digest algorithm used by the per-file --incremental
* handshake (sender computes it, receiver compares it to skip unchanged
* files) and by the basis-dir content verification. checksum_seed is passed
* to xxHash64 (and to the delta block strong hash, low 32 bits); md5 has no
* seed so it is ignored there. Both cross the wire: the receiver MUST hash
* the on-disk old file with the same algorithm and seed to reach a matching
* digest. Defaults (XXH64 / seed 0) reproduce the pre-existing behavior
* byte-for-byte. */
int checksum_algo; /* ChecksumAlgo, default CHECKSUM_ALGO_XXH64 */
uint64_t checksum_seed; /* default 0 */
char** skip_compress_suffixes;
int skip_compress_count;
bool skip_compress_set;
// Receiver-side runtime staging registry for --delay-updates. Never sent
// over the wire and never set on the sender side.
DelayUpdatesContext* delay_context;
} Config;
#define PROTOCOL_VERSION "2.3.0"
#define PROTOCOL_VERSION "2.10.0"
#define DEFAULT_CHUNK_SIZE (10 * 1024 * 1024)
/* Upper bound on total basis-dir entries (rsync caps --link-dest at 20). */
#define MAX_BASIS_DIRS 64
Config* config_create(void);
void config_delete(Config* config);
@@ -139,4 +256,22 @@ Config* config_receive(int file_descriptor);
bool config_is_remote_dest(const char* s);
void config_parse_ssh_dest(Config* config);
/* True when the negotiated delete timing performs the extra-file deletion
* BEFORE the transfer data (--delete-before / --delete-during). The flag is
* a pure function of the config and is used identically on the sender (to pick
* the manifest-first frame order) and the receiver (to delete when the early
* manifest arrives). When false the deletion is committed only after the whole
* transfer succeeded (--delete / --delete-after / --delete-delay). */
bool config_delete_timing_early(const Config* config);
/* Delete-timing sanity: with deletion enabled at most one timing flag may be
* set (none = the default delete-after commit timing); without deletion no
* timing flag may be set (each timing flag implies --delete). */
bool config_has_valid_delete_timing(const Config* config);
/* True when at least one --compare-dest/--copy-dest/--link-dest was set. */
bool config_has_basis(const Config* config);
/* Append one basis-dir entry. Returns 0 on success, -1 on allocation failure. */
int config_basis_append(Config* config, BasisDestType type, const char* path);
/* Validate a client-provided basis-dir path (relative, confined, non-empty). */
bool config_basis_path_valid(const char* path);
#endif
+4 -3
View File
@@ -1,11 +1,12 @@
#include "data.h"
#include "log.h"
#include "protocol.h"
#include <stdlib.h>
Data* data_create_empty(size_t data_size) {
/* malloc(0) is UB; allocate at least 1 byte but preserve requested size */
size_t alloc_size = data_size > 0 ? data_size : 1;
void* data = malloc(alloc_size);
void* data = protocol_alloc(alloc_size);
if (data == NULL) {
log_message(LOG_LEVEL_ERROR, "Could not allocate memory for empty data");
return NULL;
@@ -14,7 +15,7 @@ Data* data_create_empty(size_t data_size) {
}
Data* data_create_reserve(size_t size) {
Data* d = malloc(sizeof(Data));
Data* d = protocol_alloc(sizeof(Data));
if (d == NULL) {
log_message(LOG_LEVEL_ERROR, "Could not allocate memory for data");
return NULL;
@@ -26,7 +27,7 @@ Data* data_create_reserve(size_t size) {
}
Data* data_create(void* data, size_t data_size) {
Data* new_data = malloc(sizeof(Data));
Data* new_data = protocol_alloc(sizeof(Data));
if (new_data == NULL) {
log_message(LOG_LEVEL_ERROR, "Could not allocate memory for data");
free(data);
+338
View File
@@ -0,0 +1,338 @@
#include "delay_updates.h"
#include "config.h"
#include "file.h"
#include "log.h"
#include "utils.h"
#include <dirent.h>
#include <errno.h>
#include <fcntl.h>
#include <libgen.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/file.h>
#include <sys/stat.h>
#include <unistd.h>
DelayUpdatesContext* delay_updates_context_create(const char* root_directory) {
if (!root_directory)
return NULL;
DelayUpdatesContext* context = calloc(1, sizeof(DelayUpdatesContext));
if (!context)
return NULL;
context->root_directory = str_dup(root_directory);
if (!context->root_directory) {
free(context);
return NULL;
}
context->staging_root = path_cat(root_directory, DELAY_UPDATES_STAGING_DIR);
if (!context->staging_root) {
free(context->root_directory);
free(context);
return NULL;
}
context->entries = NULL;
context->count = 0;
context->capacity = 0;
context->prepared = false;
context->lock_fd = -1;
if (mtx_init(&context->mutex, mtx_plain) != thrd_success) {
free(context->staging_root);
free(context->root_directory);
free(context);
return NULL;
}
return context;
}
void delay_updates_context_destroy(DelayUpdatesContext* context) {
if (!context)
return;
mtx_destroy(&context->mutex);
if (context->lock_fd >= 0)
close(context->lock_fd);
context->lock_fd = -1;
free(context->staging_root);
free(context->root_directory);
for (size_t i = 0; i < context->count; i++) {
free(context->entries[i].staged_path);
free(context->entries[i].final_path);
free(context->entries[i].file_path);
}
free(context->entries);
free(context);
}
bool delay_updates_staging_name_conflict(const char* dir) {
if (!dir || !*dir)
return false;
size_t length = strlen(dir);
while (length > 0 && dir[length - 1] == '/')
length--;
size_t reserved_length = strlen(DELAY_UPDATES_STAGING_DIR);
if (length != reserved_length)
return false;
return strncmp(dir, DELAY_UPDATES_STAGING_DIR, length) == 0;
}
/* Recursively delete every entry inside an open directory (never following
symlinks). The directory itself is left in place. Mirrors the fd-relative
walk used by the delete code so a symlink planted inside the staging tree
can never redirect removal outside of it. */
static bool delay_wipe_dir_fd(int dirfd) {
int scanfd = dup(dirfd);
if (scanfd < 0)
return false;
DIR* dir = fdopendir(scanfd);
if (!dir) {
close(scanfd);
return false;
}
bool operation_ok = true;
const struct dirent* entry;
while ((entry = readdir(dir)) != NULL) {
if (strcmp(entry->d_name, ".") == 0 || strcmp(entry->d_name, "..") == 0)
continue;
struct stat st;
if (fstatat(dirfd, entry->d_name, &st, AT_SYMLINK_NOFOLLOW) != 0) {
if (errno != ENOENT)
operation_ok = false;
continue;
}
if (S_ISDIR(st.st_mode)) {
int childfd = openat(dirfd, entry->d_name, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
bool child_removed = false;
if (childfd >= 0) {
child_removed = delay_wipe_dir_fd(childfd);
close(childfd);
} else if (errno != ENOENT) {
operation_ok = false;
}
if (child_removed && unlinkat(dirfd, entry->d_name, AT_REMOVEDIR) != 0 && errno != ENOENT)
operation_ok = false;
} else {
if (unlinkat(dirfd, entry->d_name, 0) != 0 && errno != ENOENT)
operation_ok = false;
}
}
closedir(dir);
return operation_ok;
}
bool delay_updates_prepare(DelayUpdatesContext* context) {
if (!context)
return false;
if (context->prepared)
return true;
int fd = file_open_private_dir(context->staging_root);
if (fd < 0) {
int saved_errno = errno;
char* escaped = output_escape(context->staging_root, false);
log_message(LOG_LEVEL_ERROR, "could not create --delay-updates staging directory '%s': %s",
escaped ? escaped : "<allocation failed>", strerror(saved_errno));
free(escaped);
return false;
}
/* Hold an exclusive advisory lock on the staging directory for the whole
transfer. The staging directory name is fixed, so two simultaneous
delayed transfers to the same destination root would otherwise share it
and destroy each other's staged files. The lock makes the second session
fail cleanly instead of corrupting the first. The lock is released when
the context (and its file descriptor) is destroyed. */
if (flock(fd, LOCK_EX | LOCK_NB) != 0) {
int saved_errno = errno;
close(fd);
if (saved_errno == EWOULDBLOCK || saved_errno == EAGAIN) {
char* escaped = output_escape(context->staging_root, false);
log_message(LOG_LEVEL_ERROR,
"another --delay-updates transfer to '%s' is already in progress; refusing to "
"share the staging directory",
escaped ? escaped : "<allocation failed>");
free(escaped);
} else {
log_message(LOG_LEVEL_ERROR, "could not lock --delay-updates staging directory '%s': %s",
context->staging_root, strerror(saved_errno));
}
return false;
}
context->lock_fd = fd;
/* Only now, with exclusive ownership, wipe leftovers from an interrupted
earlier transfer; this can never race with a live session. */
bool ok = delay_wipe_dir_fd(fd);
if (!ok) {
log_message(LOG_LEVEL_ERROR, "could not clear stale --delay-updates staging files under '%s'",
context->staging_root);
close(context->lock_fd);
context->lock_fd = -1;
return false;
}
context->prepared = true;
return true;
}
bool delay_updates_record(DelayUpdatesContext* context, const char* staged_path,
const char* final_path, const char* file_path) {
if (!context || !staged_path || !final_path || !file_path)
return false;
char* staged_copy = str_dup(staged_path);
char* final_copy = str_dup(final_path);
char* file_copy = str_dup(file_path);
if (!staged_copy || !final_copy || !file_copy) {
free(staged_copy);
free(final_copy);
free(file_copy);
return false;
}
mtx_lock(&context->mutex);
bool ok = true;
if (context->count == context->capacity) {
size_t new_capacity = context->capacity == 0 ? 64 : context->capacity * 2;
if (new_capacity < context->capacity) {
ok = false;
} else {
StagedFileEntry* grown = realloc(context->entries, new_capacity * sizeof(StagedFileEntry));
if (!grown) {
ok = false;
} else {
context->entries = grown;
context->capacity = new_capacity;
}
}
}
if (ok) {
context->entries[context->count].staged_path = staged_copy;
context->entries[context->count].final_path = final_copy;
context->entries[context->count].file_path = file_copy;
context->count++;
}
mtx_unlock(&context->mutex);
if (!ok) {
free(staged_copy);
free(final_copy);
free(file_copy);
}
return ok;
}
/* Move an existing final destination file aside before the staged replacement
is installed. Deferred from stage time so the final destination is not
modified until publication. Mirrors the immediate-mode backup logic. */
static bool delay_publish_backup(const DelayUpdatesContext* context, const Config* config,
const StagedFileEntry* entry) {
bool backup_enabled = config && config->backup && !config->ignore_existing;
if (!backup_enabled)
return true;
const char* backup_suffix = (config && config->suffix) ? config->suffix : "~";
struct stat backup_stat;
if (!file_stat_secure(entry->final_path, &backup_stat))
return true; /* nothing to back up */
char* backup_path = NULL;
if (config->backup_dir) {
char* confined_backup = path_cat(context->root_directory, config->backup_dir);
if (!confined_backup)
return false;
backup_path = path_cat(confined_backup, entry->file_path);
free(confined_backup);
} else {
size_t path_len = strlen(entry->final_path);
size_t suffix_len = strlen(backup_suffix);
if (path_len > SIZE_MAX - suffix_len - 1)
return false;
backup_path = malloc(path_len + suffix_len + 1);
if (backup_path) {
memcpy(backup_path, entry->final_path, path_len);
memcpy(backup_path + path_len, backup_suffix, suffix_len + 1);
}
}
if (!backup_path)
return false;
char* parent_copy = str_dup(backup_path);
if (!parent_copy || !file_ensure_directory_secure(dirname(parent_copy))) {
free(parent_copy);
free(backup_path);
return false;
}
free(parent_copy);
bool ok = file_rename_secure(entry->final_path, backup_path);
free(backup_path);
return ok;
}
static bool delay_publish_entry(DelayUpdatesContext* context, const Config* config,
const StagedFileEntry* entry) {
if (!delay_publish_backup(context, config, entry))
return false;
if (!file_rename_secure(entry->staged_path, entry->final_path)) {
if (errno == EXDEV) {
char* escaped = output_escape(entry->final_path, false);
log_message(LOG_LEVEL_ERROR,
"staging directory is on a different filesystem than the destination; cannot "
"atomically install file (EXDEV): %s",
escaped ? escaped : "<allocation failed>");
free(escaped);
} else {
char* escaped = output_escape(entry->final_path, false);
log_message(LOG_LEVEL_ERROR, "could not install staged file '%s': %s",
escaped ? escaped : "<allocation failed>", strerror(errno));
free(escaped);
}
return false;
}
return true;
}
/* Remove the staging tree (contents plus the directory itself). Returns true
when nothing is left behind (including the case where it never existed). */
static bool delay_updates_remove_staging_tree(DelayUpdatesContext* context) {
int fd = open(context->staging_root, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
if (fd < 0)
return errno == ENOENT;
bool ok = delay_wipe_dir_fd(fd);
if (close(fd) != 0)
ok = false;
if (ok && rmdir(context->staging_root) != 0 && errno != ENOENT)
ok = false;
return ok;
}
bool delay_updates_publish(DelayUpdatesContext* context, const Config* config) {
if (!context)
return false;
mtx_lock(&context->mutex);
bool ok = true;
for (size_t i = 0; i < context->count; i++) {
if (!delay_publish_entry(context, config, &context->entries[i])) {
ok = false;
break;
}
}
mtx_unlock(&context->mutex);
/* Renaming files out of the staging tree leaves the mirrored directories
behind, and a mid-publish failure leaves the remaining staged files.
Remove whatever is left so a later run starts from a clean staging area
and no staged content can linger after a failed publish. If that cleanup
fails, tell the operator: a stale staging directory would otherwise
silently accumulate and make the next transfer's prepare-wipe fail. */
if (!delay_updates_remove_staging_tree(context)) {
log_message(LOG_LEVEL_WARNING,
"could not fully remove --delay-updates staging directory '%s' after publish; a "
"later --delay-updates transfer to this destination will try to clear it",
context->staging_root);
}
return ok;
}
void delay_updates_cleanup(DelayUpdatesContext* context) {
if (!context)
return;
/* Only a context that gained exclusive ownership may touch the shared
staging directory. If prepare never succeeded (e.g. lock contention with
another live session) the directory belongs to that other session and must
be left alone. */
if (!context->prepared)
return;
delay_updates_remove_staging_tree(context);
}
+68
View File
@@ -0,0 +1,68 @@
#ifndef DELAY_UPDATES_H
#define DELAY_UPDATES_H
#include <stdbool.h>
#include <stddef.h>
#include <threads.h>
/* Forward-declared in config.h; full type needed by file_save_to_disk. */
typedef struct Config Config;
/* One staged file awaiting publication. */
typedef struct {
char* staged_path; /* full path inside the staging tree */
char* final_path; /* full final destination path */
char* file_path; /* the file path as received on the wire */
} StagedFileEntry;
/* Receiver-side --delay-updates staging registry. All successfully written
files land under a private staging directory inside the receive root and are
atomically renamed into their final destination only at the very end of the
transfer. A single PipelineContextReceiver has exactly one writer thread,
but the registry is still mutex-protected so the same object can be safely
shared with the publish/cleanup phase that runs after the threads join. */
typedef struct DelayUpdatesContext {
char* root_directory; /* receive root the staging dir lives under */
char* staging_root; /* root_directory/<staging dir name> */
mtx_t mutex;
StagedFileEntry* entries;
size_t count;
size_t capacity;
bool prepared; /* staging dir created, wiped, and exclusively locked */
int lock_fd; /* advisory exclusive flock held on the staging dir, or -1 */
} DelayUpdatesContext;
/* Name of the private staging subdirectory created under the receive root. */
#define DELAY_UPDATES_STAGING_DIR ".fastsync-stage"
/* True when `dir` (ignoring a trailing "/") is the reserved staging directory
name. Used to reject a --backup-dir that would collide with the internal
staging area. */
bool delay_updates_staging_name_conflict(const char* dir);
/* Create an empty staging context rooted below root_directory. Does not touch
the filesystem yet. */
DelayUpdatesContext* delay_updates_context_create(const char* root_directory);
void delay_updates_context_destroy(DelayUpdatesContext* context);
/* Create the private 0700 staging directory (on first call) and wipe any
leftovers from a previously interrupted delayed transfer. Idempotent. */
bool delay_updates_prepare(DelayUpdatesContext* context);
/* Record a fully-written staged file for later publication. Copies all three
paths. Returns false on allocation failure. */
bool delay_updates_record(DelayUpdatesContext* context, const char* staged_path,
const char* final_path, const char* file_path);
/* Atomically rename every staged file into its final destination. Deferred
--backup handling runs immediately before each rename. On any failure the
remaining staged files are removed (best effort); already-published files
are not rolled back. Afterwards the staging tree is removed so a successful
or failed publish leaves no staging leftovers. */
bool delay_updates_publish(DelayUpdatesContext* context, const Config* config);
/* Best-effort removal of every staged file and the staging directory itself.
Safe to call when nothing was staged or after a successful publish. */
void delay_updates_cleanup(DelayUpdatesContext* context);
#endif
+191 -43
View File
@@ -1,5 +1,6 @@
#include "delta.h"
#include "log.h"
#include "protocol.h"
#include <stdint.h>
#include <limits.h>
#include <stdlib.h>
@@ -28,12 +29,21 @@ uint32_t delta_xxhash32(const void* data, uint32_t len) {
return XXH32(data, len, 0);
}
uint32_t delta_xxhash32_seeded(const void* data, uint32_t len, uint32_t seed) {
return XXH32(data, len, seed);
}
uint64_t delta_xxhash64(const void* data, size_t len) {
return XXH64(data, len, 0);
}
DeltaSignature* delta_signature_create(const void* old_file_data, uint64_t old_file_size,
uint32_t block_size) {
return delta_signature_create_seeded(old_file_data, old_file_size, block_size, 0);
}
DeltaSignature* delta_signature_create_seeded(const void* old_file_data, uint64_t old_file_size,
uint32_t block_size, uint32_t seed) {
if (old_file_data == NULL || old_file_size == 0 || block_size == 0)
return NULL;
@@ -43,7 +53,7 @@ DeltaSignature* delta_signature_create(const void* old_file_data, uint64_t old_f
uint32_t block_count = (uint32_t)((old_file_size + block_size - 1) / block_size);
DeltaSignature* sig = malloc(sizeof(DeltaSignature));
DeltaSignature* sig = protocol_alloc(sizeof(DeltaSignature));
if (!sig)
return NULL;
@@ -54,7 +64,7 @@ DeltaSignature* delta_signature_create(const void* old_file_data, uint64_t old_f
free(sig);
return NULL;
}
sig->blocks = malloc((size_t)block_count * sizeof(DeltaBlockSig));
sig->blocks = protocol_alloc((size_t)block_count * sizeof(DeltaBlockSig));
if (!sig->blocks) {
free(sig);
return NULL;
@@ -66,7 +76,7 @@ DeltaSignature* delta_signature_create(const void* old_file_data, uint64_t old_f
uint32_t len =
(uint32_t)((old_file_size - offset < block_size) ? (old_file_size - offset) : block_size);
sig->blocks[i].adler32 = delta_adler32(data + offset, len);
sig->blocks[i].xxhash = delta_xxhash32(data + offset, len);
sig->blocks[i].xxhash = delta_xxhash32_seeded(data + offset, len, seed);
}
return sig;
@@ -82,7 +92,7 @@ Data* delta_signature_serialize(const DeltaSignature* sig) {
total > SIZE_MAX)
return NULL;
uint8_t* buf = malloc((size_t)total);
uint8_t* buf = protocol_alloc((size_t)total);
if (!buf)
return NULL;
@@ -111,7 +121,7 @@ DeltaSignature* delta_signature_deserialize(const Data* data) {
const uint8_t* buf = (const uint8_t*)data->data;
size_t pos = 0;
DeltaSignature* sig = malloc(sizeof(DeltaSignature));
DeltaSignature* sig = protocol_alloc(sizeof(DeltaSignature));
if (!sig)
return NULL;
@@ -149,7 +159,7 @@ DeltaSignature* delta_signature_deserialize(const Data* data) {
free(sig);
return NULL;
}
sig->blocks = malloc((size_t)blocks_size);
sig->blocks = protocol_alloc((size_t)blocks_size);
if (!sig->blocks) {
free(sig);
return NULL;
@@ -178,7 +188,7 @@ static bool ensure_capacity(DeltaInstruction** instrs, uint32_t* capacity, uint3
if (*capacity > MAX_DELTA_INSTRUCTIONS / 2)
return false;
uint32_t new_cap = *capacity * 2;
DeltaInstruction* tmp = realloc(*instrs, (size_t)new_cap * sizeof(DeltaInstruction));
DeltaInstruction* tmp = protocol_realloc(*instrs, (size_t)new_cap * sizeof(DeltaInstruction));
if (!tmp)
return false;
*instrs = tmp;
@@ -195,7 +205,7 @@ static bool flush_literal(DeltaInstruction** instrs, uint32_t* capacity, uint32_
uint32_t lit_len = (uint32_t)(end - start);
if (!ensure_capacity(instrs, capacity, *count))
return false;
uint8_t* lit_data = malloc(lit_len);
uint8_t* lit_data = protocol_alloc(lit_len);
if (!lit_data)
return false;
memcpy(lit_data, data + start, lit_len);
@@ -215,8 +225,126 @@ static void free_instructions(DeltaInstruction* instrs, uint32_t count) {
free(instrs);
}
/* Sentinel meaning "no signature block" in the lookup index chains. Block
* counts are bounded well below UINT32_MAX, so it doubles as a null link. */
#define DELTA_NO_BLOCK UINT32_MAX
/* Avalanche mix for the rolling checksum so blocks do not cluster in the
* bucket table when the weak checksum has little entropy (e.g. all-zero or
* patterned files). */
static uint32_t delta_adler_mix(uint32_t h) {
h ^= h >> 16;
h *= 0x7feb352dU;
h ^= h >> 15;
h *= 0x846ca68bU;
h ^= h >> 16;
return h;
}
/* Smallest power of two >= v. v must be non-zero. */
static uint32_t delta_next_pow2(uint32_t v) {
v--;
v |= v >> 1;
v |= v >> 2;
v |= v >> 4;
v |= v >> 8;
v |= v >> 16;
return v + 1;
}
/* Build a hash index over sig->blocks keyed by the (mixed) rolling checksum.
* All blocks sharing an Adler-32 value land in the same bucket; collisions
* are chained through a single contiguous allocation:
*
* [0, bucket_count) heads (first block per bucket)
* [bucket_count, 2*bucket_count) tails (last block per bucket)
* [2*bucket_count, ...) per-block chain links
*
* Blocks are inserted in ascending index order so every bucket chain is
* ordered exactly like the historical linear scan. Returns the base pointer
* (also the heads array) or NULL when no index could be allocated; callers
* then fall back to the linear scan. */
static uint32_t* delta_build_index(const DeltaSignature* sig, uint32_t bucket_count) {
if (sig->block_count == 0 || bucket_count == 0)
return NULL;
size_t entries = (size_t)2 * bucket_count + sig->block_count;
if (entries > SIZE_MAX / sizeof(uint32_t))
return NULL;
uint32_t* index = protocol_alloc(entries * sizeof(uint32_t));
if (!index)
return NULL;
uint32_t* heads = index;
uint32_t* tails = index + bucket_count;
uint32_t* next = index + 2 * bucket_count;
uint32_t mask = bucket_count - 1;
memset(heads, 0xFF, (size_t)bucket_count * sizeof(uint32_t));
memset(tails, 0xFF, (size_t)bucket_count * sizeof(uint32_t));
for (uint32_t j = 0; j < sig->block_count; j++) {
uint32_t b = delta_adler_mix(sig->blocks[j].adler32) & mask;
if (heads[b] == DELTA_NO_BLOCK)
heads[b] = j;
else
next[tails[b]] = j;
tails[b] = j;
next[j] = DELTA_NO_BLOCK;
}
return index;
}
/* Locate the signature block matching the byte window at new_data[i].
*
* Mirrors the original per-window behaviour exactly: only a full block_size
* window can match, candidates are accepted only when the weak (Adler-32) and
* strong (xxHash32) checksums both agree, and the lowest block index wins so
* the emitted op stream is byte-identical to the linear scan. When heads is
* non-NULL the candidate set is reached through the bucket index (expected
* O(1) per window); otherwise an exact linear scan is used. */
static uint32_t delta_find_match(const uint8_t* window, uint32_t window_len, uint32_t adler,
bool full_window, const DeltaSignature* sig, const uint32_t* heads,
const uint32_t* next, uint32_t mask, uint32_t seed) {
if (!full_window || sig->block_count == 0)
return DELTA_NO_BLOCK;
if (heads) {
uint32_t b = delta_adler_mix(adler) & mask;
uint32_t window_xxh = 0;
bool have_xxh = false;
for (uint32_t j = heads[b]; j != DELTA_NO_BLOCK; j = next[j]) {
if (sig->blocks[j].adler32 != adler)
continue;
if (!have_xxh) {
window_xxh = delta_xxhash32_seeded(window, window_len, seed);
have_xxh = true;
}
if (window_xxh == sig->blocks[j].xxhash)
return j;
}
return DELTA_NO_BLOCK;
}
/* Fallback used when the index could not be allocated. */
for (uint32_t j = 0; j < sig->block_count; j++) {
if (sig->blocks[j].adler32 == adler) {
uint32_t window_xxh = delta_xxhash32_seeded(window, window_len, seed);
if (window_xxh == sig->blocks[j].xxhash)
return j;
}
}
return DELTA_NO_BLOCK;
}
Delta* delta_compute(const void* new_file_data, uint64_t new_file_size, const DeltaSignature* sig,
uint32_t block_size) {
return delta_compute_seeded(new_file_data, new_file_size, sig, block_size, 0);
}
Delta* delta_compute_seeded(const void* new_file_data, uint64_t new_file_size,
const DeltaSignature* sig, uint32_t block_size, uint32_t seed) {
if (!new_file_data || !sig || !sig->blocks || new_file_size == 0 || block_size == 0 ||
block_size > DELTA_BLOCK_SIZE_MAX || sig->block_size != block_size)
return NULL;
@@ -225,10 +353,29 @@ Delta* delta_compute(const void* new_file_data, uint64_t new_file_size, const De
uint32_t capacity = 64;
uint32_t count = 0;
DeltaInstruction* instrs = malloc((size_t)capacity * sizeof(DeltaInstruction));
DeltaInstruction* instrs = protocol_alloc((size_t)capacity * sizeof(DeltaInstruction));
if (!instrs)
return NULL;
/* Build a one-time bucket index over the signature blocks keyed by the weak
* checksum. This turns the per-byte-window candidate lookup from an
* O(block_count) linear scan into an expected O(1) probe, which dominates
* the cost for large mostly-matching files (the diff steps one byte at a
* time through changed regions). On allocation failure the probe falls back
* to the original linear scan, so behaviour is unchanged under memory
* pressure. */
uint32_t* index = NULL;
const uint32_t* chain_next = NULL;
uint32_t mask = 0;
if (sig->block_count > 0) {
uint32_t bucket_count = delta_next_pow2(sig->block_count);
index = delta_build_index(sig, bucket_count);
if (index) {
chain_next = index + 2 * bucket_count;
mask = bucket_count - 1;
}
}
uint64_t literal_start = 0;
bool has_literal = false;
@@ -263,34 +410,32 @@ Delta* delta_compute(const void* new_file_data, uint64_t new_file_size, const De
}
bool matched = false;
for (uint32_t j = 0; j < sig->block_count; j++) {
if (adler == sig->blocks[j].adler32 && full_window) {
uint32_t xxh = delta_xxhash32(new_data + i, window_len);
if (xxh == sig->blocks[j].xxhash) {
if (has_literal) {
if (!flush_literal(&instrs, &capacity, &count, new_data, literal_start, i)) {
free_instructions(instrs, count);
return NULL;
}
has_literal = false;
}
if (!ensure_capacity(&instrs, &capacity, count)) {
free_instructions(instrs, count);
return NULL;
}
instrs[count].type = DELTA_INSTR_BLOCK_MATCH;
instrs[count].match.block_index = j;
instrs[count].match.block_offset = 0;
instrs[count].match.length = window_len;
count++;
i += window_len;
rolling_valid = false;
matched = true;
break;
uint32_t match_block = delta_find_match(new_data + i, window_len, adler, full_window, sig,
index, chain_next, mask, seed);
if (match_block != DELTA_NO_BLOCK) {
if (has_literal) {
if (!flush_literal(&instrs, &capacity, &count, new_data, literal_start, i)) {
free_instructions(instrs, count);
free(index);
return NULL;
}
has_literal = false;
}
if (!ensure_capacity(&instrs, &capacity, count)) {
free_instructions(instrs, count);
free(index);
return NULL;
}
instrs[count].type = DELTA_INSTR_BLOCK_MATCH;
instrs[count].match.block_index = match_block;
instrs[count].match.block_offset = 0;
instrs[count].match.length = window_len;
count++;
i += window_len;
rolling_valid = false;
matched = true;
}
if (!matched) {
@@ -302,6 +447,8 @@ Delta* delta_compute(const void* new_file_data, uint64_t new_file_size, const De
}
}
free(index);
if (has_literal) {
if (!flush_literal(&instrs, &capacity, &count, new_data, literal_start, new_file_size)) {
free_instructions(instrs, count);
@@ -309,7 +456,7 @@ Delta* delta_compute(const void* new_file_data, uint64_t new_file_size, const De
}
}
Delta* delta = malloc(sizeof(Delta));
Delta* delta = protocol_alloc(sizeof(Delta));
if (!delta) {
free_instructions(instrs, count);
return NULL;
@@ -355,7 +502,7 @@ Data* delta_serialize(const Delta* delta) {
if (delta->delta_size > UINT64_MAX - header_size || header_size + delta->delta_size > SIZE_MAX)
return NULL;
uint64_t total = header_size + delta->delta_size;
uint8_t* buf = malloc((size_t)total);
uint8_t* buf = protocol_alloc((size_t)total);
if (!buf)
return NULL;
@@ -395,7 +542,7 @@ Delta* delta_deserialize(const Data* data) {
const uint8_t* buf = (const uint8_t*)data->data;
size_t pos = 0;
Delta* delta = malloc(sizeof(Delta));
Delta* delta = protocol_alloc(sizeof(Delta));
if (!delta)
return NULL;
@@ -412,9 +559,10 @@ Delta* delta_deserialize(const Data* data) {
return NULL;
}
delta->instructions = delta->instruction_count == 0
? NULL
: malloc((size_t)delta->instruction_count * sizeof(DeltaInstruction));
delta->instructions =
delta->instruction_count == 0
? NULL
: protocol_alloc((size_t)delta->instruction_count * sizeof(DeltaInstruction));
if (delta->instruction_count > 0 && !delta->instructions) {
free(delta);
return NULL;
@@ -465,7 +613,7 @@ Delta* delta_deserialize(const Data* data) {
free(delta);
return NULL;
}
delta->instructions[i].literal.data = malloc(lit_len ? lit_len : 1);
delta->instructions[i].literal.data = protocol_alloc(lit_len ? lit_len : 1);
if (!delta->instructions[i].literal.data) {
log_message(LOG_LEVEL_ERROR, "Failed to allocate %u bytes for literal data", lit_len);
free_instructions(delta->instructions, i);
@@ -492,7 +640,7 @@ void* delta_apply(const void* old_data, uint64_t old_size, const Delta* delta,
delta->new_file_size > DELTA_MAX_FILE_SIZE || delta->new_file_size > SIZE_MAX)
return NULL;
void* output = malloc(delta->new_file_size ? (size_t)delta->new_file_size : 1);
void* output = protocol_alloc(delta->new_file_size ? (size_t)delta->new_file_size : 1);
if (!output)
return NULL;
+11
View File
@@ -56,12 +56,22 @@ typedef struct {
DeltaSignature* delta_signature_create(const void* old_file_data, uint64_t old_file_size,
uint32_t block_size);
/* Seeded equivalent of delta_signature_create: the per-block strong (xxHash32)
* checksum uses `seed` (the low 32 bits of --checksum-seed). Passing seed 0 is
* identical to the unseeded function. */
DeltaSignature* delta_signature_create_seeded(const void* old_file_data, uint64_t old_file_size,
uint32_t block_size, uint32_t seed);
Data* delta_signature_serialize(const DeltaSignature* sig);
DeltaSignature* delta_signature_deserialize(const Data* data);
void delta_signature_destroy(DeltaSignature* sig);
Delta* delta_compute(const void* new_file_data, uint64_t new_file_size, const DeltaSignature* sig,
uint32_t block_size);
/* Seeded equivalent of delta_compute: the per-window strong (xxHash32) check
* uses `seed` (the low 32 bits of --checksum-seed). The receiver's signature
* must have been built with the same seed for matching. */
Delta* delta_compute_seeded(const void* new_file_data, uint64_t new_file_size,
const DeltaSignature* sig, uint32_t block_size, uint32_t seed);
Data* delta_serialize(const Delta* delta);
Delta* delta_deserialize(const Data* data);
void* delta_apply(const void* old_data, uint64_t old_size, const Delta* delta, uint32_t block_size);
@@ -72,6 +82,7 @@ bool delta_is_worthwhile(const Delta* delta, uint64_t new_file_size);
uint32_t delta_adler32(const void* data, uint32_t len);
uint32_t delta_xxhash32(const void* data, uint32_t len);
uint32_t delta_xxhash32_seeded(const void* data, uint32_t len, uint32_t seed);
uint64_t delta_xxhash64(const void* data, size_t len);
#endif
+493 -33
View File
@@ -1,7 +1,9 @@
#include <errno.h>
#include <dirent.h>
#include <fcntl.h>
#include <libgen.h>
#include <limits.h>
#include <stdatomic.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
@@ -14,6 +16,7 @@
#include "log.h"
#include "metadata.h"
#include "utils.h"
#include "protocol.h"
static bool write_all(int fd, const void* data, unsigned long long size) {
const unsigned char* p = data;
@@ -29,30 +32,41 @@ static bool write_all(int fd, const void* data, unsigned long long size) {
return true;
}
bool file_checksum(File* file, uint64_t* checksum) {
if (!file || !checksum || !file->data)
/* Process-wide counter for scratch temp names. A --temp-dir scratch directory
is flat: different destinations that share a basename must never race onto
the same temp name. Deriving the trailing number from a global atomic
sequence keeps every temp name unique across the whole scratch directory
even when several threads write concurrently, so the O_EXCL creation loop
below almost never needs a retry. */
static unsigned long long next_temp_sequence(void) {
static atomic_ullong sequence;
return atomic_fetch_add_explicit(&sequence, 1, memory_order_relaxed);
}
bool file_checksum(File* file, ChecksumAlgo algo, uint64_t seed, uint8_t* out, size_t out_capacity,
size_t* out_len) {
if (!file || !out || !out_len || !file->data)
return false;
if (file->data->size == 0) {
*checksum = delta_xxhash64("", 0);
return true;
return checksum_digest(algo, seed, "", 0, out, out_capacity, out_len);
}
if (!file->data->data && !file_load_data(file))
return false;
*checksum = delta_xxhash64(file->data->data, file->data->size);
return true;
return checksum_digest(algo, seed, file->data->data, file->data->size, out, out_capacity,
out_len);
}
File* file_create(const char* path) {
if (!path)
return NULL;
File* file = (File*)malloc(sizeof(File));
File* file = (File*)protocol_alloc(sizeof(File));
if (file == NULL) {
log_perror("ERROR: Could not allocate memory for file struct");
return NULL;
}
size_t path_len = strlen(path);
file->path = (char*)malloc(path_len + 1);
file->path = (char*)protocol_alloc(path_len + 1);
if (file->path == NULL) {
free(file);
return NULL;
@@ -60,6 +74,7 @@ File* file_create(const char* path) {
memcpy(file->path, path, path_len);
file->path[path_len] = '\0';
file->send_path = NULL;
file->data = data_create_reserve(0);
if (file->data == NULL) {
free(file->path);
@@ -68,6 +83,8 @@ File* file_create(const char* path) {
}
file->metadata = NULL;
file->skip = false;
file->is_dir = false;
file->basis_link = NULL;
return file;
}
@@ -81,11 +98,15 @@ void file_destroy(void* item) {
file->metadata = NULL;
free(file->path);
file->path = NULL;
free(file->send_path);
file->send_path = NULL;
free(file->basis_link);
file->basis_link = NULL;
free(file);
}
FileMetadata* file_metadata_create(const struct stat* stats) {
FileMetadata* m = malloc(sizeof(FileMetadata));
FileMetadata* m = protocol_alloc(sizeof(FileMetadata));
if (m == NULL) {
log_perror("ERROR: Could not allocate memory for file metadata");
return NULL;
@@ -112,7 +133,7 @@ bool file_load_data(File* file) {
if (file->data->data == NULL) {
if (file->data->size == 0)
return true;
file->data->data = malloc(file->data->size);
file->data->data = protocol_alloc(file->data->size);
if (file->data->data == NULL) {
log_perror("Could not allocate memory for file data");
return false;
@@ -172,8 +193,17 @@ bool file_set_authorized_root(int fd, const char* canonical_path) {
}
bool file_path_exists_secure(const char* path) {
if (!path)
return false;
char* leaf = NULL;
int parent_fd = file_open_secure_parent(path, &leaf, false);
if (parent_fd < 0)
return false;
struct stat st;
return file_stat_secure(path, &st);
bool exists = fstatat(parent_fd, leaf, &st, AT_SYMLINK_NOFOLLOW) == 0;
close(parent_fd);
free(leaf);
return exists;
}
bool file_stat_secure(const char* path, struct stat* st) {
@@ -183,15 +213,29 @@ bool file_stat_secure(const char* path, struct stat* st) {
int parent_fd = file_open_secure_parent(path, &leaf, false);
if (parent_fd < 0)
return false;
int fd = openat(parent_fd, leaf, O_RDONLY | O_NONBLOCK | O_CLOEXEC | O_NOFOLLOW);
bool exists = fd >= 0 && fstat(fd, st) == 0 && S_ISREG(st->st_mode);
if (fd >= 0)
close(fd);
bool exists = fstatat(parent_fd, leaf, st, AT_SYMLINK_NOFOLLOW) == 0 && S_ISREG(st->st_mode);
close(parent_fd);
free(leaf);
return exists;
}
static bool stat_is_newer(const struct stat* st, const FileMetadata* metadata) {
if (!st || !metadata)
return false;
#ifdef __linux__
long mtime_nsec = st->st_mtim.tv_nsec;
#else
long mtime_nsec = 0;
#endif
return st->st_mtime > metadata->mtime_sec ||
(st->st_mtime == metadata->mtime_sec && mtime_nsec > metadata->mtime_nsec);
}
bool file_destination_is_newer_secure(const char* path, const FileMetadata* metadata) {
struct stat st;
return file_stat_secure(path, &st) && stat_is_newer(&st, metadata);
}
int file_open_secure_parent(const char* path, char** leaf_out, bool create_dirs) {
char* copy = str_dup(path);
if (!copy)
@@ -268,9 +312,41 @@ int file_open_secure_parent(const char* path, char** leaf_out, bool create_dirs)
return fd;
}
/* Normalized copy of a directory path: leading '/' kept, trailing '/' removed
* ("/" and "//" both collapse to "/"). A trailing slash otherwise makes the
* last path component empty, so probing that empty leaf below its parent
* always fails. */
static char* normalize_directory_path(const char* path) {
if (!path)
return NULL;
size_t len = strlen(path);
while (len > 1 && path[len - 1] == '/')
len--;
char* norm = malloc(len + 1);
if (!norm)
return NULL;
memcpy(norm, path, len);
norm[len] = '\0';
return norm;
}
bool file_ensure_directory_secure(const char* path) {
if (!path)
return false;
char* norm = normalize_directory_path(path);
if (!norm)
return false;
/* The authorized root is already an open directory, and the filesystem root
is always present: there is no final component left to create for them. */
bool root_is_open =
authorized_root_fd >= 0 && authorized_root_path && strcmp(norm, authorized_root_path) == 0;
if (root_is_open || strcmp(norm, "/") == 0) {
free(norm);
return true;
}
char* leaf = NULL;
int parent_fd = file_open_secure_parent(path, &leaf, true);
int parent_fd = file_open_secure_parent(norm, &leaf, true);
free(norm);
if (parent_fd < 0)
return false;
@@ -287,6 +363,39 @@ bool file_ensure_directory_secure(const char* path) {
return ok;
}
/* True when `path` resolves to an existing directory below the authorized root
* (never creating anything). Used by the server to decide whether a client's
* destination root already exists. A trailing slash on `path` and a destination
* equal to the authorized root itself are normalized/handled here so both
* previously-working destination forms keep working. */
bool file_directory_exists_secure(const char* path) {
if (!path)
return false;
char* norm = normalize_directory_path(path);
if (!norm)
return false;
bool root_is_open =
authorized_root_fd >= 0 && authorized_root_path && strcmp(norm, authorized_root_path) == 0;
if (root_is_open || strcmp(norm, "/") == 0) {
free(norm);
return true;
}
char* leaf = NULL;
int parent_fd = file_open_secure_parent(norm, &leaf, false);
free(norm);
if (parent_fd < 0)
return false;
int dir_fd = openat(parent_fd, leaf, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
if (dir_fd < 0 && errno == ENOENT)
dir_fd = -1;
bool ok = dir_fd >= 0;
if (dir_fd >= 0)
close(dir_fd);
close(parent_fd);
free(leaf);
return ok;
}
bool file_rename_secure(const char* old_path, const char* new_path) {
char *old_leaf = NULL, *new_leaf = NULL;
int old_parent = file_open_secure_parent(old_path, &old_leaf, false);
@@ -302,8 +411,108 @@ bool file_rename_secure(const char* old_path, const char* new_path) {
return ok;
}
bool file_to_disk_secure(const char* path, const void* data, unsigned long long data_size,
bool inplace, bool sparse, const FileMetadata* metadata) {
/* Recursively delete every entry inside an open directory, never following a
symlink (an O_NOFOLLOW fd walk, so a symlink planted inside the tree can
never redirect removal outside of it). The directory itself is left in
place; returns false on any failure. */
static bool wipe_dir_fd(int dirfd) {
int scanfd = dup(dirfd);
if (scanfd < 0)
return false;
DIR* dir = fdopendir(scanfd);
if (!dir) {
close(scanfd);
return false;
}
bool operation_ok = true;
const struct dirent* entry;
while ((entry = readdir(dir)) != NULL) {
if (strcmp(entry->d_name, ".") == 0 || strcmp(entry->d_name, "..") == 0)
continue;
struct stat st;
if (fstatat(dirfd, entry->d_name, &st, AT_SYMLINK_NOFOLLOW) != 0) {
if (errno != ENOENT)
operation_ok = false;
continue;
}
if (S_ISDIR(st.st_mode)) {
int childfd = openat(dirfd, entry->d_name, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
bool child_removed = false;
if (childfd >= 0) {
child_removed = wipe_dir_fd(childfd);
close(childfd);
} else if (errno != ENOENT) {
operation_ok = false;
}
if (child_removed && unlinkat(dirfd, entry->d_name, AT_REMOVEDIR) != 0 && errno != ENOENT)
operation_ok = false;
} else {
/* Files and symlinks alike are removed by name, never followed. */
if (unlinkat(dirfd, entry->d_name, 0) != 0 && errno != ENOENT)
operation_ok = false;
}
}
closedir(dir);
return operation_ok;
}
/* Remove the whole directory tree at `path` (confined below the authorized
root, symlink-safe). --force uses this to clear a non-empty destination
directory that blocks an incoming regular file. Returns true when the path
no longer exists as a directory (a missing path or a non-directory at the
final component is a no-op success; the normal write path replaces files). */
bool file_remove_tree_secure(const char* path) {
if (!path)
return false;
char* leaf = NULL;
int parent_fd = file_open_secure_parent(path, &leaf, false);
if (parent_fd < 0)
return false;
int dirfd = openat(parent_fd, leaf, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
if (dirfd < 0) {
bool absent = errno == ENOENT || errno == ENOTDIR || errno == ELOOP;
close(parent_fd);
free(leaf);
return absent;
}
bool ok = wipe_dir_fd(dirfd);
close(dirfd);
if (ok && unlinkat(parent_fd, leaf, AT_REMOVEDIR) != 0 && errno != ENOENT)
ok = false;
close(parent_fd);
free(leaf);
return ok;
}
/* Open a private staging/scratch directory, creating it (and any missing path
components) on demand. dir_path is expected to already be confined below
the authorized root by the caller; file_open_secure_parent re-checks that
confinement and rejects `..` components, so a scratch directory can never be
created or opened outside the destination root. The directory itself is
created 0700 so other users cannot race on names inside it. Returns an
O_DIRECTORY|O_NOFOLLOW fd, or -1 on error. */
int file_open_private_dir(const char* dir_path) {
if (!dir_path)
return -1;
char* leaf = NULL;
int parent_fd = file_open_secure_parent(dir_path, &leaf, true);
if (parent_fd < 0)
return -1;
int fd = openat(parent_fd, leaf, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
if (fd < 0 && errno == ENOENT) {
if (mkdirat(parent_fd, leaf, 0700) == 0 || errno == EEXIST)
fd = openat(parent_fd, leaf, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
}
close(parent_fd);
free(leaf);
return fd;
}
static bool file_to_disk_secure_impl(const char* path, const void* data,
unsigned long long data_size, bool inplace, bool sparse,
const FileMetadata* metadata, bool preserve_executability,
bool update, bool no_replace, bool use_fsync,
const char* temp_dir) {
char* leaf = NULL;
int dirfd = file_open_secure_parent(path, &leaf, true);
if (dirfd < 0)
@@ -311,34 +520,155 @@ bool file_to_disk_secure(const char* path, const void* data, unsigned long long
int fd = -1;
bool ok = false;
if (inplace) {
fd = openat(dirfd, leaf, O_WRONLY | O_CREAT | O_TRUNC | O_CLOEXEC | O_NOFOLLOW, 0644);
/* --inplace writes directly into the destination; a scratch --temp-dir
does not apply and must never redirect these writes. */
fd = openat(dirfd, leaf, O_WRONLY | O_CREAT | O_CLOEXEC | O_NOFOLLOW, 0644);
if (fd >= 0) {
if (!sparse || data_size == 0 || ftruncate(fd, (off_t)data_size) == 0)
ok = write_all(fd, data, data_size);
if (ok && metadata)
ok = file_restore_metadata_fd(fd, metadata);
struct stat destination_stat;
bool newer = false;
if (update && metadata && fstat(fd, &destination_stat) == 0 &&
S_ISREG(destination_stat.st_mode)) {
newer = stat_is_newer(&destination_stat, metadata);
}
if (newer) {
ok = true;
} else {
/* In-place overwrites: pre-size sparse targets and always trim the
file to the new payload length afterwards so shorter payloads can
never leave stale trailing bytes from a previous version. */
if (sparse && data_size > 0)
ok = ftruncate(fd, (off_t)data_size) == 0;
if (ok || !sparse || data_size == 0)
ok = write_all(fd, data, data_size);
if (ok)
ok = ftruncate(fd, (off_t)data_size) == 0;
/* Normalize the mode: apply the metadata-derived safe mode when the
sender supplied metadata (setuid/setgid/sticky are never honored);
otherwise fall back to a safe default so dangerous bits on an
existing destination cannot survive an overwrite. */
if (ok) {
if (metadata)
ok = file_restore_metadata_fd(fd, metadata, preserve_executability);
else if (fchmod(fd, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH) != 0)
ok = false;
}
if (ok && use_fsync)
ok = fsync(fd) == 0;
}
}
} else {
char tmp[NAME_MAX];
for (unsigned int i = 0; i < 100 && !ok; ++i) {
snprintf(tmp, sizeof(tmp), ".%s.tmp.%ld.%u", leaf, (long)getpid(), i);
fd = openat(dirfd, tmp, O_WRONLY | O_CREAT | O_EXCL | O_CLOEXEC | O_NOFOLLOW, 0600);
/* The --update newer-destination check runs first so a skipped file never
creates an empty scratch directory behind it. */
if (update && metadata) {
/* This check protects the normal atomic path as far as possible. A
concurrent replacement can still occur before the final rename. */
struct stat destination_stat;
if (fstatat(dirfd, leaf, &destination_stat, AT_SYMLINK_NOFOLLOW) == 0 &&
S_ISREG(destination_stat.st_mode) && stat_is_newer(&destination_stat, metadata)) {
close(dirfd);
free(leaf);
return true;
}
}
/* Scratch directory for the temporary working copy. When NULL the temp
file is created in the destination directory, exactly as historically. */
int scratch_dirfd = -1;
if (temp_dir) {
scratch_dirfd = file_open_private_dir(temp_dir);
if (scratch_dirfd < 0) {
int saved_errno = errno;
log_message(LOG_LEVEL_ERROR, "could not open --temp-dir scratch directory '%s': %s",
temp_dir, strerror(saved_errno));
close(dirfd);
free(leaf);
return false;
}
}
/* Temp names can exceed NAME_MAX for basenames near the limit (leaf plus
the ".tmp.<pid>.<n>" decoration); heap-size the buffer instead of
truncating into a fixed array, which would silently collide in a flat
scratch directory. The sizing sentinel is the widest value of each
format. */
int tmp_size;
if (scratch_dirfd >= 0)
tmp_size = snprintf(NULL, 0, ".%s.tmp.%ld.%llu", leaf, (long)getpid(), ~0ULL);
else
tmp_size = snprintf(NULL, 0, ".%s.tmp.%ld.%u", leaf, (long)getpid(), 999U);
if (tmp_size < 0) {
if (scratch_dirfd >= 0)
close(scratch_dirfd);
close(dirfd);
free(leaf);
return false;
}
char* tmp = malloc((size_t)tmp_size + 1);
if (!tmp) {
if (scratch_dirfd >= 0)
close(scratch_dirfd);
close(dirfd);
free(leaf);
return false;
}
for (unsigned int i = 0; i < 100; ++i) {
/* The temp name is created inside the scratch directory (when one is
configured) and, on success, atomically renamed into the destination
directory. In a shared scratch directory the atomic sequence number
keeps the name unique even for destinations with a common basename. */
if (scratch_dirfd >= 0)
snprintf(tmp, (size_t)tmp_size + 1, ".%s.tmp.%ld.%llu", leaf, (long)getpid(),
next_temp_sequence());
else
snprintf(tmp, (size_t)tmp_size + 1, ".%s.tmp.%ld.%u", leaf, (long)getpid(), i);
fd = openat(scratch_dirfd >= 0 ? scratch_dirfd : dirfd, tmp,
O_WRONLY | O_CREAT | O_EXCL | O_CLOEXEC | O_NOFOLLOW, 0600);
if (fd < 0)
continue;
continue; /* EEXIST (or a transient open error): try a fresh name. */
if (sparse && data_size > 0)
ok = ftruncate(fd, (off_t)data_size) == 0;
if (ok || (!sparse || data_size == 0))
ok = write_all(fd, data, data_size);
if (ok && metadata)
ok = file_restore_metadata_fd(fd, metadata);
ok = file_restore_metadata_fd(fd, metadata, preserve_executability);
if (ok && use_fsync)
ok = fsync(fd) == 0;
if (close(fd) != 0)
ok = false;
fd = -1;
if (ok && renameat(dirfd, tmp, dirfd, leaf) != 0)
ok = false;
if (ok) {
if (no_replace) {
/* The probe and commit cannot be one operation. A concurrent
creator may win; EEXIST is then the requested skip. */
if (linkat(scratch_dirfd >= 0 ? scratch_dirfd : dirfd, tmp, dirfd, leaf, 0) == 0 ||
errno == EEXIST) {
if (unlinkat(scratch_dirfd >= 0 ? scratch_dirfd : dirfd, tmp, 0) != 0 &&
errno != ENOENT)
ok = false;
} else {
if (scratch_dirfd >= 0 && errno == EXDEV)
log_message(LOG_LEVEL_ERROR,
"temp dir is on a different filesystem than the destination; cannot "
"link file into place (EXDEV); no fallback copy is attempted");
ok = false;
}
} else if (renameat(scratch_dirfd >= 0 ? scratch_dirfd : dirfd, tmp, dirfd, leaf) != 0) {
if (scratch_dirfd >= 0 && errno == EXDEV)
log_message(LOG_LEVEL_ERROR,
"temp dir is on a different filesystem than the destination; cannot "
"atomically install file (EXDEV); no fallback copy is attempted");
ok = false;
}
}
if (!ok)
unlinkat(dirfd, tmp, 0);
unlinkat(scratch_dirfd >= 0 ? scratch_dirfd : dirfd, tmp, 0);
/* Once the temp fd was created the outcome is permanent: a write,
metadata, fsync, close, linkat or renameat failure will not be fixed
by retrying under a fresh name, so stop here. Only the open-failure
path above retries a new name. */
break;
}
free(tmp);
if (scratch_dirfd >= 0)
close(scratch_dirfd);
}
if (fd >= 0)
close(fd);
@@ -347,9 +677,139 @@ bool file_to_disk_secure(const char* path, const void* data, unsigned long long
return ok;
}
bool file_to_disk_secure(const char* path, const void* data, unsigned long long data_size,
bool inplace, bool sparse, const FileMetadata* metadata,
bool preserve_executability, const char* temp_dir) {
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, metadata,
preserve_executability, false, false, false, temp_dir);
}
bool file_to_disk_secure_update(const char* path, const void* data, unsigned long long data_size,
bool inplace, bool sparse, const FileMetadata* metadata,
bool preserve_executability, const char* temp_dir) {
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, metadata,
preserve_executability, true, false, false, temp_dir);
}
bool file_to_disk_secure_with_fsync(const char* path, const void* data,
unsigned long long data_size, bool inplace, bool sparse,
const FileMetadata* metadata, bool preserve_executability,
bool use_fsync, const char* temp_dir) {
return file_to_disk_secure_impl(path, data, data_size, inplace, sparse, metadata,
preserve_executability, false, false, use_fsync, temp_dir);
}
bool file_to_disk_secure_no_replace(const char* path, const void* data,
unsigned long long data_size, bool sparse,
const FileMetadata* metadata, bool preserve_executability,
const char* temp_dir) {
return file_to_disk_secure_impl(path, data, data_size, false, sparse, metadata,
preserve_executability, false, true, false, temp_dir);
}
/* Atomic --link-dest install. The destination is replaced (via a temporary
* name and a final rename) with a hard link to `basis_path`. When a hard
* link cannot be created (the basis lives on a different filesystem, the
* filesystem refuses hard links, ...) the install falls back to writing a
* local copy from `data`/`data_size`, which the caller has already verified is
* byte-identical to the basis file. `metadata` is only applied on that copy
* fallback; a successful hard link keeps the basis inode's own attributes
* (applying metadata through the shared inode would mutate the basis file).
* Returns false only when both the link and the copy fallback fail. */
bool file_to_disk_secure_link(const char* path, const char* basis_path, const void* data,
unsigned long long data_size, const FileMetadata* metadata,
bool preserve_executability, bool use_fsync, const char* temp_dir) {
if (!path || !basis_path)
return false;
char* leaf = NULL;
int dirfd = file_open_secure_parent(path, &leaf, true);
if (dirfd < 0)
return false;
int scratch_dirfd = -1;
if (temp_dir) {
scratch_dirfd = file_open_private_dir(temp_dir);
if (scratch_dirfd < 0) {
int saved_errno = errno;
log_message(LOG_LEVEL_ERROR, "could not open --temp-dir scratch directory '%s': %s", temp_dir,
strerror(saved_errno));
close(dirfd);
free(leaf);
return false;
}
}
char* basis_leaf = NULL;
int basis_dirfd = file_open_secure_parent(basis_path, &basis_leaf, false);
bool linked = false;
if (basis_dirfd >= 0 && basis_leaf != NULL) {
int tmp_size = snprintf(NULL, 0, ".%s.tmp.%ld.%llu", leaf, (long)getpid(), ~0ULL);
char* tmp = NULL;
if (tmp_size >= 0)
tmp = malloc((size_t)tmp_size + 1);
if (!tmp) {
log_message(LOG_LEVEL_ERROR, "memory allocation failed while hard-linking basis file");
} else {
for (unsigned int i = 0; i < 100 && !linked; ++i) {
if (scratch_dirfd >= 0)
snprintf(tmp, (size_t)tmp_size + 1, ".%s.tmp.%ld.%llu", leaf, (long)getpid(),
next_temp_sequence());
else
snprintf(tmp, (size_t)tmp_size + 1, ".%s.tmp.%ld.%u", leaf, (long)getpid(), i);
if (linkat(basis_dirfd, basis_leaf, scratch_dirfd >= 0 ? scratch_dirfd : dirfd, tmp, 0) ==
0) {
linked = true;
break;
}
if (errno != EEXIST)
break; /* EXDEV / EPERM / ...: give up and fall back to a copy */
}
if (linked) {
int target_dirfd = scratch_dirfd >= 0 ? scratch_dirfd : dirfd;
if (use_fsync) {
int tfd = openat(target_dirfd, tmp, O_RDONLY | O_NOFOLLOW | O_CLOEXEC);
if (tfd < 0 || fsync(tfd) != 0) {
linked = false;
if (tfd >= 0)
close(tfd);
} else {
close(tfd);
}
}
if (linked && renameat(target_dirfd, tmp, dirfd, leaf) != 0)
linked = false;
if (!linked)
unlinkat(target_dirfd, tmp, 0);
}
free(tmp);
}
}
if (basis_dirfd >= 0)
close(basis_dirfd);
free(basis_leaf);
basis_leaf = NULL;
if (!linked) {
if (scratch_dirfd >= 0)
close(scratch_dirfd);
close(dirfd);
free(leaf);
/* The basis file could not be linked in (missing, cross-device, refused
by the filesystem). Write a byte-identical local copy instead. */
return file_to_disk_secure_with_fsync(path, data, data_size, false, false, metadata,
preserve_executability, use_fsync, temp_dir);
}
if (scratch_dirfd >= 0)
close(scratch_dirfd);
close(dirfd);
free(leaf);
return true;
}
bool file_write_to_disk(const char* path, const void* data, unsigned long long data_size,
bool inplace, bool sparse) {
if (!path || (!data && data_size != 0) || has_path_traversal(path))
return false;
return file_to_disk_secure(path, data, data_size, inplace, sparse, NULL);
return file_to_disk_secure(path, data, data_size, inplace, sparse, NULL, false, NULL);
}
+50 -2
View File
@@ -4,6 +4,7 @@
#include "file_send.h"
#include "file_receive.h"
#include "file_types.h"
#include "checksum.h"
#include <stdbool.h>
#include <stdint.h>
#include <sys/stat.h>
@@ -14,7 +15,12 @@
File* file_create(const char* path);
void file_destroy(void* item);
bool file_load_data(File* file);
bool file_checksum(File* file, uint64_t* checksum);
/* Compute the whole-file content digest of `file` with the negotiated
* --checksum-choice algorithm and --checksum-seed. Writes the digest into
* `out` (capacity `out_capacity`) and its length into `*out_len`. Returns
* false on read/allocation failure or when the digest would not fit. */
bool file_checksum(File* file, ChecksumAlgo algo, uint64_t seed, uint8_t* out, size_t out_capacity,
size_t* out_len);
size_t file_content_to_buffer(File* file);
FileMetadata* file_metadata_create(const struct stat* stats);
void file_metadata_destroy(void* metadata);
@@ -27,10 +33,52 @@ bool file_set_authorized_root(int fd, const char* canonical_path);
/* Secure path/filesystem primitives (symlink-safe, O_NOFOLLOW, root-confined). */
bool file_path_exists_secure(const char* path);
bool file_stat_secure(const char* path, struct stat* st);
bool file_destination_is_newer_secure(const char* path, const FileMetadata* metadata);
int file_open_secure_parent(const char* path, char** leaf_out, bool create_dirs);
bool file_ensure_directory_secure(const char* path);
bool file_directory_exists_secure(const char* path);
bool file_rename_secure(const char* old_path, const char* new_path);
/* Remove the whole directory tree at `path` (confined, symlink-safe). Used by
--force to clear a non-empty destination directory that blocks an incoming
regular file. See the .c for the exact success semantics. */
bool file_remove_tree_secure(const char* path);
/* Open a private 0700 directory (creating it on demand) that must live below
the authorized root. Used for the --temp-dir scratch directory and the
--delay-updates staging directory. */
int file_open_private_dir(const char* dir_path);
/* The file_to_disk_secure* variants write a temporary copy in the destination
directory and atomically rename it over `path`. temp_dir is an absolute,
root-confined scratch directory (already validated by the caller): when it
is non-NULL the temporary copy is instead created there (with a name unique
across the whole scratch directory) and atomically renamed into the
destination directory once fully written and fsynced. A rename across
filesystems (EXDEV) fails the write with an error; the file is never
silently copied into place. Pass NULL for the historical same-directory
behavior. --inplace writes never use temp_dir. */
bool file_to_disk_secure(const char* path, const void* data, unsigned long long data_size,
bool inplace, bool sparse, const FileMetadata* metadata);
bool inplace, bool sparse, const FileMetadata* metadata,
bool preserve_executability, const char* temp_dir);
bool file_to_disk_secure_with_fsync(const char* path, const void* data,
unsigned long long data_size, bool inplace, bool sparse,
const FileMetadata* metadata, bool preserve_executability,
bool use_fsync, const char* temp_dir);
/* With update enabled, an existing newer destination is left untouched. The
check is descriptor-based for inplace writes; atomic replacement still has
an unavoidable final rename race without filesystem locking. */
bool file_to_disk_secure_update(const char* path, const void* data, unsigned long long data_size,
bool inplace, bool sparse, const FileMetadata* metadata,
bool preserve_executability, const char* temp_dir);
bool file_to_disk_secure_no_replace(const char* path, const void* data,
unsigned long long data_size, bool sparse,
const FileMetadata* metadata, bool preserve_executability,
const char* temp_dir);
/* Atomic --link-dest install: replace `path` with a hard link to `basis_path`
(via a temp name + rename); fall back to a byte-identical local copy from
`data` when the link is impossible (EXDEV/EPERM/unsupported filesystem).
`metadata` is applied only on the copy fallback. */
bool file_to_disk_secure_link(const char* path, const char* basis_path, const void* data,
unsigned long long data_size, const FileMetadata* metadata,
bool preserve_executability, bool use_fsync, const char* temp_dir);
#endif
+191
View File
@@ -0,0 +1,191 @@
#include "file_list.h"
#include "log.h"
#include "utils.h"
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
typedef struct {
char** items;
int count;
int capacity;
} StringList;
static void string_list_destroy(StringList* list) {
if (!list)
return;
for (int i = 0; i < list->count; i++)
free(list->items[i]);
free(list->items);
}
static bool string_list_add(StringList* list, const char* text) {
if (list->count == list->capacity) {
int new_cap = list->capacity > 0 ? list->capacity * 2 : 16;
char** grown = realloc(list->items, (size_t)new_cap * sizeof(char*));
if (!grown)
return false;
list->items = grown;
list->capacity = new_cap;
}
list->items[list->count] = str_dup(text);
if (!list->items[list->count])
return false;
list->count++;
return true;
}
/* Validate and normalize one entry. Returns:
* 1 -> added to `out`
* 0 -> blank entry, skip
* -1 -> invalid (message set in `err`)
* `strip_line_endings` trims a trailing CR/LF (line mode only); NUL mode keeps
* the entry bytes verbatim so names ending in CR/LF survive. */
static int normalize_entry(const char* raw, size_t len, bool strip_line_endings, StringList* out,
char* err, size_t err_size) {
if (strip_line_endings) {
while (len > 0 && (raw[len - 1] == '\n' || raw[len - 1] == '\r'))
len--;
}
if (len == 0)
return 0;
if (raw[0] == '/') {
snprintf(err, err_size, "absolute path entries are not allowed: '%.*s'", (int)len, raw);
return -1;
}
/* Reject NUL bytes inside a token defensively (NUL-delimited mode splits on
* them, so this only guards against embedded garbage). */
char* dup = malloc(len + 1);
if (!dup) {
snprintf(err, err_size, "memory allocation failed");
return -1;
}
memcpy(dup, raw, len);
dup[len] = '\0';
/* Rebuild the path token-by-token: skip '.' and empty segments, reject '..'. */
size_t out_len = 0;
for (const char* part = dup;;) {
const char* slash = strchr(part, '/');
size_t part_len = slash ? (size_t)(slash - part) : strlen(part);
if (part_len == 1 && part[0] == '.') {
/* skip "." segment */
} else if (part_len == 2 && part[0] == '.' && part[1] == '.') {
snprintf(err, err_size, "path traversal entry is not allowed: '%s'", dup);
free(dup);
return -1;
} else if (part_len > 0) {
if (out_len > 0)
dup[out_len++] = '/';
memmove(dup + out_len, part, part_len);
out_len += part_len;
}
if (!slash)
break;
part = slash + 1;
}
dup[out_len] = '\0';
int result;
if (out_len == 0) {
/* "." / "./" lists the source root: the whole tree is transferred. */
result = string_list_add(out, "") ? 1 : -1;
if (result < 0)
snprintf(err, err_size, "memory allocation failed");
} else {
result = string_list_add(out, dup) ? 1 : -1;
if (result < 0)
snprintf(err, err_size, "memory allocation failed");
}
free(dup);
return result;
}
static FileListSet* string_list_to_set(StringList* raw, char* err, size_t err_size) {
FileListSet* set = malloc(sizeof(FileListSet));
if (!set) {
snprintf(err, err_size, "memory allocation failed");
return NULL;
}
set->count = raw->count;
set->entries = raw->items;
raw->items = NULL;
raw->count = 0;
return set;
}
FileListSet* file_list_load(const char* path, bool null_separated, char* err, size_t err_size) {
if (err && err_size > 0)
err[0] = '\0';
if (!path || !*path) {
snprintf(err, err_size, "no file given");
return NULL;
}
FILE* fp = fopen(path, "r");
if (!fp) {
char* escaped = output_escape(path, false);
snprintf(err, err_size, "could not open '%s': %s", escaped ? escaped : path, strerror(errno));
free(escaped);
return NULL;
}
StringList raw = {0};
char* line = NULL;
size_t line_cap = 0;
ssize_t n;
bool ok = true;
char delim = null_separated ? '\0' : '\n';
while (ok && (n = getdelim(&line, &line_cap, delim, fp)) != -1) {
int r = normalize_entry(line, (size_t)n, !null_separated, &raw, err, err_size);
if (r < 0) {
ok = false;
break;
}
}
free(line);
fclose(fp);
if (!ok) {
string_list_destroy(&raw);
return NULL;
}
FileListSet* set = string_list_to_set(&raw, err, err_size);
if (!set)
string_list_destroy(&raw);
return set;
}
void file_list_destroy(FileListSet* set) {
if (!set)
return;
for (int i = 0; i < set->count; i++)
free(set->entries[i]);
free(set->entries);
free(set);
}
static bool path_has_prefix(const char* path, const char* prefix) {
size_t plen = strlen(prefix);
if (strncmp(path, prefix, plen) != 0)
return false;
return path[plen] == '/' || path[plen] == '\0';
}
bool file_list_affects(const FileListSet* set, const char* rel) {
if (!set)
return true;
if (!rel)
return false;
for (int i = 0; i < set->count; i++) {
const char* entry = set->entries[i];
if (entry[0] == '\0')
return true; /* whole tree listed */
if (strcmp(rel, entry) == 0)
return true; /* the entry itself is listed */
if (path_has_prefix(rel, entry))
return true; /* rel lives under a listed directory */
if (path_has_prefix(entry, rel))
return true; /* rel is an ancestor directory of a listed entry */
}
return false;
}
+35
View File
@@ -0,0 +1,35 @@
#ifndef FILE_LIST_H
#define FILE_LIST_H
#include <stdbool.h>
#include <stddef.h>
/* --files-from allow-set. The file lists source paths RELATIVE to the source
* root. A listed regular file is transferred; a listed directory transfers its
* whole subtree (FastSync's recursion is always on). Blank lines are ignored.
*
* Entries are normalized: leading "./" and duplicate "/" are removed, an entry
* of "." means the whole tree, absolute entries and ".." traversal are
* rejected at parse time. The set is immutable and shared read-only across
* scanner worker threads.
*/
typedef struct {
char** entries; /* normalized rel paths; "" means the whole tree */
int count;
} FileListSet;
/* Load and validate a --files-from file. When `null_separated` (-0/--from0)
* entries are delimited by NUL instead of newlines. Returns NULL with a message
* in `err` on open/validation failure. An empty file yields an empty set
* (nothing is transferred). */
FileListSet* file_list_load(const char* path, bool null_separated, char* err, size_t err_size);
void file_list_destroy(FileListSet* set);
/* True when `rel` (path relative to the source root, "" == root) is a listed
* entry, lives under a listed directory, or is an ancestor directory of a
* listed entry. Used to prune scanning: directories are descended only when
* this returns true, files are transferred only when it returns true. */
bool file_list_affects(const FileListSet* set, const char* rel);
#endif
+1060 -108
View File
File diff suppressed because it is too large. Load diff
+33 -1
View File
@@ -8,8 +8,40 @@
/* Server-side file receive/save path. */
File* file_receive(const Config* config, int file_descriptor);
File* file_receive_directory(int file_descriptor);
File* receive_incremental_check(int fd, const Config* config, bool* skipped);
int receive_manifest(int fd, const Config* config, int* next_status);
/* A received delete-manifest frame: the keep-set (`keeps`, destination-relative
paths the sender transferred/keeps) plus `protected`, destination-relative
prefixes the sender asks the receiver never to delete (paths excluded on the
source, protected at any depth). When --delete-excluded is given the sender
transmits an empty protected list so excluded destination mirrors are treated
as ordinary extras. */
typedef struct DeleteManifest {
ArrayList* keeps;
ArrayList* protected;
} DeleteManifest;
void delete_manifest_free(DeleteManifest* manifest);
/* Read a delete-manifest frame: keep count + keeps, then protected count +
protected prefixes (self-delimiting; the leading STATUS_MANIFEST code has been
consumed). Returns an owned DeleteManifest, or NULL after signalling
STATUS_ERROR on a malformed frame. */
DeleteManifest* receive_manifest_entries(int fd);
/* Remove destination entries under config->receive_root_directory that are not
in `manifest` (bounded, all-or-nothing walk; staging-dir, basis-dir and
protected-prefix skips). `--max-delete` and `--force` are honored here. The
caller decides WHEN to run it based on the negotiated delete timing. Returns
false (and the transfer fails) when the deletion cannot be committed. */
bool manifest_delete_extras(const Config* config, DeleteManifest* manifest);
/* Outcome of a single file_save_to_disk operation. The receiver needs to
distinguish "written" from "skipped" so --remove-source-files can be told
which sources were actually stored. */
typedef enum { FILE_SAVE_ERROR = 0, FILE_SAVE_WRITTEN = 1, FILE_SAVE_SKIPPED = 2 } FileSaveResult;
FileSaveResult file_save_to_disk_full(const char* root_directory, const File* file,
const Config* config);
bool file_save_to_disk(const char* root_directory, const File* file, const Config* config);
#endif
+24 -6
View File
@@ -19,19 +19,29 @@
bool file_send_single_calls(File* file, int file_descriptor, bool use_metadata,
int compression_level, bool send_path) {
return file_send_single_calls_with_skip(file, file_descriptor, use_metadata, compression_level,
send_path, NULL, -1, 0);
}
bool file_send_single_calls_with_skip(File* file, int file_descriptor, bool use_metadata,
int compression_level, bool send_path,
char* const* skip_suffixes, int skip_count,
int compression_threads) {
if (!file || !file->path || !file->data || (file->data->size != 0 && !file->data->data))
return false;
const Data* data_to_send = file->data;
Data* compressed_data = NULL;
if (compression_level > 0 && !compression_should_skip(file->path)) {
compressed_data = data_compress(file->data, compression_level);
if (compression_level > 0 &&
!compression_should_skip_with_suffixes(file->path, skip_suffixes, skip_count)) {
compressed_data =
data_compress_with_threads(file->data, compression_level, compression_threads);
if (compressed_data == NULL) {
log_message(LOG_LEVEL_ERROR, "Failed to compress file data");
return false;
}
data_to_send = compressed_data;
}
if (send_path && !send_str(file_descriptor, file->path)) {
if (send_path && !send_str(file_descriptor, file_wire_path(file))) {
data_destroy(compressed_data);
return false;
}
@@ -49,13 +59,21 @@ bool file_send_single_calls(File* file, int file_descriptor, bool use_metadata,
bool file_send_sendfile(File* file, int file_descriptor, bool use_metadata, int compression_level,
bool send_path) {
return file_send_sendfile_with_skip(file, file_descriptor, use_metadata, compression_level,
send_path, NULL, -1, 0);
}
bool file_send_sendfile_with_skip(File* file, int file_descriptor, bool use_metadata,
int compression_level, bool send_path, char* const* skip_suffixes,
int skip_count, int compression_threads) {
if (!file || !file->path || !file->data)
return false;
if (compression_level > 0)
return file_send_single_calls(file, file_descriptor, use_metadata, compression_level,
send_path);
return file_send_single_calls_with_skip(file, file_descriptor, use_metadata, compression_level,
send_path, skip_suffixes, skip_count,
compression_threads);
if (send_path && !send_str(file_descriptor, file->path))
if (send_path && !send_str(file_descriptor, file_wire_path(file)))
return false;
if (use_metadata && !metadata_send(file_descriptor, file->metadata))
return false;
+7
View File
@@ -8,7 +8,14 @@
bool file_send_single_calls(File* file, int file_descriptor, bool use_metadata,
int compression_level, bool send_path);
bool file_send_single_calls_with_skip(File* file, int file_descriptor, bool use_metadata,
int compression_level, bool send_path,
char* const* skip_suffixes, int skip_count,
int compression_threads);
bool file_send_sendfile(File* file, int file_descriptor, bool use_metadata, int compression_level,
bool send_path);
bool file_send_sendfile_with_skip(File* file, int file_descriptor, bool use_metadata,
int compression_level, bool send_path, char* const* skip_suffixes,
int skip_count, int compression_threads);
#endif
+4 -3
View File
@@ -140,7 +140,8 @@ static bool write_all(int fd, const void* data, unsigned long long size) {
}
bool file_store_write_secure(const char* path, const void* data, unsigned long long data_size,
bool inplace, bool sparse, const FileMetadata* metadata) {
bool inplace, bool sparse, const FileMetadata* metadata,
bool preserve_executability) {
char* leaf = NULL;
int dirfd = file_store_open_secure_parent(path, &leaf);
if (dirfd < 0)
@@ -153,7 +154,7 @@ bool file_store_write_secure(const char* path, const void* data, unsigned long l
if (!sparse || data_size == 0 || ftruncate(fd, (off_t)data_size) == 0)
ok = write_all(fd, data, data_size);
if (ok && metadata)
ok = file_restore_metadata_fd(fd, metadata);
ok = file_restore_metadata_fd(fd, metadata, preserve_executability);
}
} else {
int tmp_size = snprintf(NULL, 0, ".%s.tmp.%ld.%u", leaf, (long)getpid(), 99U);
@@ -178,7 +179,7 @@ bool file_store_write_secure(const char* path, const void* data, unsigned long l
if (ok || (!sparse || data_size == 0))
ok = write_all(fd, data, data_size);
if (ok && metadata)
ok = file_restore_metadata_fd(fd, metadata);
ok = file_restore_metadata_fd(fd, metadata, preserve_executability);
if (close(fd) != 0)
ok = false;
fd = -1;
+2 -1
View File
@@ -8,6 +8,7 @@ bool file_store_set_authorized_root(int fd, const char* canonical_path);
int file_store_open_secure_parent(const char* path, char** leaf_out);
bool file_store_rename_secure(const char* old_path, const char* new_path);
bool file_store_write_secure(const char* path, const void* data, unsigned long long data_size,
bool inplace, bool sparse, const FileMetadata* metadata);
bool inplace, bool sparse, const FileMetadata* metadata,
bool preserve_executability);
#endif
+21
View File
@@ -17,9 +17,30 @@ typedef struct {
typedef struct {
char* path;
/* Sender-side override for the path transmitted on the wire (and used for
* the delete manifest / change output). NULL means "use `path`". With
* -R + --files-from this holds the entry's bare relative destination path,
* while `path` stays the absolute local source path the client reads from.
* Never populated on the receiver. */
char* send_path;
Data* data;
FileMetadata* metadata;
bool skip;
/* True when this entry is an explicit directory entry (--dirs mode): the
* receiver creates the directory instead of writing a regular file. */
bool is_dir;
/* Receiver-only, --link-dest: when set, install the destination entry as a
* hard link to this absolute (root-confined) path instead of writing
* `data`. The matching code has already verified the link target's content
* equals the incoming file, and `data` is kept as the cross-filesystem
* fallback (a local copy) if the hard link cannot be created. */
char* basis_link;
} File;
/* The path that should be sent on the wire and used for the receiver-side
* destination layout (see send_path). */
static inline const char* file_wire_path(const File* file) {
return file && file->send_path ? file->send_path : (file ? file->path : NULL);
}
#endif
+427
View File
@@ -0,0 +1,427 @@
#include "filter.h"
#include "log.h"
#include "utils.h"
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
/* ---- Single rule parsing ---- */
static bool rule_text_is_unsupported_word(const char* p, size_t len) {
static const char* const words[] = {"merge", "dir-merge", "hide", "show",
"protect", "risk", "clear"};
for (size_t i = 0; i < sizeof(words) / sizeof(words[0]); i++) {
size_t wl = strlen(words[i]);
if (len == wl && strncmp(p, words[i], wl) == 0)
return true;
}
return false;
}
/* rsync include/exclude rule modifiers we do NOT implement. A rule whose +/- is
* immediately followed by one of these is rejected instead of being silently
* parsed as a literal pattern. */
static bool is_unsupported_rule_modifier(char c) {
return c == '!' || c == 'C' || c == 's' || c == 'r' || c == 'p' || c == 'x';
}
FilterRule* filter_rule_parse(const char* line, char* err, size_t err_size) {
if (err && err_size > 0)
err[0] = '\0';
if (!line)
return NULL;
char* text = str_dup(line);
if (!text) {
if (err)
snprintf(err, err_size, "memory allocation failed");
return NULL;
}
size_t len = strlen(text);
while (len > 0 && (text[len - 1] == '\n' || text[len - 1] == '\r'))
text[--len] = '\0';
const char* p = text;
while (*p == ' ' || *p == '\t')
p++;
if (*p == '\0') {
snprintf(err, err_size, "empty filter rule");
free(text);
return NULL;
}
FilterAction action = FILTER_ACTION_EXCLUDE;
if (*p == '+' || *p == '-') {
action = *p == '+' ? FILTER_ACTION_INCLUDE : FILTER_ACTION_EXCLUDE;
p++;
/* rsync attaches rule modifiers directly to the +/- (e.g. "-s foo"). Only
* the '/' anchor modifier is supported; anything else is a clear error
* rather than a silently-ignored literal. */
if (*p != ' ' && *p != '\t' && *p != '\0' && is_unsupported_rule_modifier(*p)) {
snprintf(err, err_size,
"filter rule modifier '%c' is not supported (only the '/' anchor after +/- "
"is implemented; put a space between +/- and the pattern)",
*p);
free(text);
return NULL;
}
while (*p == ' ' || *p == '\t')
p++;
} else {
/* ':' (dir-merge) and '.' (merge) are rsync filter-rule shorthands. At the
* start of a rule they mean "merge this file", so reject them instead of
* silently turning them into inert exclude patterns. */
if (*p == ':' || *p == '.' || *p == '!') {
snprintf(err, err_size,
"filter rule starting with '%c' is not supported (merge/dir-merge/list-clear "
"shorthands are not implemented; use +/- include/exclude rules)",
*p);
free(text);
return NULL;
}
const char* sp = p;
while (*sp != '\0' && *sp != ' ' && *sp != '\t')
sp++;
size_t word_len = (size_t)(sp - p);
if (rule_text_is_unsupported_word(p, word_len)) {
snprintf(err, err_size,
"'%.*s' filter directives are not supported (only +/- include/exclude rules "
"with an optional '/' anchor and trailing '/' dir marker)",
(int)word_len, p);
free(text);
return NULL;
}
if (word_len == strlen("include") && strncmp(p, "include", word_len) == 0) {
action = FILTER_ACTION_INCLUDE;
p = sp;
} else if (word_len == strlen("exclude") && strncmp(p, "exclude", word_len) == 0) {
action = FILTER_ACTION_EXCLUDE;
p = sp;
}
while (*p == ' ' || *p == '\t')
p++;
}
if (*p == '\0') {
snprintf(err, err_size, "filter rule has no pattern");
free(text);
return NULL;
}
/* A pattern beginning with '/' is anchored (either as "-/foo" or "- /foo"). */
bool anchored = false;
if (*p == '/') {
anchored = true;
p++;
while (*p == ' ' || *p == '\t')
p++;
}
if (*p == '\0') {
snprintf(err, err_size, "filter rule has no pattern after '/' anchor");
free(text);
return NULL;
}
/* Pattern runs to the end of the rule; a single trailing '/' marks dir-only. */
size_t pat_len = strlen(p);
bool dir_only = false;
if (pat_len > 1 && p[pat_len - 1] == '/') {
dir_only = true;
pat_len--;
} else if (pat_len == 1 && p[0] == '/') {
/* "//" anchored with nothing after: meaningless. */
snprintf(err, err_size, "filter rule has no pattern");
free(text);
return NULL;
}
FilterRule* rule = calloc(1, sizeof(FilterRule));
if (!rule) {
snprintf(err, err_size, "memory allocation failed");
free(text);
return NULL;
}
rule->pattern = malloc(pat_len + 1);
if (!rule->pattern) {
free(rule);
snprintf(err, err_size, "memory allocation failed");
free(text);
return NULL;
}
memcpy(rule->pattern, p, pat_len);
rule->pattern[pat_len] = '\0';
rule->action = action;
rule->anchored = anchored;
rule->dir_only = dir_only;
rule->owner = NULL;
free(text);
return rule;
}
void filter_rule_free(FilterRule* rule) {
if (!rule)
return;
free(rule->pattern);
free(rule->owner);
free(rule);
}
/* ---- Ordered rule lists ---- */
FilterRuleList* filter_rule_list_create(void) {
return calloc(1, sizeof(FilterRuleList));
}
bool filter_rule_list_add(FilterRuleList* list, FilterRule* rule) {
if (!list || !rule)
return false;
if (list->count == list->capacity) {
int new_cap = list->capacity > 0 ? list->capacity * 2 : 8;
FilterRule** grown = realloc(list->items, (size_t)new_cap * sizeof(FilterRule*));
if (!grown)
return false;
list->items = grown;
list->capacity = new_cap;
}
list->items[list->count++] = rule;
return true;
}
bool filter_rule_list_parse_append(FilterRuleList* list, const char* line, char* err,
size_t err_size) {
FilterRule* rule = filter_rule_parse(line, err, err_size);
if (!rule)
return false;
if (!filter_rule_list_add(list, rule)) {
filter_rule_free(rule);
snprintf(err, err_size, "memory allocation failed");
return false;
}
return true;
}
void filter_rule_list_free(FilterRuleList* list) {
if (!list)
return;
for (int i = 0; i < list->count; i++)
filter_rule_free(list->items[i]);
free(list->items);
free(list);
}
static bool set_rule_owner(FilterRule* rule, const char* owner) {
char* dup = str_dup(owner ? owner : "");
if (!dup)
return false;
free(rule->owner);
rule->owner = dup;
return true;
}
/* ---- CVS default excludes (-C) ---- */
typedef struct {
const char* pattern;
bool dir_only;
} CvsDefaultRule;
static const CvsDefaultRule CVS_DEFAULTS[] = {
{"RCS", false}, {"SCCS", false}, {"CVS", false}, {"CVS.adm", false},
{"RCSLOG", false}, {"cvslog.*", false}, {"tags", false}, {"TAGS", false},
{".make.state", false}, {".nse_depinfo", false}, {"*~", false}, {"#*", false},
{".#*", false}, {",*", false}, {"_$*", false}, {"*$", false},
{"*.old", false}, {"*.bak", false}, {"*.BAK", false}, {"*.orig", false},
{"*.rej", false}, {".del-*", false}, {"*.a", false}, {"*.olb", false},
{"*.o", false}, {"*.obj", false}, {"*.so", false}, {"*.exe", false},
{"*.Z", false}, {"*.elc", false}, {"*.ln", false}, {"core", false},
{".svn/", true}, {".git/", true}, {".hg/", true}, {".bzr/", true},
};
static bool cvs_rule_list_append(FilterRuleList* list) {
for (size_t i = 0; i < sizeof(CVS_DEFAULTS) / sizeof(CVS_DEFAULTS[0]); i++) {
FilterRule* rule = calloc(1, sizeof(FilterRule));
if (!rule)
return false;
rule->action = FILTER_ACTION_EXCLUDE;
rule->dir_only = CVS_DEFAULTS[i].dir_only;
size_t plen = strlen(CVS_DEFAULTS[i].pattern);
if (rule->dir_only && plen > 0 && CVS_DEFAULTS[i].pattern[plen - 1] == '/')
plen--; /* keep the cleaned pattern, matching filter_rule_parse */
rule->pattern = malloc(plen + 1);
if (!rule->pattern) {
free(rule);
return false;
}
memcpy(rule->pattern, CVS_DEFAULTS[i].pattern, plen);
rule->pattern[plen] = '\0';
if (!set_rule_owner(rule, "")) {
filter_rule_free(rule);
return false;
}
if (!filter_rule_list_add(list, rule)) {
filter_rule_free(rule);
return false;
}
}
return true;
}
FilterRuleList* filter_base_build(const char* const* rule_texts, int rule_count, bool cvs_exclude,
char* err, size_t err_size) {
if (err && err_size > 0)
err[0] = '\0';
FilterRuleList* list = filter_rule_list_create();
if (!list) {
snprintf(err, err_size, "memory allocation failed");
return NULL;
}
for (int i = 0; i < rule_count; i++) {
if (!rule_texts || !rule_texts[i])
continue;
FilterRule* rule = filter_rule_parse(rule_texts[i], err, err_size);
if (!rule) {
filter_rule_list_free(list);
return NULL;
}
if (!set_rule_owner(rule, "")) {
filter_rule_free(rule);
filter_rule_list_free(list);
snprintf(err, err_size, "memory allocation failed");
return NULL;
}
if (!filter_rule_list_add(list, rule)) {
filter_rule_free(rule);
filter_rule_list_free(list);
snprintf(err, err_size, "memory allocation failed");
return NULL;
}
}
if (cvs_exclude && !cvs_rule_list_append(list)) {
filter_rule_list_free(list);
snprintf(err, err_size, "memory allocation failed");
return NULL;
}
return list;
}
/* ---- Per-directory .rsync-filter files ---- */
FilterRuleList* filter_file_read(const char* dir_path, const char* owner_rel, bool* exists,
char* err, size_t err_size) {
if (err && err_size > 0)
err[0] = '\0';
if (exists)
*exists = false;
char* filter_path = path_cat(dir_path, ".rsync-filter");
if (!filter_path) {
snprintf(err, err_size, "memory allocation failed");
return NULL;
}
FILE* fp = fopen(filter_path, "r");
free(filter_path);
if (!fp) {
if (errno == ENOENT || errno == ENOTDIR)
return filter_rule_list_create();
log_message(LOG_LEVEL_WARNING, "Could not read .rsync-filter in %s: %s", dir_path,
strerror(errno));
return filter_rule_list_create();
}
if (exists)
*exists = true;
FilterRuleList* list = filter_rule_list_create();
if (!list) {
fclose(fp);
snprintf(err, err_size, "memory allocation failed");
return NULL;
}
char* line = NULL;
size_t line_cap = 0;
ssize_t n;
bool ok = true;
while ((n = getline(&line, &line_cap, fp)) != -1) {
const char* p = line;
while (*p == ' ' || *p == '\t')
p++;
if (*p == '\0' || *p == '\n' || *p == '\r' || *p == '#')
continue;
FilterRule* rule = filter_rule_parse(p, err, err_size);
if (!rule) {
ok = false;
break;
}
if (!set_rule_owner(rule, owner_rel)) {
filter_rule_free(rule);
snprintf(err, err_size, "memory allocation failed");
ok = false;
break;
}
if (!filter_rule_list_add(list, rule)) {
filter_rule_free(rule);
snprintf(err, err_size, "memory allocation failed");
ok = false;
break;
}
}
free(line);
fclose(fp);
if (!ok) {
filter_rule_list_free(list);
return NULL;
}
return list;
}
/* ---- Rule matching ---- */
/* Match a pattern that contains '/' (non-anchored) against the end of the
* relative path, starting at any path-component boundary. */
static bool glob_suffix_match(const char* pattern, const char* str) {
if (glob_match(pattern, str))
return true;
for (const char* slash = strchr(str, '/'); slash; slash = strchr(slash + 1, '/')) {
if (glob_match(pattern, slash + 1))
return true;
}
return false;
}
static FilterAction rule_matches(const FilterRule* rule, const char* rel_path, const char* leaf,
bool is_dir) {
if (!rule || !rule->pattern)
return FILTER_ACTION_NONE;
if (rule->dir_only && !is_dir)
return FILTER_ACTION_NONE;
/* A rule applies only to entries below its owner directory. */
const char* rel2 = rel_path;
if (rule->owner && rule->owner[0] != '\0') {
size_t owner_len = strlen(rule->owner);
if (strncmp(rule->owner, rel_path, owner_len) != 0)
return FILTER_ACTION_NONE;
if (rel_path[owner_len] != '/')
return FILTER_ACTION_NONE;
rel2 = rel_path + owner_len + 1;
}
if (rel2[0] == '\0')
return FILTER_ACTION_NONE;
bool matched;
if (rule->anchored) {
matched = glob_match(rule->pattern, rel2);
} else if (strchr(rule->pattern, '/') != NULL) {
matched = glob_suffix_match(rule->pattern, rel2);
} else {
matched = glob_match(rule->pattern, leaf);
}
return matched ? rule->action : FILTER_ACTION_NONE;
}
FilterAction filter_rules_apply(const FilterRuleList* list, const char* rel_path, const char* leaf,
bool is_dir) {
if (!list)
return FILTER_ACTION_NONE;
for (int i = 0; i < list->count; i++) {
FilterAction action = rule_matches(list->items[i], rel_path, leaf, is_dir);
if (action != FILTER_ACTION_NONE)
return action;
}
return FILTER_ACTION_NONE;
}
+83
View File
@@ -0,0 +1,83 @@
#ifndef FILTER_H
#define FILTER_H
#include <stdbool.h>
#include <stddef.h>
/* rsync-style filter rule engine (client-side file selection).
*
* Supported rule syntax (documented subset):
* [+|-] [anchored '/' prefix] pattern [trailing '/' for dir-only]
*
* "+ PATTERN" include rule (first match wins)
* "- PATTERN" exclude rule
* "PATTERN" implicit exclude rule (rsync default)
* "include PATTERN" / "exclude PATTERN" word forms
* leading '/' after the +/- anchors the pattern to its owner directory
* (the transfer root for command-line/-C rules, the directory that
* contains a .rsync-filter file for per-directory rules)
* a trailing '/' makes the rule match directories only
*
* Rejected explicitly (no silent no-ops): the rsync merge/dir-merge/list-clear
* shorthands written as a rule that starts with ':' or '.' or '!', the
* merge/dir-merge/hide/show/protect/risk/clear words, and every include/exclude
* rule modifier other than '/' (! C s r p x). The pattern must be separated
* from +/- by a space (or a single '/' anchor), exactly like rsync's
* "-s foo"/"-p ..." modifier syntax is refused.
*/
typedef enum {
FILTER_ACTION_NONE = 0, /* no rule matched */
FILTER_ACTION_EXCLUDE = -1,
FILTER_ACTION_INCLUDE = 1
} FilterAction;
typedef struct {
FilterAction action;
bool anchored; /* pattern anchored to the rule's owner directory */
bool dir_only; /* pattern had a trailing '/': matches directories only */
char* owner; /* owning directory rel path ("" == transfer root) */
char* pattern; /* cleaned glob pattern (no leading '/', no trailing '/') */
} FilterRule;
typedef struct {
FilterRule** items; /* owned array of rule pointers */
int count;
int capacity;
} FilterRuleList;
/* Parse a single filter-rule line (no trailing newline required). Returns an
* owned rule, or NULL on unsupported/invalid syntax with a message in `err`. */
FilterRule* filter_rule_parse(const char* line, char* err, size_t err_size);
void filter_rule_free(FilterRule* rule);
FilterRuleList* filter_rule_list_create(void);
/* Append a fully-parsed rule (takes ownership). Returns false on OOM. */
bool filter_rule_list_add(FilterRuleList* list, FilterRule* rule);
/* Parse `line` and append it. Returns false and fills `err` on bad syntax. */
bool filter_rule_list_parse_append(FilterRuleList* list, const char* line, char* err,
size_t err_size);
void filter_rule_list_free(FilterRuleList* list);
/* Build the command-line filter set: `rule_texts` (--filter=RULE in the order
* given, 0..rule_count) followed by the -C CVS default excludes when
* cvs_exclude is true. All rules are owned by "" (the transfer root).
* Returns NULL on unsupported rule text (message in `err`). */
FilterRuleList* filter_base_build(const char* const* rule_texts, int rule_count, bool cvs_exclude,
char* err, size_t err_size);
/* Read "<dir_path>/.rsync-filter" and return its rules, each owned by
* `owner_rel`. A missing file yields an empty list with *exists=false; an
* unreadable file is treated as missing. Returns NULL only on parse or
* allocation failure (message in `err`). */
FilterRuleList* filter_file_read(const char* dir_path, const char* owner_rel, bool* exists,
char* err, size_t err_size);
/* Evaluate an entry against one ordered rule list. Returns FILTER_ACTION_NONE
* when no rule matched, otherwise the first matching rule's action.
* `rel_path` is the entry's path relative to the transfer root ("" == root),
* `leaf` its final name, `is_dir` whether it is a directory. */
FilterAction filter_rules_apply(const FilterRuleList* list, const char* rel_path, const char* leaf,
bool is_dir);
#endif
+83 -1
View File
@@ -1,5 +1,6 @@
#include "log.h"
#include <errno.h>
#include <stdbool.h>
#include <stdarg.h>
#include <stdio.h>
#include <string.h>
@@ -7,16 +8,54 @@
static const char* log_level_strings[] = {"DEBUG", "INFO", "WARN", "ERROR"};
static LogLevel current_log_level = LOG_LEVEL_WARNING;
static uint32_t current_debug_flags = 0;
static uint32_t info_flags = 0;
static bool info_flags_explicit = false;
static FILE* log_fp = NULL;
static _Thread_local bool eight_bit_output;
static LogStderrMode stderr_mode = LOG_STDERR_ERRORS;
void set_log_level(LogLevel level) {
current_log_level = level;
}
void set_log_debug_flags(uint32_t flags) {
current_debug_flags = flags;
}
uint32_t get_log_debug_flags(void) {
return current_debug_flags;
}
void set_log_info_flags(uint32_t flags) {
info_flags = flags;
info_flags_explicit = true;
}
uint32_t get_log_info_flags(void) {
return info_flags;
}
void log_set_file(FILE* fp) {
log_fp = fp;
}
void log_set_8_bit_output(bool enabled) {
eight_bit_output = enabled;
}
bool log_get_8_bit_output(void) {
return eight_bit_output;
}
void log_set_stderr_mode(LogStderrMode mode) {
stderr_mode = mode;
}
LogStderrMode log_get_stderr_mode(void) {
return stderr_mode;
}
static inline void write_message(FILE* dest_io, LogLevel log_level, struct tm t, const char* format,
va_list args) {
fprintf(dest_io, "%04d-%02d-%02d %02d:%02d:%02d [%s]: ", t.tm_year + 1900, t.tm_mon + 1,
@@ -37,7 +76,7 @@ void log_message(LogLevel log_level, const char* format, ...) {
return;
FILE* dest_io = stdout;
if (log_level == LOG_LEVEL_ERROR) {
if (stderr_mode == LOG_STDERR_ALL || log_level == LOG_LEVEL_ERROR) {
dest_io = stderr;
}
@@ -53,6 +92,49 @@ void log_message(LogLevel log_level, const char* format, ...) {
}
}
void log_debug_message(LogDebugFlag flag, const char* format, ...) {
if (current_log_level > LOG_LEVEL_DEBUG || !(current_debug_flags & flag))
return;
time_t now = time(NULL);
struct tm t;
if (!localtime_r(&now, &t))
return;
va_list args;
va_start(args, format);
write_message(stdout, LOG_LEVEL_DEBUG, t, format, args);
va_end(args);
if (log_fp) {
va_start(args, format);
write_message(log_fp, LOG_LEVEL_DEBUG, t, format, args);
va_end(args);
}
}
void log_info_message(LogInfoFlag flag, const char* format, ...) {
if ((info_flags_explicit && (info_flags & flag) == 0) ||
(!info_flags_explicit && current_log_level > LOG_LEVEL_DEBUG))
return;
time_t now = time(NULL);
struct tm t;
if (!localtime_r(&now, &t))
return;
va_list args;
va_start(args, format);
write_message(stdout, LOG_LEVEL_INFO, t, format, args);
va_end(args);
if (log_fp) {
va_start(args, format);
write_message(log_fp, LOG_LEVEL_INFO, t, format, args);
va_end(args);
}
}
void log_perror(const char* context) {
log_message(LOG_LEVEL_ERROR, "%s: %s", context, strerror(errno));
}
+29
View File
@@ -2,12 +2,41 @@
#define LOG_H
#include <stdio.h>
#include <stdbool.h>
#include <stdint.h>
typedef enum { LOG_LEVEL_DEBUG, LOG_LEVEL_INFO, LOG_LEVEL_WARNING, LOG_LEVEL_ERROR } LogLevel;
typedef enum { LOG_STDERR_ERRORS, LOG_STDERR_ALL } LogStderrMode;
typedef enum {
LOG_DEBUG_IO = 1u << 0,
LOG_DEBUG_PROTO = 1u << 1,
LOG_DEBUG_PACK = 1u << 2,
LOG_DEBUG_UTIL = 1u << 3,
LOG_DEBUG_ALL = (1u << 4) - 1,
} LogDebugFlag;
typedef enum {
LOG_INFO_COPY = 1u << 0,
LOG_INFO_MISC = 1u << 1,
LOG_INFO_SKIP = 1u << 2,
LOG_INFO_STATS = 1u << 3,
LOG_INFO_ALL = LOG_INFO_COPY | LOG_INFO_MISC | LOG_INFO_SKIP | LOG_INFO_STATS,
} LogInfoFlag;
void log_message(LogLevel log_level, const char* message, ...);
void log_perror(const char* context);
void set_log_level(LogLevel level);
void set_log_debug_flags(uint32_t flags);
uint32_t get_log_debug_flags(void);
void log_debug_message(LogDebugFlag flag, const char* message, ...);
void set_log_info_flags(uint32_t flags);
uint32_t get_log_info_flags(void);
void log_info_message(LogInfoFlag flag, const char* message, ...);
void log_set_file(FILE* fp);
void log_set_8_bit_output(bool enabled);
bool log_get_8_bit_output(void);
void log_set_stderr_mode(LogStderrMode mode);
LogStderrMode log_get_stderr_mode(void);
#endif
+56 -10
View File
@@ -2,6 +2,7 @@
#include "file.h"
#include "log.h"
#include "protocol.h"
#include "utils.h"
#include <errno.h>
#include <fcntl.h>
#include <stdint.h>
@@ -24,6 +25,29 @@ typedef char static_assert_mode_t_fits[(sizeof(mode_t) <= sizeof(int32_t)) ? 1 :
typedef char static_assert_uid_t_fits[(sizeof(uid_t) <= sizeof(int32_t)) ? 1 : -1];
typedef char static_assert_gid_t_fits[(sizeof(gid_t) <= sizeof(int32_t)) ? 1 : -1];
bool metadata_mtime_matches(time_t left_sec, long left_nsec, time_t right_sec, long right_nsec,
int modify_window) {
int64_t left = (int64_t)left_sec;
int64_t right = (int64_t)right_sec;
int64_t seconds;
int64_t nanoseconds;
if (left > right || (left == right && left_nsec >= right_nsec)) {
seconds = left - right;
nanoseconds = (int64_t)left_nsec - (int64_t)right_nsec;
} else {
seconds = right - left;
nanoseconds = (int64_t)right_nsec - (int64_t)left_nsec;
}
if (nanoseconds < 0) {
seconds--;
nanoseconds += 1000000000LL;
}
if (modify_window == 0)
return left == right;
return seconds < modify_window || (seconds == modify_window && nanoseconds == 0);
}
void metadata_to_buf(char** buf, const FileMetadata* m) {
int32_t present = (m != NULL) ? 1 : 0;
memcpy(*buf, &present, sizeof(present));
@@ -55,7 +79,7 @@ FileMetadata* metadata_from_buf(char** buf) {
return NULL;
if (!present)
return NULL;
FileMetadata* m = malloc(sizeof(FileMetadata));
FileMetadata* m = protocol_alloc(sizeof(FileMetadata));
if (m == NULL)
return NULL;
int32_t mode;
@@ -122,7 +146,7 @@ FileMetadata* metadata_receive(int file_descriptor, int* ok) {
*ok = 0;
return NULL;
}
FileMetadata* m = malloc(sizeof(FileMetadata));
FileMetadata* m = protocol_alloc(sizeof(FileMetadata));
if (m == NULL) {
if (ok)
*ok = 0;
@@ -179,12 +203,27 @@ FileMetadata* metadata_receive(int file_descriptor, int* ok) {
return m;
}
void file_restore_metadata(const char* path, const FileMetadata* metadata) {
static mode_t metadata_mode(const FileMetadata* metadata, mode_t current_mode,
bool preserve_executability) {
const mode_t execute_bits = S_IXUSR | S_IXGRP | S_IXOTH;
if (preserve_executability)
return (current_mode & 0777 & ~execute_bits) | (metadata->mode & execute_bits);
return metadata->mode & 0777 & ~(S_IWGRP | S_IWOTH);
}
void file_restore_metadata(const char* path, const FileMetadata* metadata,
bool preserve_executability) {
if (metadata == NULL)
return;
mode_t safe_mode = metadata->mode & 0777 & ~(S_IWGRP | S_IWOTH);
if (chmod(path, safe_mode) != 0)
log_message(LOG_LEVEL_WARNING, "Failed to chmod %s: %s", path, strerror(errno));
struct stat current;
mode_t current_mode = stat(path, &current) == 0 ? current.st_mode : 0;
mode_t safe_mode = metadata_mode(metadata, current_mode, preserve_executability);
if (chmod(path, safe_mode) != 0) {
char* escaped_path = output_escape(path, log_get_8_bit_output());
log_message(LOG_LEVEL_WARNING, "Failed to chmod %s: %s",
escaped_path ? escaped_path : "<allocation failed>", strerror(errno));
free(escaped_path);
}
/* Never apply client-supplied ownership. The descriptor API below is the
receiver write path; retain this legacy API only for compatibility. */
struct timespec times[2];
@@ -192,15 +231,22 @@ void file_restore_metadata(const char* path, const FileMetadata* metadata) {
times[0].tv_nsec = UTIME_OMIT;
times[1].tv_sec = metadata->mtime_sec;
times[1].tv_nsec = metadata->mtime_nsec;
if (utimensat(AT_FDCWD, path, times, 0) != 0)
log_message(LOG_LEVEL_WARNING, "Failed to set timestamps on %s: %s", path, strerror(errno));
if (utimensat(AT_FDCWD, path, times, 0) != 0) {
char* escaped_path = output_escape(path, log_get_8_bit_output());
log_message(LOG_LEVEL_WARNING, "Failed to set timestamps on %s: %s",
escaped_path ? escaped_path : "<allocation failed>", strerror(errno));
free(escaped_path);
}
}
bool file_restore_metadata_fd(int fd, const FileMetadata* metadata) {
bool file_restore_metadata_fd(int fd, const FileMetadata* metadata, bool preserve_executability) {
if (fd < 0 || metadata == NULL)
return metadata == NULL;
bool ok = true;
mode_t safe_mode = metadata->mode & 0777 & ~(S_IWGRP | S_IWOTH);
struct stat current;
if (fstat(fd, &current) != 0)
return false;
mode_t safe_mode = metadata_mode(metadata, current.st_mode, preserve_executability);
if (fchmod(fd, safe_mode) != 0)
ok = false;
/* Client uid/gid values are deliberately not authoritative. */
+8 -2
View File
@@ -5,6 +5,7 @@
#include <stdbool.h>
#include <stdint.h>
#include <sys/stat.h>
#include <time.h>
/*
* Wire format (introduced in protocol version 2.0.0):
@@ -29,7 +30,12 @@ void metadata_to_buf(char** buf, const FileMetadata* m);
FileMetadata* metadata_from_buf(char** buf);
bool metadata_send(int file_descriptor, const FileMetadata* m);
FileMetadata* metadata_receive(int file_descriptor, int* ok);
void file_restore_metadata(const char* path, const FileMetadata* metadata);
bool file_restore_metadata_fd(int fd, const FileMetadata* metadata);
void file_restore_metadata(const char* path, const FileMetadata* metadata,
bool preserve_executability);
bool file_restore_metadata_fd(int fd, const FileMetadata* metadata, bool preserve_executability);
/* Compare timestamps using rsync's whole-second modification window. */
bool metadata_mtime_matches(time_t left_sec, long left_nsec, time_t right_sec, long right_nsec,
int modify_window);
#endif
+119 -10
View File
@@ -26,9 +26,17 @@ PipelineContextSender* pipeline_context_sender_create(Config* config, Queue* que
context->scanner_done = false;
context->loader_done = false;
context->manifest = NULL;
context->excluded_paths = NULL;
context->scan_had_io_error = false;
context->remove_source_files = NULL;
context->early_delete = false;
context->total_files = 0;
context->progress_bytes = 0;
context->total_bytes = 0;
context->sender_done = false;
atomic_init(&context->cancelled, false);
protocol_session_init(&context->allocation_session, -1, -1);
protocol_session_set_max_alloc(&context->allocation_session, config->max_alloc);
int init = 0;
if (mtx_init(&context->mutex_scanner, mtx_plain) != thrd_success)
goto fail;
@@ -76,6 +84,10 @@ void pipeline_context_sender_destroy(PipelineContextSender* context) {
if (context->manifest) {
array_list_delete(context->manifest);
}
if (context->excluded_paths)
array_list_delete(context->excluded_paths);
if (context->remove_source_files)
array_list_delete(context->remove_source_files);
config_delete(context->config);
queue_destroy(context->queue_scanner);
queue_destroy(context->queue_loader);
@@ -98,9 +110,15 @@ PipelineContextReceiver* pipeline_context_receiver_create(Config* config, Queue*
context->queue = queue;
context->file_descriptor = file_descriptor;
context->ssl = ssl;
context->outcomes.entries = NULL;
context->outcomes.count = 0;
context->outcomes.capacity = 0;
protocol_session_init(&context->session, file_descriptor, file_descriptor);
protocol_session_set_ssl(&context->session, ssl);
context->receiver_done = false;
context->queued_bytes = 0;
context->max_queue_bytes = 0;
context->deferred_manifest = NULL;
atomic_init(&context->cancelled, false);
int init = 0;
if (mtx_init(&context->mutex, mtx_plain) != thrd_success)
@@ -129,21 +147,83 @@ fail:
void pipeline_context_receiver_destroy(PipelineContextReceiver* context) {
config_delete(context->config);
if (context->deferred_manifest)
delete_manifest_free(context->deferred_manifest);
queue_destroy(context->queue);
receiver_outcomes_destroy(&context->outcomes);
mtx_destroy(&context->mutex);
cnd_destroy(&context->condition_not_full);
cnd_destroy(&context->condition_not_empty);
free(context);
}
void pipeline_context_receiver_set_queue_byte_limit(PipelineContextReceiver* context,
size_t max_bytes) {
if (context == NULL)
return;
mtx_lock(&context->mutex);
context->max_queue_bytes = max_bytes;
context->queued_bytes = 0;
cnd_broadcast(&context->condition_not_full);
mtx_unlock(&context->mutex);
}
void pipeline_context_receiver_note_bytes_released(PipelineContextReceiver* context,
size_t released_bytes) {
if (context == NULL || context->max_queue_bytes == 0 || released_bytes == 0)
return;
mtx_lock(&context->mutex);
if (released_bytes >= context->queued_bytes)
context->queued_bytes = 0;
else
context->queued_bytes -= released_bytes;
cnd_signal(&context->condition_not_full);
mtx_unlock(&context->mutex);
}
bool pipeline_context_receiver_enqueue_file(PipelineContextReceiver* context, File* file) {
if (context == NULL || file == NULL)
return false;
size_t file_bytes = file->data ? file->data->size : 0;
mtx_lock(&context->mutex);
while (!atomic_load(&context->cancelled)) {
bool blocked_by_count = queue_is_full(context->queue);
bool blocked_by_budget = false;
if (context->max_queue_bytes > 0) {
size_t budget = context->max_queue_bytes;
size_t used = context->queued_bytes;
if (used >= budget) {
blocked_by_budget = true;
} else if (file_bytes > budget - used) {
/* A single payload larger than the whole budget (not possible with
the per-file receive cap) is only admitted to an empty pipeline so
the wait can never deadlock. */
blocked_by_budget = used != 0;
}
}
if (!blocked_by_count && !blocked_by_budget)
break;
cnd_wait(&context->condition_not_full, &context->mutex);
}
if (atomic_load(&context->cancelled)) {
mtx_unlock(&context->mutex);
file_destroy(file);
return false;
}
if (!queue_enqueue(context->queue, file)) {
mtx_unlock(&context->mutex);
file_destroy(file);
return false;
}
context->queued_bytes += file_bytes;
cnd_signal(&context->condition_not_empty);
mtx_unlock(&context->mutex);
return true;
}
static bool receiver_enqueue_file(File* file, void* context_pointer) {
PipelineContextReceiver* context = context_pointer;
if (queue_enqueue_multithreaded_cancel(context->queue, file, &context->mutex,
&context->condition_not_empty,
&context->condition_not_full, &context->cancelled))
return true;
file_destroy(file);
return false;
PipelineContextReceiver* context = (PipelineContextReceiver*)context_pointer;
return pipeline_context_receiver_enqueue_file(context, file);
}
static void receiver_thread_fail(PipelineContextReceiver* context) {
@@ -163,8 +243,9 @@ int receive_thread(void* pipeline_context) {
const Config* config = context->config;
mtx_unlock(&context->mutex);
ReceiverSink sink = {receiver_enqueue_file, context, false, false};
if (receiver_process((Config*)config, file_descriptor, &sink) != 0) {
ReceiverSink sink = {receiver_enqueue_file, context, false, false, NULL};
if (receiver_process_pending((Config*)config, file_descriptor, &sink,
&context->deferred_manifest) != 0) {
receiver_thread_fail(context);
protocol_session_unbind();
return thrd_error;
@@ -179,6 +260,7 @@ int receive_thread(void* pipeline_context) {
int write_thread(void* pipeline_context) {
PipelineContextReceiver* context = (PipelineContextReceiver*)pipeline_context;
protocol_session_bind(&context->session);
mtx_lock(&context->mutex);
bool save_to_disk = context->config->save_to_disk;
char* root_directory = str_dup(context->config->receive_root_directory);
@@ -190,6 +272,7 @@ int write_thread(void* pipeline_context) {
cnd_broadcast(&context->condition_not_full);
cnd_broadcast(&context->condition_not_empty);
mtx_unlock(&context->mutex);
protocol_session_unbind();
return thrd_error;
}
@@ -199,10 +282,34 @@ int write_thread(void* pipeline_context) {
&context->condition_not_full, &context->receiver_done);
if (file == NULL) {
free(root_directory);
protocol_session_unbind();
return thrd_success;
}
if (save_to_disk && !file_save_to_disk(root_directory, file, context->config)) {
size_t file_bytes = file->data ? file->data->size : 0;
FileSaveResult result = FILE_SAVE_SKIPPED;
if (save_to_disk) {
result = file_save_to_disk_full(root_directory, file, context->config);
if (result == FILE_SAVE_ERROR) {
file_destroy(file);
pipeline_context_receiver_note_bytes_released(context, file_bytes);
mtx_lock(&context->mutex);
atomic_store(&context->cancelled, true);
context->receiver_done = true;
cnd_broadcast(&context->condition_not_full);
cnd_broadcast(&context->condition_not_empty);
mtx_unlock(&context->mutex);
free(root_directory);
protocol_session_unbind();
return thrd_error;
}
}
/* Record the per-file outcome so a --remove-source-files sender learns
which sources were actually written versus skipped on the receiver.
Explicit directory entries have no source and are never acknowledged. */
if (context->config->remove_source_files && !file->is_dir && !file->skip &&
!receiver_outcomes_append(&context->outcomes, (unsigned char)result)) {
file_destroy(file);
pipeline_context_receiver_note_bytes_released(context, file_bytes);
mtx_lock(&context->mutex);
atomic_store(&context->cancelled, true);
context->receiver_done = true;
@@ -210,8 +317,10 @@ int write_thread(void* pipeline_context) {
cnd_broadcast(&context->condition_not_empty);
mtx_unlock(&context->mutex);
free(root_directory);
protocol_session_unbind();
return thrd_error;
}
file_destroy(file);
pipeline_context_receiver_note_bytes_released(context, file_bytes);
}
}
+51
View File
@@ -9,6 +9,7 @@
#include "file.h"
#include "protocol.h"
#include "queue.h"
#include "receiver.h"
#include <openssl/ssl.h>
typedef struct {
@@ -24,10 +25,31 @@ typedef struct {
cnd_t condition_not_empty_loader;
bool loader_done;
ArrayList* manifest;
/* Protected prefixes (paths the source scan excluded by user rules) sent
with the keep-set manifest so --delete leaves them alone unless
--delete-excluded is set. NULL when not collecting. Populated by the
scanner thread (parallel workers append under mutex_scanner via the
scanner's exclusion sink) or, in the early modes, by the path-only pre-scan
on the calling thread before the pipeline starts. */
ArrayList* excluded_paths;
/* A source I/O error (unreadable directory) was recorded during the scan.
Set by the pre-scan (before the threads start) or by the scanner thread
under mutex_scanner; the caller turns it into a non-zero exit when
--ignore-errors kept the run going. */
bool scan_had_io_error;
ArrayList* remove_source_files;
/* True when --delete-before/--delete-during require the keep-set manifest to
be transmitted before any file data: context->manifest is then prebuilt by
a path-only pre-scan on the calling thread and the pipeline scanner must
not append to it. Set once before the worker threads start. */
bool early_delete;
mtx_t mutex_progress;
int total_files;
unsigned long long progress_bytes;
unsigned long long total_bytes;
bool sender_done;
atomic_bool cancelled;
ProtocolSession allocation_session;
} PipelineContextSender;
typedef struct PipelineContextReceiver {
@@ -36,11 +58,28 @@ typedef struct PipelineContextReceiver {
int file_descriptor;
SSL* ssl;
ProtocolSession session;
ReceiverOutcomes outcomes;
mtx_t mutex;
cnd_t condition_not_full;
cnd_t condition_not_empty;
bool receiver_done;
atomic_bool cancelled;
/* Aggregate payload bytes that have been received but not yet released by
the disk writer (queued or in the writer's hand). Guarded by `mutex`.
When `max_queue_bytes` is non-zero the receiver blocks before enqueuing
once this total would exceed it, so decompressed/copied file payloads
buffered ahead of a slow disk writer respect the per-connection memory
budget instead of growing without bound. */
size_t queued_bytes;
size_t max_queue_bytes;
/* Keep-set manifest for the commit-style (late) deletion
(--delete/--delete-after/--delete-delay). receive_thread parses the whole
protocol stream but hands the manifest here instead of deleting while the
disk writer may still be draining; the caller (server.c) commits the
deletion after both threads have joined, so no extra is removed unless the
transfer truly succeeded. NULL in the early delete modes (which delete at
the manifest). */
DeleteManifest* deferred_manifest;
} PipelineContextReceiver;
PipelineContextSender* pipeline_context_sender_create(Config* config, Queue* queue_scanner,
@@ -49,6 +88,18 @@ void pipeline_context_sender_destroy(PipelineContextSender* context);
PipelineContextReceiver* pipeline_context_receiver_create(Config* config, Queue* queue_receiver,
int file_descriptor, SSL* ssl);
void pipeline_context_receiver_destroy(PipelineContextReceiver* context);
/* Bound the bytes buffered ahead of the disk writer (see max_queue_bytes). */
void pipeline_context_receiver_set_queue_byte_limit(PipelineContextReceiver* context,
size_t max_bytes);
/* Blocking enqueue used by the receive pipeline sink. Blocks while the queue
is full by element count or when adding `file` would push queued_bytes over
the configured byte limit; waits until the disk writer releases bytes.
Takes ownership of `file` on success and destroys it on failure/cancel. */
bool pipeline_context_receiver_enqueue_file(PipelineContextReceiver* context, File* file);
/* Account for `released_bytes` of payload memory that has been freed by the
disk writer, unblocking a receiver that is waiting on the byte limit. */
void pipeline_context_receiver_note_bytes_released(PipelineContextReceiver* context,
size_t released_bytes);
int receive_thread(void* pipeline_context);
int write_thread(void* pipeline_context);
#endif
+133 -35
View File
@@ -1,5 +1,6 @@
#include "protocol.h"
#include "log.h"
#include "utils.h"
#include <errno.h>
#include <limits.h>
#include <openssl/ssl.h>
@@ -13,13 +14,13 @@
#define RECEIVE_TIMEOUT_SEC 60 /* 60 second per-message timeout */
#define SEND_TIMEOUT_SEC 60
#define MAX_CONNECTION_MEMORY (256ULL * 1024 * 1024) /* bounded cumulative receive budget */
static __thread int io_read_fd = -1;
static __thread int io_write_fd = -1;
static __thread SSL* io_ssl;
static __thread ProtocolSession* bound_session;
static __thread ProtocolSession legacy_io_session = {.read_fd = -1, .write_fd = -1};
static __thread ProtocolSession legacy_io_session = {
.read_fd = -1, .write_fd = -1, .max_alloc = DEFAULT_MAX_ALLOC};
static unsigned long long io_bwlimit = 0;
static mtx_t bw_mutex;
@@ -27,12 +28,30 @@ static once_flag bw_mutex_once = ONCE_FLAG_INIT;
static unsigned long long global_bwlimit(void);
static bool protocol_reserve_memory(ProtocolSession* session, size_t charge) {
unsigned long long allocated = atomic_load(&session->total_allocated_bytes);
while (true) {
if (allocated > MAX_CONNECTION_MEMORY ||
(unsigned long long)charge > MAX_CONNECTION_MEMORY - allocated)
return false;
if (atomic_compare_exchange_weak(&session->total_allocated_bytes, &allocated,
allocated + (unsigned long long)charge))
return true;
}
}
static void protocol_release_memory_for_session(ProtocolSession* session, size_t charge) {
unsigned long long allocated = atomic_load(&session->total_allocated_bytes);
while (true) {
unsigned long long remaining = (unsigned long long)charge >= allocated ? 0 : allocated - charge;
if (atomic_compare_exchange_weak(&session->total_allocated_bytes, &allocated, remaining))
break;
}
}
void protocol_release_memory(size_t charge) {
ProtocolSession* session = bound_session ? bound_session : &legacy_io_session;
if ((unsigned long long)charge >= session->total_allocated_bytes)
session->total_allocated_bytes = 0;
else
session->total_allocated_bytes -= charge;
protocol_release_memory_for_session(session, charge);
}
void io_set_fds(int read_fd, int write_fd) {
bound_session = NULL;
@@ -44,7 +63,9 @@ void io_set_fds(int read_fd, int write_fd) {
legacy_io_session.read_fd = read_fd;
legacy_io_session.write_fd = write_fd;
legacy_io_session.ssl = NULL;
legacy_io_session.total_allocated_bytes = 0;
legacy_io_session.eight_bit_output = false;
atomic_store(&legacy_io_session.total_allocated_bytes, 0);
legacy_io_session.max_alloc = DEFAULT_MAX_ALLOC;
protocol_session_set_bwlimit(&legacy_io_session, global_bwlimit());
}
@@ -54,11 +75,46 @@ void protocol_session_init(ProtocolSession* session, int read_fd, int write_fd)
memset(session, 0, sizeof(*session));
session->read_fd = read_fd;
session->write_fd = write_fd;
session->max_alloc = DEFAULT_MAX_ALLOC;
atomic_init(&session->total_allocated_bytes, 0);
protocol_session_set_bwlimit(session, global_bwlimit());
}
void protocol_session_set_max_alloc(ProtocolSession* session, unsigned long long max_alloc) {
if (!session)
session = bound_session ? bound_session : &legacy_io_session;
session->max_alloc = max_alloc;
}
static bool allocation_allowed(const ProtocolSession* session, size_t size) {
return (unsigned long long)size <= session->max_alloc;
}
static void* protocol_alloc_for_session(const ProtocolSession* session, size_t size) {
if (!allocation_allowed(session, size))
return NULL;
return malloc(size);
}
static void* protocol_realloc_for_session(const ProtocolSession* session, void* ptr, size_t size) {
if (!allocation_allowed(session, size))
return NULL;
return realloc(ptr, size);
}
void* protocol_alloc(size_t size) {
const ProtocolSession* session = bound_session ? bound_session : &legacy_io_session;
return protocol_alloc_for_session(session, size);
}
void* protocol_realloc(void* ptr, size_t size) {
const ProtocolSession* session = bound_session ? bound_session : &legacy_io_session;
return protocol_realloc_for_session(session, ptr, size);
}
void protocol_session_bind(ProtocolSession* session) {
bound_session = session;
log_set_8_bit_output(session && session->eight_bit_output);
}
void protocol_session_unbind(void) {
@@ -103,6 +159,19 @@ void protocol_session_set_bwlimit(ProtocolSession* session, unsigned long long b
session->bw_last_refill_nsec = now.tv_nsec;
}
void protocol_session_set_8_bit_output(ProtocolSession* session, bool enabled) {
if (!session)
return;
session->eight_bit_output = enabled;
if (session == bound_session)
log_set_8_bit_output(enabled);
}
void protocol_set_8_bit_output(bool enabled) {
ProtocolSession* session = bound_session ? bound_session : &legacy_io_session;
protocol_session_set_8_bit_output(session, enabled);
}
static void bw_throttle_session(ProtocolSession* session, size_t bytes_written) {
if (session->bwlimit == 0)
return;
@@ -153,7 +222,8 @@ static ProtocolSession* legacy_session(int read_fd, int write_fd) {
legacy_io_session.write_fd != target_write_fd) {
legacy_io_session.read_fd = target_read_fd;
legacy_io_session.write_fd = target_write_fd;
legacy_io_session.total_allocated_bytes = 0;
atomic_store(&legacy_io_session.total_allocated_bytes, 0);
legacy_io_session.max_alloc = DEFAULT_MAX_ALLOC;
protocol_session_set_bwlimit(&legacy_io_session, global_bwlimit());
} else if (legacy_io_session.bwlimit != global_bwlimit()) {
protocol_session_set_bwlimit(&legacy_io_session, global_bwlimit());
@@ -184,7 +254,7 @@ static int deadline_remaining_ms(const struct timespec* deadline) {
bool protocol_send_n_data(ProtocolSession* session, const void* data, size_t data_size) {
if (!data && data_size != 0)
return false;
log_message(LOG_LEVEL_DEBUG, " Sending n Data: %zu", data_size);
log_debug_message(LOG_DEBUG_IO, " Sending n Data: %zu", data_size);
if (!session)
return false;
int fd = session->write_fd;
@@ -228,19 +298,29 @@ bool protocol_send_n_data(ProtocolSession* session, const void* data, size_t dat
if (session->ssl)
wait_events = POLLOUT;
}
log_message(LOG_LEVEL_DEBUG, " Send n Data: %zu", total_bytes_send);
log_debug_message(LOG_DEBUG_IO, " Send n Data: %zu", total_bytes_send);
return true;
}
bool protocol_receive_n_data_timed(ProtocolSession* session, void* data, size_t data_size,
int timeout_sec);
bool protocol_receive_n_data(ProtocolSession* session, void* data, size_t data_size) {
log_message(LOG_LEVEL_DEBUG, " Receiving n Data: %zu", data_size);
return protocol_receive_n_data_timed(session, data, data_size, RECEIVE_TIMEOUT_SEC);
}
bool protocol_receive_n_data_timed(ProtocolSession* session, void* data, size_t data_size,
int timeout_sec) {
log_debug_message(LOG_DEBUG_IO, " Receiving n Data: %zu", data_size);
if (!session)
return false;
int fd = session->read_fd;
if (timeout_sec <= 0)
timeout_sec = RECEIVE_TIMEOUT_SEC;
struct timespec deadline;
clock_gettime(CLOCK_MONOTONIC, &deadline);
deadline.tv_sec += RECEIVE_TIMEOUT_SEC;
deadline.tv_sec += timeout_sec;
size_t total_bytes_received = 0;
short wait_events = POLLIN;
@@ -249,7 +329,7 @@ bool protocol_receive_n_data(ProtocolSession* session, void* data, size_t data_s
struct pollfd pfd = {.fd = fd, .events = wait_events};
int poll_result = poll(&pfd, 1, deadline_remaining_ms(&deadline));
if (poll_result == 0) {
log_message(LOG_LEVEL_ERROR, "Receive timeout after %ds", RECEIVE_TIMEOUT_SEC);
log_message(LOG_LEVEL_ERROR, "Receive timeout after %ds", timeout_sec);
return false;
}
if (poll_result < 0) {
@@ -287,7 +367,7 @@ bool protocol_receive_n_data(ProtocolSession* session, void* data, size_t data_s
if (session->ssl)
wait_events = POLLIN;
}
log_message(LOG_LEVEL_DEBUG, " Received n Data: %zu", total_bytes_received);
log_debug_message(LOG_DEBUG_IO, " Received n Data: %zu", total_bytes_received);
return true;
}
@@ -315,6 +395,8 @@ static const char* status_to_string(Status status) {
return "ABORT";
case STATUS_CHECK_BATCH:
return "CHECK_BATCH";
case STATUS_MKDIR:
return "MKDIR";
default:
return "UNKNOWN";
}
@@ -328,7 +410,7 @@ bool protocol_send_str(ProtocolSession* session, const char* data) {
return false;
if (!protocol_send_n_data(session, data, size))
return false;
log_message(LOG_LEVEL_DEBUG, "Send String: %s", data);
log_debug_message(LOG_DEBUG_PROTO, "Send String: %s", data);
return true;
}
@@ -336,13 +418,12 @@ char* protocol_receive_str(ProtocolSession* session) {
size_t size;
if (!protocol_receive_n_data(session, &size, sizeof(size_t)))
return NULL;
if (size > MAX_STRING_SIZE || size > SIZE_MAX - 1 ||
size + 1 > MAX_CONNECTION_MEMORY - session->total_allocated_bytes) {
if (size > MAX_STRING_SIZE || size > SIZE_MAX - 1) {
log_message(LOG_LEVEL_ERROR, "String size %zu exceeds maximum %llu", size,
(unsigned long long)MAX_STRING_SIZE);
return NULL;
}
char* data = (char*)malloc(size + 1);
char* data = (char*)protocol_alloc_for_session(session, size + 1);
if (data == NULL)
return NULL;
if (!protocol_receive_n_data(session, data, size)) {
@@ -355,8 +436,7 @@ char* protocol_receive_str(ProtocolSession* session) {
return NULL;
}
data[size] = '\0';
session->total_allocated_bytes += size + 1;
log_message(LOG_LEVEL_DEBUG, "Received String: %s", data);
log_debug_message(LOG_DEBUG_PROTO, "Received String: %s", data);
return data;
}
@@ -370,7 +450,7 @@ bool protocol_send_data(ProtocolSession* session, const Data* data) {
return false;
if (!protocol_send_n_data(session, data->data, data_size))
return false;
log_message(LOG_LEVEL_DEBUG, "Send %lld data", data_size);
log_debug_message(LOG_DEBUG_PROTO, "Send %lld data", data_size);
return true;
}
@@ -385,25 +465,29 @@ Data* protocol_receive_data_limited(ProtocolSession* session, unsigned long long
(unsigned long long)MAX_DATA_PAYLOAD_SIZE);
return NULL;
}
if (size > SIZE_MAX)
return NULL;
size_t allocation_size = size == 0 ? 1 : (size_t)size;
if (allocation_size > MAX_CONNECTION_MEMORY - session->total_allocated_bytes) {
if (!protocol_reserve_memory(session, allocation_size)) {
log_message(LOG_LEVEL_ERROR, "Per-connection memory limit exceeded (%llu + %llu > %llu)",
(unsigned long long)session->total_allocated_bytes, size,
(unsigned long long)atomic_load(&session->total_allocated_bytes), size,
(unsigned long long)MAX_CONNECTION_MEMORY);
return NULL;
}
void* data = malloc(allocation_size);
if (data == NULL)
return NULL;
if (!protocol_receive_n_data(session, data, (size_t)size)) {
free(data);
void* data = protocol_alloc_for_session(session, allocation_size);
if (data == NULL) {
protocol_release_memory_for_session(session, allocation_size);
return NULL;
}
session->total_allocated_bytes += allocation_size;
log_message(LOG_LEVEL_DEBUG, "Received %lld data", size);
if (!protocol_receive_n_data(session, data, (size_t)size)) {
free(data);
protocol_release_memory_for_session(session, allocation_size);
return NULL;
}
log_debug_message(LOG_DEBUG_PROTO, "Received %lld data", size);
Data* result = data_create(data, (size_t)size);
if (!result) {
session->total_allocated_bytes -= allocation_size;
protocol_release_memory_for_session(session, allocation_size);
return NULL;
}
result->protocol_charge = allocation_size;
@@ -417,28 +501,39 @@ Data* protocol_receive_data(ProtocolSession* session) {
bool protocol_send_int(ProtocolSession* session, int data) {
if (!protocol_send_n_data(session, &data, sizeof(int)))
return false;
log_message(LOG_LEVEL_DEBUG, "Send Int: %d", data);
log_debug_message(LOG_DEBUG_PROTO, "Send Int: %d", data);
return true;
}
bool protocol_receive_int(ProtocolSession* session, int* data) {
if (!protocol_receive_n_data(session, data, sizeof(int)))
return false;
log_message(LOG_LEVEL_DEBUG, "Received Int: %d", *data);
log_debug_message(LOG_DEBUG_PROTO, "Received Int: %d", *data);
return true;
}
bool protocol_send_status(ProtocolSession* session, Status status) {
if (!protocol_send_n_data(session, &status, sizeof(Status)))
return false;
log_message(LOG_LEVEL_DEBUG, "Send Status: %s", status_to_string(status));
log_debug_message(LOG_DEBUG_PROTO, "Send Status: %s", status_to_string(status));
return true;
}
bool protocol_receive_status(ProtocolSession* session, Status* status) {
if (!protocol_receive_n_data(session, status, sizeof(Status)))
return false;
log_message(LOG_LEVEL_DEBUG, "Received Status: %s", status_to_string(*status));
log_debug_message(LOG_DEBUG_PROTO, "Received Status: %s", status_to_string(*status));
return true;
}
/* protocol_receive_status with an explicit per-message deadline (seconds).
Used where a single reply may legitimately take far longer than the default
60 s receive window - e.g. the sender waiting for the early-delete ACK after
the receiver committed a large (up to MAX_SERVER_DELETE_COUNT) deletion. */
bool protocol_receive_status_timed(ProtocolSession* session, Status* status, int timeout_sec) {
if (!protocol_receive_n_data_timed(session, status, sizeof(Status), timeout_sec))
return false;
log_debug_message(LOG_DEBUG_PROTO, "Received Status: %s", status_to_string(*status));
return true;
}
@@ -469,3 +564,6 @@ bool send_status(int fd, Status status) {
bool receive_status(int fd, Status* status) {
return protocol_receive_status(legacy_session(fd, -1), status);
}
bool receive_status_timed(int fd, Status* status, int timeout_sec) {
return protocol_receive_status_timed(legacy_session(fd, -1), status, timeout_sec);
}
+35 -6
View File
@@ -4,20 +4,34 @@
#include "data.h"
#include <stdbool.h>
#include <stddef.h>
#include <stdatomic.h>
/* Maximum allowed string size for receive_str (64 KB) */
#define MAX_STRING_SIZE (64 * 1024)
/* Maximum allowed data payload size for receive_data (100 MB) */
#define MAX_DATA_PAYLOAD_SIZE (100ULL * 1024 * 1024)
/* Maximum uncompressed file payload accepted by the receiver. */
#define MAX_RECEIVE_FILE_SIZE (64ULL * 1024 * 1024)
/* Maximum uncompressed file payload accepted by the receiver's whole-file
* paths. A single whole file is charged against the per-connection memory
* reservation (MAX_CONNECTION_MEMORY) and against the server allocation
* ceiling (MAX_SERVER_ALLOC), so this mirrors those 256 MB bounds rather than
* the older 64 MB chunk-era cap. Chunk-serialized payloads keep their own
* 64 MB cap (MAX_CHUNK_SIZE). */
#define MAX_RECEIVE_WHOLE_FILE_SIZE (256ULL * 1024 * 1024)
/* Maximum allowed data payload size for receive_data (whole-file bound) */
#define MAX_DATA_PAYLOAD_SIZE MAX_RECEIVE_WHOLE_FILE_SIZE
/* Maximum chunk size (64 MB) — prevents unbounded allocation from the wire */
#define MAX_CHUNK_SIZE (64ULL * 1024 * 1024)
#define MAX_MANIFEST_ENTRIES (1024 * 1024)
/* Aggregate bytes retained by one received deletion manifest. */
#define MAX_MANIFEST_BYTES (16ULL * 1024 * 1024)
#define DEFAULT_MAX_ALLOC (1ULL * 1024 * 1024 * 1024)
/* Server policy ceiling for a client-provided allocation limit. */
#define MAX_SERVER_ALLOC (256ULL * 1024 * 1024)
/* Bounded cumulative per-connection receive budget. In-flight wire buffers,
decompression buffers and queued (not yet written) file payloads for a
connection must stay within this ceiling. */
#define MAX_CONNECTION_MEMORY (256ULL * 1024 * 1024)
typedef struct ssl_st SSL;
@@ -35,7 +49,9 @@ typedef struct ProtocolSession {
long long bw_tokens;
long long bw_last_refill_sec;
long bw_last_refill_nsec;
unsigned long long total_allocated_bytes;
atomic_ullong total_allocated_bytes;
bool eight_bit_output;
unsigned long long max_alloc;
} ProtocolSession;
typedef int Status;
@@ -51,7 +67,10 @@ enum NET_STATUS {
STATUS_DELTA_DATA,
STATUS_KEEPALIVE,
STATUS_ABORT,
STATUS_CHECK_BATCH
STATUS_CHECK_BATCH,
/* An explicit directory entry (--dirs): the sender transmits only the path;
* the receiver creates the directory below the receive root. */
STATUS_MKDIR
};
void io_set_fds(int read_fd, int write_fd);
@@ -65,6 +84,11 @@ void protocol_session_bind(ProtocolSession* session);
void protocol_session_unbind(void);
void protocol_session_set_ssl(ProtocolSession* session, SSL* ssl);
void protocol_session_set_bwlimit(ProtocolSession* session, unsigned long long bytes_per_sec);
void protocol_session_set_max_alloc(ProtocolSession* session, unsigned long long max_alloc);
void* protocol_alloc(size_t size);
void* protocol_realloc(void* ptr, size_t size);
void protocol_session_set_8_bit_output(ProtocolSession* session, bool enabled);
void protocol_set_8_bit_output(bool enabled);
bool protocol_send_n_data(ProtocolSession* session, const void* data, size_t data_size);
bool protocol_receive_n_data(ProtocolSession* session, void* data, size_t data_size);
bool protocol_send_str(ProtocolSession* session, const char* data);
@@ -88,5 +112,10 @@ bool send_int(int file_descriptor, int data);
bool receive_int(int file_descriptor, int* data);
bool send_status(int file_descriptor, Status status);
bool receive_status(int file_descriptor, Status* status);
/* receive_status with an explicit per-message deadline in seconds, instead of
the default RECEIVE_TIMEOUT_SEC. A reply that may legitimately take longer
(e.g. the early-delete ACK after a large receiver-side deletion) must use
this so the sender does not abort after the deletion already committed. */
bool receive_status_timed(int file_descriptor, Status* status, int timeout_sec);
#endif
+73 -15
View File
@@ -3,6 +3,7 @@
#include "utils.h"
#include <fcntl.h>
#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>
#include <string.h>
#include <sys/socket.h>
@@ -15,6 +16,12 @@ typedef struct {
char* remote_path;
} RemoteDest;
static void ssh_child_setup_failed(int status_fd) {
ssize_t wret = write(status_fd, "x", 1);
(void)wret;
_exit(1);
}
static void remote_dest_destroy(RemoteDest* r) {
free(r->user);
free(r->host);
@@ -68,10 +75,57 @@ static int parse_remote_dest(const char* dest, RemoteDest* r) {
return 0;
}
Client* client_connect_ssh(const char* destination, int port, const char* server_path) {
char* ssh_build_remote_command(const char* server_path, bool old_args) {
const char* path = server_path ? server_path : "fastsync-server";
const char* suffix = " --stdio";
size_t path_len = strlen(path);
size_t suffix_len = strlen(suffix);
if (old_args) {
if (path_len > SIZE_MAX - suffix_len - 1)
return NULL;
char* command = malloc(path_len + suffix_len + 1);
if (!command)
return NULL;
memcpy(command, path, path_len);
memcpy(command + path_len, suffix, suffix_len + 1);
return command;
}
/* Quote the executable as one remote-shell word. This is the default safety boundary. */
size_t quote_count = 0;
for (const char* p = path; *p; p++)
if (*p == '\'')
quote_count++;
if (path_len > SIZE_MAX - suffix_len - 4 ||
quote_count > (SIZE_MAX - path_len - suffix_len - 4) / 4)
return NULL;
size_t command_len = path_len + quote_count * 4 + suffix_len + 4;
char* command = malloc(command_len + 1);
if (!command)
return NULL;
char* out = command;
*out++ = '\'';
for (const char* p = path; *p; p++) {
if (*p == '\'') {
memcpy(out, "'\\''", 4);
out += 4;
} else {
*out++ = *p;
}
}
*out++ = '\'';
memcpy(out, suffix, suffix_len + 1);
return command;
}
Client* client_connect_ssh(const char* destination, int port, const char* server_path,
bool old_args) {
RemoteDest r;
if (parse_remote_dest(destination, &r) != 0) {
fprintf(stderr, "Invalid remote destination: %s\n", destination);
char* escaped = output_escape(destination, false);
fprintf(stderr, "Invalid remote destination: %s\n", escaped ? escaped : "<allocation failed>");
free(escaped);
return NULL;
}
@@ -111,11 +165,12 @@ Client* client_connect_ssh(const char* destination, int port, const char* server
if (pid == 0) {
close(sv[0]);
close(exec_pipe[0]);
fcntl(exec_pipe[1], F_SETFD, FD_CLOEXEC);
if (sv[1] != STDIN_FILENO)
dup2(sv[1], STDIN_FILENO);
if (sv[1] != STDOUT_FILENO)
dup2(sv[1], STDOUT_FILENO);
if (fcntl(exec_pipe[1], F_SETFD, FD_CLOEXEC) < 0)
ssh_child_setup_failed(exec_pipe[1]);
if (sv[1] != STDIN_FILENO && dup2(sv[1], STDIN_FILENO) < 0)
ssh_child_setup_failed(exec_pipe[1]);
if (sv[1] != STDOUT_FILENO && dup2(sv[1], STDOUT_FILENO) < 0)
ssh_child_setup_failed(exec_pipe[1]);
if (sv[1] > 1)
close(sv[1]);
@@ -126,7 +181,7 @@ Client* client_connect_ssh(const char* destination, int port, const char* server
ssh_user_len = strlen(r.host) + 1;
char* ssh_user = malloc(ssh_user_len);
if (!ssh_user)
_exit(1);
ssh_child_setup_failed(exec_pipe[1]);
if (r.user && r.user[0] != '\0')
snprintf(ssh_user, ssh_user_len, "%s@%s", r.user, r.host);
else
@@ -135,6 +190,9 @@ Client* client_connect_ssh(const char* destination, int port, const char* server
char* ssh_argv[16];
int ac = 0;
char port_str[16];
char* remote_command = ssh_build_remote_command(server_path, old_args);
if (!remote_command)
ssh_child_setup_failed(exec_pipe[1]);
ssh_argv[ac++] = "ssh";
ssh_argv[ac++] = "-o";
ssh_argv[ac++] = "Compression=no";
@@ -148,14 +206,11 @@ Client* client_connect_ssh(const char* destination, int port, const char* server
ssh_argv[ac++] = port_str;
}
ssh_argv[ac++] = ssh_user;
ssh_argv[ac++] = (char*)(server_path ? server_path : "fastsync-server");
ssh_argv[ac++] = "--stdio";
ssh_argv[ac++] = remote_command;
ssh_argv[ac] = NULL;
execvp("ssh", ssh_argv);
log_perror("exec of ssh failed");
ssize_t wret = write(exec_pipe[1], "x", 1);
(void)wret;
_exit(1);
ssh_child_setup_failed(exec_pipe[1]);
}
close(sv[1]);
@@ -165,12 +220,15 @@ Client* client_connect_ssh(const char* destination, int port, const char* server
ssize_t n = read(exec_pipe[0], &exec_status, 1);
close(exec_pipe[0]);
if (n > 0) {
if (n != 0) {
close(sv[0]);
waitpid(pid, NULL, 0);
remote_dest_destroy(&r);
const char* path = server_path ? server_path : "fastsync-server";
char* escaped = output_escape(path, false);
fprintf(stderr, "Error: could not launch '%s --stdio' on remote\n",
server_path ? server_path : "fastsync-server");
escaped ? escaped : "<allocation failed>");
free(escaped);
return NULL;
}
+3 -1
View File
@@ -3,6 +3,8 @@
#include "transport_tcp.h"
Client* client_connect_ssh(const char* destination, int port, const char* server_path);
Client* client_connect_ssh(const char* destination, int port, const char* server_path,
bool old_args);
char* ssh_build_remote_command(const char* server_path, bool old_args);
#endif
+5 -1
View File
@@ -1,6 +1,7 @@
#include "transport_tcp.h"
#include "log.h"
#include "protocol.h"
#include "utils.h"
#include <arpa/inet.h>
#include <errno.h>
#include <netdb.h>
@@ -190,7 +191,10 @@ bool tcp_connect_socket(Client* client, char* host, int port) {
int err = getaddrinfo(host, port_str, &hints, &result);
if (err != 0 || result == NULL) {
fprintf(stderr, "Could not resolve host: %s (%s)\n", host, gai_strerror(err));
char* escaped_host = output_escape(host, false);
fprintf(stderr, "Could not resolve host: %s (%s)\n",
escaped_host ? escaped_host : "<allocation failed>", gai_strerror(err));
free(escaped_host);
return false;
}
+13 -3
View File
@@ -2,6 +2,7 @@
#include "log.h"
#include "protocol.h"
#include "transport_tcp.h"
#include "utils.h"
#include <arpa/inet.h>
#include <openssl/err.h>
#include <openssl/ssl.h>
@@ -65,13 +66,19 @@ static SSL_CTX* create_ssl_ctx(bool is_server, const char* cert, const char* key
return NULL;
}
if (SSL_CTX_use_certificate_file(ctx, cert, SSL_FILETYPE_PEM) <= 0) {
log_message(LOG_LEVEL_ERROR, "Failed to load certificate: %s", cert);
char* escaped = output_escape(cert, false);
log_message(LOG_LEVEL_ERROR, "Failed to load certificate: %s",
escaped ? escaped : "<allocation failed>");
free(escaped);
log_ssl_errors();
SSL_CTX_free(ctx);
return NULL;
}
if (SSL_CTX_use_PrivateKey_file(ctx, key, SSL_FILETYPE_PEM) <= 0) {
log_message(LOG_LEVEL_ERROR, "Failed to load private key: %s", key);
char* escaped = output_escape(key, false);
log_message(LOG_LEVEL_ERROR, "Failed to load private key: %s",
escaped ? escaped : "<allocation failed>");
free(escaped);
log_ssl_errors();
SSL_CTX_free(ctx);
return NULL;
@@ -85,7 +92,10 @@ static SSL_CTX* create_ssl_ctx(bool is_server, const char* cert, const char* key
if (ca_path) {
if (!SSL_CTX_load_verify_locations(ctx, ca_path, NULL)) {
log_message(LOG_LEVEL_ERROR, "Failed to load CA: %s", ca_path);
char* escaped = output_escape(ca_path, false);
log_message(LOG_LEVEL_ERROR, "Failed to load CA: %s",
escaped ? escaped : "<allocation failed>");
free(escaped);
log_ssl_errors();
SSL_CTX_free(ctx);
return NULL;
+224 -51
View File
@@ -1,6 +1,6 @@
#include "utils.h"
#include "array_list.h"
#include "libgen.h"
#include "log.h"
#include <dirent.h>
#include <errno.h>
#include <fcntl.h>
@@ -82,45 +82,6 @@ static int open_authorized_destination(const char* dest_root) {
return dirfd;
}
bool mkdir_r(const char* path) {
if (!path || *path == '\0')
return false;
char* duplicate = str_dup(path);
if (!duplicate)
return false;
int dirfd = open(path[0] == '/' ? "/" : ".", O_RDONLY | O_DIRECTORY | O_CLOEXEC | O_NOFOLLOW);
if (dirfd < 0) {
free(duplicate);
return false;
}
bool ok = true;
char* saveptr = NULL;
char* component = strtok_r(duplicate, "/", &saveptr);
while (component) {
if (strcmp(component, "..") == 0) {
ok = false;
break;
}
if (strcmp(component, ".") != 0) {
int next = openat(dirfd, component, O_RDONLY | O_DIRECTORY | O_CLOEXEC | O_NOFOLLOW);
if (next < 0 && errno == ENOENT) {
if (mkdirat(dirfd, component, 0755) == 0 || errno == EEXIST)
next = openat(dirfd, component, O_RDONLY | O_DIRECTORY | O_CLOEXEC | O_NOFOLLOW);
}
if (next < 0) {
ok = false;
break;
}
close(dirfd);
dirfd = next;
}
component = strtok_r(NULL, "/", &saveptr);
}
close(dirfd);
free(duplicate);
return ok;
}
char* str_dup(const char* string) {
if (string == NULL)
return NULL;
@@ -132,6 +93,32 @@ char* str_dup(const char* string) {
return new_string;
}
char* output_escape(const char* string, bool eight_bit_output) {
if (!string)
return NULL;
size_t length = strlen(string);
if (length > (SIZE_MAX - 1) / 5)
return NULL;
char* escaped = malloc(length * 5 + 1);
if (!escaped)
return NULL;
size_t out = 0;
for (size_t i = 0; i < length; i++) {
unsigned char byte = (unsigned char)string[i];
if ((byte >= 32 && byte <= 126) || (eight_bit_output && byte >= 128)) {
escaped[out++] = (char)byte;
} else {
escaped[out++] = '\\';
escaped[out++] = '#';
escaped[out++] = (char)('0' + ((byte >> 6) & 7));
escaped[out++] = (char)('0' + ((byte >> 3) & 7));
escaped[out++] = (char)('0' + (byte & 7));
}
}
escaped[out] = '\0';
return escaped;
}
/* Match a glob pattern against a string. Supported wildcards:
* ? matches any single character except '/'.
* * matches any sequence of characters within one path component (no '/').
@@ -187,6 +174,25 @@ bool glob_match(const char* pattern, const char* str) {
return *str == '\0';
}
bool format_human_bytes(unsigned long long bytes, char* buffer, size_t buffer_size) {
static const char* const units[] = {"B", "KB", "MB", "GB", "TB", "PB", "EB"};
double value = (double)bytes;
size_t unit = 0;
int written;
if (!buffer || buffer_size == 0)
return false;
while (value >= 1024.0 && unit < sizeof(units) / sizeof(units[0]) - 1) {
value /= 1024.0;
unit++;
}
if (unit == 0)
written = snprintf(buffer, buffer_size, "%llu %s", bytes, units[unit]);
else
written = snprintf(buffer, buffer_size, "%.1f %s", value, units[unit]);
return written >= 0 && (size_t)written < buffer_size;
}
static bool is_dir_in_manifest(const char* rel_path, ArrayList* manifest) {
size_t len = strlen(rel_path);
for (int i = 0; i < manifest->size; i++) {
@@ -198,9 +204,134 @@ static bool is_dir_in_manifest(const char* rel_path, ArrayList* manifest) {
return false;
}
/* True when child_rel is, or lies below, a protected entry. A prefix "a"
therefore protects "a" and "a/b/c" but not "ab". Entries with top_level_only
set only protect DIRECT children of the receive root (at_root); nested
directories that share such a name stay ordinary destination content. */
static bool path_under_skip_prefix(const char* child_rel, bool at_root,
const DeleteSkipEntry* skips, int skip_count) {
for (int i = 0; i < skip_count; i++) {
if (skips[i].top_level_only && !at_root)
continue;
size_t prefix_len = strlen(skips[i].prefix);
if (strncmp(child_rel, skips[i].prefix, prefix_len) == 0 &&
(child_rel[prefix_len] == '\0' || child_rel[prefix_len] == '/'))
return true;
}
return false;
}
/* All-or-nothing max-delete needs to know BEFORE any unlink whether the run
would delete more than max_delete entries. This rehearsal pass walks the
destination with the same decisions as the delete pass but never touches the
filesystem: it counts every regular file the delete pass would unlink and
every directory it would rmdir (a directory is removed only once every entry
below it has been removed and nothing the walker leaves in place survives).
Entries the walker never removes (symlinks, manifest-listed files, protected
prefixes) mark the enclosing directory as surviving, exactly as they would
make a real rmdir fail with ENOTEMPTY. Stops early once *count reaches the
cap (sets *exceeds). Returns false on a traversal error. */
static bool count_extras_fd(int dirfd, const char* rel_path, ArrayList* manifest, size_t cap,
size_t* count, bool* exceeds, const DeleteSkipEntry* skips,
int skip_count, bool* survives) {
/* openat(dirfd, ".") opens an independent file description: a dup() would
share dirfd's file offset, and a prior rehearsal pass must not have drained
this directory's stream before the delete pass reads it again. */
int scanfd = openat(dirfd, ".", O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
if (scanfd < 0)
return false;
DIR* dir = fdopendir(scanfd);
if (!dir) {
close(scanfd);
return false;
}
bool operation_ok = true;
bool local_survives = false;
bool at_root = rel_path[0] == '\0';
const struct dirent* entry;
while ((entry = readdir(dir)) != NULL) {
if (strcmp(entry->d_name, ".") == 0 || strcmp(entry->d_name, "..") == 0)
continue;
if (*exceeds)
break;
char* child_rel = path_cat((char*)rel_path, entry->d_name);
if (!child_rel) {
operation_ok = false;
continue;
}
if (path_under_skip_prefix(child_rel, at_root, skips, skip_count)) {
local_survives = true;
free(child_rel);
continue;
}
struct stat st;
if (fstatat(dirfd, entry->d_name, &st, AT_SYMLINK_NOFOLLOW) != 0) {
if (errno != ENOENT)
operation_ok = false;
free(child_rel);
continue;
}
if (S_ISLNK(st.st_mode)) {
local_survives = true;
free(child_rel);
continue;
}
if (S_ISDIR(st.st_mode)) {
int childfd = openat(dirfd, entry->d_name, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
bool child_ok = true;
bool child_survives = true;
if (childfd >= 0) {
child_ok = count_extras_fd(childfd, child_rel, manifest, cap, count, exceeds, skips,
skip_count, &child_survives);
close(childfd);
} else if (errno != ENOENT) {
operation_ok = false;
}
if (!child_ok)
operation_ok = false;
if (is_dir_in_manifest(child_rel, manifest)) {
/* A directory with kept content below it is never removed. */
local_survives = true;
} else if (child_survives) {
/* The directory still holds entries the walker leaves in place, so an
rmdir would fail with ENOTEMPTY; the delete pass leaves it behind
rather than reporting an error (matching rsync). */
local_survives = true;
} else {
if (*count >= cap) {
*exceeds = true;
} else {
(*count)++;
}
}
} else {
bool found = false;
for (int i = 0; i < manifest->size; i++) {
if (strcmp((char*)manifest->items[i], child_rel) == 0) {
found = true;
break;
}
}
if (!found) {
if (*count >= cap) {
*exceeds = true;
} else {
(*count)++;
}
}
}
free(child_rel);
}
closedir(dir);
*survives = local_survives;
return operation_ok;
}
static bool delete_extras_fd(int dirfd, const char* rel_path, ArrayList* manifest,
size_t max_delete, size_t* deleted_count) {
int scanfd = dup(dirfd);
size_t max_delete, size_t* deleted_count, const DeleteSkipEntry* skips,
int skip_count) {
/* Independent file description (see count_extras_fd). */
int scanfd = openat(dirfd, ".", O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
if (scanfd < 0)
return false;
DIR* dir = fdopendir(scanfd);
@@ -218,6 +349,17 @@ static bool delete_extras_fd(int dirfd, const char* rel_path, ArrayList* manifes
operation_ok = false;
continue;
}
/* A --delay-updates run keeps its staging directory as a direct child of
the receive root, and basis-dir snapshots live below it too. Their
contents are not manifest entries, so descending into them would delete
every staged / basis file as an "extra". Only the staging name (a
top-level-only prefix) and the basis prefixes are protected: a nested
destination directory that happens to be called .fastsync-stage is
ordinary content. */
if (path_under_skip_prefix(child_rel, rel_path[0] == '\0', skips, skip_count)) {
free(child_rel);
continue;
}
struct stat st;
if (fstatat(dirfd, entry->d_name, &st, AT_SYMLINK_NOFOLLOW) != 0) {
if (errno != ENOENT)
@@ -234,7 +376,8 @@ static bool delete_extras_fd(int dirfd, const char* rel_path, ArrayList* manifes
int childfd = openat(dirfd, entry->d_name, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
bool child_removed = false;
if (childfd >= 0) {
child_removed = delete_extras_fd(childfd, child_rel, manifest, max_delete, deleted_count);
child_removed = delete_extras_fd(childfd, child_rel, manifest, max_delete, deleted_count,
skips, skip_count);
if (!child_removed)
operation_ok = false;
close(childfd);
@@ -246,7 +389,12 @@ static bool delete_extras_fd(int dirfd, const char* rel_path, ArrayList* manifes
operation_ok = false;
} else {
if (unlinkat(dirfd, entry->d_name, AT_REMOVEDIR) != 0) {
if (errno != ENOENT)
/* ENOENT: already gone (fine). ENOTEMPTY/EEXIST: the directory
still holds entries the walker leaves in place (a protected
excluded prefix, a kept file the manifest protects, a symlink);
rsync leaves such a directory behind, so this is not an error.
Only genuine I/O failures abort the deletion. */
if (errno != ENOENT && errno != ENOTEMPTY && errno != EEXIST)
operation_ok = false;
} else {
(*deleted_count)++;
@@ -274,7 +422,9 @@ static bool delete_extras_fd(int dirfd, const char* rel_path, ArrayList* manifes
} else {
(*deleted_count)++;
}
fprintf(stderr, " Deleted: %s\n", child_rel);
char* escaped_path = output_escape(child_rel, log_get_8_bit_output());
fprintf(stderr, " Deleted: %s\n", escaped_path ? escaped_path : "<allocation failed>");
free(escaped_path);
}
}
free(child_rel);
@@ -283,9 +433,13 @@ static bool delete_extras_fd(int dirfd, const char* rel_path, ArrayList* manifes
return operation_ok;
}
bool delete_extras_limited(const char* dest_root, ArrayList* manifest, size_t max_delete) {
DeleteWalkResult delete_extras_limited(const char* dest_root, ArrayList* manifest,
size_t max_delete, const DeleteSkipEntry* skips,
int skip_count, size_t* deleted_out) {
if (deleted_out)
*deleted_out = 0;
if (!manifest)
return false;
return DELETE_WALK_ERROR;
int rootfd;
if (authorized_root_fd >= 0) {
if (authorized_root_path)
@@ -298,16 +452,35 @@ bool delete_extras_limited(const char* dest_root, ArrayList* manifest, size_t ma
rootfd = open(dest_root, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
}
if (rootfd < 0)
return false;
return DELETE_WALK_ERROR;
if (max_delete != SIZE_MAX) {
/* Rehearse the deletion first so a run that would exceed the cap removes
nothing (rsync's all-or-nothing --max-delete contract). */
size_t count = 0;
bool exceeds = false;
bool survives = false;
bool counted_ok = count_extras_fd(rootfd, "", manifest, max_delete, &count, &exceeds, skips,
skip_count, &survives);
if (!counted_ok) {
close(rootfd);
return DELETE_WALK_ERROR;
}
if (exceeds) {
close(rootfd);
return DELETE_WALK_LIMIT_EXCEEDED;
}
}
size_t deleted_count = 0;
bool ok = delete_extras_fd(rootfd, "", manifest, max_delete, &deleted_count);
bool ok = delete_extras_fd(rootfd, "", manifest, max_delete, &deleted_count, skips, skip_count);
if (close(rootfd) != 0)
ok = false;
return ok;
if (deleted_out)
*deleted_out = deleted_count;
return ok ? DELETE_WALK_OK : DELETE_WALK_ERROR;
}
bool delete_extras(const char* dest_root, ArrayList* manifest) {
return delete_extras_limited(dest_root, manifest, SIZE_MAX);
return delete_extras_limited(dest_root, manifest, SIZE_MAX, NULL, 0, NULL) == DELETE_WALK_OK;
}
bool has_path_traversal(const char* path) {
+38 -2
View File
@@ -5,17 +5,53 @@
#include <stddef.h>
#include <stdbool.h>
bool mkdir_r(const char* path);
char* str_dup(const char* string);
char* output_escape(const char* string, bool eight_bit_output);
char* path_cat(const char* path1, const char* path2);
bool glob_match(const char* pattern, const char* str);
/* Result of a bounded extra-file deletion run. */
typedef enum {
/* Every extra entry was removed (or there were none). */
DELETE_WALK_OK = 0,
/* The destination holds more extras than the numeric cap for this run. With
the all-or-nothing max-delete semantics NOTHING was removed (the walker
counts first and refuses to start when the run would exceed the limit). */
DELETE_WALK_LIMIT_EXCEEDED,
/* A traversal or unlink failure aborted the deletion (partial removal is
possible, mirroring the delete pass). */
DELETE_WALK_ERROR
} DeleteWalkResult;
/* One protected entry for the delete walker. When top_level_only is true the
prefix is skipped only as a DIRECT child of dest_root (the --delay-updates
staging directory, which must not hide genuine extras inside a nested
destination directory that happens to share the staging name); otherwise the
prefix is skipped at any depth (the --compare-dest/--copy-dest/--link-dest
basis trees, and the sender-side protected filter-excluded prefixes, which
are never destination content). */
typedef struct {
const char* prefix;
bool top_level_only;
} DeleteSkipEntry;
/* Remove files/dirs under dest_root that are not listed in manifest without
ever descending into a protected prefix (see DeleteSkipEntry). When
max_delete is not SIZE_MAX the run is all-or-nothing: extras are counted
first and DELETE_WALK_LIMIT_EXCEEDED is returned (with nothing removed) when
the count would exceed the cap. `deleted_out` optionally receives the number
of entries actually removed. The all-or-nothing guarantee holds only while
the destination tree is not being concurrently modified: the rehearsal pass
and the delete pass are two separate walks, so a concurrent change between
them (another process adding/removing entries) can make the second pass
delete a different set than the first one counted. */
DeleteWalkResult delete_extras_limited(const char* dest_root, ArrayList* manifest,
size_t max_delete, const DeleteSkipEntry* skips,
int skip_count, size_t* deleted_out);
bool delete_extras(const char* dest_root, ArrayList* manifest);
bool delete_extras_limited(const char* dest_root, ArrayList* manifest, size_t max_delete);
bool utils_set_authorized_root(int fd, const char* canonical_path);
/* The fd-only compatibility form is fail-closed for path-based operations;
* callers should use utils_set_authorized_root with the canonical identity. */
void utils_set_authorized_root_fd(int fd);
bool has_path_traversal(const char* path);
bool utils_valid_batch_path(const char* path);
bool format_human_bytes(unsigned long long bytes, char* buffer, size_t buffer_size);
#endif
+73
View File
@@ -6,6 +6,7 @@ import socket
import subprocess
import sys
import tempfile
import threading
import time
PROJECT_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
@@ -53,6 +54,78 @@ class ServerManager:
self.stop()
class CountingProxy:
"""One-shot TCP forwarder that counts the bytes flowing in each direction
between one client and the real server.
Client output and --stats report SOURCE lengths, so a delta/fuzzy transfer
that moves only a few percent of the file is invisible in normal output.
Routing the client through this proxy makes the actual wire usage
observable: client_to_server counts every byte the client sent (config,
paths, and file/delta payloads), server_to_client counts the reply bytes
(including the receiver's delta signatures).
"""
def __init__(self, target_port):
self.target_port = target_port
self._listener = socket.socket()
self._listener.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self._listener.bind(("127.0.0.1", 0))
self._listener.listen(1)
self._listener.settimeout(30)
self.port = self._listener.getsockname()[1]
self.client_to_server = 0
self.server_to_client = 0
@staticmethod
def _pump(src, dst, counter):
while True:
try:
data = src.recv(65536)
except OSError:
return
if not data:
try:
dst.shutdown(socket.SHUT_WR)
except OSError:
pass
return
try:
dst.sendall(data)
except OSError:
return
counter[0] += len(data)
def run(self, cmd):
"""Forward one client run (the full command list) to the real server and
return the CompletedProcess after the counts have settled."""
def serve():
try:
client_sock, _ = self._listener.accept()
server_sock = socket.create_connection(("127.0.0.1", self.target_port),
timeout=10)
except OSError:
self._listener.close()
return
c2s, s2c = [0], [0]
a = threading.Thread(target=self._pump, args=(client_sock, server_sock, c2s))
b = threading.Thread(target=self._pump, args=(server_sock, client_sock, s2c))
a.start()
b.start()
a.join()
b.join()
self.client_to_server = c2s[0]
self.server_to_client = s2c[0]
self._listener.close()
thread = threading.Thread(target=serve)
thread.start()
result = subprocess.run(cmd, capture_output=True, text=True, timeout=180)
thread.join(20)
return result
def run_client(source_dir, dest_dir, flags=None, port=None, extra_args=None):
"""Run the client and return (result, duration)."""
cmd = CLIENT_CMD + ["--source-dir", source_dir, "--dest-dir", dest_dir, "--save-to-disk"]
File diff suppressed because it is too large. Load diff
+5
View File
@@ -71,6 +71,11 @@ class TestTCPFlags:
r = _run_tcp_test("Compression (-c)", shared_server.port, ["-c"])
assert r["status"] == "Success", r["error"]
def test_compression_threads(self, shared_server):
r = _run_tcp_test("Compression threads (-c --compress-threads=2)", shared_server.port,
["-c", "--compress-threads=2"])
assert r["status"] == "Success", r["error"]
def test_chunk_serialization(self, shared_server):
r = _run_tcp_test("Chunk Serialization (-s)", shared_server.port, ["-s"])
assert r["status"] == "Success", r["error"]
+6
View File
@@ -1,9 +1,12 @@
#include "test_array_list.h"
#include "test_chunk.h"
#include "test_change_list.h"
#include "test_checksum.h"
#include "test_client_cli.h"
#include "test_compression.h"
#include "test_config.h"
#include "test_data.h"
#include "test_delay_updates.h"
#include "test_delta.h"
#include "test_file.h"
#include "test_file_sendfile.h"
@@ -40,15 +43,18 @@ int main() {
RUN_TEST(test_array_list);
RUN_TEST(test_shared_utils);
RUN_TEST(test_chunk);
RUN_TEST(test_change_list);
RUN_TEST(test_config);
RUN_TEST(test_compression);
RUN_TEST(test_scanner);
RUN_TEST(test_checksum);
RUN_TEST(test_delta);
RUN_TEST(test_data);
RUN_TEST(test_protocol);
RUN_TEST(test_metadata);
RUN_TEST(test_glob);
RUN_TEST(test_file);
RUN_TEST(test_delay_updates);
RUN_TEST(test_file_sendfile);
RUN_TEST(test_multiprocessing);
RUN_TEST(test_log);
+101
View File
@@ -0,0 +1,101 @@
#include "test_change_list.h"
#include "change_list.h"
#include "test_utils.h"
#include "utils.h"
#include <stdlib.h>
#include <string.h>
#include <time.h>
static ChangeEvent sample_event(void) {
ChangeEvent event;
memset(&event, 0, sizeof(event));
event.path = "/srv/root/sub/file.txt";
event.decision = CHANGE_SENT;
event.is_directory = false;
event.size = 12345;
event.bytes_sent = 999;
event.mtime_sec = 1700000000;
return event;
}
static void test_format_tokens() {
ChangeEvent event = sample_event();
char* line = change_render_format("%f %n %l %b %M %%", &event);
EXPECT_NOT_NULL(line);
EXPECT_EQ_STR(line, "/srv/root/sub/file.txt file.txt 12345 999 1700000000 %");
free(line);
}
static void test_format_unknown_tokens_preserved() {
ChangeEvent event = sample_event();
char* line = change_render_format("x%q=%f%z", &event);
EXPECT_NOT_NULL(line);
EXPECT_EQ_STR(line, "x%q=/srv/root/sub/file.txt%z");
free(line);
}
static void test_format_leaf_name() {
ChangeEvent event = sample_event();
event.path = "bare.txt";
char* line = change_render_format("%n|%f", &event);
EXPECT_NOT_NULL(line);
EXPECT_EQ_STR(line, "bare.txt|bare.txt");
free(line);
}
static void test_render_itemize_sent_file() {
ChangeEvent event = sample_event();
char* line = change_render_itemize(&event);
EXPECT_NOT_NULL(line);
EXPECT_EQ_STR(line, ">f+++++++++ /srv/root/sub/file.txt");
free(line);
}
static void test_render_itemize_up_to_date_is_empty() {
ChangeEvent event = sample_event();
event.decision = CHANGE_UP_TO_DATE;
char* line = change_render_itemize(&event);
EXPECT_NOT_NULL(line);
EXPECT_EQ_STR(line, "");
free(line);
}
static void test_render_list_line() {
char* line = change_render_list_line(0100644, 4096, 1700000000, "/srv/x.txt");
EXPECT_NOT_NULL(line);
EXPECT_TRUE(strncmp(line, "-rw-r--r--", 10) == 0);
EXPECT_TRUE(strstr(line, "4096") != NULL);
EXPECT_TRUE(strstr(line, "/srv/x.txt") != NULL);
free(line);
}
static void test_change_list_enabled() {
Config* config = config_create();
EXPECT_NOT_NULL(config);
EXPECT_FALSE(change_list_enabled(config));
config->itemize_changes = true;
EXPECT_TRUE(change_list_enabled(config));
config->itemize_changes = false;
config->out_format = str_dup("%f");
EXPECT_TRUE(change_list_enabled(config));
free(config->out_format);
config->out_format = NULL;
EXPECT_FALSE(change_list_enabled(config));
/* config_delete() closes log_file, so use a throwaway tmpfile. */
config->log_file = tmpfile();
EXPECT_NOT_NULL(config->log_file);
EXPECT_FALSE(change_list_enabled(config)); /* needs a format too */
config->log_file_format = str_dup("%n");
EXPECT_TRUE(change_list_enabled(config));
config_delete(config); /* closes config->log_file */
}
void test_change_list() {
test_format_tokens();
test_format_unknown_tokens_preserved();
test_format_leaf_name();
test_render_itemize_sent_file();
test_render_itemize_up_to_date_is_empty();
test_render_list_line();
test_change_list_enabled();
}
+6
View File
@@ -0,0 +1,6 @@
#ifndef TEST_CHANGE_LIST_H
#define TEST_CHANGE_LIST_H
void test_change_list(void);
#endif
+146
View File
@@ -0,0 +1,146 @@
#include "test_checksum.h"
#include "checksum.h"
#include "test_utils.h"
#include <string.h>
/* Known xxHash64 vector (seed 0) for the empty string and a literal.
* The md5 vectors are the standard NIST/RFC1321 test strings. These pin the
* digest selection to genuinely distinct algorithm outputs so a --checksum-
* choice change is observable, not a silent no-op. */
static void test_checksum_xxh64_seed0() {
uint8_t out[CHECKSUM_MAX_DIGEST_LEN];
size_t len = 0;
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_XXH64, 0, "hello", 5, out, sizeof(out), &len));
EXPECT_TRUE(len == (size_t)8);
/* Hard-coded: XXH64("hello", 5, 0). */
uint8_t expect[8] = {0xa3, 0x6d, 0x9f, 0x88, 0x7d, 0x82, 0xc7, 0x26};
for (int i = 0; i < 8; i++)
EXPECT_EQ_INT(out[i], expect[i]);
}
static void test_checksum_xxh64_empty() {
uint8_t out[CHECKSUM_MAX_DIGEST_LEN];
size_t len = 0;
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_XXH64, 0, "", 0, out, sizeof(out), &len));
EXPECT_TRUE(len == (size_t)8);
/* XXH64("", 0, 0). */
uint8_t expect[8] = {0x99, 0xe9, 0xd8, 0x51, 0x37, 0xdb, 0x46, 0xef};
for (int i = 0; i < 8; i++)
EXPECT_EQ_INT(out[i], expect[i]);
}
/* A nonzero seed must change the xxh64 digest: the algorithm is genuinely
* seed-aware, deterministic, and distinct from seed 0. */
static void test_checksum_xxh64_seed_changes_digest() {
uint8_t a[CHECKSUM_MAX_DIGEST_LEN], b[CHECKSUM_MAX_DIGEST_LEN];
size_t alen = 0, blen = 0;
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_XXH64, 7, "payload", 7, a, sizeof(a), &alen));
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_XXH64, 0, "payload", 7, b, sizeof(b), &blen));
EXPECT_TRUE(alen == blen);
EXPECT_TRUE(memcmp(a, b, alen) != 0);
}
static void test_checksum_xxh64_seed_deterministic() {
uint8_t a[CHECKSUM_MAX_DIGEST_LEN], b[CHECKSUM_MAX_DIGEST_LEN];
size_t alen = 0, blen = 0;
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_XXH64, 12345, "same", 4, a, sizeof(a), &alen));
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_XXH64, 12345, "same", 4, b, sizeof(b), &blen));
EXPECT_TRUE(alen == blen);
EXPECT_TRUE(memcmp(a, b, alen) == 0);
}
static void test_checksum_md5_vectors() {
uint8_t out[CHECKSUM_MAX_DIGEST_LEN];
size_t len = 0;
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_MD5, 0, "", 0, out, sizeof(out), &len));
EXPECT_TRUE(len == (size_t)16);
uint8_t expect_empty[16] = {0xd4, 0x1d, 0x8c, 0xd9, 0x8f, 0x00, 0xb2, 0x04,
0xe9, 0x80, 0x09, 0x98, 0xec, 0xf8, 0x42, 0x7e};
EXPECT_TRUE(memcmp(out, expect_empty, 16) == 0);
/* MD5("abc") */
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_MD5, 0, "abc", 3, out, sizeof(out), &len));
uint8_t expect_abc[16] = {0x90, 0x01, 0x50, 0x98, 0x3c, 0xd2, 0x4f, 0xb0,
0xd6, 0x96, 0x3f, 0x7d, 0x28, 0xe1, 0x7f, 0x72};
EXPECT_TRUE(memcmp(out, expect_abc, 16) == 0);
}
/* md5 is 16 bytes and differs from the 8-byte xxh64 for the same input, so the
* choice is observably different both in length and in content. */
static void test_checksum_algo_lengths_distinct() {
EXPECT_EQ_INT((int)checksum_digest_len(CHECKSUM_ALGO_XXH64), 8);
EXPECT_EQ_INT((int)checksum_digest_len(CHECKSUM_ALGO_MD5), 16);
uint8_t x[CHECKSUM_MAX_DIGEST_LEN], m[CHECKSUM_MAX_DIGEST_LEN];
size_t xl = 0, ml = 0;
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_XXH64, 0, "same content", 12, x, sizeof(x), &xl));
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_MD5, 0, "same content", 12, m, sizeof(m), &ml));
EXPECT_TRUE(xl == (size_t)8);
EXPECT_TRUE(ml == (size_t)16);
EXPECT_TRUE(memcmp(x, m, 8) != 0);
}
/* md5 has no seed: two distinct seeds give the same md5 digest (documented);
* the seed is only honored by xxh64 and the delta block hash (low 32 bits). */
static void test_checksum_md5_seed_ignored() {
uint8_t a[CHECKSUM_MAX_DIGEST_LEN], b[CHECKSUM_MAX_DIGEST_LEN];
size_t alen = 0, blen = 0;
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_MD5, 0, "data", 4, a, sizeof(a), &alen));
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_MD5, 99, "data", 4, b, sizeof(b), &blen));
EXPECT_TRUE(memcmp(a, b, alen) == 0);
}
static void test_checksum_algo_name_mapping() {
EXPECT_EQ_INT(checksum_algo_from_name("xxh64"), (int)CHECKSUM_ALGO_XXH64);
EXPECT_EQ_INT(checksum_algo_from_name("XXH64"), (int)CHECKSUM_ALGO_XXH64);
EXPECT_EQ_INT(checksum_algo_from_name("xxhash"), (int)CHECKSUM_ALGO_XXH64);
EXPECT_EQ_INT(checksum_algo_from_name("XXHASH"), (int)CHECKSUM_ALGO_XXH64);
EXPECT_EQ_INT(checksum_algo_from_name("md5"), (int)CHECKSUM_ALGO_MD5);
EXPECT_EQ_INT(checksum_algo_from_name("MD5"), (int)CHECKSUM_ALGO_MD5);
EXPECT_TRUE(checksum_algo_from_name("sha256") < 0);
EXPECT_TRUE(checksum_algo_from_name("crc32") < 0);
EXPECT_TRUE(checksum_algo_from_name("none") < 0);
EXPECT_TRUE(checksum_algo_from_name("") < 0);
EXPECT_TRUE(checksum_algo_from_name(NULL) < 0);
EXPECT_TRUE(checksum_algo_valid((int)CHECKSUM_ALGO_XXH64));
EXPECT_TRUE(checksum_algo_valid((int)CHECKSUM_ALGO_MD5));
EXPECT_FALSE(checksum_algo_valid(99));
EXPECT_EQ_STR(checksum_algo_name(CHECKSUM_ALGO_XXH64), "xxh64");
EXPECT_EQ_STR(checksum_algo_name(CHECKSUM_ALGO_MD5), "md5");
}
static void test_checksum_truncated_buffer_rejected() {
uint8_t small[4];
size_t len = 0;
/* The digest cannot fit in a 4-byte buffer. */
EXPECT_FALSE(checksum_digest(CHECKSUM_ALGO_XXH64, 0, "x", 1, small, sizeof(small), &len));
EXPECT_FALSE(checksum_digest(CHECKSUM_ALGO_MD5, 0, "x", 1, small, sizeof(small), &len));
EXPECT_FALSE(checksum_digest(CHECKSUM_ALGO_XXH64, 0, NULL, 5, small, sizeof(small), &len));
EXPECT_FALSE(checksum_digest(CHECKSUM_ALGO_XXH64, 0, "x", 1, NULL, 0, &len));
EXPECT_FALSE(checksum_digest(CHECKSUM_ALGO_XXH64, 0, "x", 1, small, sizeof(small), NULL));
}
/* A NULL data pointer with size 0 is the empty input, not an error. */
static void test_checksum_null_empty_digest() {
uint8_t a[CHECKSUM_MAX_DIGEST_LEN], b[CHECKSUM_MAX_DIGEST_LEN];
size_t alen = 0, blen = 0;
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_XXH64, 0, NULL, 0, a, sizeof(a), &alen));
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_XXH64, 0, "", 0, b, sizeof(b), &blen));
EXPECT_TRUE(alen == blen);
EXPECT_TRUE(memcmp(a, b, alen) == 0);
}
void test_checksum(void) {
test_checksum_xxh64_seed0();
test_checksum_xxh64_empty();
test_checksum_xxh64_seed_changes_digest();
test_checksum_xxh64_seed_deterministic();
test_checksum_md5_vectors();
test_checksum_algo_lengths_distinct();
test_checksum_md5_seed_ignored();
test_checksum_algo_name_mapping();
test_checksum_truncated_buffer_rejected();
test_checksum_null_empty_digest();
}
+6
View File
@@ -0,0 +1,6 @@
#ifndef TEST_CHECKSUM_H
#define TEST_CHECKSUM_H
void test_checksum(void);
#endif
+71
View File
@@ -89,7 +89,78 @@ static void test_chunk_operations() {
unlink(path2);
}
/* A chunk mixing a regular file and an explicit directory entry (--dirs, with
* or without metadata) must round-trip through serialize/deserialize with the
* is_dir flag and the entry type marker preserved. */
static void test_chunk_dir_entry_roundtrip() {
const char* file_path = "temp_chunk_dir_file.txt";
const char* dir_path = "temp_chunk_dir_entry";
const char* content = "regular file payload";
/* A failed earlier run can leave artifacts behind; start clean. */
rmdir(dir_path);
unlink(file_path);
file_write_to_disk(file_path, content, strlen(content), false, false);
EXPECT_EQ_INT(mkdir(dir_path, 0755), 0);
for (int use_metadata = 0; use_metadata <= 1; use_metadata++) {
struct stat st;
EXPECT_EQ_INT(stat(file_path, &st), 0);
File* reg = file_create(file_path);
EXPECT_NOT_NULL(reg);
reg->data->size = (unsigned long long)st.st_size;
EXPECT_TRUE(file_load_data(reg));
File* dir = file_create(dir_path);
EXPECT_NOT_NULL(dir);
dir->is_dir = true;
if (use_metadata) {
reg->metadata = file_metadata_create(&st);
EXPECT_NOT_NULL(reg->metadata);
struct stat dst;
EXPECT_EQ_INT(stat(dir_path, &dst), 0);
dir->metadata = file_metadata_create(&dst);
EXPECT_NOT_NULL(dir->metadata);
}
File* files[2] = {reg, dir};
Chunk* chunk = chunk_create(files, 2);
EXPECT_NOT_NULL(chunk);
Data* serialized = chunk_serialize(chunk, use_metadata != 0);
EXPECT_NOT_NULL(serialized);
Chunk* deserialized = chunk_deserialize(serialized, use_metadata != 0);
EXPECT_NOT_NULL(deserialized);
EXPECT_EQ_INT(deserialized->element_count, 2);
EXPECT_FALSE(deserialized->items[0]->is_dir);
EXPECT_EQ_STR(deserialized->items[0]->path, file_path);
EXPECT_EQ_INT((int)deserialized->items[0]->data->size, (int)strlen(content));
EXPECT_EQ_INT(memcmp(deserialized->items[0]->data->data, content, strlen(content)), 0);
EXPECT_TRUE(deserialized->items[1]->is_dir);
EXPECT_EQ_STR(deserialized->items[1]->path, dir_path);
EXPECT_EQ_INT((int)deserialized->items[1]->data->size, 0);
if (use_metadata) {
EXPECT_NOT_NULL(deserialized->items[0]->metadata);
EXPECT_NOT_NULL(deserialized->items[1]->metadata);
} else {
EXPECT_NULL(deserialized->items[0]->metadata);
EXPECT_NULL(deserialized->items[1]->metadata);
}
data_destroy(serialized);
chunk_destroy(deserialized);
chunk_destroy(chunk); /* frees reg and dir */
}
unlink(file_path);
rmdir(dir_path);
}
void test_chunk() {
test_file_operations();
test_chunk_operations();
test_chunk_dir_entry_roundtrip();
}
+1664 -26
View File
File diff suppressed because it is too large. Load diff
+27
View File
@@ -55,6 +55,31 @@ static void test_data_compress_decompress_large() {
data_destroy(decompressed);
}
static void test_skip_compress_suffix_matching() {
char* suffixes[] = {".ZIP", ".GZ"};
EXPECT_TRUE(compression_should_skip_with_suffixes("archive.zip", suffixes, 2));
EXPECT_TRUE(compression_should_skip_with_suffixes("backup.TAR.GZ", suffixes, 2));
EXPECT_FALSE(compression_should_skip_with_suffixes("notes.txt", suffixes, 2));
EXPECT_FALSE(compression_should_skip_with_suffixes("archive.zip", suffixes, 0));
}
static void test_data_compress_with_threads_roundtrip() {
const size_t size = 8 * 1024 * 1024;
Data* input = data_create_empty(size);
EXPECT_NOT_NULL(input);
for (size_t i = 0; i < size; i++)
((char*)input->data)[i] = (char)((i / 4096) % 7);
Data* compressed = data_compress_with_threads(input, 3, 2);
EXPECT_NOT_NULL(compressed);
Data* decompressed = data_decompress(compressed);
EXPECT_NOT_NULL(decompressed);
EXPECT_EQ_INT((int)decompressed->size, (int)size);
EXPECT_EQ_INT(memcmp(decompressed->data, input->data, size), 0);
data_destroy(input);
data_destroy(compressed);
data_destroy(decompressed);
}
static void test_chunk_compress_decompress_roundtrip() {
char* path1 = "temp_comp_test_1.txt";
char* content1 = "chunk compression test file 1";
@@ -115,5 +140,7 @@ static void test_chunk_compress_decompress_roundtrip() {
void test_compression() {
test_data_compress_decompress_roundtrip();
test_data_compress_decompress_large();
test_skip_compress_suffix_matching();
test_data_compress_with_threads_roundtrip();
test_chunk_compress_decompress_roundtrip();
}
+566 -4
View File
@@ -95,6 +95,7 @@ static void test_pipeline_sender_lifecycle() {
EXPECT_EQ_INT(pcs->queue_loader->capacity, 15);
EXPECT_FALSE(pcs->scanner_done);
EXPECT_FALSE(pcs->loader_done);
EXPECT_EQ_INT((int)pcs->allocation_session.max_alloc, (int)cfg->max_alloc);
pipeline_context_sender_destroy(pcs);
}
@@ -121,11 +122,29 @@ static void test_config_send_receive() {
send_cfg->receive_root_directory = str_dup("/send/dst");
send_cfg->save_to_disk = true;
send_cfg->use_multithreading = true;
send_cfg->use_chunk_serialization = true;
send_cfg->use_chunk_serialization = false;
send_cfg->use_compression = true;
send_cfg->use_metadata = true;
send_cfg->use_executability = true;
send_cfg->use_delta = true;
send_cfg->whole_file = true;
send_cfg->fuzzy = true;
send_cfg->ignore_times = true;
send_cfg->size_only = true;
send_cfg->compression_level = 5;
send_cfg->chunk_size = 1024;
send_cfg->eight_bit_output = true;
send_cfg->modify_window = 4;
send_cfg->existing = true;
send_cfg->ignore_existing = true;
send_cfg->delay_updates = true;
send_cfg->relative = true;
send_cfg->mkpath = true;
send_cfg->skip_compress_set = true;
send_cfg->skip_compress_count = 1;
send_cfg->skip_compress_suffixes = calloc(1, sizeof(char*));
send_cfg->skip_compress_suffixes[0] = str_dup(".zip");
send_cfg->max_alloc = MAX_SERVER_ALLOC + 1;
/* Use socketpair for bidirectional communication */
int p[2];
@@ -154,12 +173,41 @@ static void test_config_send_receive() {
ok = false;
if (!recv_cfg->use_multithreading)
ok = false;
if (!recv_cfg->use_chunk_serialization)
if (recv_cfg->use_chunk_serialization)
ok = false;
if (recv_cfg->compression_level != 5)
ok = false;
if (recv_cfg->chunk_size != 1024)
ok = false;
if (!recv_cfg->use_executability)
ok = false;
if (!recv_cfg->size_only)
ok = false;
if (!recv_cfg->ignore_times)
ok = false;
if (!recv_cfg->eight_bit_output)
ok = false;
if (recv_cfg->use_delta)
ok = false;
if (!recv_cfg->fuzzy)
ok = false;
if (recv_cfg->modify_window != 4)
ok = false;
if (!recv_cfg->existing)
ok = false;
if (!recv_cfg->ignore_existing)
ok = false;
if (!recv_cfg->delay_updates)
ok = false;
if (!recv_cfg->relative)
ok = false;
if (!recv_cfg->mkpath)
ok = false;
if (!recv_cfg->skip_compress_set || recv_cfg->skip_compress_count != 1 ||
strcmp(recv_cfg->skip_compress_suffixes[0], ".zip") != 0)
ok = false;
if (recv_cfg->max_alloc != MAX_SERVER_ALLOC)
ok = false;
}
config_delete(recv_cfg);
close(p[0]);
@@ -185,11 +233,11 @@ static void test_config_send_receive() {
}
static void test_config_send_receive_version_mismatch() {
/* Create a config with a different protocol version */
/* A peer using the previous wire format must be rejected. */
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
free(cfg->version);
cfg->version = str_dup("0.0");
cfg->version = str_dup("2.3.0");
cfg->send_directory = str_dup("/src");
cfg->receive_root_directory = str_dup("/dst");
@@ -233,6 +281,8 @@ static void test_config_receive_truncated() {
/* A valid prefix exercises cleanup after allocated wire strings and a
* partially received scalar field. */
EXPECT_TRUE(send_str(p[1], PROTOCOL_VERSION));
unsigned long long max_alloc = DEFAULT_MAX_ALLOC;
EXPECT_TRUE(send_n_data(p[1], &max_alloc, sizeof(max_alloc)));
EXPECT_TRUE(send_str(p[1], "/src"));
EXPECT_TRUE(send_str(p[1], "/dst"));
EXPECT_TRUE(send_int(p[1], 1));
@@ -244,6 +294,444 @@ static void test_config_receive_truncated() {
close(p[1]);
}
static bool config_string_roundtrip_matches(const Config* send_cfg, Config* recv) {
/* The sender serializes NULL strings as "" on the wire. Receivers must
canonicalize those empty values back to NULL for the options whose client
default is NULL (backup_dir, temp_dir, partial_dir, suffix), while a real
non-empty value round-trips unchanged. */
const char* fields[4];
char* const* recv_fields[4];
fields[0] = send_cfg->backup_dir;
recv_fields[0] = &recv->backup_dir;
fields[1] = send_cfg->temp_dir;
recv_fields[1] = &recv->temp_dir;
fields[2] = send_cfg->partial_dir;
recv_fields[2] = &recv->partial_dir;
fields[3] = send_cfg->suffix;
recv_fields[3] = &recv->suffix;
for (int i = 0; i < 4; i++) {
const char* sent = fields[i];
const char* got = *recv_fields[i];
if (sent == NULL || sent[0] == '\0') {
if (got != NULL)
return false;
} else if (got == NULL || strcmp(sent, got) != 0) {
return false;
}
}
return true;
}
static bool roundtrip_config_ok(const Config* send_cfg) {
int p[2];
if (socketpair(AF_UNIX, SOCK_STREAM, 0, p) != 0)
return false;
pid_t pid = fork();
if (pid == 0) {
close(p[1]);
io_set_fds(p[0], p[0]);
Config* recv = config_receive(p[0]);
bool ok = recv != NULL;
if (ok) {
ok = recv->version != NULL && strcmp(recv->version, PROTOCOL_VERSION) == 0;
ok = ok && recv->send_directory && recv->receive_root_directory;
ok = ok && config_string_roundtrip_matches(send_cfg, recv);
}
config_delete(recv);
close(p[0]);
_exit(ok ? 0 : 1);
} else {
close(p[0]);
io_set_fds(p[1], p[1]);
bool sent = config_send(p[1], send_cfg);
int status;
waitpid(pid, &status, 0);
close(p[1]);
return sent && WIFEXITED(status) && WEXITSTATUS(status) == 0;
}
}
/* Issue #252: NULL-vs-empty must survive the wire for backup_dir, temp_dir,
partial_dir, and suffix. NULL and explicitly-empty client values are both
serialized as "" and must be reconstructed as NULL so plain --backup (with
no --suffix/--backup-dir) works exactly like the client configured it. */
static void test_config_string_null_vs_empty_roundtrip() {
if (is_running_under_valgrind())
return;
/* NULL values on the wire must come back as NULL. */
Config* a = config_create();
EXPECT_NOT_NULL(a);
a->send_directory = str_dup("/src");
a->receive_root_directory = str_dup("/dst");
EXPECT_TRUE(roundtrip_config_ok(a));
config_delete(a);
/* Explicitly empty strings (indistinguishable on the wire from NULL) must
be canonicalized to NULL by the receiver. */
Config* b = config_create();
EXPECT_NOT_NULL(b);
b->send_directory = str_dup("/src");
b->receive_root_directory = str_dup("/dst");
b->backup_dir = str_dup("");
b->temp_dir = str_dup("");
b->partial_dir = str_dup("");
b->suffix = str_dup("");
EXPECT_TRUE(roundtrip_config_ok(b));
config_delete(b);
/* Non-empty values must round-trip unchanged. */
Config* c = config_create();
EXPECT_NOT_NULL(c);
c->send_directory = str_dup("/src");
c->receive_root_directory = str_dup("/dst");
c->backup_dir = str_dup("backups");
c->temp_dir = str_dup("/tmp/fast");
c->partial_dir = str_dup(".partial");
c->suffix = str_dup(".bak");
EXPECT_TRUE(roundtrip_config_ok(c));
config_delete(c);
}
/* A --temp-dir value must survive config_send/config_receive unchanged on the
receive side (round-trips through the resume-options wire block). */
static void test_config_temp_dir_roundtrip() {
if (is_running_under_valgrind())
return;
Config* c = config_create();
EXPECT_NOT_NULL(c);
c->send_directory = str_dup("/src");
c->receive_root_directory = str_dup("/dst");
c->temp_dir = str_dup("scratch");
EXPECT_TRUE(roundtrip_config_ok(c));
config_delete(c);
/* An empty-STRING wire value is canonicalized back to NULL (never an empty
scratch-dir name). */
c = config_create();
EXPECT_NOT_NULL(c);
c->send_directory = str_dup("/src");
c->receive_root_directory = str_dup("/dst");
c->temp_dir = str_dup("");
EXPECT_TRUE(roundtrip_config_ok(c));
config_delete(c);
}
static void test_config_delay_updates_reserved_backup_rejected() {
if (is_running_under_valgrind())
return;
Config* c = config_create();
EXPECT_NOT_NULL(c);
c->send_directory = str_dup("/src");
c->receive_root_directory = str_dup("/dst");
c->delay_updates = true;
c->backup_dir = str_dup(".fastsync-stage");
/* The receiver-side wire validation must reject a --backup-dir that collides
with the internal delay-updates staging directory. */
EXPECT_FALSE(roundtrip_config_ok(c));
config_delete(c);
c = config_create();
EXPECT_NOT_NULL(c);
c->send_directory = str_dup("/src");
c->receive_root_directory = str_dup("/dst");
c->delay_updates = true;
c->backup_dir = str_dup("backups");
EXPECT_TRUE(roundtrip_config_ok(c));
config_delete(c);
}
static void test_config_delete_timing_early_helper() {
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
EXPECT_FALSE(config_delete_timing_early(cfg));
EXPECT_TRUE(config_has_valid_delete_timing(cfg));
cfg->use_delete = true;
EXPECT_TRUE(config_has_valid_delete_timing(cfg));
EXPECT_FALSE(config_delete_timing_early(cfg));
config_delete(cfg);
cfg = config_create();
cfg->use_delete = true;
cfg->delete_before = true;
EXPECT_TRUE(config_delete_timing_early(cfg));
EXPECT_TRUE(config_has_valid_delete_timing(cfg));
config_delete(cfg);
cfg = config_create();
cfg->use_delete = true;
cfg->delete_during = true;
EXPECT_TRUE(config_delete_timing_early(cfg));
EXPECT_TRUE(config_has_valid_delete_timing(cfg));
config_delete(cfg);
cfg = config_create();
cfg->use_delete = true;
cfg->delete_delay = true;
EXPECT_FALSE(config_delete_timing_early(cfg));
EXPECT_TRUE(config_has_valid_delete_timing(cfg));
config_delete(cfg);
cfg = config_create();
cfg->use_delete = true;
cfg->delete_after = true;
EXPECT_FALSE(config_delete_timing_early(cfg));
EXPECT_TRUE(config_has_valid_delete_timing(cfg));
config_delete(cfg);
/* Two simultaneous timings are invalid. */
cfg = config_create();
cfg->use_delete = true;
cfg->delete_before = true;
cfg->delete_after = true;
EXPECT_TRUE(config_delete_timing_early(cfg));
EXPECT_FALSE(config_has_valid_delete_timing(cfg));
config_delete(cfg);
/* A timing flag without deletion is invalid. */
cfg = config_create();
cfg->delete_delay = true;
EXPECT_FALSE(config_has_valid_delete_timing(cfg));
EXPECT_FALSE(config_delete_timing_early(cfg));
config_delete(cfg);
}
/* New delete-timing fields must survive config_send/config_receive unchanged,
and a config carrying two conflicting timings must be rejected. */
static void test_config_delete_timing_wire_roundtrip() {
if (is_running_under_valgrind())
return;
struct {
bool before, during, delay, after;
} cases[] = {
{false, false, false, false}, {true, false, false, false}, {false, true, false, false},
{false, false, true, false}, {false, false, false, true},
};
for (size_t i = 0; i < sizeof(cases) / sizeof(cases[0]); i++) {
int p[2];
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
pid_t pid = fork();
if (pid == 0) {
close(p[1]);
io_set_fds(p[0], p[0]);
Config* recv = config_receive(p[0]);
bool ok = recv != NULL;
if (ok) {
ok = recv->use_delete && recv->delete_before == cases[i].before &&
recv->delete_during == cases[i].during && recv->delete_delay == cases[i].delay &&
recv->delete_after == cases[i].after;
}
config_delete(recv);
close(p[0]);
_exit(ok ? 0 : 1);
} else {
close(p[0]);
io_set_fds(p[1], p[1]);
Config* send_cfg = config_create();
EXPECT_NOT_NULL(send_cfg);
send_cfg->send_directory = str_dup("/src");
send_cfg->receive_root_directory = str_dup("/dst");
send_cfg->use_delete = true;
send_cfg->delete_before = cases[i].before;
send_cfg->delete_during = cases[i].during;
send_cfg->delete_delay = cases[i].delay;
send_cfg->delete_after = cases[i].after;
bool sent = config_send(p[1], send_cfg);
int status;
waitpid(pid, &status, 0);
close(p[1]);
config_delete(send_cfg);
EXPECT_TRUE(sent);
EXPECT_TRUE(WIFEXITED(status) && WEXITSTATUS(status) == 0);
}
}
}
/* The receiver-side wire validation rejects a keep-set config with two
conflicting delete-timing flags. */
static void test_config_delete_timing_conflict_rejected() {
if (is_running_under_valgrind())
return;
Config* c = config_create();
EXPECT_NOT_NULL(c);
c->send_directory = str_dup("/src");
c->receive_root_directory = str_dup("/dst");
c->use_delete = true;
c->delete_before = true;
c->delete_delay = true;
EXPECT_FALSE(roundtrip_config_ok(c));
config_delete(c);
}
/* The deletion-policy fields that cross the wire survive a config round trip:
--force (force_delete), --delete-excluded, --prune-empty-dirs and the
--max-delete number (default -1 == no client limit). */
static void test_config_delete_policy_wire_roundtrip() {
if (is_running_under_valgrind())
return;
struct {
bool force_delete, delete_excluded, prune_empty_dirs;
int max_delete;
} cases[] = {
{false, false, false, -1},
{true, false, false, 0},
{false, true, true, 7},
};
for (size_t i = 0; i < sizeof(cases) / sizeof(cases[0]); i++) {
int p[2];
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
pid_t pid = fork();
if (pid == 0) {
close(p[1]);
io_set_fds(p[0], p[0]);
Config* recv = config_receive(p[0]);
bool ok = recv != NULL;
if (ok) {
ok = recv->force_delete == cases[i].force_delete &&
recv->delete_excluded == cases[i].delete_excluded &&
recv->prune_empty_dirs == cases[i].prune_empty_dirs &&
recv->max_delete == cases[i].max_delete;
}
config_delete(recv);
close(p[0]);
_exit(ok ? 0 : 1);
} else {
close(p[0]);
io_set_fds(p[1], p[1]);
Config* send_cfg = config_create();
EXPECT_NOT_NULL(send_cfg);
send_cfg->send_directory = str_dup("/src");
send_cfg->receive_root_directory = str_dup("/dst");
send_cfg->force_delete = cases[i].force_delete;
send_cfg->delete_excluded = cases[i].delete_excluded;
send_cfg->prune_empty_dirs = cases[i].prune_empty_dirs;
send_cfg->max_delete = cases[i].max_delete;
bool sent = config_send(p[1], send_cfg);
int status;
waitpid(pid, &status, 0);
close(p[1]);
config_delete(send_cfg);
EXPECT_TRUE(sent);
EXPECT_TRUE(WIFEXITED(status) && WEXITSTATUS(status) == 0);
}
}
}
/* Basis-dir lists survive the config wire: each entry's type and path must
round-trip unchanged. */
static void test_config_basis_roundtrip() {
if (is_running_under_valgrind())
return;
Config* send_cfg = config_create();
EXPECT_NOT_NULL(send_cfg);
send_cfg->send_directory = str_dup("/send/src");
send_cfg->receive_root_directory = str_dup("/send/dst");
EXPECT_EQ_INT(config_basis_append(send_cfg, BASIS_DEST_LINK, "prior"), 0);
EXPECT_EQ_INT(config_basis_append(send_cfg, BASIS_DEST_COMPARE, "snap/2026-01"), 0);
EXPECT_EQ_INT(config_basis_append(send_cfg, BASIS_DEST_COPY, "copy"), 0);
int p[2];
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
io_set_fds(p[0], p[1]);
io_set_bwlimit(0);
pid_t pid = fork();
if (pid == 0) {
close(p[1]);
io_set_fds(p[0], p[0]);
Config* recv = config_receive(p[0]);
bool ok = recv != NULL && recv->basis_count == 3 && recv->basis_dirs != NULL;
if (ok) {
ok = recv->basis_dirs[0].type == BASIS_DEST_LINK &&
strcmp(recv->basis_dirs[0].path, "prior") == 0;
ok = ok && recv->basis_dirs[1].type == BASIS_DEST_COMPARE &&
strcmp(recv->basis_dirs[1].path, "snap/2026-01") == 0;
ok = ok && recv->basis_dirs[2].type == BASIS_DEST_COPY &&
strcmp(recv->basis_dirs[2].path, "copy") == 0;
}
config_delete(recv);
close(p[0]);
close(p[1]);
_exit(ok ? 0 : 1);
} else {
close(p[0]);
io_set_fds(p[1], p[1]);
bool sent = config_send(p[1], send_cfg);
int status;
waitpid(pid, &status, 0);
close(p[1]);
config_delete(send_cfg);
EXPECT_TRUE(sent);
EXPECT_TRUE(WIFEXITED(status) && WEXITSTATUS(status) == 0);
}
}
/* The receiver must reject a basis-dir path that would escape the destination
root. The values are injected directly (bypassing the client-side append
validator) so the receiver-side wire validation is what is exercised. */
static void test_config_basis_wire_rejects_escaping() {
if (is_running_under_valgrind())
return;
Config* c = config_create();
EXPECT_NOT_NULL(c);
c->send_directory = str_dup("/src");
c->receive_root_directory = str_dup("/dst");
c->basis_count = 1;
c->basis_dirs = calloc(1, sizeof(BasisDest));
c->basis_dirs[0].type = BASIS_DEST_LINK;
c->basis_dirs[0].path = str_dup("../../etc");
EXPECT_FALSE(roundtrip_config_ok(c));
config_delete(c);
c = config_create();
EXPECT_NOT_NULL(c);
c->send_directory = str_dup("/src");
c->receive_root_directory = str_dup("/dst");
c->basis_count = 1;
c->basis_dirs = calloc(1, sizeof(BasisDest));
c->basis_dirs[0].type = BASIS_DEST_LINK;
c->basis_dirs[0].path = str_dup("/abs");
EXPECT_FALSE(roundtrip_config_ok(c));
config_delete(c);
/* A well-formed list still round-trips even with a manually built struct. */
c = config_create();
EXPECT_NOT_NULL(c);
c->send_directory = str_dup("/src");
c->receive_root_directory = str_dup("/dst");
c->basis_count = 1;
c->basis_dirs = calloc(1, sizeof(BasisDest));
c->basis_dirs[0].type = BASIS_DEST_COPY;
c->basis_dirs[0].path = str_dup("safe");
EXPECT_TRUE(roundtrip_config_ok(c));
config_delete(c);
}
/* Basis-dir paths are canonicalized on the way in: trailing slashes and
interior empty / "." components are dropped so validation, the delete-walker
prefix and the receiver lookup all agree on one stored form. */
static void test_config_basis_normalization() {
Config* c = config_create();
EXPECT_NOT_NULL(c);
EXPECT_EQ_INT(config_basis_append(c, BASIS_DEST_LINK, "prior/"), 0);
EXPECT_EQ_INT(config_basis_append(c, BASIS_DEST_LINK, "a//b"), 0);
EXPECT_EQ_INT(config_basis_append(c, BASIS_DEST_LINK, "./x/./y/"), 0);
EXPECT_EQ_INT(c->basis_count, 3);
EXPECT_EQ_STR(c->basis_dirs[0].path, "prior");
EXPECT_EQ_STR(c->basis_dirs[1].path, "a/b");
EXPECT_EQ_STR(c->basis_dirs[2].path, "x/y");
/* Degenerate values that normalize away to nothing stay rejected. */
EXPECT_EQ_INT(config_basis_append(c, BASIS_DEST_LINK, "."), -1);
EXPECT_EQ_INT(config_basis_append(c, BASIS_DEST_LINK, ".."), -1);
EXPECT_EQ_INT(config_basis_append(c, BASIS_DEST_LINK, "/abs"), -1);
EXPECT_EQ_INT(config_basis_append(c, BASIS_DEST_LINK, "a/../b"), -1);
EXPECT_EQ_INT(config_basis_append(c, BASIS_DEST_LINK, ""), -1);
config_delete(c);
}
static void test_config_is_remote_dest() {
/* Valid SSH-style destinations */
EXPECT_TRUE(config_is_remote_dest("user@host:/path"));
@@ -266,6 +754,68 @@ static void test_config_is_remote_dest() {
EXPECT_TRUE(config_is_remote_dest("user@host:"));
}
/* --checksum-choice/--cc and --checksum-seed cross the wire intact so the
receiver hashes the on-disk old file with the same algorithm and seed. */
static void test_config_checksum_options_wire_roundtrip() {
if (is_running_under_valgrind())
return;
struct {
int algo;
unsigned long long seed;
} cases[] = {
{CHECKSUM_ALGO_XXH64, 0},
{CHECKSUM_ALGO_XXH64, 42},
{CHECKSUM_ALGO_MD5, 7},
{CHECKSUM_ALGO_MD5, 0},
};
for (size_t i = 0; i < sizeof(cases) / sizeof(cases[0]); i++) {
int p[2];
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
pid_t pid = fork();
if (pid == 0) {
close(p[1]);
io_set_fds(p[0], p[0]);
Config* recv = config_receive(p[0]);
bool ok = recv != NULL && recv->checksum_algo == cases[i].algo &&
recv->checksum_seed == cases[i].seed;
config_delete(recv);
close(p[0]);
_exit(ok ? 0 : 1);
} else {
close(p[0]);
io_set_fds(p[1], p[1]);
Config* send_cfg = config_create();
EXPECT_NOT_NULL(send_cfg);
send_cfg->send_directory = str_dup("/src");
send_cfg->receive_root_directory = str_dup("/dst");
send_cfg->checksum_algo = cases[i].algo;
send_cfg->checksum_seed = cases[i].seed;
bool sent = config_send(p[1], send_cfg);
int status;
waitpid(pid, &status, 0);
close(p[1]);
config_delete(send_cfg);
EXPECT_TRUE(sent);
EXPECT_TRUE(WIFEXITED(status) && WEXITSTATUS(status) == 0);
}
}
}
/* An out-of-range algorithm id on the wire must be rejected on receive, never
accepted as-is (prevents mixing unsupported digests on a path). */
static void test_config_receive_rejects_invalid_checksum_algo() {
if (is_running_under_valgrind())
return;
Config* c = config_create();
EXPECT_NOT_NULL(c);
c->send_directory = str_dup("/src");
c->receive_root_directory = str_dup("/dst");
c->checksum_algo = 99;
EXPECT_FALSE(roundtrip_config_ok(c));
config_delete(c);
}
void test_config() {
test_config_lifecycle();
test_config_ssh_dest();
@@ -277,6 +827,18 @@ void test_config() {
test_config_send_receive();
test_config_send_receive_version_mismatch();
test_config_receive_truncated();
test_config_string_null_vs_empty_roundtrip();
test_config_temp_dir_roundtrip();
test_config_delay_updates_reserved_backup_rejected();
test_config_delete_timing_wire_roundtrip();
test_config_delete_timing_conflict_rejected();
test_config_delete_policy_wire_roundtrip();
test_config_basis_roundtrip();
test_config_basis_wire_rejects_escaping();
test_config_basis_normalization();
test_config_checksum_options_wire_roundtrip();
test_config_receive_rejects_invalid_checksum_algo();
}
test_config_delete_timing_early_helper();
test_config_is_remote_dest();
}
+296
View File
@@ -0,0 +1,296 @@
#include "test_delay_updates.h"
#include "config.h"
#include "delay_updates.h"
#include "file.h"
#include "file_receive.h"
#include "test_utils.h"
#include "utils.h"
#include <dirent.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <unistd.h>
/* Recursively remove a test tree (never follows symlinks). */
static void remove_tree(const char* path) {
struct stat st;
if (lstat(path, &st) != 0)
return;
if (S_ISDIR(st.st_mode)) {
DIR* dir = opendir(path);
if (!dir)
return;
const struct dirent* entry;
while ((entry = readdir(dir)) != NULL) {
if (strcmp(entry->d_name, ".") == 0 || strcmp(entry->d_name, "..") == 0)
continue;
char* child = path_cat(path, entry->d_name);
if (child) {
remove_tree(child);
free(child);
}
}
closedir(dir);
rmdir(path);
} else {
unlink(path);
}
}
/* Build a File that carries `content`. */
static File* make_file(const char* path, const char* content) {
File* f = file_create(path);
if (!f)
return NULL;
f->data->data = malloc(strlen(content));
if (!f->data->data) {
file_destroy(f);
return NULL;
}
memcpy(f->data->data, content, strlen(content));
f->data->size = strlen(content);
return f;
}
static char* read_all(const char* path) {
FILE* fp = fopen(path, "rb");
if (!fp)
return NULL;
char buf[256] = {0};
size_t n = fread(buf, 1, sizeof(buf) - 1, fp);
fclose(fp);
char* out = malloc(n + 1);
if (!out)
return NULL;
memcpy(out, buf, n);
out[n] = '\0';
return out;
}
static void test_delay_updates_no_final_before_publish() {
const char* root = "test_delay_tmp";
remove_tree(root);
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
cfg->delay_updates = true;
File* f = make_file("sub/file.txt", "staged payload");
EXPECT_NOT_NULL(f);
// cppcheck-suppress knownConditionTrueFalse
if (!cfg || !f)
goto out;
EXPECT_EQ_INT(file_save_to_disk_full(root, f, cfg), FILE_SAVE_WRITTEN);
EXPECT_NOT_NULL(cfg->delay_context);
const char* final_path = "test_delay_tmp/sub/file.txt";
/* Before publication the final destination must not contain the file. */
EXPECT_FALSE(file_path_exists_secure(final_path));
/* The complete staged copy must live inside the staging tree. */
char* staged = path_cat("test_delay_tmp/.fastsync-stage", "/sub/file.txt");
EXPECT_NOT_NULL(staged);
// cppcheck-suppress knownConditionTrueFalse
if (staged) {
char* content = read_all(staged);
EXPECT_NOT_NULL(content);
// cppcheck-suppress knownConditionTrueFalse
if (content) {
EXPECT_EQ_STR(content, "staged payload");
free(content);
}
free(staged);
}
out:
file_destroy(f);
config_delete(cfg);
remove_tree(root);
}
static void test_delay_updates_publish_installs_files() {
const char* root = "test_delay_pub_tmp";
remove_tree(root);
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
cfg->delay_updates = true;
File* f = make_file("sub/file.txt", "published payload");
EXPECT_NOT_NULL(f);
// cppcheck-suppress knownConditionTrueFalse
if (!cfg || !f)
goto out;
EXPECT_EQ_INT(file_save_to_disk_full(root, f, cfg), FILE_SAVE_WRITTEN);
const char* final_path = "test_delay_pub_tmp/sub/file.txt";
EXPECT_FALSE(file_path_exists_secure(final_path));
EXPECT_TRUE(delay_updates_publish(cfg->delay_context, cfg));
/* After a successful publish the file is installed and staging is gone. */
char* content = read_all(final_path);
EXPECT_NOT_NULL(content);
// cppcheck-suppress knownConditionTrueFalse
if (content) {
EXPECT_EQ_STR(content, "published payload");
free(content);
}
EXPECT_FALSE(file_path_exists_secure("test_delay_pub_tmp/.fastsync-stage"));
out:
file_destroy(f);
config_delete(cfg);
remove_tree(root);
}
/* The staged tree is cleaned on the error/abort path and final files that were
never published do not appear at the destination. */
static void test_delay_updates_cleanup_removes_staged() {
const char* root = "test_delay_clean_tmp";
remove_tree(root);
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
cfg->delay_updates = true;
File* f = make_file("sub/file.txt", "never installed");
EXPECT_NOT_NULL(f);
// cppcheck-suppress knownConditionTrueFalse
if (!cfg || !f)
goto out;
EXPECT_EQ_INT(file_save_to_disk_full(root, f, cfg), FILE_SAVE_WRITTEN);
EXPECT_TRUE(file_path_exists_secure("test_delay_clean_tmp/.fastsync-stage/sub/file.txt"));
delay_updates_cleanup(cfg->delay_context);
EXPECT_FALSE(file_path_exists_secure("test_delay_clean_tmp/.fastsync-stage"));
EXPECT_FALSE(file_path_exists_secure("test_delay_clean_tmp/sub/file.txt"));
out:
file_destroy(f);
config_delete(cfg);
remove_tree(root);
}
/* With --backup the previous version is only moved aside at publication. */
static void test_delay_updates_backup_deferred_to_publish() {
const char* root = "test_delay_bak_tmp";
remove_tree(root);
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
cfg->delay_updates = true;
cfg->backup = true;
EXPECT_TRUE(file_write_to_disk("test_delay_bak_tmp/file.txt", "AAAA", 4, false, false));
File* f = make_file("file.txt", "BBBB");
EXPECT_NOT_NULL(f);
// cppcheck-suppress knownConditionTrueFalse
if (!cfg || !f)
goto out;
EXPECT_EQ_INT(file_save_to_disk_full(root, f, cfg), FILE_SAVE_WRITTEN);
/* Stage time must not touch the final file or create the backup yet. */
char* before = read_all("test_delay_bak_tmp/file.txt");
EXPECT_NOT_NULL(before);
// cppcheck-suppress knownConditionTrueFalse
if (before) {
EXPECT_EQ_STR(before, "AAAA");
free(before);
}
EXPECT_FALSE(file_path_exists_secure("test_delay_bak_tmp/file.txt~"));
EXPECT_TRUE(delay_updates_publish(cfg->delay_context, cfg));
char* after = read_all("test_delay_bak_tmp/file.txt");
char* backup = read_all("test_delay_bak_tmp/file.txt~");
EXPECT_NOT_NULL(after);
EXPECT_NOT_NULL(backup);
// cppcheck-suppress knownConditionTrueFalse
if (after) {
EXPECT_EQ_STR(after, "BBBB");
free(after);
}
// cppcheck-suppress knownConditionTrueFalse
if (backup) {
EXPECT_EQ_STR(backup, "AAAA");
free(backup);
}
out:
file_destroy(f);
config_delete(cfg);
remove_tree(root);
}
/* Skip/update policy checks run against the final path at stage time, matching
what an immediate run would decide. */
static void test_delay_updates_skip_semantics() {
const char* root = "test_delay_skip_tmp";
remove_tree(root);
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
cfg->delay_updates = true;
/* --existing: final destination missing -> skipped, nothing staged. */
File* missing = make_file("missing.txt", "new");
EXPECT_NOT_NULL(missing);
// cppcheck-suppress knownConditionTrueFalse
if (!cfg || !missing)
goto out;
cfg->existing = true;
EXPECT_EQ_INT(file_save_to_disk_full(root, missing, cfg), FILE_SAVE_SKIPPED);
cfg->existing = false;
/* --ignore-existing: final destination present -> skipped. */
EXPECT_TRUE(file_write_to_disk("test_delay_skip_tmp/existing.txt", "old", 3, false, false));
File* present = make_file("existing.txt", "new");
EXPECT_NOT_NULL(present);
// cppcheck-suppress knownConditionTrueFalse
if (!present)
goto out;
cfg->ignore_existing = true;
EXPECT_EQ_INT(file_save_to_disk_full(root, present, cfg), FILE_SAVE_SKIPPED);
cfg->ignore_existing = false;
/* Without a skip flag the file is staged and later published. */
File* fresh = make_file("fresh.txt", "content");
EXPECT_NOT_NULL(fresh);
// cppcheck-suppress knownConditionTrueFalse
if (!fresh)
goto out;
EXPECT_EQ_INT(file_save_to_disk_full(root, fresh, cfg), FILE_SAVE_WRITTEN);
EXPECT_TRUE(delay_updates_publish(cfg->delay_context, cfg));
char* content = read_all("test_delay_skip_tmp/fresh.txt");
EXPECT_NOT_NULL(content);
// cppcheck-suppress knownConditionTrueFalse
if (content) {
EXPECT_EQ_STR(content, "content");
free(content);
}
out:
file_destroy(missing);
file_destroy(present);
file_destroy(fresh);
config_delete(cfg);
remove_tree(root);
}
/* The reserved staging name must be recognizable for validation, including
with a trailing slash. */
static void test_delay_updates_reserved_name_helper() {
EXPECT_TRUE(delay_updates_staging_name_conflict(".fastsync-stage"));
EXPECT_TRUE(delay_updates_staging_name_conflict(".fastsync-stage/"));
EXPECT_TRUE(delay_updates_staging_name_conflict(".fastsync-stage///"));
EXPECT_FALSE(delay_updates_staging_name_conflict(NULL));
EXPECT_FALSE(delay_updates_staging_name_conflict(""));
EXPECT_FALSE(delay_updates_staging_name_conflict("backups"));
EXPECT_FALSE(delay_updates_staging_name_conflict(".fastsync-stage.bak"));
}
void test_delay_updates() {
test_delay_updates_reserved_name_helper();
test_delay_updates_no_final_before_publish();
test_delay_updates_publish_installs_files();
test_delay_updates_cleanup_removes_staged();
test_delay_updates_backup_deferred_to_publish();
test_delay_updates_skip_semantics();
}
+6
View File
@@ -0,0 +1,6 @@
#ifndef TEST_DELAY_UPDATES_H
#define TEST_DELAY_UPDATES_H
void test_delay_updates(void);
#endif
+347
View File
@@ -343,12 +343,356 @@ static void test_delta_apply_rejects_output_overflow() {
EXPECT_TRUE(delta_apply(old_data, sizeof(old_data), &delta, 1) == NULL);
}
/* ---------------------------------------------------------------------------
* Hash-index lookup differential tests.
*
* delta_compute buckets signature blocks by their weak checksum. These tests
* prove the bucket-indexed candidate lookup is behaviour-identical to the
* original per-window linear scan: the emitted instruction stream (types,
* lengths, literal bytes and chosen block indices) must match a naive linear
* reference exactly, and the delta must reconstruct the new buffer.
* ------------------------------------------------------------------------- */
#define REF_NO_MATCH UINT32_MAX
typedef struct {
DeltaInstruction* items;
uint32_t count;
uint32_t cap;
} RefDelta;
static void ref_delta_free(RefDelta* ref) {
if (!ref->items)
return;
for (uint32_t i = 0; i < ref->count; i++)
if (ref->items[i].type == DELTA_INSTR_LITERAL)
free(ref->items[i].literal.data);
free(ref->items);
ref->items = NULL;
ref->count = 0;
ref->cap = 0;
}
static bool ref_delta_push(RefDelta* ref, DeltaInstruction instr) {
if (ref->count == ref->cap) {
uint32_t new_cap = ref->cap ? ref->cap * 2 : 16;
DeltaInstruction* tmp = realloc(ref->items, (size_t)new_cap * sizeof(DeltaInstruction));
if (!tmp)
return false;
ref->items = tmp;
ref->cap = new_cap;
}
ref->items[ref->count++] = instr;
return true;
}
static bool ref_delta_flush_literal(RefDelta* ref, const uint8_t* data, uint64_t start,
uint64_t end) {
if (start >= end)
return true;
uint8_t* lit = malloc((size_t)(end - start));
if (!lit)
return false;
memcpy(lit, data + start, (size_t)(end - start));
DeltaInstruction instr = {
.type = DELTA_INSTR_LITERAL,
.literal = {.data = lit, .length = (uint32_t)(end - start)},
};
return ref_delta_push(ref, instr);
}
/* Naive O(windows x blocks) re-implementation of the historical delta_compute
* candidate scan: only full windows may match, a candidate needs both the weak
* (Adler-32) and strong (xxHash32) checksums to agree, and the lowest matching
* block index is selected. */
static bool ref_delta_build(RefDelta* ref, const uint8_t* new_data, uint64_t new_size,
const DeltaSignature* sig) {
uint32_t block_size = sig->block_size;
uint64_t i = 0;
uint64_t literal_start = 0;
bool has_literal = false;
while (i < new_size) {
uint32_t window_len = (uint32_t)((new_size - i < block_size) ? (new_size - i) : block_size);
bool full_window = (window_len == block_size);
uint32_t matched = REF_NO_MATCH;
if (full_window) {
uint32_t adler = delta_adler32(new_data + i, window_len);
for (uint32_t j = 0; j < sig->block_count; j++) {
if (sig->blocks[j].adler32 == adler &&
delta_xxhash32(new_data + i, window_len) == sig->blocks[j].xxhash) {
matched = j;
break;
}
}
}
if (matched != REF_NO_MATCH) {
if (has_literal) {
if (!ref_delta_flush_literal(ref, new_data, literal_start, i))
return false;
has_literal = false;
}
DeltaInstruction instr = {
.type = DELTA_INSTR_BLOCK_MATCH,
.match = {.block_index = matched, .block_offset = 0, .length = window_len},
};
if (!ref_delta_push(ref, instr))
return false;
i += window_len;
} else {
if (!has_literal) {
literal_start = i;
has_literal = true;
}
i++;
}
}
if (has_literal && !ref_delta_flush_literal(ref, new_data, literal_start, new_size))
return false;
return true;
}
static bool ref_delta_matches(const RefDelta* ref, const Delta* delta) {
if (ref->count != delta->instruction_count)
return false;
for (uint32_t i = 0; i < ref->count; i++) {
const DeltaInstruction* a = &ref->items[i];
const DeltaInstruction* b = &delta->instructions[i];
if (a->type != b->type)
return false;
if (a->type == DELTA_INSTR_BLOCK_MATCH) {
if (a->match.block_index != b->match.block_index ||
a->match.block_offset != b->match.block_offset || a->match.length != b->match.length)
return false;
} else {
if (a->literal.length != b->literal.length ||
memcmp(a->literal.data, b->literal.data, a->literal.length) != 0)
return false;
}
}
return true;
}
static void expect_linear_reference_match(const uint8_t* old_data, uint64_t old_size,
const uint8_t* new_data, uint64_t new_size,
uint32_t block_size, const char* label) {
DeltaSignature* sig = delta_signature_create(old_data, old_size, block_size);
if (!sig) {
printf(" [FAIL] %s: signature creation failed\n", label);
EXPECT_NOT_NULL(sig);
return;
}
Delta* delta = delta_compute(new_data, new_size, sig, block_size);
if (!delta) {
printf(" [FAIL] %s: delta_compute returned NULL\n", label);
delta_signature_destroy(sig);
EXPECT_NOT_NULL(delta);
return;
}
RefDelta ref = {0};
bool ok = ref_delta_build(&ref, new_data, new_size, sig);
if (ok)
ok = ref_delta_matches(&ref, delta);
if (!ok) {
printf(" [FAIL] %s: instruction stream differs from linear reference "
"(linear=%u indexed=%u)\n",
label, ref.count, delta->instruction_count);
}
ref_delta_free(&ref);
delta_destroy(delta);
delta_signature_destroy(sig);
EXPECT_TRUE(ok);
}
static void fill_delta_pattern(uint8_t* buf, uint64_t size, uint32_t seed) {
uint32_t x = seed ? seed : 1;
for (uint64_t i = 0; i < size; i++) {
x ^= x << 13;
x ^= x >> 17;
x ^= x << 5;
buf[i] = (uint8_t)(x >> 24);
}
}
static void test_delta_hash_index_matches_linear_reference() {
/* Identical file (full block alignment). */
uint8_t old_a[32768];
uint8_t new_a[32768];
fill_delta_pattern(old_a, sizeof(old_a), 42);
memcpy(new_a, old_a, sizeof(old_a));
expect_linear_reference_match(old_a, sizeof(old_a), new_a, sizeof(new_a), 2048,
"identical 32KiB @ 2KiB");
/* Scattered single-byte edits in the middle of each block. */
uint8_t new_b[32768];
memcpy(new_b, old_a, sizeof(old_a));
for (size_t p = 100; p < sizeof(new_b); p += 4096)
new_b[p] ^= 0x5A;
expect_linear_reference_match(old_a, sizeof(old_a), new_b, sizeof(new_b), 2048,
"32KiB scattered single-byte edits @ 2KiB");
/* Non-aligned old file (partial final block) with a single edit. */
uint8_t old_c[30000];
uint8_t new_c[30000];
fill_delta_pattern(old_c, sizeof(old_c), 7);
memcpy(new_c, old_c, sizeof(old_c));
new_c[15000] ^= 0x3C;
expect_linear_reference_match(old_c, sizeof(old_c), new_c, sizeof(new_c), 2048,
"30KiB partial-tail single edit @ 2KiB");
/* Growth: appended data after an identical prefix. */
uint8_t old_d[24576];
uint8_t new_d[34576];
fill_delta_pattern(old_d, sizeof(old_d), 11);
memcpy(new_d, old_d, sizeof(old_d));
fill_delta_pattern(new_d + sizeof(old_d), sizeof(new_d) - sizeof(old_d), 23);
expect_linear_reference_match(old_d, sizeof(old_d), new_d, sizeof(new_d), 2048,
"24KiB -> 34KiB appended @ 2KiB");
/* Insertion shifting everything after the edit point (rsync re-sync). */
uint8_t old_e[65536];
uint8_t new_e[65536 + 3000];
fill_delta_pattern(old_e, sizeof(old_e), 99);
memcpy(new_e, old_e, 20000);
fill_delta_pattern(new_e + 20000, 3000, 101);
memcpy(new_e + 23000, old_e + 20000, sizeof(old_e) - 20000);
expect_linear_reference_match(old_e, sizeof(old_e), new_e, sizeof(new_e), 2048,
"64KiB + 3KiB insertion @ 2KiB");
/* Deletion shrinking the file. */
uint8_t new_f[sizeof(old_e) - 5000];
memcpy(new_f, old_e, 30000);
memcpy(new_f + 30000, old_e + 35000, sizeof(old_e) - 35000);
expect_linear_reference_match(old_e, sizeof(old_e), new_f, sizeof(new_f), 2048,
"64KiB - 5KiB deletion @ 2KiB");
/* Repeated identical blocks must resolve to the lowest block index. */
uint8_t old_g[4 * 4096];
uint8_t new_g[4 * 4096];
for (uint32_t b = 0; b < 4; b++)
fill_delta_pattern(old_g + b * 4096, 4096, b % 2 == 0 ? 500 : 501); /* block0==block2 */
memcpy(new_g, old_g, sizeof(old_g));
new_g[4096 + 5] ^= 0x11; /* edit inside the second (duplicated) chunk */
expect_linear_reference_match(old_g, sizeof(old_g), new_g, sizeof(new_g), 4096,
"duplicated chunks @ 4KiB");
/* Block larger than the file: nothing can match, all literal. */
uint8_t old_h[1000];
uint8_t new_h[1000];
fill_delta_pattern(old_h, sizeof(old_h), 3);
memcpy(new_h, old_h, sizeof(old_h));
expect_linear_reference_match(old_h, sizeof(old_h), new_h, sizeof(new_h), 4096,
"1KiB file @ 4KiB block");
}
static void test_delta_hash_index_large_mostly_matching() {
const uint64_t size = 4ULL * 1024 * 1024;
const uint32_t block_size = 8192;
uint8_t* old_data = malloc((size_t)size);
uint8_t* new_data = malloc((size_t)size);
EXPECT_TRUE(old_data != NULL && new_data != NULL);
fill_delta_pattern(old_data, size, 1234);
memcpy(new_data, old_data, (size_t)size);
/* Scattered single-byte changes across the whole buffer. Each change forces
* the diff to re-synchronise by walking one byte at a time through the
* affected block, which is exactly the case that used to cost O(bytes x
* blocks) with the linear scan. */
const uint64_t nchanges = 64;
for (uint64_t c = 0; c < nchanges; c++) {
uint64_t pos = (c * (size / nchanges)) + (c % 17);
new_data[pos] ^= (uint8_t)(0xA0 + (c % 16));
}
DeltaSignature* sig = delta_signature_create(old_data, size, block_size);
EXPECT_NOT_NULL(sig);
EXPECT_EQ_INT((int)sig->block_count, (int)(size / block_size));
Delta* delta = delta_compute(new_data, size, sig, block_size);
EXPECT_NOT_NULL(delta);
EXPECT_EQ_INT((int)delta->new_file_size, (int)size);
uint32_t match_count = 0;
for (uint32_t i = 0; i < delta->instruction_count; i++)
if (delta->instructions[i].type == DELTA_INSTR_BLOCK_MATCH)
match_count++;
EXPECT_TRUE(match_count > 0);
void* result = delta_apply(old_data, size, delta, block_size);
EXPECT_NOT_NULL(result);
EXPECT_EQ_INT(memcmp(result, new_data, (size_t)size), 0);
free(result);
delta_destroy(delta);
delta_signature_destroy(sig);
free(old_data);
free(new_data);
}
/* --checksum-seed: the delta strong (block) hash is genuinely seed-aware. A
* nonzero seed changes the per-block xxHash32, and a signature + delta computed
* with the same seed still reconstruct the file exactly (symmetric), while a
* mismatched seed produces a delta that does not match the signature blocks. */
static void test_delta_xxhash32_seeded() {
const char* data = "seedme";
uint32_t a = delta_xxhash32(data, 6);
uint32_t b = delta_xxhash32_seeded(data, 6, 42);
uint32_t c = delta_xxhash32_seeded(data, 6, 42);
EXPECT_TRUE(a != b);
EXPECT_EQ_INT((int)b, (int)c);
/* Unseeded == seeded with 0 (default reproduces today's behavior). */
EXPECT_EQ_INT((int)delta_xxhash32(data, 6), (int)delta_xxhash32_seeded(data, 6, 0));
}
static void test_delta_seeded_signature_compute_matches() {
uint32_t block_size = 1024;
/* Identical old/new data with a non-zero seed: the receiver builds a seeded
signature and the sender computes a seeded delta over the same bytes, so
every block matches and applying the delta rebuilds the file exactly. */
char data[4096];
for (int i = 0; i < 4096; i++)
data[i] = (char)(i % 256);
DeltaSignature* sig = delta_signature_create_seeded(data, 4096, block_size, 99);
EXPECT_NOT_NULL(sig);
Delta* delta = delta_compute_seeded(data, 4096, sig, block_size, 99);
EXPECT_NOT_NULL(delta);
void* rebuilt = delta_apply(data, 4096, delta, block_size);
EXPECT_NOT_NULL(rebuilt);
EXPECT_TRUE(memcmp(rebuilt, data, 4096) == 0);
free(rebuilt);
delta_destroy(delta);
delta_signature_destroy(sig);
/* A MISMATCHED seed means the sender's window xxHash32 never equals the
receiver's signature-block xxHash32: no block can match, so the delta is
not worthwhile / has no block matches. This proves the seed really gates
the block comparison rather than being an inert parameter. */
sig = delta_signature_create_seeded(data, 4096, block_size, 99);
EXPECT_NOT_NULL(sig);
delta = delta_compute_seeded(data, 4096, sig, block_size, 7);
EXPECT_NOT_NULL(delta);
bool any_match = false;
for (uint32_t i = 0; i < delta->instruction_count; i++)
if (delta->instructions[i].type == DELTA_INSTR_BLOCK_MATCH)
any_match = true;
EXPECT_FALSE(any_match);
delta_destroy(delta);
delta_signature_destroy(sig);
}
void test_delta() {
test_adler32_basic();
test_adler32_different_data();
test_xxhash32_basic();
test_xxhash32_different_data();
test_xxhash64_different_data();
test_delta_xxhash32_seeded();
test_signature_roundtrip();
test_delta_identical_files();
test_delta_small_edit();
@@ -360,4 +704,7 @@ void test_delta() {
test_is_worthwhile();
test_large_file_delta();
test_delta_apply_rejects_output_overflow();
test_delta_hash_index_matches_linear_reference();
test_delta_hash_index_large_mostly_matching();
test_delta_seeded_signature_compute_matches();
}
+482
View File
@@ -1,13 +1,16 @@
#include "test_file.h"
#include "file.h"
#include "data.h"
#include "config.h"
#include "utils.h"
#include "protocol.h"
#include "test_utils.h"
#include <fcntl.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <sys/wait.h>
#include <time.h>
#include <unistd.h>
static void test_file_create() {
@@ -88,6 +91,272 @@ static void test_file_save_to_disk() {
rmdir("test_save_tmp");
}
static void test_file_save_to_disk_with_fsync_config() {
File* f = file_create("saved_file_fsync.txt");
EXPECT_NOT_NULL(f);
const char* content = "Save to disk with fsync";
f->data->data = malloc(strlen(content));
EXPECT_NOT_NULL(f->data->data);
memcpy(f->data->data, content, strlen(content));
f->data->size = strlen(content);
Config* config = config_create();
EXPECT_NOT_NULL(config);
config->use_fsync = true;
EXPECT_TRUE(file_save_to_disk("test_save_fsync_tmp", f, config));
struct stat st;
EXPECT_EQ_INT(stat("test_save_fsync_tmp/saved_file_fsync.txt", &st), 0);
EXPECT_EQ_INT((int)st.st_size, (int)strlen(content));
file_destroy(f);
config_delete(config);
unlink("test_save_fsync_tmp/saved_file_fsync.txt");
rmdir("test_save_fsync_tmp");
}
static void test_file_save_to_disk_existing() {
const char* root = "test_existing_tmp";
const char* existing_path = "test_existing_tmp/existing.txt";
const char* missing_path = "test_existing_tmp/missing.txt";
EXPECT_TRUE(file_write_to_disk(existing_path, "old", 3, false, false));
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
cfg->existing = true;
File* existing = file_create("existing.txt");
EXPECT_NOT_NULL(existing);
existing->data->data = malloc(3);
EXPECT_NOT_NULL(existing->data->data);
memcpy(existing->data->data, "new", 3);
existing->data->size = 3;
EXPECT_TRUE(file_save_to_disk(root, existing, cfg));
file_destroy(existing);
File* missing = file_create("missing.txt");
EXPECT_NOT_NULL(missing);
missing->data->data = malloc(7);
EXPECT_NOT_NULL(missing->data->data);
memcpy(missing->data->data, "skipped", 7);
missing->data->size = 7;
EXPECT_TRUE(file_save_to_disk(root, missing, cfg));
file_destroy(missing);
FILE* fp = fopen(existing_path, "rb");
char content[4] = {0};
EXPECT_NOT_NULL(fp);
// cppcheck-suppress knownConditionTrueFalse
if (fp) {
EXPECT_EQ_INT((int)fread(content, 1, 3, fp), 3);
fclose(fp);
}
EXPECT_EQ_STR(content, "new");
EXPECT_EQ_INT(access(missing_path, F_OK), -1);
config_delete(cfg);
unlink(existing_path);
rmdir("test_existing_tmp");
}
static void test_file_save_to_disk_ignore_existing() {
const char* path = "test_ignore_existing_tmp/existing.txt";
EXPECT_TRUE(file_write_to_disk(path, "old", 3, false, false));
File* file = file_create("existing.txt");
EXPECT_NOT_NULL(file);
file->data->data = malloc(3);
EXPECT_NOT_NULL(file->data->data);
memcpy(file->data->data, "new", 3);
file->data->size = 3;
Config* config = config_create();
EXPECT_NOT_NULL(config);
config->ignore_existing = true;
EXPECT_TRUE(file_save_to_disk("test_ignore_existing_tmp", file, config));
FILE* stream = fopen(path, "rb");
char content[4] = {0};
EXPECT_NOT_NULL(stream);
// cppcheck-suppress knownConditionTrueFalse
if (stream) {
EXPECT_EQ_INT((int)fread(content, 1, 3, stream), 3);
fclose(stream);
}
EXPECT_EQ_STR(content, "old");
file_destroy(file);
config_delete(config);
unlink(path);
rmdir("test_ignore_existing_tmp");
}
static void test_file_save_to_disk_ignore_existing_entry_types() {
const char* root = "test_ignore_existing_entries_tmp";
const char* directory = "test_ignore_existing_entries_tmp/directory";
const char* link = "test_ignore_existing_entries_tmp/link";
const char* target = "test_ignore_existing_entries_tmp/target";
const char* backup = "test_ignore_existing_entries_tmp/backup.txt~";
const char* backup_file = "test_ignore_existing_entries_tmp/backup.txt";
Config* config = config_create();
File* file = file_create("unused");
unlink(link);
unlink(target);
unlink(backup);
unlink(backup_file);
rmdir(directory);
rmdir(root);
EXPECT_NOT_NULL(config);
EXPECT_NOT_NULL(file);
// cppcheck-suppress knownConditionTrueFalse
if (!config || !file)
return;
config->ignore_existing = true;
config->backup = true;
file->data->data = malloc(3);
EXPECT_NOT_NULL(file->data->data);
// cppcheck-suppress knownConditionTrueFalse
if (!file->data->data) {
file_destroy(file);
config_delete(config);
return;
}
memcpy(file->data->data, "new", 3);
file->data->size = 3;
EXPECT_EQ_INT(mkdir(root, 0755), 0);
EXPECT_EQ_INT(mkdir(directory, 0755), 0);
EXPECT_TRUE(file_write_to_disk(target, "old", 3, false, false));
EXPECT_EQ_INT(symlink("target", link), 0);
free(file->path);
file->path = str_dup("directory");
EXPECT_TRUE(file_save_to_disk(root, file, config));
free(file->path);
file->path = str_dup("link");
EXPECT_TRUE(file_save_to_disk(root, file, config));
free(file->path);
file->path = str_dup("backup.txt");
EXPECT_TRUE(file_write_to_disk(backup_file, "old", 3, false, false));
EXPECT_TRUE(file_save_to_disk(root, file, config));
EXPECT_TRUE(file_path_exists_secure(backup_file));
EXPECT_FALSE(file_path_exists_secure(backup));
file_destroy(file);
config_delete(config);
unlink(link);
unlink(target);
unlink(backup_file);
rmdir(directory);
rmdir(root);
}
/* Issue #253: with --partial --partial-dir a completed write must be installed
at the real destination rather than left under the partial directory. */
static void test_file_save_to_disk_partial_install() {
const char* root = "test_partial_install_tmp";
const char* dest_file = "test_partial_install_tmp/file.txt";
const char* partial_file = "test_partial_install_tmp/.partial/file.txt";
unlink(dest_file);
unlink(partial_file);
rmdir("test_partial_install_tmp/.partial");
rmdir(root);
File* f = file_create("file.txt");
EXPECT_NOT_NULL(f);
const char* content = "partial-dir content";
f->data->data = malloc(strlen(content));
EXPECT_NOT_NULL(f->data->data);
memcpy(f->data->data, content, strlen(content));
f->data->size = strlen(content);
Config* config = config_create();
EXPECT_NOT_NULL(config);
config->partial = true;
config->partial_dir = str_dup(".partial");
EXPECT_EQ_INT(file_save_to_disk_full(root, f, config), FILE_SAVE_WRITTEN);
FILE* fp = fopen(dest_file, "rb");
EXPECT_NOT_NULL(fp);
// cppcheck-suppress knownConditionTrueFalse
if (fp) {
char buf[64] = {0};
size_t nread = fread(buf, 1, sizeof(buf) - 1, fp);
fclose(fp);
EXPECT_EQ_INT((int)nread, (int)strlen(content));
EXPECT_EQ_INT(memcmp(buf, content, strlen(content)), 0);
}
/* A completed transfer must not linger under the partial dir. */
EXPECT_EQ_INT(access(partial_file, F_OK), -1);
file_destroy(f);
config_delete(config);
unlink(dest_file);
rmdir(root);
}
/* Issue #251: file_save_to_disk_full must distinguish receiver-side skips
(--existing/--ignore-existing/--update) from real writes so the sender can
decide whether --remove-source-files may unlink its source. */
static void test_file_save_to_disk_reports_skips() {
const char* root = "test_save_skip_tmp";
const char* existing_path = "test_save_skip_tmp/existing.txt";
unlink(existing_path);
rmdir(root);
EXPECT_TRUE(file_write_to_disk(existing_path, "old", 3, false, false));
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
File* new_file = file_create("missing.txt");
EXPECT_NOT_NULL(new_file);
new_file->data->data = malloc(7);
EXPECT_NOT_NULL(new_file->data->data);
memcpy(new_file->data->data, "skipped", 7);
new_file->data->size = 7;
/* --existing: destination is missing -> skipped, not an error. */
cfg->existing = true;
EXPECT_EQ_INT(file_save_to_disk_full(root, new_file, cfg), FILE_SAVE_SKIPPED);
cfg->existing = false;
/* --ignore-existing: destination present -> skipped. */
File* present = file_create("existing.txt");
EXPECT_NOT_NULL(present);
present->data->data = malloc(3);
EXPECT_NOT_NULL(present->data->data);
memcpy(present->data->data, "new", 3);
present->data->size = 3;
cfg->ignore_existing = true;
EXPECT_EQ_INT(file_save_to_disk_full(root, present, cfg), FILE_SAVE_SKIPPED);
cfg->ignore_existing = false;
/* A normal overwrite of an existing file is a real write. */
EXPECT_EQ_INT(file_save_to_disk_full(root, present, cfg), FILE_SAVE_WRITTEN);
/* --update: a newer destination is skipped. */
struct stat st;
EXPECT_EQ_INT(stat(existing_path, &st), 0);
time_t now = time(NULL);
FileMetadata metadata = {.mode = st.st_mode,
.uid = st.st_uid,
.gid = st.st_gid,
.mtime_sec = now - 100,
.mtime_nsec = 0};
present->metadata = &metadata;
cfg->update = true;
EXPECT_EQ_INT(file_save_to_disk_full(root, present, cfg), FILE_SAVE_SKIPPED);
present->metadata = NULL;
file_destroy(new_file);
file_destroy(present);
config_delete(cfg);
unlink(existing_path);
rmdir(root);
}
static void test_file_write_to_disk_basic() {
const char* content = "Basic file_write_to_disk test";
EXPECT_TRUE(file_write_to_disk("test_file_write_to_disk_basic.txt", content, strlen(content),
@@ -108,6 +377,17 @@ static void test_file_write_to_disk_basic() {
unlink("test_file_write_to_disk_basic.txt");
}
static void test_file_write_to_disk_with_fsync() {
const char* path = "test_file_write_to_disk_fsync.txt";
const char* content = "fsync file content";
EXPECT_TRUE(file_to_disk_secure_with_fsync(path, content, strlen(content), false, false, NULL,
false, true, NULL));
struct stat st;
EXPECT_EQ_INT(stat(path, &st), 0);
EXPECT_EQ_INT((int)st.st_size, (int)strlen(content));
unlink(path);
}
static void test_file_write_to_disk_creates_dirs() {
const char* content = "Nested dir test";
EXPECT_TRUE(file_write_to_disk("test_nested_tmp/nested/file.txt", content, strlen(content), false,
@@ -451,6 +731,197 @@ static void test_file_send_single_calls_metadata_and_path() {
}
}
static void test_inplace_overwrite_clears_special_mode_bits() {
const char* root = "test_inplace_tmp";
const char* path = "test_inplace_tmp/priv.txt";
const char* content = "olddata";
unlink(path);
rmdir(root);
EXPECT_EQ_INT(mkdir(root, 0700), 0);
/* Create a destination carrying setuid + sticky bits. */
int fd = open(path, O_WRONLY | O_CREAT | O_TRUNC | O_CLOEXEC, 0644);
EXPECT_TRUE(fd >= 0);
// cppcheck-suppress knownConditionTrueFalse
if (fd < 0) {
rmdir(root);
return;
}
EXPECT_EQ_INT((int)write(fd, content, strlen(content)), (int)strlen(content));
EXPECT_EQ_INT(fchmod(fd, S_ISUID | S_ISVTX | 0755), 0);
EXPECT_EQ_INT(close(fd), 0);
/* Overwrite in place without metadata: the mode must be normalized to a
safe default (0644) and the setuid/sticky bits must be gone. */
File* f = file_create("priv.txt");
EXPECT_NOT_NULL(f);
const char* new_content = "newdata";
f->data->data = malloc(strlen(new_content));
EXPECT_NOT_NULL(f->data->data);
memcpy(f->data->data, new_content, strlen(new_content));
f->data->size = strlen(new_content);
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
cfg->inplace = true;
EXPECT_TRUE(file_save_to_disk(root, f, cfg));
file_destroy(f);
config_delete(cfg);
struct stat st;
EXPECT_EQ_INT(stat(path, &st), 0);
EXPECT_EQ_INT((int)(st.st_mode & (S_ISUID | S_ISGID | S_ISVTX)), 0);
EXPECT_EQ_INT((int)(st.st_mode & 0777), 0644);
FILE* stream = fopen(path, "rb");
char buf[16] = {0};
EXPECT_NOT_NULL(stream);
// cppcheck-suppress knownConditionTrueFalse
if (stream) {
size_t nread = fread(buf, 1, sizeof(buf) - 1, stream);
fclose(stream);
EXPECT_EQ_INT((int)nread, (int)strlen(new_content));
}
EXPECT_EQ_STR(buf, new_content);
unlink(path);
rmdir(root);
}
static void test_inplace_overwrite_metadata_strips_special_bits() {
const char* root = "test_inplace_meta_tmp";
const char* path = "test_inplace_meta_tmp/meta.txt";
const char* source = "test_inplace_meta_source.txt";
unlink(path);
unlink(source);
rmdir(root);
EXPECT_EQ_INT(mkdir(root, 0700), 0);
/* Existing destination with setuid+sticky set. */
int fd = open(path, O_WRONLY | O_CREAT | O_TRUNC | O_CLOEXEC, 0644);
EXPECT_TRUE(fd >= 0);
// cppcheck-suppress knownConditionTrueFalse
if (fd < 0) {
rmdir(root);
return;
}
EXPECT_EQ_INT((int)write(fd, "olddata", 7), 7);
EXPECT_EQ_INT(fchmod(fd, S_ISUID | S_ISVTX | 0755), 0);
EXPECT_EQ_INT(close(fd), 0);
/* Build source metadata carrying a plain executable mode (no specials). */
EXPECT_TRUE(file_write_to_disk(source, "source", 6, false, false));
EXPECT_EQ_INT(chmod(source, 0755), 0);
struct stat source_st;
EXPECT_EQ_INT(stat(source, &source_st), 0);
File* f = file_create("meta.txt");
EXPECT_NOT_NULL(f);
const char* new_content = "meta";
f->data->data = malloc(strlen(new_content));
EXPECT_NOT_NULL(f->data->data);
memcpy(f->data->data, new_content, strlen(new_content));
f->data->size = strlen(new_content);
f->metadata = file_metadata_create(&source_st);
EXPECT_NOT_NULL(f->metadata);
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
cfg->inplace = true;
EXPECT_TRUE(file_save_to_disk(root, f, cfg));
file_destroy(f);
config_delete(cfg);
unlink(source);
struct stat st;
EXPECT_EQ_INT(stat(path, &st), 0);
/* Metadata-derived mode is applied and never includes setuid/setgid/sticky. */
EXPECT_EQ_INT((int)(st.st_mode & (S_ISUID | S_ISGID | S_ISVTX)), 0);
EXPECT_EQ_INT((int)(st.st_mode & 0777), 0755);
unlink(path);
rmdir(root);
}
static void test_inplace_overwrite_truncates_shorter_payload() {
const char* root = "test_inplace_trunc_tmp";
const char* path = "test_inplace_trunc_tmp/big.txt";
unlink(path);
rmdir(root);
EXPECT_EQ_INT(mkdir(root, 0700), 0);
const char* old_content = "0123456789abcdef"; /* 16 bytes */
EXPECT_TRUE(file_write_to_disk(path, old_content, strlen(old_content), false, false));
File* f = file_create("big.txt");
EXPECT_NOT_NULL(f);
const char* new_content = "hi";
f->data->data = malloc(strlen(new_content));
EXPECT_NOT_NULL(f->data->data);
memcpy(f->data->data, new_content, strlen(new_content));
f->data->size = strlen(new_content);
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
cfg->inplace = true;
EXPECT_TRUE(file_save_to_disk(root, f, cfg));
file_destroy(f);
config_delete(cfg);
/* A shorter payload must truncate the file: no stale trailing bytes. */
struct stat st;
EXPECT_EQ_INT(stat(path, &st), 0);
EXPECT_EQ_INT((int)st.st_size, (int)strlen(new_content));
FILE* stream = fopen(path, "rb");
char buf[32] = {0};
EXPECT_NOT_NULL(stream);
// cppcheck-suppress knownConditionTrueFalse
if (stream) {
size_t nread = fread(buf, 1, sizeof(buf) - 1, stream);
fclose(stream);
EXPECT_EQ_INT((int)nread, (int)strlen(new_content));
}
EXPECT_EQ_STR(buf, new_content);
unlink(path);
rmdir(root);
}
/* Explicit directory entries (--dirs) create the directory under the receive
root through the same save funnel, creating parents as needed, and reject
traversal the same way a file path does. */
static void test_dir_entry_save_to_disk() {
const char* root = "test_dir_entry_root";
EXPECT_EQ_INT(mkdir(root, 0755), 0);
Config* config = config_create();
EXPECT_NOT_NULL(config);
File* dir = file_create("alpha/beta/gamma");
EXPECT_NOT_NULL(dir);
dir->is_dir = true;
EXPECT_EQ_INT(file_save_to_disk_full(root, dir, config), FILE_SAVE_WRITTEN);
EXPECT_EQ_INT(file_save_to_disk_full(root, dir, config), FILE_SAVE_WRITTEN);
file_destroy(dir);
struct stat st;
EXPECT_EQ_INT(stat("test_dir_entry_root/alpha/beta/gamma", &st), 0);
EXPECT_TRUE(S_ISDIR(st.st_mode));
/* The directory-entry save path never follows or escapes. */
File* evil = file_create("../dir_entry_escape");
EXPECT_NOT_NULL(evil);
evil->is_dir = true;
EXPECT_EQ_INT(file_save_to_disk_full(root, evil, config), FILE_SAVE_ERROR);
file_destroy(evil);
EXPECT_EQ_INT(lstat("../dir_entry_escape", &st), -1);
config_delete(config);
rmdir("test_dir_entry_root/alpha/beta/gamma");
rmdir("test_dir_entry_root/alpha/beta");
rmdir("test_dir_entry_root/alpha");
rmdir(root);
}
void test_file() {
test_file_create();
test_file_destroy_null();
@@ -458,12 +929,20 @@ void test_file() {
test_file_load_data();
test_file_load_data_missing_file();
test_file_save_to_disk();
test_file_save_to_disk_with_fsync_config();
test_file_save_to_disk_existing();
test_file_save_to_disk_ignore_existing();
test_file_save_to_disk_ignore_existing_entry_types();
test_file_save_to_disk_partial_install();
test_file_save_to_disk_reports_skips();
test_file_write_to_disk_basic();
test_file_write_to_disk_with_fsync();
test_file_write_to_disk_creates_dirs();
test_file_write_to_disk_does_not_follow_symlink();
test_file_content_to_buffer();
test_file_save_to_disk_path_traversal();
test_file_save_to_disk_deep_traversal();
test_dir_entry_save_to_disk();
if (!is_running_under_valgrind()) {
// Fork tests are skipped under valgrind because the parent process runs
// orders of magnitude slower than the child (parent is instrumented, child
@@ -476,4 +955,7 @@ void test_file() {
test_file_send_single_calls_metadata_and_path();
}
test_file_metadata_create();
test_inplace_overwrite_clears_special_mode_bits();
test_inplace_overwrite_metadata_strips_special_bits();
test_inplace_overwrite_truncates_shorter_payload();
}
+26
View File
@@ -1,6 +1,8 @@
#include "test_log.h"
#include "log.h"
#include "test_utils.h"
#include <string.h>
#include <unistd.h>
/* Test default log level: WARNING and ERROR should print, DEBUG and INFO should not.
* We can't easily capture stderr in unit tests, so we verify the functions don't crash
@@ -90,6 +92,29 @@ static void test_log_filtering() {
EXPECT_TRUE(true);
}
static void test_log_stderr_mode_all() {
int pipe_fds[2];
EXPECT_EQ_INT(pipe(pipe_fds), 0);
int saved_stderr = dup(STDERR_FILENO);
EXPECT_TRUE(saved_stderr >= 0);
EXPECT_TRUE(dup2(pipe_fds[1], STDERR_FILENO) >= 0);
close(pipe_fds[1]);
set_log_level(LOG_LEVEL_WARNING);
log_set_stderr_mode(LOG_STDERR_ALL);
log_message(LOG_LEVEL_WARNING, "warning routed to stderr");
fflush(stderr);
EXPECT_TRUE(dup2(saved_stderr, STDERR_FILENO) >= 0);
close(saved_stderr);
char output[128] = {0};
ssize_t length = read(pipe_fds[0], output, sizeof(output) - 1);
close(pipe_fds[0]);
EXPECT_TRUE(length > 0);
EXPECT_TRUE(strstr(output, "warning routed to stderr") != NULL);
log_set_stderr_mode(LOG_STDERR_ERRORS);
}
/* Test that log_message handles various format strings */
static void test_log_message_formats() {
set_log_level(LOG_LEVEL_DEBUG);
@@ -112,5 +137,6 @@ void test_log() {
test_log_set_level_info();
test_log_set_level_error();
test_log_filtering();
test_log_stderr_mode_all();
test_log_message_formats();
}
+45 -1
View File
@@ -123,6 +123,18 @@ static void test_metadata_rejects_invalid_values() {
close(p[1]);
}
static void test_metadata_mtime_window() {
EXPECT_TRUE(metadata_mtime_matches(100, 100000000, 101, 600000000, 2));
EXPECT_FALSE(metadata_mtime_matches(100, 100000000, 102, 600000000, 2));
EXPECT_TRUE(metadata_mtime_matches(100, 100000000, 102, 100000000, 2));
EXPECT_TRUE(metadata_mtime_matches(100, 900000000, 102, 100000000, 2));
EXPECT_FALSE(metadata_mtime_matches(100, 100000000, 102, 900000000, 2));
EXPECT_TRUE(metadata_mtime_matches(100, 900000000, 102, 900000000, 2));
EXPECT_FALSE(metadata_mtime_matches(100, 900000000, 101, 100000001, 0));
EXPECT_TRUE(metadata_mtime_matches(100, 100000000, 100, 100000001, 0));
EXPECT_TRUE(metadata_mtime_matches(100, 100000000, 100, 100000000, 0));
}
static void test_file_restore_metadata() {
const char* path = "temp_meta_restore_test.txt";
const char* content = "test content";
@@ -135,7 +147,7 @@ static void test_file_restore_metadata() {
m.mtime_sec = 1234567890;
m.mtime_nsec = 0;
file_restore_metadata(path, &m);
file_restore_metadata(path, &m, false);
struct stat st;
EXPECT_EQ_INT(stat(path, &st), 0);
@@ -145,6 +157,35 @@ static void test_file_restore_metadata() {
unlink(path);
}
static void test_file_restore_executability_only() {
const char* path = "temp_exec_restore_test.txt";
EXPECT_TRUE(file_write_to_disk(path, "x", 1, false, false));
EXPECT_EQ_INT(chmod(path, 0644), 0);
FileMetadata m = {
.mode = 0751, .uid = getuid(), .gid = getgid(), .mtime_sec = 0, .mtime_nsec = 0};
file_restore_metadata(path, &m, true);
struct stat st;
EXPECT_EQ_INT(stat(path, &st), 0);
EXPECT_EQ_INT(st.st_mode & 0777, 0755);
unlink(path);
}
static void test_directory_restore_executability_only() {
const char* path = "temp_exec_restore_test_dir";
EXPECT_EQ_INT(mkdir(path, 0700), 0);
FileMetadata m = {
.mode = 0755, .uid = getuid(), .gid = getgid(), .mtime_sec = 0, .mtime_nsec = 0};
file_restore_metadata(path, &m, true);
struct stat st;
EXPECT_EQ_INT(stat(path, &st), 0);
EXPECT_EQ_INT(st.st_mode & 0777, 0711);
rmdir(path);
}
static void test_chmod_changes() {
mode_t result;
EXPECT_TRUE(chmod_apply(0777, "u=rw,go=r", &result));
@@ -173,6 +214,9 @@ void test_metadata() {
test_metadata_send_receive_roundtrip();
test_metadata_send_null();
test_metadata_rejects_invalid_values();
test_metadata_mtime_window();
test_file_restore_metadata();
test_file_restore_executability_only();
test_directory_restore_executability_only();
test_chmod_changes();
}
+83
View File
@@ -250,6 +250,88 @@ static void test_write_thread_done() {
config_delete(cfg);
}
typedef struct {
PipelineContextReceiver* context;
File* file;
atomic_bool* done;
atomic_bool* result;
} ByteBudgetEnqueueArg;
static int byte_budget_enqueue_worker(void* arg) {
ByteBudgetEnqueueArg* worker = arg;
bool ok = pipeline_context_receiver_enqueue_file(worker->context, worker->file);
atomic_store(worker->result, ok);
atomic_store(worker->done, true);
return thrd_success;
}
/* A receiver must not buffer more decompressed/copied payload bytes ahead of
the (slow) disk writer than the configured byte budget: an enqueue that
would exceed the budget blocks until the writer releases bytes. */
static void test_receiver_enqueue_byte_budget() {
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
free(cfg->version);
cfg->version = str_dup(PROTOCOL_VERSION);
cfg->send_directory = str_dup("/src");
cfg->receive_root_directory = str_dup("/dst");
cfg->save_to_disk = false;
Queue* q = queue_create(16, file_destroy);
EXPECT_NOT_NULL(q);
PipelineContextReceiver* ctx = pipeline_context_receiver_create(cfg, q, -1, NULL);
EXPECT_NOT_NULL(ctx);
pipeline_context_receiver_set_queue_byte_limit(ctx, 3000);
ctx->receiver_done = false;
File* first = file_create("budget_file_1");
EXPECT_NOT_NULL(first);
first->data->size = 2000;
EXPECT_TRUE(pipeline_context_receiver_enqueue_file(ctx, first));
EXPECT_EQ_INT((int)ctx->queued_bytes, 2000);
/* Second 2000-byte payload would push the pipeline to 4000 > 3000 budget,
so the enqueue must block until the first payload is released. */
File* second = file_create("budget_file_2");
EXPECT_NOT_NULL(second);
second->data->size = 2000;
atomic_bool done;
atomic_bool result;
atomic_init(&done, false);
atomic_init(&result, false);
ByteBudgetEnqueueArg arg = {ctx, second, &done, &result};
thrd_t enqueuer;
EXPECT_EQ_INT(thrd_create(&enqueuer, byte_budget_enqueue_worker, &arg), thrd_success);
/* Give a broken (unbounded) implementation every chance to enqueue. */
struct timespec wait = {0, 200 * 1000000L};
thrd_sleep(&wait, NULL);
EXPECT_FALSE(atomic_load(&done));
EXPECT_EQ_INT((int)ctx->queued_bytes, 2000); /* budget still honored */
/* Simulate the disk writer: dequeue + destroy + release the first file.
Releasing bytes unblocks the waiting enqueuer, which then admits the
second payload, so only the post-join state (below) is deterministic. */
File* drained = queue_dequeue_multithreaded(q, &ctx->mutex, &ctx->condition_not_empty,
&ctx->condition_not_full, &ctx->receiver_done);
EXPECT_NOT_NULL(drained);
file_destroy(drained);
pipeline_context_receiver_note_bytes_released(ctx, 2000);
EXPECT_EQ_INT(thrd_join(enqueuer, NULL), thrd_success);
EXPECT_TRUE(atomic_load(&done));
EXPECT_TRUE(atomic_load(&result));
EXPECT_EQ_INT((int)ctx->queued_bytes, 2000); /* second payload now in flight */
/* Tear down: the second file is still queued and is freed by queue_destroy. */
mtx_destroy(&ctx->mutex);
cnd_destroy(&ctx->condition_not_full);
cnd_destroy(&ctx->condition_not_empty);
free(ctx);
queue_destroy(q);
config_delete(cfg);
}
void test_multiprocessing() {
test_sender_create_destroy();
test_receiver_create_destroy();
@@ -261,4 +343,5 @@ void test_multiprocessing() {
test_receive_thread_failure_wakes_writer();
}
test_write_thread_done();
test_receiver_enqueue_byte_budget();
}
+253
View File
@@ -3,6 +3,60 @@
#include <limits.h>
#include <string.h>
#include <unistd.h>
#include <threads.h>
typedef struct {
ProtocolSession* session;
bool allocation_allowed;
} AllocationWorkerArg;
static int allocation_worker(void* arg) {
AllocationWorkerArg* worker = arg;
protocol_session_bind(worker->session);
void* allocation = protocol_alloc(8);
worker->allocation_allowed = allocation != NULL;
free(allocation);
protocol_session_unbind();
return thrd_success;
}
typedef struct {
ProtocolSession* session;
int read_fd;
bool released;
} AccountingWorkerArg;
typedef struct {
ProtocolSession* session;
atomic_int* ready;
atomic_bool* release;
bool received;
} ConcurrentAccountingWorkerArg;
static int accounting_worker(void* arg) {
AccountingWorkerArg* worker = arg;
protocol_session_bind(worker->session);
Data* data = protocol_receive_data_limited(worker->session, 8);
if (data) {
data_destroy(data);
worker->released = atomic_load(&worker->session->total_allocated_bytes) == 0;
}
protocol_session_unbind();
return data ? thrd_success : thrd_error;
}
static int concurrent_accounting_worker(void* arg) {
ConcurrentAccountingWorkerArg* worker = arg;
protocol_session_bind(worker->session);
Data* data = protocol_receive_data_limited(worker->session, 8);
worker->received = data != NULL;
atomic_fetch_add(worker->ready, 1);
while (!atomic_load(worker->release))
thrd_yield();
data_destroy(data);
protocol_session_unbind();
return thrd_success;
}
static void test_send_receive_n_data() {
int p[2];
@@ -187,6 +241,196 @@ static void test_receive_str_truncated() {
close(p[0]);
}
static void test_max_alloc_rejects_single_buffer() {
int p[2];
EXPECT_EQ_INT(pipe(p), 0);
ProtocolSession session;
protocol_session_init(&session, p[0], p[1]);
protocol_session_set_max_alloc(&session, 4);
protocol_session_bind(&session);
char payload[8] = {0};
EXPECT_TRUE(write(p[1], &(size_t){sizeof(payload)}, sizeof(size_t)) == sizeof(size_t));
EXPECT_NULL(protocol_receive_str(&session));
protocol_session_unbind();
close(p[0]);
close(p[1]);
}
static void test_explicit_session_max_alloc_cannot_be_bypassed() {
int p[2];
EXPECT_EQ_INT(pipe(p), 0);
ProtocolSession explicit_session;
ProtocolSession unrelated_session;
protocol_session_init(&explicit_session, p[0], p[1]);
protocol_session_init(&unrelated_session, p[0], p[1]);
protocol_session_set_max_alloc(&explicit_session, 4);
protocol_session_set_max_alloc(&unrelated_session, 64);
protocol_session_bind(&unrelated_session);
unsigned long long size = 8;
EXPECT_EQ_INT((int)write(p[1], &size, sizeof(size)), (int)sizeof(size));
EXPECT_EQ_INT((int)write(p[1], "12345678", 8), 8);
EXPECT_NULL(protocol_receive_data_limited(&explicit_session, 8));
EXPECT_EQ_INT((int)atomic_load(&explicit_session.total_allocated_bytes), 0);
protocol_session_unbind();
close(p[0]);
close(p[1]);
}
static void test_max_alloc_allows_configured_buffer() {
ProtocolSession session;
protocol_session_init(&session, -1, -1);
protocol_session_set_max_alloc(&session, 4);
protocol_session_bind(&session);
void* allowed = protocol_alloc(4);
const void* rejected = protocol_alloc(5);
EXPECT_NOT_NULL(allowed);
EXPECT_NULL(rejected);
free(allowed);
protocol_session_unbind();
}
static void test_max_alloc_is_bound_in_worker_threads() {
enum { WORKER_COUNT = 4 };
ProtocolSession sessions[WORKER_COUNT];
AllocationWorkerArg args[WORKER_COUNT] = {0};
thrd_t threads[WORKER_COUNT];
for (int i = 0; i < WORKER_COUNT; i++) {
protocol_session_init(&sessions[i], -1, -1);
protocol_session_set_max_alloc(&sessions[i], 4);
args[i].session = &sessions[i];
EXPECT_EQ_INT(thrd_create(&threads[i], allocation_worker, &args[i]), thrd_success);
}
for (int i = 0; i < WORKER_COUNT; i++) {
int result;
EXPECT_EQ_INT(thrd_join(threads[i], &result), thrd_success);
EXPECT_EQ_INT(result, thrd_success);
EXPECT_FALSE(args[i].allocation_allowed);
}
}
static void test_protocol_accounting_is_released_in_worker_threads() {
enum { WORKER_COUNT = 4 };
ProtocolSession sessions[WORKER_COUNT];
AccountingWorkerArg args[WORKER_COUNT] = {0};
thrd_t threads[WORKER_COUNT];
for (int i = 0; i < WORKER_COUNT; i++) {
int p[2];
EXPECT_EQ_INT(pipe(p), 0);
protocol_session_init(&sessions[i], p[0], p[1]);
protocol_session_set_max_alloc(&sessions[i], 64);
unsigned long long size = 8;
EXPECT_EQ_INT((int)write(p[1], &size, sizeof(size)), (int)sizeof(size));
EXPECT_EQ_INT((int)write(p[1], "12345678", 8), 8);
close(p[1]);
args[i].session = &sessions[i];
args[i].read_fd = p[0];
EXPECT_EQ_INT(thrd_create(&threads[i], accounting_worker, &args[i]), thrd_success);
}
for (int i = 0; i < WORKER_COUNT; i++) {
int result;
EXPECT_EQ_INT(thrd_join(threads[i], &result), thrd_success);
EXPECT_EQ_INT(result, thrd_success);
EXPECT_TRUE(args[i].released);
EXPECT_EQ_INT((int)atomic_load(&sessions[i].total_allocated_bytes), 0);
close(args[i].read_fd);
}
}
static void test_protocol_accounting_reservation_is_atomic() {
enum { WORKER_COUNT = 8 };
int p[2];
EXPECT_EQ_INT(pipe(p), 0);
ProtocolSession session;
protocol_session_init(&session, p[0], p[1]);
protocol_session_set_max_alloc(&session, 64);
const unsigned long long budget_before = MAX_SERVER_ALLOC - 8;
atomic_store(&session.total_allocated_bytes, budget_before);
for (int i = 0; i < WORKER_COUNT; i++) {
unsigned long long size = 8;
EXPECT_EQ_INT((int)write(p[1], &size, sizeof(size)), (int)sizeof(size));
EXPECT_EQ_INT((int)write(p[1], "12345678", 8), 8);
}
close(p[1]);
atomic_int ready;
atomic_bool release;
atomic_init(&ready, 0);
atomic_init(&release, false);
ConcurrentAccountingWorkerArg args[WORKER_COUNT] = {0};
thrd_t threads[WORKER_COUNT];
for (int i = 0; i < WORKER_COUNT; i++) {
args[i].session = &session;
args[i].ready = &ready;
args[i].release = &release;
EXPECT_EQ_INT(thrd_create(&threads[i], concurrent_accounting_worker, &args[i]), thrd_success);
}
while (atomic_load(&ready) != WORKER_COUNT)
thrd_yield();
bool budget_ok = atomic_load(&session.total_allocated_bytes) == budget_before + 8;
atomic_store(&release, true);
int received = 0;
for (int i = 0; i < WORKER_COUNT; i++) {
int result;
EXPECT_EQ_INT(thrd_join(threads[i], &result), thrd_success);
EXPECT_EQ_INT(result, thrd_success);
received += args[i].received ? 1 : 0;
}
EXPECT_EQ_INT(received, 1);
EXPECT_TRUE(budget_ok);
EXPECT_EQ_INT((int)atomic_load(&session.total_allocated_bytes), (int)budget_before);
close(p[0]);
}
static void test_protocol_string_accounting_is_transient() {
int p[2];
EXPECT_EQ_INT(pipe(p), 0);
ProtocolSession session;
protocol_session_init(&session, p[0], p[1]);
protocol_session_set_max_alloc(&session, 64);
EXPECT_TRUE(protocol_send_str(&session, "temporary"));
char* received = protocol_receive_str(&session);
EXPECT_NOT_NULL(received);
EXPECT_EQ_STR(received, "temporary");
EXPECT_EQ_INT((int)atomic_load(&session.total_allocated_bytes), 0);
free(received);
close(p[0]);
close(p[1]);
}
static void test_protocol_accounting_release_does_not_underflow() {
ProtocolSession session;
protocol_session_init(&session, -1, -1);
atomic_store(&session.total_allocated_bytes, 4);
protocol_session_bind(&session);
protocol_release_memory(8);
EXPECT_EQ_INT((int)atomic_load(&session.total_allocated_bytes), 0);
protocol_release_memory(1);
EXPECT_EQ_INT((int)atomic_load(&session.total_allocated_bytes), 0);
protocol_session_unbind();
}
static void test_send_receive_status_timed() {
int p[2];
EXPECT_EQ_INT(pipe(p), 0);
io_set_fds(p[0], p[1]);
io_set_bwlimit(0);
/* The extended-deadline variant must read an ordinary status just like the
default window, and must fail cleanly on EOF rather than block. */
EXPECT_TRUE(send_status(0, STATUS_OK));
Status received = -1;
EXPECT_TRUE(receive_status_timed(0, &received, 5));
EXPECT_EQ_INT((int)received, (int)STATUS_OK);
close(p[1]);
EXPECT_FALSE(receive_status_timed(0, &received, 5));
close(p[0]);
}
void test_protocol() {
test_send_receive_n_data();
test_send_receive_n_data_zero();
@@ -196,6 +440,15 @@ void test_protocol() {
test_send_receive_data();
test_send_receive_int();
test_send_receive_status();
test_send_receive_status_timed();
test_receive_n_data_truncated();
test_receive_str_truncated();
test_max_alloc_rejects_single_buffer();
test_explicit_session_max_alloc_cannot_be_bypassed();
test_max_alloc_allows_configured_buffer();
test_max_alloc_is_bound_in_worker_threads();
test_protocol_accounting_is_released_in_worker_threads();
test_protocol_accounting_reservation_is_atomic();
test_protocol_string_accounting_is_transient();
test_protocol_accounting_release_does_not_underflow();
}
+15
View File
@@ -109,6 +109,20 @@ static void test_delta_deserialize_garbage() {
data_destroy(d);
}
static void test_delta_deserialize_respects_max_alloc() {
unsigned char serialized[sizeof(uint64_t) + sizeof(uint32_t)] = {0};
Data data = {.data = serialized, .size = sizeof(serialized)};
ProtocolSession session;
protocol_session_init(&session, -1, -1);
protocol_session_set_max_alloc(&session, sizeof(Delta) - 1);
protocol_session_bind(&session);
const Delta* result = delta_deserialize(&data);
EXPECT_NULL(result);
protocol_session_unbind();
}
static void test_delta_signature_deserialize_truncated() {
char old_data[4096];
for (int i = 0; i < 4096; i++)
@@ -206,6 +220,7 @@ void test_robustness() {
test_delta_deserialize_truncated();
test_delta_deserialize_empty();
test_delta_deserialize_garbage();
test_delta_deserialize_respects_max_alloc();
test_delta_deserialize_truncated_instructions();
test_delta_signature_deserialize_truncated();
test_delta_apply_null();
+871 -3
View File
@@ -1,7 +1,10 @@
#include "test_utils.h"
#include "scanner.h"
#include "file.h"
#include "file_list.h"
#include "filter.h"
#include "utils.h"
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <unistd.h>
@@ -394,9 +397,9 @@ static void test_parallel_scanner_root_chunks_without_workers() {
create_test_file(file1, "a");
create_test_file(file2, "b");
ScannerOptions options = {false, 1, NULL, 0, NULL, 0, 0, 0,
0, 0, false, false, false, false, false};
ParallelScanner* scanner = parallel_scanner_create_with_options(dir, &options);
ScannerOptions options = {0};
options.chunk_size = 1;
ParallelScanner* scanner = parallel_scanner_create_with_options(dir, &options, NULL);
EXPECT_NOT_NULL(scanner);
int total_files = 0;
@@ -414,6 +417,851 @@ static void test_parallel_scanner_root_chunks_without_workers() {
rmdir(dir);
}
/* --one-file-system (-x) decision is a pure device comparison. */
static void test_scanner_one_file_system_decision() {
/* Option disabled: every device is allowed (unchanged default behavior). */
EXPECT_TRUE(scanner_same_filesystem(false, 0, 123));
EXPECT_TRUE(scanner_same_filesystem(false, 7, 999));
/* Option enabled: only entries on the root device may be descended into. */
EXPECT_TRUE(scanner_same_filesystem(true, 7, 7));
EXPECT_FALSE(scanner_same_filesystem(true, 7, 8));
}
/* With -x over an ordinary tree (all one device) nothing may be skipped. */
static void test_scanner_one_file_system_same_device() {
const char* root = "test_scan_ofs";
const char* sub = "test_scan_ofs/sub";
const char* deeper = "test_scan_ofs/sub/deeper";
const char* root_file = "test_scan_ofs/root.txt";
const char* sub_file = "test_scan_ofs/sub/inner.txt";
const char* deep_file = "test_scan_ofs/sub/deeper/deep.txt";
EXPECT_EQ_INT(mkdir(root, 0755), 0);
EXPECT_EQ_INT(mkdir(sub, 0755), 0);
EXPECT_EQ_INT(mkdir(deeper, 0755), 0);
create_test_file(root_file, "root");
create_test_file(sub_file, "inner");
create_test_file(deep_file, "deep");
ScannerOptions options = {0};
options.one_file_system = true;
DirectoryScanner* scanner = directory_scanner_create_with_options(root, &options);
EXPECT_NOT_NULL(scanner);
int total_files = 0;
Chunk* chunk;
while ((chunk = directory_scanner_next(scanner)) != NULL) {
total_files += chunk->element_count;
chunk_destroy(chunk);
}
EXPECT_EQ_INT(total_files, 3);
EXPECT_FALSE(directory_scanner_failed(scanner));
directory_scanner_destroy(scanner);
unlink(root_file);
unlink(sub_file);
unlink(deep_file);
rmdir(deeper);
rmdir(sub);
rmdir(root);
}
/* Multithreaded (-m) scan with -x over a single-device tree must match the
* single-threaded result. */
static void test_parallel_scanner_one_file_system_same_device() {
const char* root = "test_parallel_scan_ofs";
const char* sub = "test_parallel_scan_ofs/sub";
const char* sub2 = "test_parallel_scan_ofs/sub2";
const char* root_file = "test_parallel_scan_ofs/root.txt";
const char* sub_file = "test_parallel_scan_ofs/sub/inner.txt";
const char* sub2_file = "test_parallel_scan_ofs/sub2/inner2.txt";
EXPECT_EQ_INT(mkdir(root, 0755), 0);
EXPECT_EQ_INT(mkdir(sub, 0755), 0);
EXPECT_EQ_INT(mkdir(sub2, 0755), 0);
create_test_file(root_file, "root");
create_test_file(sub_file, "inner");
create_test_file(sub2_file, "inner2");
ScannerOptions options = {0};
options.one_file_system = true;
options.num_threads = 2;
ParallelScanner* scanner = parallel_scanner_create_with_options(root, &options, NULL);
EXPECT_NOT_NULL(scanner);
int total_files = 0;
Chunk* chunk;
while ((chunk = parallel_scanner_next(scanner)) != NULL) {
total_files += chunk->element_count;
chunk_destroy(chunk);
}
EXPECT_EQ_INT(total_files, 3);
EXPECT_FALSE(parallel_scanner_failed(scanner));
parallel_scanner_destroy(scanner);
unlink(root_file);
unlink(sub_file);
unlink(sub2_file);
rmdir(sub);
rmdir(sub2);
rmdir(root);
}
/* Scan a tree with copy_links semantics, collecting every emitted path.
* Returns 0 on success, -1 on scanner failure. */
static int collect_directory_scan(const char* root, bool one_file_system, const char* needle,
bool* found, int* total) {
ScannerOptions options = {0};
options.copy_links = true;
options.one_file_system = one_file_system;
DirectoryScanner* scanner = directory_scanner_create_with_options(root, &options);
if (!scanner)
return -1;
*found = false;
*total = 0;
Chunk* chunk;
while ((chunk = directory_scanner_next(scanner)) != NULL) {
for (int i = 0; i < chunk->element_count; i++) {
(*total)++;
if (strstr(chunk->items[i]->path, needle) != NULL)
*found = true;
}
chunk_destroy(chunk);
}
bool failed = directory_scanner_failed(scanner);
directory_scanner_destroy(scanner);
return failed ? -1 : 0;
}
static int collect_parallel_scan(const char* root, bool one_file_system, const char* needle,
bool* found, int* total) {
ScannerOptions options = {0};
options.copy_links = true;
options.one_file_system = one_file_system;
options.num_threads = 2;
ParallelScanner* scanner = parallel_scanner_create_with_options(root, &options, NULL);
if (!scanner)
return -1;
*found = false;
*total = 0;
Chunk* chunk;
while ((chunk = parallel_scanner_next(scanner)) != NULL) {
for (int i = 0; i < chunk->element_count; i++) {
(*total)++;
if (strstr(chunk->items[i]->path, needle) != NULL)
*found = true;
}
chunk_destroy(chunk);
}
bool failed = parallel_scanner_failed(scanner);
parallel_scanner_destroy(scanner);
return failed ? -1 : 0;
}
/* Rootless cross-filesystem test: a symlink nested under the scan root points
* at a directory on another device (typically /dev/shm, a tmpfs distinct from
* the build filesystem). With --copy-links semantics the scanner resolves the
* link and must descend into it only when -x is off. The nested placement
* exercises the skip decision in the sequential walker and in the parallel
* worker (depth > 1). Skips when no cross-device target is available. */
static void test_scanner_one_file_system_cross_device() {
struct stat local_stat;
if (stat(".", &local_stat) != 0)
return;
char shm_dir[64] = "/dev/shm/fastsync_ofs_shm_XXXXXX";
if (mkdtemp(shm_dir) == NULL)
return;
struct stat shm_stat;
if (stat(shm_dir, &shm_stat) != 0 || shm_stat.st_dev == local_stat.st_dev) {
rmdir(shm_dir);
return;
}
char root_dir[64] = "./fastsync_ofs_root_XXXXXX";
if (mkdtemp(root_dir) == NULL) {
rmdir(shm_dir);
return;
}
char nested[96];
snprintf(nested, sizeof(nested), "%s/nested", root_dir);
char link_path[128];
snprintf(link_path, sizeof(link_path), "%s/link", nested);
char root_file[96];
snprintf(root_file, sizeof(root_file), "%s/keep.txt", root_dir);
char shm_file[96];
snprintf(shm_file, sizeof(shm_file), "%s/inside.txt", shm_dir);
bool ready = mkdir(nested, 0755) == 0 && symlink(shm_dir, link_path) == 0;
if (ready) {
create_test_file(root_file, "keep");
create_test_file(shm_file, "cross");
}
int seq_off_rc, seq_off_total, seq_on_rc, seq_on_total;
bool seq_off_found, seq_on_found;
int par_off_rc, par_off_total, par_on_rc, par_on_total;
bool par_off_found, par_on_found;
if (!ready) {
seq_off_rc = seq_on_rc = par_off_rc = par_on_rc = -1;
seq_off_total = seq_on_total = par_off_total = par_on_total = 0;
seq_off_found = seq_on_found = par_off_found = par_on_found = false;
} else {
int rc, total;
bool found;
rc = collect_directory_scan(root_dir, false, "inside.txt", &found, &total);
seq_off_rc = rc;
seq_off_total = total;
seq_off_found = found;
rc = collect_directory_scan(root_dir, true, "inside.txt", &found, &total);
seq_on_rc = rc;
seq_on_total = total;
seq_on_found = found;
rc = collect_parallel_scan(root_dir, false, "inside.txt", &found, &total);
par_off_rc = rc;
par_off_total = total;
par_off_found = found;
rc = collect_parallel_scan(root_dir, true, "inside.txt", &found, &total);
par_on_rc = rc;
par_on_total = total;
par_on_found = found;
}
/* Hermetic cleanup regardless of scan outcome, before any assertions. */
unlink(shm_file);
rmdir(shm_dir);
unlink(link_path);
unlink(root_file);
rmdir(nested);
rmdir(root_dir);
if (!ready)
return;
/* Sequential: without -x the symlinked foreign subtree is included. */
EXPECT_EQ_INT(seq_off_rc, 0);
EXPECT_TRUE(seq_off_found);
EXPECT_EQ_INT(seq_off_total, 2);
/* Sequential: with -x the cross-device subtree is dropped, keep.txt remains. */
EXPECT_EQ_INT(seq_on_rc, 0);
EXPECT_FALSE(seq_on_found);
EXPECT_EQ_INT(seq_on_total, 1);
/* Parallel: same behavior, worker path (depth > 1). */
EXPECT_EQ_INT(par_off_rc, 0);
EXPECT_TRUE(par_off_found);
EXPECT_EQ_INT(par_off_total, 2);
EXPECT_EQ_INT(par_on_rc, 0);
EXPECT_FALSE(par_on_found);
EXPECT_EQ_INT(par_on_total, 1);
}
/* Collect emitted file paths (relative to `root`) from a sequential scan.
* Returns 0 on success with *out and *count set (caller frees *out). */
static int collect_files(const char* root, const ScannerOptions* options, char*** out,
int* out_count) {
DirectoryScanner* scanner = directory_scanner_create_with_options(root, options);
if (!scanner)
return -1;
size_t root_len = strlen(root);
while (root_len > 0 && root[root_len - 1] == '/')
root_len--;
int cap = 16;
int count = 0;
char** paths = malloc((size_t)cap * sizeof(char*));
if (!paths) {
directory_scanner_destroy(scanner);
return -1;
}
Chunk* chunk;
while ((chunk = directory_scanner_next(scanner)) != NULL) {
for (int i = 0; i < chunk->element_count; i++) {
const char* rel = chunk->items[i]->path + root_len;
if (*rel == '/')
rel++;
if (count == cap) {
cap *= 2;
char** grown = realloc(paths, (size_t)cap * sizeof(char*));
if (!grown) {
for (int k = 0; k < count; k++)
free(paths[k]);
free(paths);
chunk_destroy(chunk);
directory_scanner_destroy(scanner);
return -1;
}
paths = grown;
}
paths[count++] = str_dup(rel);
}
chunk_destroy(chunk);
}
bool failed = directory_scanner_failed(scanner);
directory_scanner_destroy(scanner);
if (failed) {
for (int k = 0; k < count; k++)
free(paths[k]);
free(paths);
return -1;
}
*out = paths;
*out_count = count;
return 0;
}
static int collect_files_parallel(const char* root, const ScannerOptions* options, char*** out,
int* out_count) {
ParallelScanner* scanner = parallel_scanner_create_with_options(root, options, NULL);
if (!scanner)
return -1;
size_t root_len = strlen(root);
while (root_len > 0 && root[root_len - 1] == '/')
root_len--;
int cap = 16;
int count = 0;
char** paths = malloc((size_t)cap * sizeof(char*));
if (!paths) {
parallel_scanner_destroy(scanner);
return -1;
}
Chunk* chunk;
while ((chunk = parallel_scanner_next(scanner)) != NULL) {
for (int i = 0; i < chunk->element_count; i++) {
const char* rel = chunk->items[i]->path + root_len;
if (*rel == '/')
rel++;
if (count == cap) {
cap *= 2;
char** grown = realloc(paths, (size_t)cap * sizeof(char*));
if (!grown) {
for (int k = 0; k < count; k++)
free(paths[k]);
free(paths);
chunk_destroy(chunk);
parallel_scanner_destroy(scanner);
return -1;
}
paths = grown;
}
paths[count++] = str_dup(rel);
}
chunk_destroy(chunk);
}
bool failed = parallel_scanner_failed(scanner);
parallel_scanner_destroy(scanner);
if (failed) {
for (int k = 0; k < count; k++)
free(paths[k]);
free(paths);
return -1;
}
*out = paths;
*out_count = count;
return 0;
}
static bool has_path(char** paths, int count, const char* rel) {
for (int i = 0; i < count; i++)
if (strcmp(paths[i], rel) == 0)
return true;
return false;
}
static void free_paths(char** paths, int count) {
for (int i = 0; i < count; i++)
free(paths[i]);
free(paths);
}
static const char* FILE_LIST_PATH = "test_scan_files_from.txt";
/* --files-from: only the listed files (and the subtree of a listed directory)
* are emitted; unrelated files and directories are pruned. */
static void test_files_from_subset(bool parallel) {
const char* root = "test_scan_ff";
const char* sub = "test_scan_ff/sub";
const char* other = "test_scan_ff/other";
EXPECT_EQ_INT(mkdir(root, 0755), 0);
EXPECT_EQ_INT(mkdir(sub, 0755), 0);
EXPECT_EQ_INT(mkdir(other, 0755), 0);
create_test_file("test_scan_ff/root.txt", "root");
create_test_file("test_scan_ff/sub/keep.txt", "keep");
create_test_file("test_scan_ff/sub/skip.bin", "skip");
create_test_file("test_scan_ff/other/unrelated.txt", "unrelated");
/* List a root file and a file under sub: sub is descended but its other file
* is not listed, and the whole `other` directory is pruned. */
create_test_file(FILE_LIST_PATH, "root.txt\nsub/keep.txt\n");
char err[160];
FileListSet* set = file_list_load(FILE_LIST_PATH, false, err, sizeof(err));
EXPECT_NOT_NULL(set);
ScannerOptions options = {0};
options.file_list = set;
if (parallel)
options.num_threads = 2;
char** paths = NULL;
int count = 0;
int rc = parallel ? collect_files_parallel(root, &options, &paths, &count)
: collect_files(root, &options, &paths, &count);
EXPECT_EQ_INT(rc, 0);
EXPECT_EQ_INT(count, 2);
EXPECT_TRUE(has_path(paths, count, "root.txt"));
EXPECT_TRUE(has_path(paths, count, "sub/keep.txt"));
EXPECT_FALSE(has_path(paths, count, "sub/skip.bin"));
EXPECT_FALSE(has_path(paths, count, "other/unrelated.txt"));
free_paths(paths, count);
file_list_destroy(set);
remove(FILE_LIST_PATH);
/* Listing a directory transfers its whole subtree. */
create_test_file(FILE_LIST_PATH, "sub\n");
set = file_list_load(FILE_LIST_PATH, false, err, sizeof(err));
EXPECT_NOT_NULL(set);
options.file_list = set;
rc = parallel ? collect_files_parallel(root, &options, &paths, &count)
: collect_files(root, &options, &paths, &count);
EXPECT_EQ_INT(rc, 0);
EXPECT_EQ_INT(count, 2);
EXPECT_TRUE(has_path(paths, count, "sub/keep.txt"));
EXPECT_TRUE(has_path(paths, count, "sub/skip.bin"));
EXPECT_FALSE(has_path(paths, count, "root.txt"));
EXPECT_FALSE(has_path(paths, count, "other/unrelated.txt"));
free_paths(paths, count);
file_list_destroy(set);
remove(FILE_LIST_PATH);
unlink("test_scan_ff/root.txt");
unlink("test_scan_ff/sub/keep.txt");
unlink("test_scan_ff/sub/skip.bin");
unlink("test_scan_ff/other/unrelated.txt");
rmdir(other);
rmdir(sub);
rmdir(root);
}
/* Filter layer: '-' excludes, first-match-wins ordering with '+', anchored
* rules, and dir-only rules all prune during the scan. */
static void test_filter_rules(bool parallel) {
const char* root = "test_scan_filter";
EXPECT_EQ_INT(mkdir(root, 0755), 0);
create_test_file("test_scan_filter/a.txt", "a");
create_test_file("test_scan_filter/b.tmp", "b");
create_test_file("test_scan_filter/c.txt", "c");
/* - *.tmp excludes only the tmp file; other files remain (default include). */
const char* exclude_only[] = {"- *.tmp"};
char err[160];
FilterRuleList* base = filter_base_build(exclude_only, 1, false, err, sizeof(err));
EXPECT_NOT_NULL(base);
ScannerOptions options = {0};
options.base_filters = base;
if (parallel)
options.num_threads = 2;
char** paths = NULL;
int count = 0;
int rc = parallel ? collect_files_parallel(root, &options, &paths, &count)
: collect_files(root, &options, &paths, &count);
EXPECT_EQ_INT(rc, 0);
EXPECT_EQ_INT(count, 2);
EXPECT_TRUE(has_path(paths, count, "a.txt"));
EXPECT_TRUE(has_path(paths, count, "c.txt"));
EXPECT_FALSE(has_path(paths, count, "b.tmp"));
free_paths(paths, count);
filter_rule_list_free(base);
/* Anchored include then exclude-all: only root-level keep* survives. */
const char* anchored[] = {"+ /a.txt", "- *"};
base = filter_base_build(anchored, 2, false, err, sizeof(err));
EXPECT_NOT_NULL(base);
options.base_filters = base;
rc = parallel ? collect_files_parallel(root, &options, &paths, &count)
: collect_files(root, &options, &paths, &count);
EXPECT_EQ_INT(rc, 0);
EXPECT_EQ_INT(count, 1);
EXPECT_TRUE(has_path(paths, count, "a.txt"));
free_paths(paths, count);
filter_rule_list_free(base);
unlink("test_scan_filter/a.txt");
unlink("test_scan_filter/b.tmp");
unlink("test_scan_filter/c.txt");
rmdir(root);
}
/* Anchored dir-only rules prune a whole subtree. */
static void test_filter_dir_only_and_anchored(bool parallel) {
const char* root = "test_scan_filter_dir";
const char* sub = "test_scan_filter_dir/sub";
EXPECT_EQ_INT(mkdir(root, 0755), 0);
EXPECT_EQ_INT(mkdir(sub, 0755), 0);
create_test_file("test_scan_filter_dir/sub/inner.txt", "x");
create_test_file("test_scan_filter_dir/keep.txt", "keep");
const char* rules[] = {"- /sub/"};
char err[160];
FilterRuleList* base = filter_base_build(rules, 1, false, err, sizeof(err));
EXPECT_NOT_NULL(base);
ScannerOptions options = {0};
options.base_filters = base;
if (parallel)
options.num_threads = 2;
char** paths = NULL;
int count = 0;
int rc = parallel ? collect_files_parallel(root, &options, &paths, &count)
: collect_files(root, &options, &paths, &count);
EXPECT_EQ_INT(rc, 0);
EXPECT_EQ_INT(count, 1);
EXPECT_TRUE(has_path(paths, count, "keep.txt"));
EXPECT_FALSE(has_path(paths, count, "sub/inner.txt"));
free_paths(paths, count);
filter_rule_list_free(base);
unlink("test_scan_filter_dir/sub/inner.txt");
unlink("test_scan_filter_dir/keep.txt");
rmdir(sub);
rmdir(root);
}
/* -C default CVS excludes prune .git/ directories and *.o files. */
static void test_cvs_defaults(bool parallel) {
const char* root = "test_scan_cvs";
const char* git = "test_scan_cvs/.git";
EXPECT_EQ_INT(mkdir(root, 0755), 0);
EXPECT_EQ_INT(mkdir(git, 0755), 0);
create_test_file("test_scan_cvs/.git/config", "cfg");
create_test_file("test_scan_cvs/object.o", "o");
create_test_file("test_scan_cvs/keep.txt", "keep");
char err[160];
FilterRuleList* base = filter_base_build(NULL, 0, true, err, sizeof(err));
EXPECT_NOT_NULL(base);
ScannerOptions options = {0};
options.base_filters = base;
if (parallel)
options.num_threads = 2;
char** paths = NULL;
int count = 0;
int rc = parallel ? collect_files_parallel(root, &options, &paths, &count)
: collect_files(root, &options, &paths, &count);
EXPECT_EQ_INT(rc, 0);
EXPECT_EQ_INT(count, 1);
EXPECT_TRUE(has_path(paths, count, "keep.txt"));
EXPECT_FALSE(has_path(paths, count, ".git/config"));
EXPECT_FALSE(has_path(paths, count, "object.o"));
free_paths(paths, count);
filter_rule_list_free(base);
unlink("test_scan_cvs/.git/config");
unlink("test_scan_cvs/object.o");
unlink("test_scan_cvs/keep.txt");
rmdir(git);
rmdir(root);
}
/* -F: a .rsync-filter placed in a directory governs its subtree and the file
* itself is never transferred. */
static void test_per_dir_filter(bool parallel) {
const char* root = "test_scan_perdir";
const char* sub = "test_scan_perdir/sub";
EXPECT_EQ_INT(mkdir(root, 0755), 0);
EXPECT_EQ_INT(mkdir(sub, 0755), 0);
create_test_file("test_scan_perdir/drop.tmp", "tmp");
create_test_file("test_scan_perdir/keep.txt", "keep");
create_test_file("test_scan_perdir/sub/nested.tmp", "tmp");
create_test_file("test_scan_perdir/.rsync-filter", "- *.tmp\n");
ScannerOptions options = {0};
options.per_dir_filters = true;
if (parallel)
options.num_threads = 2;
char** paths = NULL;
int count = 0;
int rc = parallel ? collect_files_parallel(root, &options, &paths, &count)
: collect_files(root, &options, &paths, &count);
EXPECT_EQ_INT(rc, 0);
EXPECT_EQ_INT(count, 1);
EXPECT_TRUE(has_path(paths, count, "keep.txt"));
EXPECT_FALSE(has_path(paths, count, "drop.tmp"));
EXPECT_FALSE(has_path(paths, count, "sub/nested.tmp"));
EXPECT_FALSE(has_path(paths, count, ".rsync-filter"));
free_paths(paths, count);
unlink("test_scan_perdir/drop.tmp");
unlink("test_scan_perdir/keep.txt");
unlink("test_scan_perdir/sub/nested.tmp");
unlink("test_scan_perdir/.rsync-filter");
rmdir(sub);
rmdir(root);
}
/* scanner_path_relative maps an on-disk path to its transfer-relative path,
* including the "/" transfer-root edge case (regression: children of "/" used
* to abort the scan because the suffix was mis-read). */
static void test_scanner_path_relative() {
char* rel = NULL;
rel = scanner_path_relative("/", "/");
EXPECT_NOT_NULL(rel);
EXPECT_EQ_STR(rel, "");
free(rel);
rel = scanner_path_relative("/", "/etc");
EXPECT_NOT_NULL(rel);
EXPECT_EQ_STR(rel, "etc");
free(rel);
rel = scanner_path_relative("/", "/etc/passwd");
EXPECT_NOT_NULL(rel);
EXPECT_EQ_STR(rel, "etc/passwd");
free(rel);
/* Normal roots: with and without a trailing slash on the root. */
rel = scanner_path_relative("/tmp/foo", "/tmp/foo");
EXPECT_NOT_NULL(rel);
EXPECT_EQ_STR(rel, "");
free(rel);
rel = scanner_path_relative("/tmp/foo", "/tmp/foo/bar");
EXPECT_NOT_NULL(rel);
EXPECT_EQ_STR(rel, "bar");
free(rel);
rel = scanner_path_relative("/tmp/foo/", "/tmp/foo/bar/baz.txt");
EXPECT_NOT_NULL(rel);
EXPECT_EQ_STR(rel, "bar/baz.txt");
free(rel);
/* A path outside the root maps to NULL. */
EXPECT_NULL(scanner_path_relative("/tmp/foo", "/tmp"));
EXPECT_NULL(scanner_path_relative("/tmp/foo", "/tmp/foobar"));
}
/* rsync precedence: a deeper .rsync-filter overrides a shallower one, so an
* inner "+ *.tmp" re-includes what the outer "- *.tmp" excluded. */
static void test_per_dir_filter_override(bool parallel) {
const char* root = "test_scan_perdir_ovr";
const char* sub = "test_scan_perdir_ovr/sub";
EXPECT_EQ_INT(mkdir(root, 0755), 0);
EXPECT_EQ_INT(mkdir(sub, 0755), 0);
create_test_file("test_scan_perdir_ovr/.rsync-filter", "- *.tmp\n");
create_test_file("test_scan_perdir_ovr/sub/.rsync-filter", "+ *.tmp\n");
create_test_file("test_scan_perdir_ovr/top.tmp", "x");
create_test_file("test_scan_perdir_ovr/keep.txt", "keep");
create_test_file("test_scan_perdir_ovr/sub/inside.tmp", "x");
ScannerOptions options = {0};
options.per_dir_filters = true;
if (parallel)
options.num_threads = 2;
char** paths = NULL;
int count = 0;
int rc = parallel ? collect_files_parallel(root, &options, &paths, &count)
: collect_files(root, &options, &paths, &count);
EXPECT_EQ_INT(rc, 0);
/* top.tmp is still excluded by the root file; inside.tmp is re-included by
* the subdir file; .rsync-filter files are never transferred. */
EXPECT_EQ_INT(count, 2);
EXPECT_TRUE(has_path(paths, count, "keep.txt"));
EXPECT_TRUE(has_path(paths, count, "sub/inside.tmp"));
EXPECT_FALSE(has_path(paths, count, "top.tmp"));
EXPECT_FALSE(has_path(paths, count, ".rsync-filter"));
EXPECT_FALSE(has_path(paths, count, "sub/.rsync-filter"));
free_paths(paths, count);
unlink("test_scan_perdir_ovr/top.tmp");
unlink("test_scan_perdir_ovr/keep.txt");
unlink("test_scan_perdir_ovr/sub/inside.tmp");
unlink("test_scan_perdir_ovr/.rsync-filter");
unlink("test_scan_perdir_ovr/sub/.rsync-filter");
rmdir(sub);
rmdir(root);
}
typedef struct {
char rel[512];
char send[512];
bool is_dir;
} ScanInfo;
/* Collect every scanner entry below `root` into `out` (at most `max`), mapping
* paths to their root-relative form and capturing send_path and is_dir. */
static int collect_scan_info(const char* root, const ScannerOptions* options, ScanInfo out[],
int max) {
DirectoryScanner* scanner = directory_scanner_create_with_options(root, options);
if (!scanner)
return -1;
size_t root_len = strlen(root);
while (root_len > 0 && root[root_len - 1] == '/')
root_len--;
int count = 0;
Chunk* chunk;
while ((chunk = directory_scanner_next(scanner)) != NULL) {
for (int i = 0; i < chunk->element_count && count < max; i++) {
const File* f = chunk->items[i];
const char* rel = f->path + root_len;
if (*rel == '/')
rel++;
snprintf(out[count].rel, sizeof(out[count].rel), "%s", rel);
snprintf(out[count].send, sizeof(out[count].send), "%s", f->send_path ? f->send_path : "");
out[count].is_dir = f->is_dir;
count++;
}
chunk_destroy(chunk);
}
bool failed = directory_scanner_failed(scanner);
directory_scanner_destroy(scanner);
return failed ? -1 : count;
}
static bool scan_info_present(const ScanInfo* infos, int count, const char* rel, bool is_dir,
const char* send) {
for (int i = 0; i < count; i++) {
if (strcmp(infos[i].rel, rel) == 0 && infos[i].is_dir == is_dir &&
strcmp(infos[i].send, send ? send : "") == 0)
return true;
}
return false;
}
/* Parallel variant of collect_scan_info; drains `scanner` fully and destroys
* it. */
static int collect_scan_info_parallel(ParallelScanner* scanner, const char* root, ScanInfo out[],
int max) {
if (!scanner)
return -1;
size_t root_len = strlen(root);
while (root_len > 0 && root[root_len - 1] == '/')
root_len--;
int count = 0;
Chunk* chunk;
while ((chunk = parallel_scanner_next(scanner)) != NULL) {
for (int i = 0; i < chunk->element_count && count < max; i++) {
const File* f = chunk->items[i];
const char* rel = f->path + root_len;
if (*rel == '/')
rel++;
snprintf(out[count].rel, sizeof(out[count].rel), "%s", rel);
snprintf(out[count].send, sizeof(out[count].send), "%s", f->send_path ? f->send_path : "");
out[count].is_dir = f->is_dir;
count++;
}
chunk_destroy(chunk);
}
bool failed = parallel_scanner_failed(scanner);
parallel_scanner_destroy(scanner);
return failed ? -1 : count;
}
/* -d without --files-from emits exactly the source-root directory (empty) and
* never descends. */
static void test_dirs_no_descent() {
const char* root = "test_scan_dirs_root";
EXPECT_EQ_INT(mkdir(root, 0755), 0);
EXPECT_EQ_INT(mkdir("test_scan_dirs_root/sub", 0755), 0);
create_test_file("test_scan_dirs_root/a.txt", "a");
create_test_file("test_scan_dirs_root/sub/b.txt", "b");
ScannerOptions options = {0};
options.dirs = true;
ScanInfo infos[8];
int count = collect_scan_info(root, &options, infos, 8);
EXPECT_EQ_INT(count, 1);
EXPECT_TRUE(scan_info_present(infos, count, "", true, NULL));
EXPECT_FALSE(scan_info_present(infos, count, "a.txt", false, ""));
EXPECT_FALSE(scan_info_present(infos, count, "sub/b.txt", false, ""));
unlink("test_scan_dirs_root/a.txt");
unlink("test_scan_dirs_root/sub/b.txt");
rmdir("test_scan_dirs_root/sub");
rmdir(root);
}
/* -d with --files-from transfers exactly the listed directory (empty) and the
* listed file; nothing is descended into. */
static void test_dirs_files_from() {
const char* root = "test_scan_dirs_ff";
const char* list_path = "test_scan_dirs_ff.list";
EXPECT_EQ_INT(mkdir(root, 0755), 0);
EXPECT_EQ_INT(mkdir("test_scan_dirs_ff/sub", 0755), 0);
create_test_file("test_scan_dirs_ff/sub/keep.txt", "keep");
create_test_file("test_scan_dirs_ff/sub/skip.bin", "skip");
create_test_file("test_scan_dirs_ff/top.txt", "top");
char err[160];
create_test_file(list_path, "sub\nsub/keep.txt\n");
FileListSet* set = file_list_load(list_path, false, err, sizeof(err));
EXPECT_NOT_NULL(set);
for (int relative = 0; relative <= 1; relative++) {
ScannerOptions options = {0};
options.dirs = true;
options.file_list = set;
options.relative = relative != 0;
ScanInfo infos[8];
int count = collect_scan_info(root, &options, infos, 8);
EXPECT_EQ_INT(count, 2);
if (relative) {
EXPECT_TRUE(scan_info_present(infos, count, "sub", true, "sub"));
EXPECT_TRUE(scan_info_present(infos, count, "sub/keep.txt", false, "sub/keep.txt"));
} else {
EXPECT_TRUE(scan_info_present(infos, count, "sub", true, NULL));
EXPECT_TRUE(scan_info_present(infos, count, "sub/keep.txt", false, NULL));
}
EXPECT_FALSE(scan_info_present(infos, count, "sub/skip.bin", false, ""));
EXPECT_FALSE(scan_info_present(infos, count, "top.txt", false, ""));
}
file_list_destroy(set);
remove(list_path);
unlink("test_scan_dirs_ff/sub/keep.txt");
unlink("test_scan_dirs_ff/sub/skip.bin");
unlink("test_scan_dirs_ff/top.txt");
rmdir("test_scan_dirs_ff/sub");
rmdir(root);
}
/* -R with --files-from (no -d): every file keeps its bare relative path as the
* send_path while the local scan path stays absolute-under-root. */
static void test_files_from_relative_send_path() {
const char* root = "test_scan_rel_ff";
const char* list_path = "test_scan_rel_ff.list";
EXPECT_EQ_INT(mkdir(root, 0755), 0);
EXPECT_EQ_INT(mkdir("test_scan_rel_ff/sub", 0755), 0);
create_test_file("test_scan_rel_ff/root.txt", "root");
create_test_file("test_scan_rel_ff/sub/keep.txt", "keep");
char err[160];
create_test_file(list_path, "root.txt\nsub/keep.txt\n");
FileListSet* set = file_list_load(list_path, false, err, sizeof(err));
EXPECT_NOT_NULL(set);
for (int parallel = 0; parallel <= 1; parallel++) {
ScannerOptions options = {0};
options.file_list = set;
options.relative = true;
if (parallel)
options.num_threads = 2;
ScanInfo infos[8];
int count;
if (parallel) {
ParallelScanner* scanner = parallel_scanner_create_with_options(root, &options, NULL);
EXPECT_NOT_NULL(scanner);
count = collect_scan_info_parallel(scanner, root, infos, 8);
} else {
count = collect_scan_info(root, &options, infos, 8);
}
EXPECT_EQ_INT(count, 2);
EXPECT_TRUE(scan_info_present(infos, count, "root.txt", false, "root.txt"));
EXPECT_TRUE(scan_info_present(infos, count, "sub/keep.txt", false, "sub/keep.txt"));
}
file_list_destroy(set);
remove(list_path);
unlink("test_scan_rel_ff/root.txt");
unlink("test_scan_rel_ff/sub/keep.txt");
rmdir("test_scan_rel_ff/sub");
rmdir(root);
}
void test_scanner() {
test_scanner_single_file();
test_scanner_multiple_files();
@@ -429,4 +1277,24 @@ void test_scanner() {
test_scanner_mixed_patterns();
test_scanner_no_patterns();
test_parallel_scanner_root_chunks_without_workers();
test_scanner_one_file_system_decision();
test_scanner_one_file_system_same_device();
test_parallel_scanner_one_file_system_same_device();
test_scanner_one_file_system_cross_device();
test_files_from_subset(false);
test_files_from_subset(true);
test_filter_rules(false);
test_filter_rules(true);
test_filter_dir_only_and_anchored(false);
test_filter_dir_only_and_anchored(true);
test_cvs_defaults(false);
test_cvs_defaults(true);
test_per_dir_filter(false);
test_per_dir_filter(true);
test_scanner_path_relative();
test_per_dir_filter_override(false);
test_per_dir_filter_override(true);
test_dirs_no_descent();
test_dirs_files_from();
test_files_from_relative_send_path();
}
+379 -1
View File
@@ -1,14 +1,18 @@
#include "test_server.h"
#include "config.h"
#include "delta.h"
#include "file.h"
#include "protocol.h"
#include "test_utils.h"
#include "utils.h"
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/socket.h>
#include <sys/stat.h>
#include <sys/wait.h>
#include <time.h>
#include <unistd.h>
#include "receiver.h"
@@ -176,7 +180,374 @@ static void test_receive_manifest_rejects_traversal() {
io_set_fds(p[0], p[1]);
EXPECT_TRUE(send_int(p[1], 1));
EXPECT_TRUE(send_str(p[1], "../outside"));
EXPECT_EQ_INT(receive_manifest(p[0], cfg, NULL), -1);
EXPECT_NULL(receive_manifest_entries(p[0]));
Status status;
EXPECT_TRUE(receive_status(p[1], &status));
EXPECT_EQ_INT(status, STATUS_ERROR);
close(p[0]);
close(p[1]);
config_delete(cfg);
}
static void test_receive_incremental_check_rejects_invalid_nanoseconds() {
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
cfg->receive_root_directory = str_dup("/tmp/dst");
int p[2];
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
io_set_fds(p[0], p[1]);
EXPECT_TRUE(send_str(p[1], "file.txt"));
unsigned long long size = 0;
long long mtime = 100;
long long mtime_nsec = 1000000000LL;
EXPECT_TRUE(send_n_data(p[1], &size, sizeof(size)));
EXPECT_TRUE(send_n_data(p[1], &mtime, sizeof(mtime)));
EXPECT_TRUE(send_n_data(p[1], &mtime_nsec, sizeof(mtime_nsec)));
bool skipped = false;
EXPECT_NULL(receive_incremental_check(p[0], cfg, &skipped));
Status status;
EXPECT_TRUE(receive_status(p[1], &status));
EXPECT_EQ_INT(status, STATUS_ERROR);
EXPECT_FALSE(skipped);
close(p[0]);
close(p[1]);
config_delete(cfg);
}
static char* make_check_root(const char* tag) {
char tmpl[128];
snprintf(tmpl, sizeof(tmpl), "/tmp/fastsync_%s_XXXXXX", tag);
char* path = str_dup(tmpl);
if (!path)
return NULL;
if (!mkdtemp(path)) {
free(path);
return NULL;
}
return path;
}
static void write_check_file(const char* dir, const char* name, const char* content) {
char path[1024];
snprintf(path, sizeof(path), "%s/%s", dir, name);
int fd = open(path, O_WRONLY | O_CREAT | O_TRUNC, 0644);
if (fd >= 0) {
size_t len = strlen(content);
if (write(fd, content, len) != (ssize_t)len) {
/* intentionally ignored in tests */
}
close(fd);
}
}
/* Issue #255: a same-size/mtime match is decided from metadata alone, so the
receiver answers STATUS_OK (skip) and never asks for a data body. */
static void test_incremental_check_quick_skip_by_mtime() {
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
char* root = make_check_root("qskip");
EXPECT_NOT_NULL(root);
cfg->receive_root_directory = str_dup(root);
write_check_file(root, "file.txt", "0123456789abcdef");
char path[1024];
snprintf(path, sizeof(path), "%s/file.txt", root);
struct stat st;
EXPECT_EQ_INT(stat(path, &st), 0);
int p[2];
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
io_set_fds(p[0], p[1]);
io_set_bwlimit(0);
pid_t pid = fork();
if (pid == 0) {
alarm(30);
close(p[1]);
io_set_fds(p[0], p[0]);
bool skipped = false;
File* file = receive_incremental_check(p[0], cfg, &skipped);
bool ok = file == NULL && skipped;
file_destroy(file);
config_delete(cfg);
close(p[0]);
_exit(ok ? 0 : 1);
} else {
close(p[0]);
io_set_fds(p[1], p[1]);
EXPECT_TRUE(send_str(p[1], "file.txt"));
unsigned long long size = (unsigned long long)st.st_size;
long long mtime = (long long)st.st_mtime;
long long mtime_nsec = 0;
#ifdef __linux__
mtime_nsec = (long long)st.st_mtim.tv_nsec;
#endif
EXPECT_TRUE(send_n_data(p[1], &size, sizeof(size)));
EXPECT_TRUE(send_n_data(p[1], &mtime, sizeof(mtime)));
EXPECT_TRUE(send_n_data(p[1], &mtime_nsec, sizeof(mtime_nsec)));
Status s;
EXPECT_TRUE(receive_status(p[1], &s));
EXPECT_EQ_INT(s, STATUS_OK);
int status;
waitpid(pid, &status, 0);
close(p[1]);
config_delete(cfg);
unlink(path);
rmdir(root);
free(root);
EXPECT_TRUE(WIFEXITED(status) && WEXITSTATUS(status) == 0);
}
}
/* Issue #255: a size mismatch cannot be a skip, so the receiver answers
STATUS_NEXT and consumes the full data body that follows. */
static void test_incremental_check_size_mismatch_full_transfer() {
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
char* root = make_check_root("qnext");
EXPECT_NOT_NULL(root);
cfg->receive_root_directory = str_dup(root);
write_check_file(root, "file.txt", "0123456789abcdef");
char path[1024];
snprintf(path, sizeof(path), "%s/file.txt", root);
struct stat st;
EXPECT_EQ_INT(stat(path, &st), 0);
int p[2];
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
io_set_fds(p[0], p[1]);
io_set_bwlimit(0);
pid_t pid = fork();
if (pid == 0) {
alarm(30);
close(p[1]);
io_set_fds(p[0], p[0]);
bool skipped = false;
File* file = receive_incremental_check(p[0], cfg, &skipped);
bool ok = file != NULL && !skipped && file->path != NULL && strcmp(file->path, "file.txt") == 0;
file_destroy(file);
config_delete(cfg);
close(p[0]);
_exit(ok ? 0 : 1);
} else {
close(p[0]);
io_set_fds(p[1], p[1]);
EXPECT_TRUE(send_str(p[1], "file.txt"));
unsigned long long size = (unsigned long long)st.st_size + 1;
long long mtime = (long long)st.st_mtime;
long long mtime_nsec = 0;
#ifdef __linux__
mtime_nsec = (long long)st.st_mtim.tv_nsec;
#endif
EXPECT_TRUE(send_n_data(p[1], &size, sizeof(size)));
EXPECT_TRUE(send_n_data(p[1], &mtime, sizeof(mtime)));
EXPECT_TRUE(send_n_data(p[1], &mtime_nsec, sizeof(mtime_nsec)));
Status s;
EXPECT_TRUE(receive_status(p[1], &s));
EXPECT_EQ_INT(s, STATUS_NEXT);
Data* body = data_create_reserve(8);
EXPECT_NOT_NULL(body);
body->data = malloc(8);
EXPECT_NOT_NULL(body->data);
memcpy(body->data, "replaced", 8);
body->size = 8;
EXPECT_TRUE(send_data(p[1], body));
data_destroy(body);
int status;
waitpid(pid, &status, 0);
close(p[1]);
config_delete(cfg);
unlink(path);
rmdir(root);
free(root);
EXPECT_TRUE(WIFEXITED(status) && WEXITSTATUS(status) == 0);
}
}
/* Issue #256: when a received delta claims a result above the whole-file cap,
receive_delta_file must mark the operation failed so the caller aborts with
STATUS_ERROR instead of emitting STATUS_NEXT and waiting for a body that
never arrives. */
static void test_incremental_check_delta_oversize_reports_failure() {
Config* cfg = config_create();
EXPECT_NOT_NULL(cfg);
char* root = make_check_root("qdelta");
EXPECT_NOT_NULL(root);
cfg->receive_root_directory = str_dup(root);
cfg->use_delta = true;
char content[20000];
memset(content, 'a', sizeof(content));
content[sizeof(content) - 1] = '\0';
write_check_file(root, "file.txt", content);
char path[1024];
snprintf(path, sizeof(path), "%s/file.txt", root);
struct stat st;
EXPECT_EQ_INT(stat(path, &st), 0);
EXPECT_EQ_INT((int)st.st_size, 19999);
int p[2];
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
io_set_fds(p[0], p[1]);
io_set_bwlimit(0);
pid_t pid = fork();
if (pid == 0) {
alarm(30);
close(p[1]);
io_set_fds(p[0], p[0]);
bool skipped = false;
File* file = receive_incremental_check(p[0], cfg, &skipped);
bool ok = file == NULL && !skipped;
if (ok)
send_status(p[0], STATUS_ERROR); /* mirror the server error path */
file_destroy(file);
config_delete(cfg);
close(p[0]);
_exit(ok ? 0 : 1);
} else {
close(p[0]);
io_set_fds(p[1], p[1]);
EXPECT_TRUE(send_str(p[1], "file.txt"));
unsigned long long size = (unsigned long long)st.st_size;
long long mtime = 1; /* different from the file mtime: force a transfer */
long long mtime_nsec = 0;
EXPECT_TRUE(send_n_data(p[1], &size, sizeof(size)));
EXPECT_TRUE(send_n_data(p[1], &mtime, sizeof(mtime)));
EXPECT_TRUE(send_n_data(p[1], &mtime_nsec, sizeof(mtime_nsec)));
Status s;
EXPECT_TRUE(receive_status(p[1], &s));
EXPECT_EQ_INT(s, STATUS_DELTA_SIGNATURE);
Data* sig_data = receive_data(p[1]);
EXPECT_NOT_NULL(sig_data);
DeltaSignature* sig = delta_signature_deserialize(sig_data);
EXPECT_NOT_NULL(sig);
delta_signature_destroy(sig);
data_destroy(sig_data);
/* Send a delta whose claimed output size exceeds the whole-file cap. */
Delta delta;
memset(&delta, 0, sizeof(delta));
delta.new_file_size = MAX_RECEIVE_WHOLE_FILE_SIZE + 1;
Data* bogus = delta_serialize(&delta);
EXPECT_NOT_NULL(bogus);
EXPECT_TRUE(send_status(p[1], STATUS_DELTA_DATA));
EXPECT_TRUE(bogus != NULL && send_data(p[1], bogus));
data_destroy(bogus);
/* The receiver must answer with an error, never with STATUS_NEXT. */
EXPECT_TRUE(receive_status(p[1], &s));
EXPECT_EQ_INT(s, STATUS_ERROR);
int status;
waitpid(pid, &status, 0);
close(p[1]);
config_delete(cfg);
unlink(path);
rmdir(root);
free(root);
EXPECT_TRUE(WIFEXITED(status) && WEXITSTATUS(status) == 0);
}
}
/* Late-timing keep-set leak guard: a manifest parked by the commit path must
be freed on every error exit, never leaked. These tests drive
receiver_process_pending() through an error AFTER the manifest was parked and
are exercised under ASan/valgrind to prove the list is released. */
static Config* make_late_delete_config(const char* root) {
Config* cfg = config_create();
if (!cfg)
return NULL;
cfg->send_directory = str_dup("/src");
cfg->receive_root_directory = str_dup(root);
cfg->use_delete = true;
cfg->delete_after = true;
return cfg;
}
static int run_pending_receiver(Config* cfg, int fd, DeleteManifest** pending) {
ReceiverSink sink = {0};
return receiver_process_pending(cfg, fd, &sink, pending);
}
static void test_late_manifest_abort_frees_keepset() {
Config* cfg = make_late_delete_config("/tmp/fastsync_late_abort");
EXPECT_NOT_NULL(cfg);
int p[2];
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
io_set_fds(p[0], p[1]);
io_set_bwlimit(0);
EXPECT_TRUE(send_status(p[1], STATUS_MANIFEST));
EXPECT_TRUE(send_int(p[1], 1));
EXPECT_TRUE(send_str(p[1], "keep.txt"));
EXPECT_TRUE(send_int(p[1], 0)); /* protected-prefix section is empty */
EXPECT_TRUE(send_status(p[1], STATUS_ABORT));
DeleteManifest* pending = NULL;
EXPECT_EQ_INT(run_pending_receiver(cfg, p[0], &pending), -1);
EXPECT_NULL(pending);
close(p[0]);
close(p[1]);
config_delete(cfg);
}
static void test_late_manifest_eof_frees_keepset() {
Config* cfg = make_late_delete_config("/tmp/fastsync_late_eof");
EXPECT_NOT_NULL(cfg);
int p[2];
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
io_set_fds(p[0], p[1]);
io_set_bwlimit(0);
EXPECT_TRUE(send_status(p[1], STATUS_MANIFEST));
EXPECT_TRUE(send_int(p[1], 1));
EXPECT_TRUE(send_str(p[1], "keep.txt"));
EXPECT_TRUE(send_int(p[1], 0)); /* protected-prefix section is empty */
shutdown(p[1], SHUT_WR);
DeleteManifest* pending = NULL;
EXPECT_EQ_INT(run_pending_receiver(cfg, p[0], &pending), -1);
EXPECT_NULL(pending);
close(p[0]);
close(p[1]);
config_delete(cfg);
}
static void test_late_second_manifest_frees_both() {
Config* cfg = make_late_delete_config("/tmp/fastsync_late_second");
EXPECT_NOT_NULL(cfg);
int p[2];
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
io_set_fds(p[0], p[1]);
io_set_bwlimit(0);
EXPECT_TRUE(send_status(p[1], STATUS_MANIFEST));
EXPECT_TRUE(send_int(p[1], 1));
EXPECT_TRUE(send_str(p[1], "first.txt"));
EXPECT_TRUE(send_int(p[1], 0)); /* protected-prefix section is empty */
EXPECT_TRUE(send_status(p[1], STATUS_MANIFEST));
EXPECT_TRUE(send_int(p[1], 1));
EXPECT_TRUE(send_str(p[1], "second.txt"));
EXPECT_TRUE(send_int(p[1], 0)); /* protected-prefix section is empty */
DeleteManifest* pending = NULL;
EXPECT_EQ_INT(run_pending_receiver(cfg, p[0], &pending), -1);
EXPECT_NULL(pending);
close(p[0]);
close(p[1]);
config_delete(cfg);
@@ -188,5 +559,12 @@ void test_server() {
test_receive_files_single_file();
test_receive_files_abort();
test_receive_manifest_rejects_traversal();
test_receive_incremental_check_rejects_invalid_nanoseconds();
test_incremental_check_quick_skip_by_mtime();
test_incremental_check_size_mismatch_full_transfer();
test_incremental_check_delta_oversize_reports_failure();
test_late_manifest_abort_frees_keepset();
test_late_manifest_eof_frees_keepset();
test_late_second_manifest_frees_both();
}
}
+308
View File
@@ -1,10 +1,318 @@
#include "test_shared_utils.h"
#include "utils.h"
#include "protocol.h"
#include "test_utils.h"
#include <dirent.h>
#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <threads.h>
#include <unistd.h>
/* ---- delete-walker tests ---- */
static char* make_walk_root(const char* tag) {
char* path = malloc(256);
if (!path)
return NULL;
snprintf(path, 256, "/tmp/fastsync_walk_%s_%d", tag, (int)getpid());
rmdir(path);
if (mkdir(path, 0755) != 0) {
free(path);
return NULL;
}
return path;
}
static bool write_file_at(const char* dir, const char* name, const char* content) {
char* path = path_cat(dir, name);
if (!path)
return false;
int fd = open(path, O_WRONLY | O_CREAT | O_TRUNC, 0644);
bool ok = fd >= 0;
if (fd >= 0) {
if (content) {
const char* p = content;
size_t remaining = strlen(content);
while (remaining > 0) {
ssize_t n = write(fd, p, remaining);
if (n <= 0) {
ok = false;
break;
}
p += n;
remaining -= (size_t)n;
}
}
close(fd);
}
free(path);
return ok;
}
static bool file_exists(const char* dir, const char* name) {
char* path = path_cat(dir, name);
bool exists = path && access(path, F_OK) == 0;
free(path);
return exists;
}
static bool dir_exists(const char* dir, const char* name) {
char* path = path_cat(dir, name);
struct stat st;
bool exists = path && stat(path, &st) == 0 && S_ISDIR(st.st_mode);
free(path);
return exists;
}
static int make_subdir(const char* root, const char* name) {
char* path = path_cat(root, name);
int rc = -1;
if (path) {
rc = mkdir(path, 0755);
free(path);
}
return rc;
}
static void remove_walk_tree(const char* path) {
DIR* dir = opendir(path);
if (!dir) {
rmdir(path);
return;
}
const struct dirent* entry;
while ((entry = readdir(dir)) != NULL) {
if (strcmp(entry->d_name, ".") == 0 || strcmp(entry->d_name, "..") == 0)
continue;
char* child = path_cat(path, entry->d_name);
if (child) {
struct stat st;
if (lstat(child, &st) == 0 && S_ISDIR(st.st_mode))
remove_walk_tree(child);
else
unlink(child);
free(child);
}
}
closedir(dir);
rmdir(path);
}
static ArrayList* make_manifest_strings(const char* const* entries, int count) {
ArrayList* manifest = array_list_create(free);
if (!manifest)
return NULL;
for (int i = 0; i < count; i++) {
char* dup = str_dup(entries[i]);
if (!dup || !array_list_add(manifest, dup)) {
free(dup);
array_list_delete(manifest);
return NULL;
}
}
return manifest;
}
static void test_walker_removes_extras_keeps_manifest_and_protected() {
char* root = make_walk_root("basic");
EXPECT_NOT_NULL(root);
EXPECT_TRUE(write_file_at(root, "a.txt", "extra"));
EXPECT_TRUE(write_file_at(root, "keep.txt", "kept"));
EXPECT_EQ_INT(make_subdir(root, "d"), 0);
EXPECT_TRUE(write_file_at(root, "d/e.txt", "extra"));
EXPECT_TRUE(write_file_at(root, "d/k.txt", "kept"));
EXPECT_EQ_INT(make_subdir(root, "prot"), 0);
EXPECT_TRUE(write_file_at(root, "prot/f.txt", "untouched"));
const char* keeps[] = {"keep.txt", "d/k.txt"};
ArrayList* manifest = make_manifest_strings(keeps, 2);
EXPECT_NOT_NULL(manifest);
DeleteSkipEntry skip = {"prot", false};
size_t deleted = 0;
DeleteWalkResult result = delete_extras_limited(root, manifest, 100000, &skip, 1, &deleted);
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_OK);
EXPECT_FALSE(file_exists(root, "a.txt"));
EXPECT_TRUE(file_exists(root, "keep.txt"));
EXPECT_FALSE(file_exists(root, "d/e.txt"));
EXPECT_TRUE(file_exists(root, "d/k.txt"));
EXPECT_TRUE(dir_exists(root, "d"));
EXPECT_TRUE(file_exists(root, "prot/f.txt"));
EXPECT_TRUE(deleted >= 2);
array_list_delete(manifest);
remove_walk_tree(root);
free(root);
}
static void test_walker_max_delete_exceeded_deletes_nothing() {
char* root = make_walk_root("maxdel");
EXPECT_NOT_NULL(root);
EXPECT_TRUE(write_file_at(root, "a.txt", "extra"));
EXPECT_TRUE(write_file_at(root, "b.txt", "extra"));
EXPECT_TRUE(write_file_at(root, "c.txt", "extra"));
const char* keeps[1] = {NULL};
ArrayList* manifest = make_manifest_strings(keeps, 0);
EXPECT_NOT_NULL(manifest);
size_t deleted = 999;
DeleteWalkResult result = delete_extras_limited(root, manifest, 2, NULL, 0, &deleted);
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_LIMIT_EXCEEDED);
EXPECT_EQ_INT((int)deleted, 0);
EXPECT_TRUE(file_exists(root, "a.txt"));
EXPECT_TRUE(file_exists(root, "b.txt"));
EXPECT_TRUE(file_exists(root, "c.txt"));
array_list_delete(manifest);
remove_walk_tree(root);
free(root);
}
static void test_walker_max_delete_exact_bound_deletes() {
char* root = make_walk_root("maxdel2");
EXPECT_NOT_NULL(root);
EXPECT_TRUE(write_file_at(root, "a.txt", "extra"));
EXPECT_TRUE(write_file_at(root, "b.txt", "extra"));
const char* keeps[1] = {NULL};
ArrayList* manifest = make_manifest_strings(keeps, 0);
EXPECT_NOT_NULL(manifest);
size_t deleted = 0;
DeleteWalkResult result = delete_extras_limited(root, manifest, 2, NULL, 0, &deleted);
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_OK);
EXPECT_EQ_INT((int)deleted, 2);
EXPECT_FALSE(file_exists(root, "a.txt"));
EXPECT_FALSE(file_exists(root, "b.txt"));
array_list_delete(manifest);
remove_walk_tree(root);
free(root);
}
static void test_walker_unlimited_deletes_all() {
char* root = make_walk_root("unlim");
EXPECT_NOT_NULL(root);
EXPECT_TRUE(write_file_at(root, "a.txt", "extra"));
EXPECT_TRUE(write_file_at(root, "b.txt", "extra"));
EXPECT_EQ_INT(make_subdir(root, "emptydir"), 0);
const char* keeps[1] = {NULL};
ArrayList* manifest = make_manifest_strings(keeps, 0);
EXPECT_NOT_NULL(manifest);
EXPECT_TRUE(delete_extras(root, manifest));
EXPECT_FALSE(file_exists(root, "a.txt"));
EXPECT_FALSE(file_exists(root, "b.txt"));
EXPECT_FALSE(dir_exists(root, "emptydir"));
array_list_delete(manifest);
remove_walk_tree(root);
free(root);
}
/* The 100000-entry server hard bound (MAX_SERVER_DELETE_COUNT, which this test
exercises through a literal to avoid reaching into file_receive.c) is also
all-or-nothing: a destination holding more extras than the bound must be left
completely untouched. Skipped under valgrind: 100k file creations would be
far too slow under instrumentation. */
static void test_walker_hard_bound_all_or_nothing() {
if (is_running_under_valgrind())
return;
enum { HARD_BOUND = 100000 };
char* root = make_walk_root("hardbound");
EXPECT_NOT_NULL(root);
int rootfd = open(root, O_RDONLY | O_DIRECTORY | O_CLOEXEC);
EXPECT_TRUE(rootfd >= 0);
bool created = true;
for (int i = 0; created && i < HARD_BOUND + 1; i++) {
char name[32];
snprintf(name, sizeof(name), "f%d", i);
int fd = openat(rootfd, name, O_WRONLY | O_CREAT | O_TRUNC, 0644);
if (fd < 0)
created = false;
else
close(fd);
}
EXPECT_TRUE(created);
const char* keeps[1] = {NULL};
ArrayList* manifest = make_manifest_strings(keeps, 0);
EXPECT_NOT_NULL(manifest);
size_t deleted = 999;
DeleteWalkResult result = delete_extras_limited(root, manifest, HARD_BOUND, NULL, 0, &deleted);
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_LIMIT_EXCEEDED);
EXPECT_EQ_INT((int)deleted, 0);
EXPECT_TRUE(file_exists(root, "f0"));
EXPECT_TRUE(file_exists(root, "f100000"));
array_list_delete(manifest);
/* Fast cleanup: unlink every created name through the still-open root fd. */
if (rootfd >= 0) {
for (int i = 0; i < HARD_BOUND + 1; i++) {
char name[32];
snprintf(name, sizeof(name), "f%d", i);
(void)unlinkat(rootfd, name, 0);
}
close(rootfd);
}
rmdir(root);
free(root);
}
typedef struct {
bool eight_bit_output;
const char* expected;
int failed;
} EscapeThreadArgs;
static int escape_thread(void* arg) {
EscapeThreadArgs* args = arg;
for (int i = 0; i < 1000; i++) {
char* escaped = output_escape("x\xc3\xa9\n", args->eight_bit_output);
if (!escaped || strcmp(escaped, args->expected) != 0)
args->failed = 1;
free(escaped);
}
return 0;
}
void test_shared_utils() {
test_walker_removes_extras_keeps_manifest_and_protected();
test_walker_max_delete_exceeded_deletes_nothing();
test_walker_max_delete_exact_bound_deletes();
test_walker_unlimited_deletes_all();
test_walker_hard_bound_all_or_nothing();
char formatted[32];
EXPECT_TRUE(format_human_bytes(0, formatted, sizeof(formatted)));
EXPECT_EQ_STR(formatted, "0 B");
EXPECT_TRUE(format_human_bytes(1024, formatted, sizeof(formatted)));
EXPECT_EQ_STR(formatted, "1.0 KB");
EXPECT_TRUE(format_human_bytes(1536 * 1024, formatted, sizeof(formatted)));
EXPECT_EQ_STR(formatted, "1.5 MB");
EXPECT_FALSE(format_human_bytes(1024, formatted, 4));
char high_bit[] = {'a', (char)0xc3, (char)0xa9, '\n', '\0'};
char* escaped = output_escape(high_bit, false);
EXPECT_EQ_STR(escaped, "a\\#303\\#251\\#012");
free(escaped);
escaped = output_escape(high_bit, true);
EXPECT_EQ_STR(escaped, "a\xc3\xa9\\#012");
free(escaped);
ProtocolSession safe_session;
ProtocolSession eight_bit_session;
protocol_session_init(&safe_session, -1, -1);
protocol_session_init(&eight_bit_session, -1, -1);
protocol_session_set_8_bit_output(&safe_session, false);
protocol_session_set_8_bit_output(&eight_bit_session, true);
EXPECT_FALSE(safe_session.eight_bit_output);
EXPECT_TRUE(eight_bit_session.eight_bit_output);
EscapeThreadArgs safe_args = {false, "x\\#303\\#251\\#012", 0};
EscapeThreadArgs eight_bit_args = {true, "x\xc3\xa9\\#012", 0};
thrd_t safe_thread;
thrd_t eight_bit_thread;
EXPECT_EQ_INT(thrd_create(&safe_thread, escape_thread, &safe_args), thrd_success);
EXPECT_EQ_INT(thrd_create(&eight_bit_thread, escape_thread, &eight_bit_args), thrd_success);
EXPECT_EQ_INT(thrd_join(safe_thread, NULL), thrd_success);
EXPECT_EQ_INT(thrd_join(eight_bit_thread, NULL), thrd_success);
EXPECT_FALSE(safe_args.failed);
EXPECT_FALSE(eight_bit_args.failed);
// Test str_dup
const char* dup_null = str_dup(NULL);
EXPECT_NULL(dup_null);
+33 -13
View File
@@ -4,34 +4,39 @@
static void test_ssh_connect_invalid_dest_no_colon() {
/* cppcheck-suppress constVariablePointer */
Client* client = client_connect_ssh("invalid-destination-no-colon", 22, NULL);
Client* client = client_connect_ssh("invalid-destination-no-colon", 22, NULL, false);
EXPECT_NULL(client);
}
static void test_ssh_connect_invalid_dest_empty() {
/* cppcheck-suppress constVariablePointer */
Client* client = client_connect_ssh("", 22, NULL);
Client* client = client_connect_ssh("", 22, NULL, false);
EXPECT_NULL(client);
}
/* Test client_connect_ssh with malformed destination (just a colon).
* parse_remote_dest succeeds, ssh is exec'd and fails, but the function
* creates a Client that must be cleaned up. */
/* A child that cannot exec ssh must not be returned as a successful client. */
static void test_ssh_connect_malformed() {
Client* client = client_connect_ssh(":", 22, NULL);
/* ssh binary exists, so exec succeeds; the function returns a Client.
* We just verify it doesn't crash and clean up properly. */
if (client != NULL) {
client_disconnect(client);
client_delete(client);
const char* old_path = getenv("PATH");
char* saved_path = old_path ? strdup(old_path) : NULL;
setenv("PATH", "", 1);
/* cppcheck-suppress constVariablePointer */
Client* client = client_connect_ssh(":", 22, NULL, false);
if (saved_path) {
setenv("PATH", saved_path, 1);
free(saved_path);
} else {
unsetenv("PATH");
}
EXPECT_TRUE(true);
EXPECT_NULL(client);
}
/* Test client_connect_ssh with valid format but unreachable host.
* The function launches ssh which will fail to connect, returns a Client. */
static void test_ssh_connect_unreachable() {
Client* client = client_connect_ssh("nonexistent.invalid:/remote/path", 22, NULL);
Client* client = client_connect_ssh("nonexistent.invalid:/remote/path", 22, NULL, false);
if (client != NULL) {
client_disconnect(client);
client_delete(client);
@@ -39,9 +44,24 @@ static void test_ssh_connect_unreachable() {
EXPECT_TRUE(true);
}
static void test_ssh_remote_command_argument_modes() {
char* command = ssh_build_remote_command("fast sync; touch /tmp/pwned", false);
EXPECT_EQ_STR(command, "'fast sync; touch /tmp/pwned' --stdio");
free(command);
command = ssh_build_remote_command("fast'sync", false);
EXPECT_EQ_STR(command, "'fast'\\''sync' --stdio");
free(command);
command = ssh_build_remote_command("fast sync; touch /tmp/pwned", true);
EXPECT_EQ_STR(command, "fast sync; touch /tmp/pwned --stdio");
free(command);
}
void test_transport_ssh() {
test_ssh_connect_invalid_dest_no_colon();
test_ssh_connect_invalid_dest_empty();
test_ssh_connect_malformed();
test_ssh_connect_unreachable();
test_ssh_remote_command_argument_modes();
}