Commit Graph
585 Commits
Author SHA1 Message Date
TapTap b031e73ab0 fix: treat an unreadable source root as fatal even under --ignore-errors
A sequential scanner records an opendir failure of its seed/root directory as a
skippable io_error and would complete an EMPTY scan, whose keep-set manifest
would then delete every destination entry.  The seed directory that maps to the
transfer root (relative path "") is now fatal regardless of --ignore-errors;
only subdirectories discovered during an otherwise-successful root scan are
skippable.  The -m path never had this hole (its root open failure aborts
scanner creation), so sequential and -m now agree.
2026-09-06 23:10:27 +02:00
TapTap 5f4c7ce638 fix: free walker-test root path after cleanup (ASan leak)
CI / lint (pull_request) Successful in 31s
CI / sanitizers (address) (pull_request) Successful in 40s
CI / sanitizers (undefined) (pull_request) Successful in 39s
CI / fuzz-build (pull_request) Successful in 18s
CI / coverage (pull_request) Successful in 36s
CI / valgrind (pull_request) Successful in 36s
CI / build-and-test (pull_request) Successful in 10m49s
make_walk_root() roots were removed from disk but never freed, leaking under
the address/valgrind sanitizer jobs.
2026-09-06 22:09:25 +02:00
TapTap 356b5592e0 feat: add confined symlink-safe directory-tree removal helper
CI / lint (pull_request) Successful in 32s
CI / sanitizers (address) (pull_request) Failing after 42s
CI / sanitizers (undefined) (pull_request) Successful in 43s
CI / fuzz-build (pull_request) Successful in 18s
CI / coverage (pull_request) Successful in 35s
CI / valgrind (pull_request) Failing after 36s
CI / build-and-test (pull_request) Successful in 10m49s
file_remove_tree_secure() opens the final component O_NOFOLLOW below the
authorized root and recursively wipes it with an fd-relative walk (symlinks are
removed by name, never followed); --force uses it to clear a destination
directory that blocks an incoming regular file.
2026-09-06 21:50:25 +02:00
TapTap 6e835fd9f7 docs: mark the delete-policy family implemented in RSYNC_COMPAT
--delete-excluded/--max-delete/--ignore-errors/--force/--prune-empty-dirs now
✅ with precise notes: the rsync-parity default (plain --delete protects
filter-excluded destination mirrors), the -m divergence (FastSync -m stays
multithreading, so --prune-empty-dirs is long-only), the all-or-nothing
max-delete/hard-bound error model, the 2.8.0 -> 2.9.0 protocol bump, and the
new two-section STATUS_MANIFEST frame.  Summary counts left for the orchestrator
to recount after the wave.
2026-09-06 21:50:16 +02:00
TapTap 1de2677bb1 test: delete-policy unit and integration coverage
Unit: walker all-or-nothing bounds (exceeded -> nothing removed + distinct
result; exact bound -> deletes), protected-prefix skipping, config wire
round-trip for force_delete/delete_excluded/prune_empty_dirs/max_delete, CLI
parse/validation for the new flags.  Integration (TestDeletePolicy): default
delete-excluded protection and --delete-excluded opt-out (single-thread, -m,
early --delete-before, excluded-dir subtrees), --max-delete all-or-nothing over
and at the limit, --force file-over-nonempty-dir replacement, --prune-empty-dirs
(--dirs mode + recursion-mode parity), and --ignore-errors keeping deletion
active across a genuine scan I/O error (run as an unprivileged user).
2026-09-06 21:50:12 +02:00
TapTap 0b25bacb18 feat: protect filter-excluded destination mirrors by default (--delete-excluded opt-in), --ignore-errors scan continuation, --prune-empty-dirs
The scanners now record every entry pruned by user-selection rules
(--filter/-C/per-dir, --exclude/--include, --max-size/--min-size) as a
destination-relative protected path on a caller-supplied sink (thread-safe in
the parallel scanner); --files-from subset pruning and -R relative wire paths
are never recorded.  The sender transmits these as manifest protected prefixes,
giving rsync's default --delete behavior (excluded mirrors survive) with
--delete-excluded opting back into deleting them.  --ignore-errors makes an
unreadable source directory a recorded, non-fatal scan error: the run continues,
the deletion still runs, and the exit code reports the ignored error.
--prune-empty-dirs omits an empty source directory's explicit --dirs entry.
Empty directories were never transferred by recursive scans (rsync -m parity).
2026-09-06 21:50:07 +02:00
TapTap c4e0de8f08 feat: split delete-manifest frame into keep-set + protected prefixes; enforce --max-delete and --force on the receiver
The STATUS_MANIFEST frame now carries two count-delimited sections: the kept
paths and a protected-prefix list (excluded-on-source paths the walker must not
delete unless --delete-excluded opted out).  The receiver's DeleteManifest is
passed through the commit/early paths unchanged.  manifest_delete_extras
honors a client --max-delete (all-or-nothing) and produces a distinct error for
it versus the 100000-entry server bound.  --force clears a non-empty directory
that blocks an incoming regular file (confined, symlink-safe) via a new
file_remove_tree_secure helper.
2026-09-06 21:50:02 +02:00
TapTap 1c9b660ac0 feat: all-or-nothing bounded delete walker
delete_extras_limited now rehearses a finite-capped deletion before unlinking
anything (an identical fd-relative walk that counts files and directories) and
returns DELETE_WALK_LIMIT_EXCEEDED with nothing removed when the run would
exceed the cap, so --max-delete is enforced per run instead of truncating the
deletion.  A directory that still holds entries the walker leaves in place
(protected excluded prefix, manifest-kept file, symlink) is left behind rather
than failing the whole deletion, matching rsync's leave-non-empty-dirs
behavior.  Rehearsal/delete each open an independent file description so a
prior pass cannot drain the directory stream.
2026-09-06 21:49:57 +02:00
TapTap 3631df0a3e feat: add delete-policy config fields and CLI flags
Adds ignore_errors (client-only) and force_delete (wire) booleans, makes
max_delete default -1 (no client limit), and parses --delete-excluded,
--max-delete=NUM, --ignore-errors, --force, --prune-empty-dirs.  Bumps
PROTOCOL_VERSION 2.8.0 -> 2.9.0 for the new on-the-wire force_delete field.
2026-09-06 21:49:53 +02:00
TapTap 3815b82306 docs: recount RSYNC_COMPAT summary after Phase-3 wave A
CI / lint (push) Successful in 32s
CI / sanitizers (undefined) (push) Successful in 41s
CI / sanitizers (address) (push) Successful in 42s
CI / fuzz-build (push) Successful in 16s
CI / coverage (push) Successful in 34s
CI / valgrind (push) Successful in 37s
CI / build-and-test (push) Successful in 7m13s
2026-09-06 20:11:39 +02:00
TapTap 01a5a93089 Merge feat/p3-basis-dest: alternate basis dirs (--compare-dest/--copy-dest/--link-dest) 2026-09-06 20:10:40 +02:00
TapTap 265b1e7669 Merge feat/p3-delete-timing: rsync delete timing (--delete-before/--delete-during/--del/--delete-after/--delete-delay) 2026-09-06 19:58:40 +02:00
TapTap 329fc60b14 test: make remove-source incremental-skip test deterministic
CI / lint (pull_request) Successful in 32s
CI / sanitizers (address) (pull_request) Successful in 42s
CI / sanitizers (undefined) (pull_request) Successful in 40s
CI / fuzz-build (pull_request) Successful in 17s
CI / coverage (pull_request) Successful in 35s
CI / valgrind (pull_request) Successful in 36s
CI / build-and-test (pull_request) Successful in 3m29s
The seed run did not preserve timestamps, so the destination copy's mtime was
the write time; the incremental --remove-source-files rerun only skipped the
file when both writes happened to land in the same whole second, making the
test flaky (observed intermittently in local full-suite runs and on CI).  Seed
with -M so the destination stores the source's exact mtime.
2026-09-06 19:53:22 +02:00
TapTap d6d502fbb4 docs: precise basis-dir caveats (shared-inode --inplace, attribute provenance, 256MiB limit)
CI / lint (pull_request) Successful in 32s
CI / sanitizers (address) (pull_request) Successful in 42s
CI / sanitizers (undefined) (pull_request) Successful in 40s
CI / fuzz-build (pull_request) Successful in 18s
CI / coverage (pull_request) Successful in 35s
CI / valgrind (pull_request) Successful in 36s
CI / build-and-test (pull_request) Failing after 3m31s
- --link-dest destination entries share the basis inode: a later --inplace run
  against such a path mutates the basis snapshot through the shared inode
  (recommend --copy-dest when the destination must stay independently writable).
- basis-hit files take mode/uid/gid and mtime from the basis file, not the
  sender's metadata (with --size-only the mtime can differ from the source).
- --remove-source-files sources satisfied by a basis dir are retained.
- basis runs refuse files above the 256 MiB whole-file limit up front (FastSync
  caps every whole-file payload path at 256 MiB; rsync supports arbitrary sizes);
  the config frame always carries a basis-count field (protocol 2.8.0).
2026-09-06 19:43:25 +02:00
TapTap e0e0ea6eac test: xxHash equal-size gate, basis priority, size-only/ignore-times, oversize
- unit: config basis-path normalization (trailing slash, a//b, ./x/./y collapse;
  degenerate inputs rejected).
- integration: same-size/same-mtime/different-content fixtures prove the xxHash
  gate -- the basis changed.txt now has the SAME byte size as the source so the
  size short-circuit can no longer mask the hash comparison, plus a dedicated
  parametrized same-size mismatch test asserting link/copy never use a
  content-mismatched basis and compare-dest transfers.
- basis-dir priority is first-match-wins: two link-dest dirs (inode of the
  first), and compare-dest before link-dest stays sparse while the reverse
  order hard-links.
- --size-only links a same-content basis file with a different mtime;
  --ignore-times never links even an exact match.
- --delay-updates + --delete removes extras inside a nested .fastsync-stage
  dir (regression guard) while keeping the real staging dir and basis tree.
- a basis run containing a file above the whole-file limit fails up front with
  a clear error and transfers nothing.
2026-09-06 19:43:21 +02:00
TapTap 4799d12e25 fix: refuse basis runs containing an over-limit file before any transfer
Basis dirs imply the per-file incremental check, which (like every whole-file
payload path in FastSync) is bounded by MAX_RECEIVE_WHOLE_FILE_SIZE.  A source
tree with a larger file used to abort the whole run mid-stream on the receiver
with no client-side diagnostic.  With basis dirs configured the client now
preflights the scan (respecting filters/size rules) and fails up front with a
clear error naming the offending file before connecting, matching the
documented 256 MiB whole-file limit instead of aborting silently.
2026-09-06 19:43:15 +02:00
TapTap 4bf4da37e5 fix: free basis path on copy-dest hits; keep --delete staging skip top-level-only
- copy-dest basis hits leaked the heap-allocated basis path: BASIS_DEST_COPY
  did not transfer it (only LINK does) and returned before the basis cleanup.
  basis_match_free is now called on every materialization return path (success
  and send-failure) after content/link ownership is transferred.
- the --delete walker regression: the delay-updates staging name must be
  protected only as a DIRECT child of the receive root, while basis dirs may
  be skipped at any depth.  delete_extras_limited now takes DeleteSkipEntry
  entries carrying a top_level_only flag instead of a flat prefix list, so a
  nested destination directory named .fastsync-stage is ordinary content again
  (its extras are deleted) and a basis tree is still never removed.
2026-09-06 19:43:11 +02:00
TapTap 08a5815ca7 refactor: unify basis-dir path validation and normalization
config_basis_path_valid and config_basis_append now share one normalizer
(basis_path_normalize): interior empty components (a//b) collapse, '.'
components and trailing slashes are dropped, and the stored form is exactly
the canonical relative path used by validation, the delete-walker prefix match
and the receiver's basis lookup.  Degenerate inputs (empty, absolute, '..',
'.' that normalizes to nothing) stay rejected.
2026-09-06 19:43:06 +02:00
TapTap 02679fe335 docs: clarify --del alias, early keep-set caps, ACK wait, --no-delete conflict
CI / lint (pull_request) Successful in 25s
CI / sanitizers (undefined) (pull_request) Successful in 41s
CI / sanitizers (address) (pull_request) Successful in 42s
CI / fuzz-build (pull_request) Successful in 16s
CI / coverage (pull_request) Successful in 33s
CI / valgrind (pull_request) Successful in 36s
CI / build-and-test (pull_request) Successful in 6m51s
Usage help now gives --delete-during a complete description with --del on its
own line, and notes that timing flags imply --delete while timing+--no-delete
is rejected regardless of argument order. RSYNC_COMPAT.md documents: the
receiver's MAX_MANIFEST_ENTRIES/MAX_MANIFEST_BYTES caps now abort an early-mode
run before any data (previously only the deletion step failed), the extended
early-delete ACK deadline, and the order-independent flag-conflict policy.
2026-09-06 19:35:28 +02:00
TapTap c0c315cf48 test: cover -m late-deletion failure, receiver leak exits, timed ACK read
- Unit (leak guards): drive receiver_process_pending() past a parked keep-set
  into STATUS_ABORT, EOF, and a second manifest frame; each must return -1 with
  no manifest handed out. Verified leak-free under ASan.
- Unit: receive_status_timed reads a status and fails cleanly on EOF.
- Integration: test_late_flags_commit_only_after_success now parametrizes the
  -m path, proving a failed -m late-timing run preserves every extra and that
  the deferred manifest is dropped (never applied) when the writer fails.
2026-09-06 19:35:25 +02:00
TapTap ebfaced5c2 fix: wait for the early-delete ACK with an extended deadline
The receiver performs the whole bounded deletion walk (up to
MAX_SERVER_DELETE_COUNT unlinks) before answering the delete-before/during
manifest, so its STATUS_OK reply can take far longer than the default 60 s
per-message receive window. Waiting with the default would make the sender
abort AFTER the deletion had already committed on the receiver. Add a timed
receive variant (receive_status_timed / protocol_receive_n_data_timed) and use
it for the early-manifest ACK with a 1 h explicit deadline; connection errors
and EOF still abort immediately.
2026-09-06 19:35:21 +02:00
TapTap bbf982dc0b fix: free the parked delete manifest on every receiver error exit
The late/commit path keeps the received keep-set in a local list until
STATUS_FINISHED. Error exits after it was parked (STATUS_ABORT, a failing
receive_status / non-FINISHED status, a second manifest frame, or a later
file/chunk/store failure) previously dropped the only reference and leaked up
to ~16 MB of path strings + pointer array per connection. Both failure labels
now discard the parked list exactly once; the successful FINISHED path still
hands ownership to *pending_manifest (the -m caller) without freeing it.
2026-09-06 19:35:17 +02:00
TapTap 36c2c04910 docs: mark rsync delete-timing family implemented
CI / lint (pull_request) Successful in 26s
CI / sanitizers (address) (pull_request) Successful in 39s
CI / fuzz-build (pull_request) Successful in 16s
CI / sanitizers (undefined) (pull_request) Successful in 37s
CI / coverage (pull_request) Successful in 34s
CI / valgrind (pull_request) Successful in 37s
CI / build-and-test (pull_request) Successful in 6m22s
RSYNC_COMPAT.md: flip --delete-before, --del/--delete-during, --delete-delay
and --delete-after to Implemented with precise notes (default-under--delete,
safety model, 2.7.0 -> 2.8.0 protocol bump, exact divergences from rsync).
README option tables list the new flags and the delete-after default.
2026-09-06 18:53:28 +02:00
TapTap 7eacf7c180 test: cover rsync delete-timing flags, config wire, and semantics
- CLI: each timing flag (+ --del alias) accepted and implies --delete;
  conflicting timings and a timing with --no-delete are rejected.
- Config: delete_during/delete_delay survive config_send/config_receive;
  two simultaneous timings are rejected by the receiver-side validation;
  config_delete_timing_early() mapping is unit-tested.
- Integration (TCP, single- and multithreaded): every flag removes extras on
  a successful transfer; early modes (--delete-before/--delete-during/--del)
  delete before data is applied so a destination file blocking a nested
  write is removed and the transfer succeeds, while plain --delete /
  --delete-after / --delete-delay keep it and fail with every extra intact
  (commit-style). Early timing also completes (without deleting) when the
  server refuses deletion.
2026-09-06 18:53:24 +02:00
TapTap 4e725517f0 feat: implement rsync delete timing (--delete-before/--delete-during/--delete-delay/--delete-after)
Deletion timing is now real and selected by the four rsync flags plus the
plain --delete default. Wire protocol bumps to 2.8.0: two new config
booleans (delete_during, delete_delay) are serialized and validated, joining
the existing delete_before/delete_after.

- Early modes (--delete-before, --delete-during/--del): the sender pre-scans
  the whole tree (paths only), transmits the keep-set manifest BEFORE any
  file data, and the receiver removes extras and acks STATUS_OK; the sender
  only streams data after the deletion committed. Deletion is thus performed
  even if a later transfer phase fails (rsync delete-before/during are
  destructive by definition). FastSync streams in a single scan so it cannot
  interleave per-directory like rsync delete-during; --delete-during selects
  the same engine mode as --delete-before (documented divergence).
- Late/commit modes (plain --delete, --delete-after, --delete-delay): the
  manifest closes the data stream and deletion is committed only after
  STATUS_FINISHED proves the whole transfer succeeded, preserving FastSync's
  commit-style safety. --delete-delay converges with --delete-after because
  FastSync never snapshots the destination during data flow (documented).
- The STATUS_MANIFEST frame is now self-delimiting and position-independent.
  Single-threaded receivers delete before the success frame; the -m receiver
  hands the keep-set to server.c, which commits the deletion only after the
  disk writer thread has drained (fixes a delete-vs-in-flight-temp race).
- Every timing flag implies --delete; at most one timing flag is allowed.
- Each timing flag implies --delete, matching rsync; conflicts are rejected.
2026-09-06 18:53:20 +02:00
TapTap 196a27689f docs: mark --compare-dest/--copy-dest/--link-dest implemented
CI / lint (pull_request) Successful in 33s
CI / sanitizers (address) (pull_request) Successful in 45s
CI / sanitizers (undefined) (pull_request) Successful in 43s
CI / fuzz-build (pull_request) Successful in 18s
CI / coverage (pull_request) Successful in 35s
CI / valgrind (pull_request) Successful in 36s
CI / build-and-test (pull_request) Successful in 3m20s
Document receiver-side basis semantics, relative-to-destination-root
confinement, content-verified matching, hard-link vs copy vs compare-only
behavior, cross-filesystem copy fallback, repetition/priority, --delete
exclusion, the implied --incremental and -s incompatibility, and each exact
divergence from rsync (no dest deletion on compare-dest stale entries, no
attribute re-application on basis hits, no re-linking of already-up-to-date
dest files, sources matched from basis dirs kept under --remove-source-files).
2026-09-06 18:47:41 +02:00
TapTap 0d18b7fc87 test: integration coverage for compare/copy/link-dest basis directories
- compare-dest skips an exact basis match (leaving a sparse destination) and
  still transfers files the basis cannot satisfy; a content mismatch forces a
  normal transfer.
- copy-dest materializes the unchanged file as a real local copy (distinct
  inode) and transfers content mismatches.
- link-dest hard-links (asserted same inode/nlink to the DIR file) and falls
  back to a normal transfer on content mismatch.
- a missing basis dir is a clean full-transfer no-op for all three flags.
- link-dest works through -m multithreading and --delay-updates (staged and
  published as a real link, staging cleaned up).
- --delete removes genuine extras while leaving the basis dir untouched.
2026-09-06 18:47:37 +02:00
TapTap 1fc0cacc32 test: unit tests for basis-dir CLI parsing and config wire round-trip
- parse_args accepts each flag in both forms, keeps repetition order/types,
  implies --incremental + metadata, and rejects absolute/escaping/degenerate
  paths; validate_config rejects basis dirs combined with -s.
- config wire round-trips a mixed basis list and rejects escaping/absolute
  paths on the receiver side.
2026-09-06 18:47:33 +02:00
TapTap 8f846a43b8 feat: exclude basis directories from --delete
Generalize the delete walker's protected-root-child skip into a prefix list.
The receiver now passes both the --delay-updates staging directory and every
basis-dir path, so a --delete run can never treat a basis snapshot (which a
--link-dest run just linked from) as destination content to remove.
2026-09-06 18:47:29 +02:00
TapTap d38920c972 feat: receiver-side basis matching with link/copy materialization
The receiver's per-file incremental check now consults the ordered basis-dir
list whenever the destination is not already up to date.  An exact basis match
requires equal size, equal mtime (unless --size-only; --ignore-times disables
basis matching like rsync), and an equal content xxHash64 -- the sender sends
its xxHash for every file whenever basis dirs are configured (not only under
--checksum), so a hard link or local copy is only ever made from byte-identical
content.

On a match:
  - compare-dest: reply STATUS_OK and skip data only when the destination does
    not already hold the file (sparse, rsync parity).  A destination that holds
    a DIFFERENT version falls back to a normal transfer instead of rsync's
    delete, keeping the mirror complete.
  - copy-dest: reply STATUS_OK and hand a synthetic File (bytes read from the
    basis file, basis metadata) to the normal store sink, so the file is
    installed as a real local copy through the existing atomic temp+rename
    engine and honors --existing/--ignore-existing/--update/--backup/
    --delay-updates/--partial-dir unchanged.
  - link-dest: same, but File.basis_link records the basis path and the store
    engine calls the new file_to_disk_secure_link(): an atomic temp hard link +
    rename.  Cross-filesystem/refused links fall back to a byte-identical local
    copy (never a corrupt or partial file); the copy fallback applies metadata,
    while a successful link keeps the basis inode's own attributes so the basis
    file is never mutated.

Basis-materialized files carry File.skip so they are not acknowledged to a
--remove-source-files sender (the sender already saw STATUS_OK and keeps the
source).  The no-match path is byte-for-byte identical to the existing delta /
full-data transfer.
2026-09-06 18:47:26 +02:00
TapTap df890f76ea feat: basis-dir config/CLI for --compare-dest/--copy-dest/--link-dest
Replace the vestigial single compare_dest/copy_dest/link_dest Config fields with
an ordered BasisDest list (type + path per entry) that is serialized to the
receiver and interpreted relative to the destination root.  Paths must be
relative with no '.'/'..' components (confined like --backup-dir); trailing
slashes are normalized.  Wire layout changes, so PROTOCOL_VERSION -> 2.8.0.

CLI: each flag is parsed in both --flag=DIR and --flag DIR forms, is
repeatable, and keeps command-line order as basis priority.  Supplying any
basis dir implies --incremental (and therefore metadata) on the sender because
the unchanged decision is receiver-side; combining basis dirs with -s chunk
serialization is rejected in validate_config.  Usage text updated.
2026-09-06 18:47:20 +02:00
TapTap 4cf34026e7 test: drop racy queued_bytes==0 assert in byte-budget test
CI / lint (push) Successful in 25s
CI / sanitizers (address) (push) Successful in 40s
CI / sanitizers (undefined) (push) Successful in 40s
CI / fuzz-build (push) Successful in 17s
CI / coverage (push) Successful in 34s
CI / valgrind (push) Successful in 36s
CI / build-and-test (push) Successful in 3m6s
After the writer releases bytes, the blocked enqueuer may already have
admitted the next payload, so the transient queued_bytes==0 read is
scheduling-dependent (lost the race under ASan). The post-join state
(covers unblocking + budget re-limit) is deterministic.
2026-09-06 16:27:54 +02:00
TapTap b04ffa608b fix: free rel path on parallel root-scan entries without -R
CI / lint (push) Successful in 26s
CI / sanitizers (undefined) (push) Successful in 41s
CI / sanitizers (address) (push) Failing after 41s
CI / fuzz-build (push) Successful in 16s
CI / coverage (push) Successful in 35s
CI / valgrind (push) Successful in 37s
CI / build-and-test (push) Successful in 3m6s
scan_root_entry str_dup'd the entry name for every root-level file but only
consumed it on the -R + --files-from path, leaking 1 string per root file in
normal parallel scans (found by CI ASan/valgrind).
2026-09-06 16:21:23 +02:00
TapTap 4146814aee Merge feat/p2-relative-dirs-mkpath: -R/--no-implied-dirs/--dirs/--mkpath
CI / lint (push) Successful in 25s
CI / sanitizers (undefined) (push) Successful in 42s
CI / sanitizers (address) (push) Failing after 42s
CI / fuzz-build (push) Successful in 16s
CI / coverage (push) Successful in 34s
CI / valgrind (push) Failing after 37s
CI / build-and-test (push) Successful in 3m8s
Final Phase-2 row: -R+files-from dest layout (bare relative wire path),
--no-implied-dirs ancestor checks, --dirs/-d explicit directory entries
(STATUS_MKDIR + chunk dir marker), --mkpath server create-root (absent
root now rejected without it). Protocol 2.7.0. c-review REQUEST CHANGES
-> blocker (trailing-slash/root-equal dest) + warnings/nits fixed; PR #266.
2026-09-06 16:10:23 +02:00
TapTap 3275b6c978 fix: address c-review for -R/--dirs/--mkpath row
B1: destination-root existence/creation mishandled two legitimate forms.
file_directory_exists_secure/file_ensure_directory_secure now normalize a
trailing-slash destination (so the last component is never an empty leaf)
and treat a destination equal to the already-open authorized root as present
(no spurious <root>/<basename> nested dir with --mkpath). Confinement and
O_NOFOLLOW probing are unchanged. Regression integration tests: existing
dest with trailing slash works with and without --mkpath; dest == authorized
root works and creates no stray nested dir.

W1: chunk serialize/deserialize round-trip unit test for a directory entry
mixed with a regular file, with and without metadata; --dirs coverage under
-s and -s -m.
W2: --list-only and --dry-run now print file_wire_path() so all outputs show
the -R transformed relative name, matching -i/--out-format.
W3: RSYNC_COMPAT -d/--dirs note: dirs are created immediately under
--delay-updates (only regular files are staged).
W4: --dirs generator flushes chunks by element count too, so a long
--files-from list of empty directories cannot exceed the per-chunk file cap.
N1: STATUS_MKDIR added to status_to_string.
N2: removed dead TestMkpath._transfer.
N3: removed redundant --no-implied-dirs OPTION_TABLE row.
N4: integration tests: --dirs --delete keeps the just-created empty dir (and
deletes extras); a listed dir colliding with a regular file at the dest fails
cleanly.
2026-09-06 16:10:00 +02:00
TapTap c2cf231158 feat: implement -R/--relative, --no-implied-dirs, --dirs/-d, --mkpath
CI / lint (pull_request) Successful in 26s
CI / sanitizers (address) (pull_request) Failing after 41s
CI / sanitizers (undefined) (pull_request) Successful in 40s
CI / fuzz-build (pull_request) Successful in 18s
CI / coverage (pull_request) Successful in 35s
CI / valgrind (pull_request) Failing after 37s
CI / build-and-test (pull_request) Failing after 2m35s
RSYNC_COMPAT Phase-2 row M: path-list construction and destination
directory creation while preserving traversal safety.

- -R/--relative with --files-from: transmit each listed entry under its
  bare relative destination path (no source-root mirror). Files keep an
  absolute local read path plus a separate wire/dest path (File.send_path);
  manifest, incremental quick-check and change output follow the wire path,
  so --delete and --remove-source-files stay consistent. -R without
  --files-from is unchanged (full mirror).
- --no-implied-dirs: client-only, only meaningful with -R + --files-from.
  A listed file whose parent dir is not itself (or via an ancestor)
  explicitly listed cannot be placed; the run fails up front with a clear
  error. No effect otherwise.
- --dirs/-d + --old-dirs/--old-d aliases: -d <dir> transmits the source
  root as an explicit empty directory entry (STATUS_MKDIR frame); with
  --files-from listed dirs are created empty and listed files transferred,
  never descending. Works single-threaded, -m (sequential scanner in the
  -m scan thread) and chunk-serialization (per-file type marker).
  Directory entries appear in the delete manifest.
- --mkpath: new wire bool; server creates the destination root (and missing
  leading components under its authorized root) at connection start. A
  missing destination root is now rejected by default.
- Protocol bumped to 2.7.0 (mkpath wire field + STATUS_MKDIR + chunk type
  marker). All receive paths funnel through file_save_to_disk_full which
  creates directories via the secure confined mkdir engine; dir entries are
  excluded from --remove-source-files outcome acknowledgements on both ends.
- Unit coverage: CLI parse (relative/dirs aliases/mkpath/no-implied-dirs),
  config round-trip (relative + mkpath), scanner --dirs non-recursion and
  -R send_path (sequential + parallel), receiver dir-entry save.
- Integration coverage: TestRelativeFilesFrom, TestNoImpliedDirs, TestDirs,
  TestMkpath (single and -m).
- RSYNC_COMPAT: 4 rows move to Implemented (Summary 67/3/5/1/71 = 147).
2026-09-06 15:28:32 +02:00
TapTap a196522010 style: clang-format test_client_cli.c after merge resolution
CI / lint (push) Successful in 23s
CI / sanitizers (undefined) (push) Successful in 42s
CI / sanitizers (address) (push) Successful in 43s
CI / fuzz-build (push) Successful in 17s
CI / coverage (push) Successful in 34s
CI / valgrind (push) Successful in 36s
CI / build-and-test (push) Successful in 1m45s
2026-09-06 14:29:28 +02:00
TapTap 342dd152ef Merge feat/p2-files-from-filter: files-from/from0/filter/-F/-C
Client-side file-list selection and filter-rule layer (client-only; no wire
change). rsync-consistent inner-first per-dir filter precedence; listed-but-
missing files-from entries hard-error; root '/' scan regression fixed;
Summary recounted (63 implemented / 75 not).
c-review REQUEST CHANGES -> blockers fixed; PR #265.
2026-09-06 14:25:51 +02:00
TapTap a6c982cd86 Merge feat/p2-delay-updates: implement --delay-updates
Receiver stages writes and publishes only after the full transfer succeeds
(single and -m, before the outcomes frame). delete walker skips staging;
backup-dir name reserved; flock serializes concurrent delayed sessions;
protocol bump to 2.6.0. c-review REQUEST CHANGES -> blockers fixed; PR #264.
2026-09-06 14:19:38 +02:00
TapTap 4295fefaa3 fix: --delay-updates delete/backup collisions, publish-failure test, staging lock
Review fixes for --delay-updates:

- --delete no longer deletes the staged files: the delete walker gains a
  skip_root_child parameter and receive_manifest passes DELAY_UPDATES_STAGING_DIR
  when delay_updates is active, so deletion removes genuine extras while the
  staging dir (a direct child of the receive root) is left for publication in
  both single and -m modes.
- --backup-dir is rejected when it collides with the reserved internal staging
  name .fastsync-stage (trailing slash normalized), in client validation and in
  the received-config wire validation, preventing old backups from being
  silently installed as new files.
- Staging dir is now held under an exclusive advisory flock for the whole
  transfer (context lifetime): two simultaneous delayed transfers to one
  destination root no longer share/destroy each other's staged data - the
  second fails cleanly.  Cleanup only touches the staging dir when this context
  owns the lock, so a lock-contention failure cannot wipe a live session.
- Post-publish staging cleanup now returns/logs instead of discarding failures
  (warning when the staging dir cannot be fully removed).
- Reworked the publish-failure integration test to exercise real mid-publish
  semantics (top-level file published, nested rename fails, no rollback,
  sources retained under --remove-source-files) and added integration tests for
  --delete + --delay-updates ordering and reserved --backup-dir rejection.
- RSYNC_COMPAT note documents delete ordering, the reserved-name hazard, and
  the concurrency guard.
2026-09-06 14:19:18 +02:00
TapTap 7f2180ef90 fix: filter precedence, files-from errors, NUL/CRLF and filter-rule rejections
Address an independent c-review of the files-from/filter feature:

- .rsync-filter precedence now matches rsync: evaluate the innermost
  (current) directory's rules first, then ancestors, then the command-line
  base (--filter/-C), so a deeper file's '+' can re-include what a shallower
  '-' excluded (regression tests in both scan modes; single-thread and -m).
- --files-from: a listed entry missing on disk and an empty list are now hard
  errors surfaced pre-transfer in send_files, send_files_multithreaded,
  dry-run and --list-only; '.' (whole tree) and empty listed dirs stay valid.
- scanner_path_relative now handles a transfer root of / (previously the
  scanner aborted on children of /).
- Reject unsupported rsync filter syntax explicitly (no silent no-ops):
  +/- modifiers other than '/' (! C s r p x) and rules beginning with ':'/'.'
  /'!' (merge/dir-merge/list-clear shorthands). Docs updated.
- -0/--from0 NUL mode preserves entry bytes (no CR/LF trimming); only newline
  mode trims. Absolute-entry error message no longer includes the newline.
- --no-from0/--no-cvs-exclude registered as negatable booleans.
- RSYNC_COMPAT rows updated for the precedence, rejection list, NUL-mode
  detail and the documented O(entries x files) scalability bound of the
  allow-set (Summary unchanged: 62/3/5/1/76 = 147).
2026-09-06 14:16:12 +02:00
TapTap f4c15a7b78 feat: add --files-from/-0, --filter, -C and -F filter layer
CI / lint (pull_request) Successful in 23s
CI / sanitizers (address) (pull_request) Successful in 40s
CI / sanitizers (undefined) (pull_request) Successful in 39s
CI / fuzz-build (pull_request) Successful in 17s
CI / coverage (pull_request) Successful in 34s
CI / build-and-test (pull_request) Successful in 1m28s
CI / valgrind (pull_request) Successful in 36s
Implement the RSYNC_COMPAT Phase-2 filter/parser feature group:
- --files-from=FILE (repeatable) plus -0/--from0 NUL delimiters: parse the
  source file list relative to the source root into a shared read-only
  allow-set; the scanner transfers listed files and the whole subtree of
  listed directories and prunes everything else in single- and
  multithreaded mode. Absolute/'..' entries and missing files are hard
  CLI errors.
- --filter=RULE: rsync-style +/- rules (anchored '/', dir-only trailing '/',
  word include/exclude forms) evaluated first-match-wins with a default of
  include, as an independent layer from legacy --exclude/--include.
  Unsupported directives (merge/hide/... ) are rejected explicitly. -f stays
  sendfile.
- -C/--cvs-exclude: well-known rsync CVS default exclude set.
- -F: per-directory .rsync-filter files read during traversal and applied to
  the owning directory's subtree (single + parallel), never transferred.
- Delete manifest still derives from what was actually sent.

Client-only config fields; no wire/protocol change. Adds unit coverage
(CLI parse, allow-set and filter scanning single+parallel) and integration
tests (TestFilesFrom, TestFilters). RSYNC_COMPAT matrix rows updated:
5 rows move to Implemented (Summary 62/3/5/1/76 = 147).
2026-09-06 13:43:34 +02:00
TapTap a2a82dd856 feat: implement --delay-updates receiver staging and publication
CI / lint (pull_request) Failing after 22s
CI / build-and-test (pull_request) Skipped
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
Stage every successfully written file under a private 0700 .fastsync-stage
directory inside the receive root and atomically publish all staged files
only after the whole protocol stream (manifest/delete handling included)
has completed, immediately before the success/outcome frame.  On any
abort/error before publication nothing is installed and staging is removed;
a publish failure aborts the transfer with best-effort cleanup of the
remainder (already-published files are not rolled back).  Crash leftovers
are wiped when the next delayed transfer starts.

Wire: new delay_updates config flag (selection-options block), protocol
version bumped to 2.6.0, client/server validation rejects --inplace.
CLI/usage/validation updated.  Works in single-threaded and -m modes
(exactly one write_thread stages files; the staged-file registry is
mutex-protected; publication runs once after both threads join).
--existing/--ignore-existing/--update decide against the final destination
at stage time; --backup is deferred to publication.  remove_source_files
outcomes are only sent after publication so skipped/unpublished sources are
never deleted.  Default (no flag) behavior is unchanged.

Tests: config wire round-trip, CLI parse, --inplace rejection, new
test_delay_updates unit suite (27 suites total), and integration
TestDelayUpdates covering single/-m parity, incremental reruns, remove
source files, receiver-skip ordering, and a deterministic publish-failure
abort path.
2026-09-06 13:20:03 +02:00
TapTap 8577f95550 fix: cppcheck cleanups (printf sentinel type, scanner test style)
CI / lint (push) Successful in 22s
CI / sanitizers (address) (push) Successful in 40s
CI / sanitizers (undefined) (push) Successful in 39s
CI / fuzz-build (push) Successful in 16s
CI / coverage (push) Successful in 34s
CI / build-and-test (push) Successful in 1m23s
CI / valgrind (push) Successful in 36s
2026-09-06 12:56:54 +02:00
TapTap 2931bb97b4 docs: recount RSYNC_COMPAT summary after Phase-2 wave A
6 rows flipped to implemented across the three merges; total stays 147.
2026-09-06 12:53:50 +02:00
TapTap 88ab4f65cb Merge feat/p2-itemize-output: structured change output (-i, --list-only, --out-format, --log-file-format)
Client-only per-file event/format model (change_list.c). Reviewed
(c-review APPROVE WITH NITS, all fixed); PR #263.
2026-09-06 12:52:46 +02:00
TapTap 9fbb86ca68 Merge feat/p2-temp-dir: implement --temp-dir
Receiver writes temps into a confined scratch dir and atomically renames
into place; EXDEV aborts; inplace/partial bypass; thread-safe temp names.
Uses existing wire field; no protocol bump. Reviewed (c-review APPROVE
WITH NITS, all fixed); PR #262.
2026-09-06 12:52:11 +02:00
TapTap f91ca70eda Merge feat/p2-one-file-system: implement -x/--one-file-system
Sender-side scanner stays within the source filesystem (-x), single and
multithreaded; default unchanged; client-only, no wire change. Reviewed
(c-review APPROVE WITH NITS, all fixed); PR #261.
2026-09-06 12:51:12 +02:00
TapTap 1b67f5ffcf docs: clarify %b semantics, temper thread-safety claim, harden %M scan; add -m coverage
CI / lint (pull_request) Successful in 20s
CI / sanitizers (undefined) (pull_request) Successful in 41s
CI / sanitizers (address) (pull_request) Successful in 41s
CI / fuzz-build (pull_request) Successful in 16s
CI / coverage (pull_request) Successful in 35s
CI / build-and-test (pull_request) Successful in 1m22s
CI / valgrind (pull_request) Successful in 36s
Address c-review nits on the itemize/output feature:
- RSYNC_COMPAT.md: state that %b is the source length (always == %l) because
  no wire-byte counter exists; keep Summary equal to the matrix (recounted:
  54 implemented / 84 not-implemented, 147 rows total - four rows flipped).
- change_list.h/.c: document bytes_sent == size; note itemize/out-format lines
  never interleave with each other but may interleave with legacy log
  messages sharing the stream; mark the %M stat() path best-effort.
- change_render_format scan in format_uses_mtime now mirrors the tokenizer
  (skips '%%' and unknown '%X' pairs) so a literal '%%M' no longer triggers
  the stat() fallback.
- Integration tests: --list-only under -m; a changed file on a second
  --incremental run emits exactly one '>f' line while unchanged files print
  nothing; --log-file + --log-file-format under -m.
2026-09-06 12:49:49 +02:00
TapTap 2d841405e6 test: cover -x cross-device skip without root; harden OneFileSystem tests
CI / lint (pull_request) Failing after 20s
CI / build-and-test (pull_request) Skipped
CI / sanitizers (address) (pull_request) Skipped
CI / sanitizers (undefined) (pull_request) Skipped
CI / fuzz-build (pull_request) Skipped
CI / coverage (pull_request) Skipped
CI / valgrind (pull_request) Skipped
Add a rootless unit test that reaches the actual st_dev skip branch in both
the sequential and parallel (-m) scanners: a symlink nested under the scan
root points at a directory on /dev/shm (a different device than the build
fs) and, under --copy-links semantics, -x must drop that subtree while a
plain scan includes it. Skips only when no cross-device target exists.
Integration OneFileSystem test now cleans both dest dirs up front and
reports a busy test mountpoint instead of ignoring the umount result.
RSYNC_COMPAT.md notes that cross-filesystem mount-point subdirectories are
dropped entirely (rsync parity).
2026-09-06 12:49:26 +02:00