diff --git a/CHANGELOG.md b/CHANGELOG.md index 658e241..beed8f1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,14 @@ The rsync-parity cycle 2.29 (no wire change; `PROTOCOL_VERSION` stays 2.28.0). `RSYNC_COMPAT.md` moves from **116 ✅ / 14 ⚠️ / 27 ❌** to **120 ✅ / 10 ⚠️ / 27 ❌** of 157 rows. +An audit cycle follows on the same wire version (`PROTOCOL_VERSION` stays +2.28.0): a security-and-correctness pass over the parity-2.29 baseline. It fixes +a `--temp-dir` symlink escape, gates client-controlled special permission bits, +corrects `--partial-dir`/`--bwlimit`/`-z` behavior, rejects unsupported filter +modifiers, and tightens client and wire validation. No parity row changes +classification, so the matrix stays **120 ✅ / 10 ⚠️ / 27 ❌** of 157 rows; the +affected rows' notes and the summary tally in `RSYNC_COMPAT.md` were updated. + ### Changed - **rsync-exact traversal order.** The sequential scanner now walks each @@ -47,6 +55,64 @@ The rsync-parity cycle 2.29 (no wire change; `PROTOCOL_VERSION` stays 2.28.0). (a general whole-file limit, not basis-specific). - `--stats` byte totals and `--msgs2stderr` stay documented divergences. +### Security + +- **`--temp-dir` symlink escape fixed.** The receiver's scratch directory was + opened with a bare `open()`, so a symlink planted under the receive root could + redirect receiver scratch files outside the authorized root. The opened + directory is now judged by the real path of its fd (`/proc/self/fd` via + `realpath`) and an escaping target is refused (`EACCES`, logged); an in-root + link to another filesystem (the `EXDEV` fallback case) still works. +- **Client-controlled special bits masked when super-user activities are not + permitted.** Setuid/setgid/sticky bits (`--perms`, `--chmod`, the symlink and + special-node paths, and deferred directory modes) are now stripped when the + connection forbids super activities (`--no-super`, a non-opted daemon module, + a privileged listener without `--allow-super`); exact rsync semantics are + preserved wherever super activities are permitted. +- **Daemon umask no longer forced to `0`.** `daemonize()` now sets the + conventional `022`, so implied parent directories created without `-p` are no + longer world-writable `0777`. +- **Credentials and signal handling hardened.** Secret files are opened with + `O_NOFOLLOW|O_NONBLOCK` (while allowing fd-backed store paths), and signal + handlers use `sigaction` with async-signal-safe bodies. + +### Fixed + +- **`-z` on 100–256 MiB files.** The decompressor's internal ceiling was 100 MiB + while the receiver advertises and the sender compresses whole files up to + `MAX_RECEIVE_WHOLE_FILE_SIZE` (256 MiB), so `-z` on a 100–256 MiB regular file + failed with `Declared decompressed size exceeds 104857600 bytes`. The ceiling + is now defined in terms of the protocol whole-file bound (still an + allocation-clamped bomb guard). +- **`--bwlimit` now paces `--sendfile`.** The plaintext-TCP `--sendfile` fast + path bypassed the protocol's token bucket, so the limit was ignored there. It + now throttles through the same per-session leaky bucket as the TLS path. +- **`--partial-dir` implies `--partial`.** Matching rsync 3.4.1 (which sets + `keep_partial` after option parsing), `--partial-dir=DIR` alone retains an + interrupted transfer's partial and wins over an explicit `--no-partial`; + `--inplace` still bypasses the partial machinery. +- **Unsupported filter modifiers rejected.** The `x` xattr-name modifier and the + merge-only `e`/`n`/`w` modifiers are rejected with a clear error instead of + being silently ignored (`x` on merge/dir-merge rules) or folded into the + pattern (producing misleading merge-file errors). Glued patterns (`-newfile`, + `-e2e`) and mixed tokens (`H,!secret`) keep their historical parsing. +- **Miscellaneous correctness fixes:** `--filter` rule count is checked + client-side against `MAX_FILTER_RULES` before any network I/O (the receiver + still re-checks the expanded count); unknown wire `Status` values are rejected + as protocol errors; a mutex leak on an init-failure path, an `errno` read + after `free()` in deferred delete application, `log_perror` misuse for + non-`errno` conditions, and a `NULL` `server_host`/`ssh_destination` + allocation path were fixed; `SSL_read` length is clamped and `sendfile` + `poll()` retries on `EINTR`. + +### Refactored / Docs + +- Dropped dead `filter_rules_apply` and dead `--old-args` plumbing, unified + `set_error`, deduplicated `path_is_within` and shared constants, and added + printf format attributes (fixing format mismatches). `RSYNC_COMPAT.md`, + `CHANGELOG.md` and `HANDOFF.md` were updated for the audit cycle; the + `RSYNC_COMPAT.md` summary tally was corrected to match the rows. + ## [2.28.0] - 2026-09-20 The rsync-parity cycle. `PROTOCOL_VERSION` moves `2.26.0 → 2.27.0 → 2.28.0`; diff --git a/HANDOFF.md b/HANDOFF.md index 73dba0d..e2931dc 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -1,22 +1,30 @@ -# FastSync — Session Handoff (2026-09-20) +# FastSync — Session Handoff (2026-09-21) ## Current status -- **Release `v2.28.0`** is tagged and merged to `main` (PR #304, `b4d54504`). - `dev` is at `558782d` (the incremental-check flake fix). +- **Release `v2.28.0`** is tagged and merged to `main`: tag `v2.28.0` points at + `ee6523a`, and the PR #304 merge commit `b4d54504` is on `main`. +- **`dev` is at `0fbb9de`** — the merge of parity cycle 2.29 (PR #305). The old + `558782d` (incremental-check flake fix) is an ancestor. - **`PROTOCOL_VERSION` = `"2.28.0"`** (`src/shared/config.h`); CMake `project(FastFileTransfer VERSION 2.28.0)`. -- **Parity cycle 2.29 on branch `feat/parity-2.29`** (from `dev` @ `558782d`), - no wire change. It closes the scanner-order, delete-timing, relative-basis and - fuzzy-eligibility residuals and improves the `--info`/`--stats`/`--debug` - partials. Parity matrix: **120 ✅ / 10 ⚠️ / 27 ❌ = 157** (was 116/14/27). - Remaining ⚠️ rows: `--info`, `--debug`, `--msgs2stderr`, `--stats`, - `--progress`, `--delete-before`, `--compare-dest`/`--copy-dest`/`--link-dest` - (over-256-MiB basis MISS), `-y`/`--fuzzy` (256 MiB buffer cap). -- **Deferred (needs a wire bump):** the `--progress`/`--info` receiver→sender - event channel (root `./` line, ancestor suppression, `skip`/`backup` echo, - symlink/empty-dir quick-check); `--delete-before` phase-0 keep-set; and the - general >256 MiB single-file streaming limit (B4). -- Feature branch `feat/parity-2.29`; integration PR to `dev` pending. +- **Parity cycle 2.29 is merged to `dev`** (PR #305), no wire change. It closed + the scanner-order, delete-timing, relative-basis and fuzzy-eligibility + residuals and improved the `--info`/`--stats`/`--debug` partials. Parity + matrix: **120 ✅ / 10 ⚠️ / 27 ❌ = 157**. Remaining ⚠️ rows: `--info`, + `--debug`, `--msgs2stderr`, `--stats`, `--progress`, `--delete-before`, the + three basis-dir options, and `-y`/`--fuzzy`. +- **Audit cycle complete on branch `fix/audit-cycle`** (branched from `dev` @ + `0fbb9de`), integration PR to `dev` pending. No wire change + (`PROTOCOL_VERSION` stays 2.28.0). It lands the receiver/client security and + correctness fixes — `--temp-dir` symlink-escape confinement, special-bit + masking under a super-off policy, daemon `umask(022)`, the `-z` decompression + ceiling raised to the 256 MiB whole-file bound, `--bwlimit` pacing the + plaintext `--sendfile` path, `--partial-dir` implying `--partial`, rejection + of unsupported filter modifiers (`x`/`e`/`n`/`w`), client-side + `MAX_FILTER_RULES` enforcement, unknown wire `Status` rejection, and the + accompanying refactors/docs. The parity matrix is unchanged at + **120 ✅ / 10 ⚠️ / 27 ❌ = 157**; this docs pass (worktree `fix/audit-docs2`) + corrects the `RSYNC_COMPAT.md` summary tally to match the rows. ## What landed this session @@ -102,7 +110,8 @@ uptodate` plus the leading `./` root name line for `--info=name` (only the root-line trigger condition and receiver-side `skip` wording remain). Matrix now **111 ✅ / 14 ⚠️ / 32 ❌ = 157**; differential + unit tests added in - `test_features.py`, `test_option_parity.py`, `test_delete_plan.c`, + `test_features.py`, `test_option_parity.py`, the unit test + `tests/test_delete_plan.c`, `test_delete_delay_budget_parity.py`, `test_delete_timing_parity.py`. 12. **No-wire parity track 2b** on `feat/parity-2.28` (no protocol change): `--progress`/`-P`/`--info=progress` (when not `--quiet`) now run an opt-in @@ -199,17 +208,43 @@ **116 ✅ / 14 ⚠️ / 27 ❌ = 157** (the `--delete`/`--delete-during` rows stay ⚠️ for the abort boundary; `--delete-after` stays ✅). +17. **Audit cycle** on `fix/audit-cycle` (from `dev` @ `0fbb9de`; + `PROTOCOL_VERSION` stays `2.28.0`): a security/correctness pass over the + parity-2.29 baseline. It raises the decompression ceiling to the 256 MiB + protocol whole-file bound (`-z` on 100–256 MiB files now works), paces the + plaintext-TCP `--sendfile` path with `--bwlimit`, confines the `--temp-dir` + scratch dir by the fd's real path (symlink escape refused), masks + client-controlled setuid/setgid/sticky bits when super activities are not + permitted, sets the daemon umask to `022`, makes `--partial-dir` imply + `--partial`, rejects the unsupported filter modifiers (`x`/`e`/`n`/`w`), + enforces `MAX_FILTER_RULES` client-side, rejects unknown wire `Status` + values, and hardens credentials/signal handling (with the accompanying + refactors and docs). No row changes classification, so the matrix stays + **120 ✅ / 10 ⚠️ / 27 ❌ = 157**. This docs pass is on `fix/audit-docs2`. + ## Next steps -1. **Merge PR #284** (`dev` -> `main`) once reviewed (protected branch). -2. **Deferred security items** (documented, not implemented): - - Pre-auth config/daemon-auth handshake has no aggregate wall-clock deadline - (per-message timeout only) — slowloris holds connection slots. - - Per-source registry fails open when the shared table is full (per-module/global - caps and host ACLs still apply); consider fail-closed or larger/evicting table. - - SCRAM-like daemon auth has no TLS channel binding (and is not RFC 5802). - - `cleanup()` signal handler calls non-async-signal-safe teardown; daemon `umask(0)`. - - Wire protocol assumes homogeneous word size/endianness (lengths are native - `size_t`) — document or move to fixed-width framing. +1. **Open and merge the audit-cycle PR** (`fix/audit-cycle`, including this + `fix/audit-docs2` docs pass) into `dev` once reviewed. `dev` is the default + branch; all PRs target `dev`, never `main` directly. +2. **Remaining deferred items:** + - **Large structural refactors:** delete-engine consolidation + (`delete_extras_fd`/`manifest_delete_extras`/the delete-plan path), + god-function splits, and translation-unit splits. + - **`--progress`/`--info` receiver→sender event channel:** the root `./` + line, ancestor-directory suppression, receiver-side `skip`/`backup` echo, + and symlink/empty-dir quick-check feedback. + - **`--delete-before` phase-0 keep-set** (rsync fixes the file list before + the data pass; FastSync keeps its pre-scan snapshot race). + - **>256 MiB single-file streaming** (B4, the general whole-file limit). + - **Wire native-size framing:** lengths are native `size_t` and the protocol + assumes homogeneous word size/endianness — document or move to fixed-width + framing. + - **SCRAM-like daemon auth channel binding:** no TLS channel binding today + (and it is not RFC 5802). + - Still-open security nits: the pre-auth config/daemon-auth handshake has no + aggregate wall-clock deadline (per-message timeout only — slowloris holds + connection slots); the per-source registry fails open when the shared table + is full (per-module/global caps and host ACLs still apply). 3. **Out of scope / intentional:** pull (remote source) mode is **not** planned — FastSync is push-only; see `RSYNC_COMPAT.md#direction`. diff --git a/RSYNC_COMPAT.md b/RSYNC_COMPAT.md index 11aa01b..437b113 100644 --- a/RSYNC_COMPAT.md +++ b/RSYNC_COMPAT.md @@ -6,8 +6,8 @@ This document maps rsync's full feature set to FastSync's current implementation | Status | Count | Description | |--------|-------|-------------| -| ✅ Parity | 116 | Reproduces rsync's semantics for this option's scope | -| ⚠️ Caveat | 14 | Wired and tested, but carries a documented behavioral difference from rsync (named in the row and/or the wave notes) | +| ✅ Parity | 120 | Reproduces rsync's semantics for this option's scope | +| ⚠️ Caveat | 10 | Wired and tested, but carries a documented behavioral difference from rsync (named in the row and/or the wave notes) | | ❌ Divergent | 27 | Rejected, an accepted no-op, deliberately non-rsync (native config/auth/batch, privileged namespaces, safe-subset privilege), or impossible on any portable filesystem call | | **Total** | **157** | One row per rsync option/feature group; a row may name several spellings | @@ -62,9 +62,21 @@ matrix is **111 ✅ / 13 ⚠️ / 33 ❌ = 157**. - **Fuzzy eligibility.** The `-y/--fuzzy` candidate search no longer inherits the ordinary delta engine's 16 KiB minimum or 10× ratio bound, so an oversized or sub-16-KiB sibling is reused as rsync reuses it (`test_parity_basis_fuzzy.py`). - **Output partials.** `--info=mount`/`--info=stats`, the `--stats` `dir:` breakdown under `-r`, and real `--debug` output for `flist`/`del`/`hash`/`deltasum`/`recv`/`filter`/`send` were added (`test_parity_info_mount_stats.py`, `test_output_parity.py`, `test_parity_debug.py`); those rows stay ⚠️ for their remaining documented residuals. `--delete-before`'s phase-0 late-file divergence and the `--progress` root/ancestor/symlink feedback remain open (they need a receiver→sender event channel), and the >256 MiB single-file streaming limit (B4) was not addressed. The matrix is now **120 ✅ / 10 ⚠️ / 27 ❌ = 157**. +**Audit cycle (no wire change; `PROTOCOL_VERSION` stays 2.28.0).** A security-and-correctness audit pass ran against the parity-2.29 baseline; none of the fixes changes a row's classification, so the matrix stays **120 ✅ / 10 ⚠️ / 27 ❌ = 157**. The affected rows (`-z`/`--compress`, `--bwlimit`, `-T`/`--temp-dir`, `-p`/`--chmod`, `--partial-dir`, `--filter`) had their notes updated in place: + +- **Decompression ceiling.** `MAX_DECOMPRESSED_SIZE` was 100 MiB while the receiver advertises and the sender compresses whole files up to `MAX_RECEIVE_WHOLE_FILE_SIZE` (256 MiB), so `-z` on a 100–256 MiB regular file failed with `Declared decompressed size exceeds 104857600 bytes`. The ceiling is now defined in terms of the protocol whole-file bound (still a real allocation-clamped bomb guard), so the two cannot drift; `-z` on 100–256 MiB files now works. +- **`--bwlimit` with `--sendfile`.** The plaintext-TCP `--sendfile` fast path wrote through `sendfile(2)` without passing through the protocol's token bucket, so `--bwlimit` was ignored on that path. It is now paced through the same per-session leaky bucket, so TLS and plaintext transports share identical `--bwlimit` semantics. +- **`--temp-dir` confinement.** The receiver's scratch dir was opened with a bare `open()`, so a client-planted symlink under the receive root could redirect receiver scratch files outside the authorized root. The opened directory is now judged by the real path of its fd (`/proc/self/fd` via `realpath`), and an escaping target is refused (`EACCES`, logged); an in-root link to another filesystem (the `EXDEV` fallback case) still works. +- **Special-bit masking and daemon umask.** Setuid/setgid/sticky bits from the client (`--perms`, `--chmod`, symlink and special-node paths, deferred directory modes) were applied even when the connection forbade super-user activities. They are now stripped when the super policy is off (`FileAttrPolicy.super_permitted`), and exact rsync semantics are preserved when permitted. The daemon's forced `umask(0)` is now `umask(022)`, so implied parent directories are no longer world-writable `0777`. +- **`--partial-dir` implies `--partial`.** Matching rsync 3.4.1 (which sets `keep_partial` after option parsing), `--partial-dir=DIR` alone now retains an interrupted transfer's partial and wins over an explicit `--no-partial`; `--inplace` still bypasses the partial machinery. +- **Filter modifiers.** The `x` xattr-name modifier and the merge-only `e`/`n`/`w` modifiers are now rejected with a clear error instead of being silently ignored (`x` on merge/dir-merge rules) or folded into the pattern (producing misleading merge-file errors). Glued patterns (`-newfile`, `-e2e`) keep their historical parsing. +- **Bounds and wire validation.** `--filter` rule count is now checked client-side against `MAX_FILTER_RULES` (with an actionable message before any network I/O) rather than surfacing as an opaque receiver protocol error; `send_protect_entries()` still re-checks the expanded count. Unknown wire `Status` values are rejected as protocol errors (`status_is_valid()`), and the audit also fixed a mutex leak on an init-failure path, an `errno`-after-`free()` in deferred delete application, `log_perror` misuse for non-`errno` conditions, `SSL_read` length clamping, `sendfile` `poll` `EINTR` retry, and printf-format/attribute issues. + **Parity completion wave (protocol 2.23.0 → 2.26.0).** This wave closed the remaining gaps the rsync-parity wave left open (delete timing, wire counters and -output, codec breadth, general `-R`/`-d`, the full filter grammar, receiver-side +output, codec breadth, general `-R`/`-d`, the filter grammar (the unsupported +`x` xattr-name and merge-only `e`/`n`/`w` modifiers are explicitly rejected, not +silently accepted), receiver-side name resolution, absolute basis dirs, and the remaining client quick wins) and reclassified the inherently non-rsync rows as **divergent** (native daemon config/auth, the non-interoperable batch container, `--fake-super`'s xattr @@ -122,7 +134,7 @@ Every one of those has an entry below with its remaining caveats. |------|-------------------|-----------------|-------| | `--exclude-from=FILE` | Read exclude patterns from file | ✅ Parity | Reads patterns from file | | `--include-from=FILE` | Read include patterns from file | ✅ Parity | Reads patterns from file | -| `--filter=RULE` | Add file-filtering rule | ✅ Parity | The short `-f` **is** bound to `--filter` (the old FastSync sendfile conflict is gone; sendfile is long-only `--sendfile`), and `-f RULE`, `-f=RULE`, `--filter=RULE` and the two-argument form all parse. Protocol 2.26.0 implements rsync's filter grammar: `+`/`-`, `include`/`exclude`, a leading `/` anchor (to the transfer root or a `.rsync-filter` file's directory), a trailing `/` dir-only rule, and the `merge`/`.`, `dir-merge`/`:`, `hide`/`H`, `show`/`S`, `protect`/`P`, `risk`/`R` and `clear`/`!` words, including the `:`/`.` modifiers. First match wins; the filter layer is independent of `--exclude`/`--include`. **Track 4a (protocol 2.28.0) adds the receiver filter engine:** the sender compiles its root-level rules exactly as the scanner does (`filter_base_build`) and streams them as one bounded, self-describing config-frame block; the receiver reconstructs them and re-applies first-match-wins to every extraneous destination path during deletion, so a `P *.log` rule protects a destination-only `extra.log` (differential `filter_protect`/`filter_protect_during`/`filter_protect_delay` vs rsync 3.4.1, plus the `-n` would-delete enumeration) — matching rsync's dual-sided engine for the command-line rule set. **Remaining residual:** per-directory merge (`:`/`.`, and therefore `-F`) is not yet re-derived on the receiver; a destination-only entry that matches ONLY a per-directory merge rule is still protected only through the sender-derived source-mirror prefixes, not by the received base rule list | +| `--filter=RULE` | Add file-filtering rule | ✅ Parity | The short `-f` **is** bound to `--filter` (the old FastSync sendfile conflict is gone; sendfile is long-only `--sendfile`), and `-f RULE`, `-f=RULE`, `--filter=RULE` and the two-argument form all parse. Protocol 2.26.0 implements rsync's filter grammar: `+`/`-`, `include`/`exclude`, a leading `/` anchor (to the transfer root or a `.rsync-filter` file's directory), a trailing `/` dir-only rule, and the `merge`/`.`, `dir-merge`/`:`, `hide`/`H`, `show`/`S`, `protect`/`P`, `risk`/`R` and `clear`/`!` words, including the `:`/`.` modifiers. The xattr-name `x` modifier and the merge-only `e`/`n`/`w` modifiers are **explicitly rejected with a clear error** (audit-cycle fix: the list parser used by `--filter`/`-f` previously silently ignored `x` on merge/dir-merge rules and folded `e`/`n`/`w` into the pattern, producing misleading failures); a token made up solely of modifier characters that names an unsupported modifier is rejected, while glued patterns (`-newfile`, `-e2e`) and mixed tokens (`H,!secret`) keep their historical parsing. First match wins; the filter layer is independent of `--exclude`/`--include`. **Track 4a (protocol 2.28.0) adds the receiver filter engine:** the sender compiles its root-level rules exactly as the scanner does (`filter_base_build`) and streams them as one bounded, self-describing config-frame block; the receiver reconstructs them and re-applies first-match-wins to every extraneous destination path during deletion, so a `P *.log` rule protects a destination-only `extra.log` (differential `filter_protect`/`filter_protect_during`/`filter_protect_delay` vs rsync 3.4.1, plus the `-n` would-delete enumeration) — matching rsync's dual-sided engine for the command-line rule set. **Remaining residual:** per-directory merge (`:`/`.`, and therefore `-F`) is not yet re-derived on the receiver; a destination-only entry that matches ONLY a per-directory merge rule is still protected only through the sender-derived source-mirror prefixes, not by the received base rule list | | `--files-from=FILE` | Read source file list from file | ✅ Parity | Entries are paths relative to the source root (leading `./` stripped, `..`/absolute rejected at parse time, blank lines ignored; NUL-delimited with `-0`). A listed regular file is transferred; a listed directory transfers its whole subtree (FastSync recursion is always on). Non-listed paths are pruned by the scanner; the delete manifest is scoped to the listed directory subtrees. A listed entry that does not exist is a hard error unless `--ignore-missing-args`/`--delete-missing-args` is given. **An empty list is a zero-transfer success (exit 0), matching rsync 3.4.1** — the earlier claim that rsync reports "no source files specified" was wrong. Scalability note: `file_list_affects` is O(list size) per scanned entry, so a very large list against a huge tree is quadratic (the documented bound) | | `-0`, `--from0` | Delimit *-from files with NULs | ✅ Parity | `--files-from` entries become NUL-delimited; the flag may appear before or after `--files-from` on the command line. NUL mode preserves entry bytes exactly (trailing CR/LF are part of the name; only newline mode trims them) | | `--max-size=SIZE` | Skip files larger than SIZE | ✅ Parity | `max_size` in scanner | @@ -168,9 +180,9 @@ Every one of those has an entry below with its remaining caveats. | `--backup-dir=DIR` | Backup directory hierarchy | ✅ Parity | `backup_dir` config field | | `--suffix=SUFFIX` | Backup suffix (default ~) | ✅ Parity | `suffix` config field | | `--delay-updates` | Put updated files in place at end | ❌ Divergent | Successfully received files are staged under a private 0700 `.fastsync-stage` dir inside the receive root and atomically renamed into their final destinations only after the whole transfer (manifest/delete handling included) succeeds, just before the success/outcome frame is sent. The delete walker deliberately skips the staging dir at the receive root, so `--delete` removes genuine extras but never the staged files (deletion runs before publication; rsync's delete-after ordering is not implemented). `--existing`/`--ignore-existing`/`--update` decide against the final destination path at stage time; `--backup` moves the old file aside at publication, and **`--force` is honored at publication** (protocol 2.23.0): a staged regular file or symlink may replace a destination directory that blocks it. Incompatible with `--inplace` and with `--backup-dir=.fastsync-stage` (the internal staging name is reserved; both are rejected). The staging dir name is fixed, so two simultaneous delayed transfers to the same destination root are serialized with an exclusive advisory lock held for the whole transfer: the second session fails cleanly instead of corrupting the first. Aborting or failing before publication installs nothing and removes the staging tree; a crash between stage and publish leaves staged leftovers that the next delayed run wipes at start (process death releases the lock). A stage→publish failure aborts the transfer (best-effort cleanup of the not-yet-published staged files; already-published files are not rolled back). **Reclassified Divergent (differential evidence):** the staging name is fixed and a delayed run wipes a pre-existing destination tree of that name at start even without `--delete`, whereas rsync uses its own internal temp name and leaves a genuine destination entry named `.fastsync-stage` untouched (`test_delay_updates_staging_name_collision_residual`); deletion also runs before publication while rsync's `--delay-updates` implies `--delete-after`. Works in single-threaded and `-j`/`--threads` modes | -| `-T`, `--temp-dir=DIR` | Create temporary files in DIR | ❌ Divergent | `--temp-dir` with the rsync short `-T` (the timeout alias moved to long-only `--timeout`). A **relative** dir matches rsync exactly: it is resolved below the receive/destination root and must already exist (differentially verified: `rsync -a --temp-dir=scratch src/ dst/` and FastSync produce identical trees and an empty scratch dir). **Reclassified as a deliberate divergence because an absolute `--temp-dir` is rejected by the receiver** — it is resolved verbatim by rsync standalone (which will use `/tmp` or any other absolute directory, including one outside the destination), but FastSync's security-reviewed receiver confines the scratch dir to the authorized receive root and rejects any absolute path or one containing `..`. A differential test confirms rsync exits 0 using an absolute scratch dir while FastSync refuses before writing anything into it (the scratch dir stays empty). Its daemon mode also confines relative to the module, but standalone rsync's absolute-temp-dir behavior is not reproduced because it would let a client place receiver scratch files outside the sandbox. Temp copies use a unique name in the scratch dir and are atomically renamed into place; **on `EXDEV` (scratch dir and destination on different filesystems, reachable via a confined relative symlink) the receiver falls back to a non-atomic copy instead of aborting**, matching rsync. `--inplace` and `--partial-dir` writes bypass the scratch dir | +| `-T`, `--temp-dir=DIR` | Create temporary files in DIR | ❌ Divergent | `--temp-dir` with the rsync short `-T` (the timeout alias moved to long-only `--timeout`). A **relative** dir matches rsync exactly: it is resolved below the receive/destination root and must already exist (differentially verified: `rsync -a --temp-dir=scratch src/ dst/` and FastSync produce identical trees and an empty scratch dir). **Reclassified as a deliberate divergence because an absolute `--temp-dir` is rejected by the receiver** — it is resolved verbatim by rsync standalone (which will use `/tmp` or any other absolute directory, including one outside the destination), but FastSync's security-reviewed receiver confines the scratch dir to the authorized receive root and rejects any absolute path or one containing `..`. **Audit-cycle hardening:** the opened dir is additionally judged by the real path of its fd (`/proc/self/fd`), so a client-planted symlink under the receive root cannot redirect receiver scratch files outside the authorized root (an escaping target is refused with `EACCES`), while an in-root symlink to another filesystem — the `EXDEV` fallback case — still works. A differential test confirms rsync exits 0 using an absolute scratch dir while FastSync refuses before writing anything into it (the scratch dir stays empty). Its daemon mode also confines relative to the module, but standalone rsync's absolute-temp-dir behavior is not reproduced because it would let a client place receiver scratch files outside the sandbox. Temp copies use a unique name in the scratch dir and are atomically renamed into place; **on `EXDEV` (scratch dir and destination on different filesystems, reachable via a confined relative symlink) the receiver falls back to a non-atomic copy instead of aborting**, matching rsync. `--inplace` and `--partial-dir` writes bypass the scratch dir | | `--partial` | Keep partially transferred files | ✅ Parity | On a failed/interrupted write the already-written temp file is retained at the destination path (best-effort rename instead of unlink) so a later `--append`/`--append-verify` run can resume it. Retention never runs when no data was actually written or under `--ignore-existing`/`--existing` (the destination is not ours to overwrite), and it only ever renames the already-written temp. A failed rename falls back to the normal unlink | -| `--partial-dir=DIR` | Keep partial files in DIR | ✅ Parity | With `--partial`, the working file is written under the confined partial directory (a relative dir below the receive root) and atomically renamed into place once complete, so an interrupted transfer leaves a resumable copy there and completed transfers do not linger under it. `--inplace` bypasses the partial dir (rsync parity). Requires `--partial` | +| `--partial-dir=DIR` | Keep partial files in DIR | ✅ Parity | The working file is written under the confined partial directory (a relative dir below the receive root) and atomically renamed into place once complete, so an interrupted transfer leaves a resumable copy there and completed transfers do not linger under it. `--inplace` bypasses the partial dir (rsync parity). **Implies `--partial`** (audit-cycle fix, matching rsync 3.4.1, which sets `keep_partial` after option parsing): `--partial-dir=DIR` alone retains an interrupted transfer's partial, and the implication wins over an explicit `--no-partial` regardless of order. `--inplace` is the exception — it writes the destination in place with no partial staging, so the implication is skipped | ## 7. Deletion @@ -316,12 +328,12 @@ why plain `--append` works on the normal atomic path, not only with `--inplace`. | Flag | Rsync Description | FastSync Status | Notes | |------|-------------------|-----------------|-------| | `--preserve` | (FastSync alias, not an rsync flag) | ✅ Parity | **FastSync-only alias** for `-p` + `-t` (mode + mtime), long-form only. It is not rsync's `--preserve` (rsync has no such option); the short `-M` that used to spell it is now rsync's `--remote-option`. The wire metadata also carries uid/gid for `-o`/`-g`/`-a`, and ownership is applied via `-o`/`-g`, `-a`, or an explicit identity flag (`--numeric-ids`/`--usermap`/`--groupmap`/`--chown`/`--copy-as`) | -| `-p`, `--perms` | Preserve permissions | ✅ Parity | Real per-attribute flag (protocol 2.22.0): `preserve_perms` applies the source mode independently of times/owner/group. **Strict rsync parity (protocol 2.23.0): the source mode is copied exactly, including setuid/setgid/sticky and group/other-write bits — there is no masking.** Without `-p`, a new file gets `source_mode & ~umask` when metadata is present (else the historical fixed `0644`); new directories without `-p` still use FastSync's `0755` creation default, because directory metadata is only applied when a directory attribute is requested. `-A/--acls` implies `-p`; `--chmod` does **not** imply `-p` (rsync parity) and applies its own unsanitized changes to the new mode. `-X/--xattrs` does not imply `-p`. The SSH port moved to `--ssh-port`. rsync-parity short form | +| `-p`, `--perms` | Preserve permissions | ✅ Parity | Real per-attribute flag (protocol 2.22.0): `preserve_perms` applies the source mode independently of times/owner/group. **Strict rsync parity when super-user activities are permitted (protocol 2.23.0): the source mode is copied exactly, including setuid/setgid/sticky and group/other-write bits.** **Audit-cycle fix:** when the connection forbids super-user activities (`--no-super`, a non-opted daemon module, or a privileged standalone listener without `--allow-super`), the setuid/setgid/sticky bits are masked from the applied mode (the other bits are unaffected); exact rsync semantics are preserved wherever super activities are permitted. Without `-p`, a new file gets `source_mode & ~umask` when metadata is present (else the historical fixed `0644`); new directories without `-p` still use FastSync's `0755` creation default, because directory metadata is only applied when a directory attribute is requested. **Audit-cycle fix:** the daemon no longer forces `umask(0)` (which made implied parent directories world-writable `0777`); it uses the conventional `022`, and `-p`/`-a` still restore the exact source mode via `fchmod`. `-A/--acls` implies `-p`; `--chmod` does **not** imply `-p` (rsync parity) and applies its own unsanitized changes to the new mode. `-X/--xattrs` does not imply `-p`. The SSH port moved to `--ssh-port`. rsync-parity short form | | `-o`, `--owner` | Preserve owner | ✅ Parity | Real per-attribute flag (`preserve_owner`): preserve the source uid, resolved on the receiver by name against its own user database with a raw-numeric fallback (only numeric ids cross the wire). `--usermap`/`--chown=USER` imply it. Application follows the `--super`/`--no-super` policy; a non-opted daemon module applies no ownership (see the Daemon Mode notes) | | `-g`, `--group` | Preserve group | ✅ Parity | Real per-attribute flag (`preserve_group`): preserve the source gid, resolved by name on the receiver with a raw-numeric fallback. `--groupmap`/`--chown=:GROUP` imply it. Same privilege/super-policy gating as `-o` | | `-t`, `--times` | Preserve modification times | ✅ Parity | Real per-attribute flag (`preserve_times`): apply the source mtime independently of the other attributes. `-O/--omit-dir-times` suppresses directories only and `-J/--omit-link-times` suppresses symlinks only; `-U`/`-N` do not imply it. `--preserve`/`-a` imply it, and `--incremental`/`--delta` auto-enable it unless `--no-times`/`--no-preserve` | | `-E`, `--executability` | Preserve executability | ✅ Parity | Preserves executable permission bits (implies metadata preservation) | -| `--chmod=CHMOD` | Affect file permissions | ✅ Parity | Faithful port of rsync 3.4.1's `parse_chmod`/`tweak_mode`: numeric octal and symbolic `ugo`/`rwx` changes, `D`/`F` directory/file selectors, `X` (execute only on directories or already-executable files), `s`/`t` setuid/setgid/sticky, and append semantics — repeated clauses and repeated `--chmod` options accumulate in order (joined with commas). The changes are applied to the new mode **without sanitization** (matching rsync) and `--chmod` does **not** imply `-p` (rsync parity). Applied to files and directories on the receiver | +| `--chmod=CHMOD` | Affect file permissions | ✅ Parity | Faithful port of rsync 3.4.1's `parse_chmod`/`tweak_mode`: numeric octal and symbolic `ugo`/`rwx` changes, `D`/`F` directory/file selectors, `X` (execute only on directories or already-executable files), `s`/`t` setuid/setgid/sticky, and append semantics — repeated clauses and repeated `--chmod` options accumulate in order (joined with commas). The changes are applied to the new mode **without sanitization** (matching rsync), except that setuid/setgid/sticky are masked when the connection forbids super-user activities (audit-cycle fix, see `-p`), and `--chmod` does **not** imply `-p` (rsync parity). Applied to files and directories on the receiver | | `-A`, `--acls` | Preserve ACLs | ✅ Parity | Implemented on Linux via the POSIX-ACL xattr representation: the sender captures the `system.posix_acl_access` / `system.posix_acl_default` xattrs and the receiver re-applies them fd-relative. A differential test with `setfacl` confirms the complete access and default ACL sets (including `mask`) are identical to rsync's on a directory. libacl is not required; a `fsetxattr` an unprivileged receiver may not perform is logged and skipped, never fatal. Only the `system.posix_acl_*` namespaces plus `user.*` are ever applied; privileged namespaces are never applied. Implies metadata transmission | | `-X`, `--xattrs` | Preserve extended attributes | ❌ Divergent | Deliberately restricted to unprivileged `user.*` extended attributes plus the two POSIX ACL xattrs; `security.*` (SELinux, capabilities, ...) and `trusted.*` are **never** captured or applied — a client can never force a privileged attribute onto the destination, and the receiver independently re-validates every incoming name against the whitelist. This is a security-policy divergence from rsync, which can preserve the privileged namespaces with the needed privilege; implementing them would defeat FastSync's privilege-escalation guard. `user.*` capture/apply matches rsync in a differential test. Payloads are bounded on both ends. Incompatible with `-s` | | `-H`, `--hard-links` | Preserve hard links | ✅ Parity | Files on the source that share an inode (`st_dev`+`st_ino`, e.g. a `cp -al` tree) are re-created as hard links to one another on the destination, so duplicate links stay deduplicated and only the first member's data is sent (later members are transmitted as payload-less `STATUS_HARDLINK` frames). The receiver links each sibling to the first member's installed file with an atomic link + rename; on `link()` failure it falls back to a byte-identical local copy of the first member, never a partial/corrupt file. Requires the sequential scan for ordering (the first member is always emitted and installed before any sibling is linked). Works single-threaded and under `-j`/`--threads`, `--inplace`, `--delay-updates` (links staged and published by rename) and `--partial`. Crosses the wire (`preserve_hard_links` bool; `PROTOCOL_VERSION` bumped **2.11.0 → 2.12.0**, peers must match). Incompatible with `-s` (chunk serialization) and `--append`/`--append-verify`, rejected up front with a distinct error. See the Phase-4 hard-links notes below | @@ -683,7 +695,7 @@ targets verbatim, matching rsync. | Flag | Rsync Description | FastSync Status | Notes | |------|-------------------|-----------------|-------| -| `-z`, `--compress` | Compress file data | ✅ Parity | Streaming compression. **Protocol 2.26.0 implements rsync 3.4.1's codec set** (`zstd` default, `lz4`, `zlib`, `zlibx`, `none`), selectable via `--compress-choice`/`--zc` and negotiated with `auto`. `-z` is the compression short form; `-c` is rsync's `--checksum`. `--skip-compress` applies rsync 3.4.1's default suffix list when no list is given. **Track 3a closes the codec caveats:** `zlibx` is no longer a divergence — FastSync's zlib stream already carries only the delta/token (literal) bytes, which is exactly rsync's zlibx semantics, so `--zc=zlib` and `--zc=zlibx` land the same tree/stdout/exit (differential `test_compress_codec_matches_rsync_bytes`) and the zlib/zlibx aliasing is only an implementation detail. Each codec now uses rsync's own default `--compress-level` (zstd 3, zlib/zlibx 6, lz4 ignored) and `auto` consults `RSYNC_COMPRESS_LIST` before the compiled-in order; the deterministic same-build resolution needs no peer probe | +| `-z`, `--compress` | Compress file data | ✅ Parity | Streaming compression. **Protocol 2.26.0 implements rsync 3.4.1's codec set** (`zstd` default, `lz4`, `zlib`, `zlibx`, `none`), selectable via `--compress-choice`/`--zc` and negotiated with `auto`. `-z` is the compression short form; `-c` is rsync's `--checksum`. `--skip-compress` applies rsync 3.4.1's default suffix list when no list is given. **Track 3a closes the codec caveats:** `zlibx` is no longer a divergence — FastSync's zlib stream already carries only the delta/token (literal) bytes, which is exactly rsync's zlibx semantics, so `--zc=zlib` and `--zc=zlibx` land the same tree/stdout/exit (differential `test_compress_codec_matches_rsync_bytes`) and the zlib/zlibx aliasing is only an implementation detail. Each codec now uses rsync's own default `--compress-level` (zstd 3, zlib/zlibx 6, lz4 ignored) and `auto` consults `RSYNC_COMPRESS_LIST` before the compiled-in order; the deterministic same-build resolution needs no peer probe. **Audit-cycle fix:** the decompressor's internal ceiling is now defined by the protocol whole-file bound (`MAX_RECEIVE_WHOLE_FILE_SIZE`, 256 MiB) instead of a separate 100 MiB constant, so `-z` on a 100–256 MiB regular file no longer fails with `Declared decompressed size exceeds 104857600 bytes` | | `--compress-choice=STR`, `--zc=STR` | Choose compression algorithm | ✅ Parity | Protocol 2.26.0 accepts rsync 3.4.1's compiled-in choices — `zstd` (default), `lz4`, `zlib`, `zlibx`, `none`, `auto` — and rejects an unknown name with exit 4 like rsync. The negotiated codec id crosses the wire (`compression_algo`), so the receiver decodes with the sender's codec. `--zc` is the alias. `auto` now resolves through `RSYNC_COMPRESS_LIST` (whitespace-separated; unknown names skipped, first supported wins, all-unknown is exit 4) and then the compiled-in order, and an explicit `--zc` wins; the deterministic same-build resolution needs no peer probe. `zlib`/`zlibx` share FastSync's literal-only zlib path, which is rsync's zlibx behavior and is observably identical for both, so `zlibx` is not a divergence (the aliasing is an implementation detail) | | `--compress-level=NUM`, `--zl=NUM` | Set compression level | ✅ Parity | Accepted range 1-22. When omitted, rsync 3.4.1's **per-codec default** applies: zstd 3 (`ZSTD_CLEVEL_DEFAULT`), zlib/zlibx 6 (`Z_DEFAULT_COMPRESSION` resolved), lz4 ignored (no tunable level; FastSync keeps a positive gate value and `lz4_compress` ignores it, so the bytes match rsync). An explicit level is clamped per codec like rsync's `init_compression_level()`: zstd 1-22, zlib/zlibx 1-9, lz4 ignored. Verified against `rsync --debug=NSTR1`, which reports the same effective level per codec | | `--compress-threads=NUM` | Set compression threads | ✅ Parity | `compression_threads` config field (client-only; does not cross the wire). Sets the number of worker threads used by the zstd compression pool to NUM (1..64; 0/garbage/oversized rejected up front). Accepted in both `--compress-threads=NUM` and two-argument `--compress-threads NUM` forms. Composes with `-z`/compression; under the `-j`/`--threads` multithreaded pipeline it parallelizes compressed chunk encoding. See test_tcp.py `-z --compress-threads=2` and test_client_cli.c | @@ -704,7 +716,7 @@ targets verbatim, matching rsync. | `-4`, `--ipv4` | Prefer IPv4 | ✅ Parity | Forces `AF_INET` in the `getaddrinfo` hints for client destination/source resolution and the server bind (see the Phase 5, Wave B note). Mutually exclusive with `-6` | | `-6`, `--ipv6` | Prefer IPv6 | ✅ Parity | Forces `AF_INET6` in the `getaddrinfo` hints for client destination/source resolution and the server bind. Mutually exclusive with `-4` | | `--remote-option=OPT`, `-M` | Send an option only to the remote side | ❌ Divergent | Each value is appended to the remote server invocation over SSH as an individually single-quote-escaped shell word in `ssh_build_remote_command()`. Values are validated (non-empty, no control characters) and shell metacharacters cannot break out of the quoting (`;`, `&`, `\|`, `, `$`, `(`, `)`, quotes are neutralized), so a value cannot inject an arbitrary remote command and a subsequent `--` on the client line cannot be turned into one. The short `-M` form (`-M OPT`, `-M=OPT`, and rsync-style attached `-MOPT`) is available, matching rsync; metadata mode moved to long-only `--preserve`. **Reclassified because the daemon/TCP case cannot be reproduced:** `-M` is only meaningful for the SSH transport (`user@host:path`); a daemon (`host::module/path`) or local TCP destination **rejects** it, whereas rsync forwards it to its own remote process on every transport. A differential test starts a real rsync daemon and shows `-M--totally-bogus` reaching the remote parser (`unknown option`) while a valid `-M--safe-links` is accepted. FastSync's daemon handshake is a fixed binary config frame with no per-connection argv channel; adding one would let a client set arbitrary server-side options (the same class of divergence as the native daemon config/auth), so the safe subset stays SSH-only | -| `--bwlimit=RATE` | Limit I/O bandwidth | ✅ Parity | A faithful port of rsync 3.4.1's `parse_size_arg(bwlimit_arg, 'K', "bwlimit", 512, -1, True)`: a bare value is KiB/s, `K`/`M`/`G`/`T`/`P` are binary suffixes, `KB`/`MB` are decimal, `KiB`/`MiB` are binary, decimals are accepted and quantized to whole KiB exactly like rsync's `(size + 512) / 1024`, `0` (or an empty value) means "no limit", and any other value below the 512-byte floor is rejected. The token bucket's burst capacity is ~100 ms of bandwidth, matching the point at which rsync's leaky bucket starts sleeping, so a throttled transfer paces like rsync (4 MiB at `--bwlimit=1024`/`2048` matches rsync within ~4%). Differential-tested: the accept/reject matrix and the wall-clock rate both match rsync 3.4.1. The limit is a local I/O concern and is not negotiated on the wire | +| `--bwlimit=RATE` | Limit I/O bandwidth | ✅ Parity | A faithful port of rsync 3.4.1's `parse_size_arg(bwlimit_arg, 'K', "bwlimit", 512, -1, True)`: a bare value is KiB/s, `K`/`M`/`G`/`T`/`P` are binary suffixes, `KB`/`MB` are decimal, `KiB`/`MiB` are binary, decimals are accepted and quantized to whole KiB exactly like rsync's `(size + 512) / 1024`, `0` (or an empty value) means "no limit", and any other value below the 512-byte floor is rejected. The token bucket's burst capacity is ~100 ms of bandwidth, matching the point at which rsync's leaky bucket starts sleeping, so a throttled transfer paces like rsync (4 MiB at `--bwlimit=1024`/`2048` matches rsync within ~4%). Differential-tested: the accept/reject matrix and the wall-clock rate both match rsync 3.4.1. **Audit-cycle fix:** the plaintext-TCP `--sendfile` fast path now passes its writes through the same token bucket, so `--bwlimit` also paces it (previously the `sendfile(2)` path bypassed the limiter entirely); the TLS and plaintext transports therefore share identical throttling. The limit is a local I/O concern and is not negotiated on the wire | ## 14. Daemon Mode @@ -740,7 +752,7 @@ targets verbatim, matching rsync. | Flag | Rsync Description | FastSync Status | Notes | |------|-------------------|-----------------|-------| | Path escape detection | Ensure files stay within root | ✅ Parity | `has_path_traversal()` + realpath | -| Symlink-safe delete | Skip symlinks in delete walk | ✅ Parity | `delete_extras_walk()` | +| Symlink-safe delete | Skip symlinks in delete walk | ✅ Parity | `delete_extras_fd()` (`src/shared/utils.c`) and `manifest_delete_extras()` (`src/shared/file_receive.c`) | | Protocol version check | Verify compatible versions | ✅ Parity | `config_receive()` | | Max data/string/chunk sizes | Prevent OOM attacks | ✅ Parity | Per-message limits | | Per-connection memory limit | Cap memory per connection | ✅ Parity | `MAX_CONNECTION_MEMORY` is **256 MiB per connection** (256 * 1024 * 1024 bytes), charged across protocol reservations and decompression/chunk allocations. This is a FastSync-internal bound with no direct rsync analogue | @@ -1153,7 +1165,10 @@ wire protocol three times (full rationale in `src/shared/config.h`): - **Filter grammar:** `merge`/`.`, `dir-merge`/`:`, `hide`/`H`, `show`/`S`, `protect`/`P`, `risk`/`R`, `clear`/`!`, include/exclude and the `:`/`.` modifiers; `-f` is bound to `--filter`; a single `-F` transfers - `.rsync-filter` and `-FF` excludes it. + `.rsync-filter` and `-FF` excludes it. The xattr-name `x` modifier and the + merge-only `e`/`n`/`w` modifiers are **not implemented** and are rejected with + a clear error (audit cycle) instead of being silently ignored or folded into + the pattern. - **Absolute basis directories** are used verbatim (rsync semantics) and **`--link-dest`** relinks an already up-to-date destination.