FastSync/HANDOFF.md

# FastSync — Session Handoff (2026-09-21)

## Current status
- **Release `v2.28.0`** is tagged and merged to `main`: tag `v2.28.0` points at
  `ee6523a`, and the PR #304 merge commit `b4d54504` is on `main`.
- **`dev` is at `0fbb9de`** — the merge of parity cycle 2.29 (PR #305). The old
  `558782d` (incremental-check flake fix) is an ancestor.
- **`PROTOCOL_VERSION` = `"2.28.0"`** (`src/shared/config.h`); CMake
  `project(FastFileTransfer VERSION 2.28.0)`.
- **Parity cycle 2.29 is merged to `dev`** (PR #305), no wire change. It closed
  the scanner-order, delete-timing, relative-basis and fuzzy-eligibility
  residuals and improved the `--info`/`--stats`/`--debug` partials. Parity
  matrix: **120 ✅ / 10 ⚠️ / 27 ❌ = 157**. Remaining ⚠️ rows: `--info`,
  `--debug`, `--msgs2stderr`, `--stats`, `--progress`, `--delete-before`, the
  three basis-dir options, and `-y`/`--fuzzy`.
- **Audit cycle complete on branch `fix/audit-cycle`** (branched from `dev` @
  `0fbb9de`), integration PR to `dev` pending. No wire change
  (`PROTOCOL_VERSION` stays 2.28.0). It lands the receiver/client security and
  correctness fixes — `--temp-dir` symlink-escape confinement, special-bit
  masking under a super-off policy, daemon `umask(022)`, the `-z` decompression
  ceiling raised to the 256 MiB whole-file bound, `--bwlimit` pacing the
  plaintext `--sendfile` path, `--partial-dir` implying `--partial`, rejection
  of unsupported filter modifiers (`x`/`e`/`n`/`w`), client-side
  `MAX_FILTER_RULES` enforcement, unknown wire `Status` rejection, and the
  accompanying refactors/docs. The parity matrix is unchanged at
  **120 ✅ / 10 ⚠️ / 27 ❌ = 157**; this docs pass (worktree `fix/audit-docs2`)
  corrects the `RSYNC_COMPAT.md` summary tally to match the rows.


## What landed this session
1. **Wave 8 (refactors):** Config X-macro wire table; single-owner `authorized_root`;
   daemon per-module/per-host caps + cross-process auth lockout (`daemon_limits.[ch]`);
   `Data` charge returns to its owning `ProtocolSession`.
2. **Wave 9 (protocol 2.21.0):** optional `STATUS_ERROR_DETAIL` rejection reasons;
   server-contacting `--dry-run` (`STATUS_DRY_RUN_TRANSFER`, receiver mutates nothing).
3. **Security wave:** ran 5 parallel audits (wire parsing; daemon/transport/TLS/auth;
   receiver confinement; client/CLI/SSH; crypto/memory/limits). Fixed all HIGH and the
   confirmed MEDIUMs:
   - SSH `-o ProxyCommand=…` argument injection (RCE) — reject leading `-`, insert `--`.
   - Truncated zstd frame infinite CPU loop (remote DoS).
   - FIFO receiver opens lacked `O_NONBLOCK` (indefinite hang).
   - `--inplace` could write a FIFO/device (bypass of `--write-devices` gate).
   - `--force` not gated by server `--allow-delete`.
   - Privileged standalone server defaulted super activities on; added `--allow-super`
     (never honored with `--stdio`).
   - `--dry-run` content/hash oracle on `read only`/basis files removed.
   - Empty `hosts allow`/`deny`/`auth users` now rejected.
   - TLS: AEAD-only 1.2 + server preference, TOCTOU-safe key load, IP-SAN verify,
     CN-truncation guard. Glob backtracking bounded; line reads bounded; ACL xattrs
     gated on `--acls`; decompression/chunk memory charged; pre-auth `basis_count`
     NULL-deref fixed.
4. **Tooling:** benchmark accuracy (data mix, verification, percentiles, `tc`,
   `build-bench/`, `--warm` mode); `shell.nix` full toolchain and no build-on-entry;
   docs state push-only / remote-source unsupported.
5. **Preserve-attribute split (protocol 2.22.0)** landed on `feat/preserve-attr-split`: per-attribute `-p/-t/-o/-g` + `--no-*` negations, `-a` = `-rlptgoD`, and the 2.21.0 → 2.22.0 wire bump.
6. **Rsync-parity wave (protocol 2.23.0)** on `feat/rsync-parity`: rsync short options/clustering/attached values (`-r`/`-b`/`-L`/`-B`, `-av`, `-aAX`, `-B1000`, `-essh`, `-MOPT`), `-c` checksum quick-check, `--checksum-choice`/`--compress-choice` validation and seed randomization, rsync timeout/max-alloc defaults, temp-dir confinement + `EXDEV` fallback, ownership/mapping parity (numeric-ids modifier, map ranges/`*`/empty-FROM, `--chown`+map conflicts, fake-super resolved-owner record), verbatim symlink storage with rsync `--safe-links`/`--munge-links`, socket recreation under `--specials`, `--chmod` 3.4.1 semantics, and delete scoping + `--max-delete` partial/exit-25. Wire: appended delete-manifest synchronized-directory section and `STATUS_DELETE_LIMIT`.
7. **Parity-completion wave (protocol 2.24.0 → 2.26.0)** on `feat/parity-completion`: per-directory delete plans (`STATUS_DELETE_PLAN`) for `--delete-during`/`--delete-delay`; receiver `STATUS_STATS` counters feeding `--stats`/`--progress` and `--out-format %b/%c/%C`, plus `-n --delete` lines; `lz4`/`zlib`/`zlibx` compression and `md4`/`sha1`/`none` checksums with `auto` negotiation (default `xxh128`/`zstd`); general `-R`/`--no-implied-dirs`/`-d`; the full filter grammar (`merge`/`dir-merge`/`hide`/`show`/`protect`/`risk`/`clear` + modifiers) and corrected `-F`/`-FF`; receiver-side `--chown`/map TO-name resolution; absolute basis dirs + `--link-dest` relink; receiver-side `--ignore-existing` short-circuit; `--preallocate` over `--sparse` via `fallocate(2)`; `--iconv=.`/`-`/`--no-iconv`; lone `-h` help; aliases `--ignore-non-existing`/`--protect-args`/`--msgs2stderr`; and the full `--info`/`--debug` vocabulary. `RSYNC_COMPAT.md` reclassifies the matrix to 106 ✅ / 27 ⚠️ / 23 ❌; the later rsync-parity-stats pass (`fix/parity-stats`) moves it to 107 ✅ / 25 ⚠️ / 24 ❌ (see item 8).
8. **rsync-parity-stats pass** on `fix/parity-stats` (no wire change, `PROTOCOL_VERSION` stays `2.26.0`): `--delete-delay` now reports only entries it actually removes, while the `--max-delete` budget is charged at plan/snapshot time (`planned`, via `defer_add`) to bound the deferred list (a refilled deferred directory that survives `ENOTEMPTY` is not reported but still consumes budget); `--stats` gained the `(reg/dir/link/special)` `Number of files` breakdown and now counts only regular files actually stored for `Number of regular files transferred`/transferred size/literal data (up-to-date re-runs report 0); `Total file size` includes symlink target lengths; `--progress` prints the leading `./` root line and counts it in `to-chk` so a single-file transfer matches rsync; and `%C` uses the selected transfer checksum with `checksum_digest_file` supporting md4/sha1/none, byte-identical to rsync for every algorithm. `--out-format` reclassified ❌ (`%b`/delta-`%c` are protocol-specific). Differential + regression tests added; full suite + ASan + clang-format + cppcheck clean.
9. **Option-parity wave (protocol 2.26.0 → 2.27.0, on `fix/parity-options`):**
   `--bwlimit` now ports rsync 3.4.1's units/quantization and paces like its
   leaky bucket; `--ignore-errors` reproduces rsync's default (an I/O error
   skips deletion unless the flag is set; the readable tree still transfers and
   the run exits 23) across every delete timing; the `--info` categories with a
   FastSync event (`name`/`flist`/`del`/`remove`/`nonreg`/`progress`) emit
   rsync's line format, with real-run `deleting`/`*deleting` lines carried over
   the new trailing config bool `report_deletes` (golden wire updated by
   `tests/test_config.c`). Two residuals were reclassified **divergent**: `-M`
   over daemon/TCP (no argv channel in FastSync's binary config handshake;
   rsync-daemon differential pins the rsync behavior) and receiver-side
    `protect`/`risk` re-derivation for destination-only entries (would need a
    receiver filter engine; differential pins the divergence — **reversed by
    track 4a below**, which adds that engine). The options pass
    stands at **110 ✅ / 21 ⚠️ / 26 ❌**. New `tests/integration/test_option_parity.py`
   holds the rsync differentials (bwlimit parse+rate, info lines, real-setpriv
   `--ignore-errors`, rsync-daemon `-M`, filter-protect pin).

10. **rsync-parity-fs pass** on `fix/parity-fs` (no wire change of its own; integrated
   on top of the 2.27.0 options wave): recursive transfers now recreate empty source directories (and
   `-m/--prune-empty-dirs` still suppresses them), a directory entry replaces a
   blocking destination regular file, and `-R --no-implied-dirs --files-from`
   places a listed file under a missing implied parent with default attributes
   instead of refusing (real rsync 3.4.1 parity, differential-tested). `--iconv`
   now reproduces rsync's push direction (destination charset = the spec's REMOTE
   half; a server `--iconv` overrides), and `-T/--temp-dir` relative semantics are
   confirmed identical while the absolute-path confinement is a deliberate
   divergence. The basis-dir options, `--delay-updates` and `--dry-run` were
   reclassified to ❌ after a differential test reproduced each exact residual
   (basis content verification, fixed staging-name collision, and dry-run
   would-delete over-report). `--fuzzy` was also reclassified to ❌ (deterministic
   heuristic with a 10× size window, not rsync's matcher), but its residual is the
   candidate-selection heuristic itself: the final tree is byte-exact by design, so
   it is pinned by the `TestFuzzy` threshold suite rather than a byte-level rsync
   differential. (Track 5b later found the name heuristic is rsync's own and moved
   the row ❌ → ⚠️, leaving only the narrower delta size window; see entry 15.) The parity-review pass then moved `--delete-delay` to ⚠️ (the
   plan-time `--max-delete` charge and non-recursive deferred removal differ from
   rsync when a snapshotted entry fails removal). Differential-gate allowlist
   entries `min_size`/`empty_dirs_recursive`/`dirs_plain` were removed. The
   integrated stats+options+fs branch stands at **111 ✅ / 13 ⚠️ / 33 ❌ = 157**;
   full suite + ASan + clang-format + cppcheck clean.

11. **No-wire parity track 1** on `feat/parity-2.28` (no protocol change):
    `-n --delete` now sends the same filter-excluded + size-pruned protected
    prefixes and synchronized-directory scope as a real run (dry-run would-delete
    matches rsync for source-derived protections; the destination-only exclude
    residual was later closed by track 4a, readdir ordering remains);
    `--delete-delay` now charges
    `--max-delete` on actual removals and re-scans a queued directory at commit
    to remove content created after the plan, with an independent deferred-list
    cap (only partial-delete ordering remains); and `--info=name2` emits `NAME is
    uptodate` plus the leading `./` root name line for `--info=name` (only the
    root-line trigger condition and receiver-side `skip` wording remain). Matrix
    now **111 ✅ / 14 ⚠️ / 32 ❌ = 157**; differential + unit tests added in
    `test_features.py`, `test_option_parity.py`, the unit test
    `tests/test_delete_plan.c`,
    `test_delete_delay_budget_parity.py`, `test_delete_timing_parity.py`.
12. **No-wire parity track 2b** on `feat/parity-2.28` (no protocol change):
    `--progress`/`-P`/`--info=progress` (when not `--quiet`) now run an opt-in
    paths-only metadata pre-count (no file reads/hashing) that supplies rsync's
    full file-list total for the `to-chk` denominator and the directory names,
    and emits per-directory/symlink/special name lines, in both the sequential
    and `--threads` paths. `--delete-during`/`--delete-delay` reuse their
    keep-set pre-scan instead of a second walk; non-progress runs are
    unaffected. Differential tests (`progress`/`progress_threads` over a new
    `multidir` corpus) match rsync's name set and `to-chk` denominator on a
    fresh transfer, and the single-file byte-identical test still passes;
    emission order (rsync's sorted depth-first vs FastSync's readdir/BFS stream)
    plus re-run over-naming (unconditional `./`, ancestor dirs named with a
    transferred child, and no quick-check for symlinks/empty dirs) remain the
    caveats, so the row stays ⚠️ and the matrix is unchanged at
    **111 ✅ / 14 ⚠️ / 32 ❌ = 157**.

13. **Wire parity track 4a** on `feat/parity-2.28` (`PROTOCOL_VERSION` stays
    `2.28.0`): the receiver now has a delete-time filter engine. The sender
    compiles its root-level selection rules exactly as the scanner does
    (`filter_base_build`) and streams them as one bounded, self-describing
    config-frame block (action, sides, anchored, dir-only, negate, owner,
    pattern; bounded rule count and pattern bytes, unknown action/sides is a
    protocol error). The receiver reconstructs `protect_rules` and applies them
    first-match-wins to each extraneous destination path in every delete timing
    (the whole-tree commit walker, the `--delete-during`/`--delete-delay`
    per-directory plans, and the `-n` would-delete enumeration), so a
    `P *.log` rule protects a destination-only `extra.log` like rsync (with
    `risk` cancelling); the sender-derived protected-prefix behavior is
    preserved when no rules are sent and `--delete-excluded` semantics are
    unchanged. Per-directory merge (`:`/`.`) receiver re-derivation remains the
    residual. `TestFilterProtect` (real + dry-run) plus differential cases
    `filter_protect`, `filter_protect_during`, `filter_protect_delay` added and
    the `--filter=RULE` row moves ❌ → ✅: matrix now
    **115 ✅ / 11 ⚠️ / 31 ❌ = 157**; unit tests, the three named integration
    files, clang-format and cppcheck clean.

14. **Wire parity track 5a** on `feat/parity-2.28` (`PROTOCOL_VERSION` stays
    `2.28.0` by project decision): the three basis-dir options now default to
    rsync's metadata quick-check (equal size + equal mtime, or size alone under
    `--size-only`; `-I` disables matching) instead of FastSync's historical
    xxHash64 content equality, so a same-size/different-content basis is trusted
    exactly as rsync trusts it. A new FastSync-only, long-only `--verify-basis`
    flag restores the strict whole-file content equality; its bool is appended to
    the basis block of the config frame (golden wire frame 882 → 886 bytes).
    `--verify-basis` streams the confined basis descriptor to hash it, and a
    basis hit is no longer capped at the 256 MiB whole-file payload bound:
    `--copy-dest` streams the basis through a bounded buffer and `--link-dest`'s
    copy fallback streams from the basis, so an over-limit hit materializes (a
    basis MISS still falls back to the normal transfer and keeps its own bound).
    A `--copy-dest` hit re-applies the SOURCE attributes (the sender transmits
    the source metadata with the basis check frame), matching rsync's
    "copy then fix attributes"; a `--link-dest` success keeps the shared inode's
    attributes (writing through it would mutate the basis). Differential cases
    `copy_dest` and `verify_basis` added; `test_basis_dir_size_only_content_residual`
    converted to a passing parity assertion; `TestBasisDestDirs` updated for the
    new default + `--verify-basis`; unit tests cover the quick-check/verify
    decision and the same-size/different-content handshake. The
    `--compare-dest`/`--copy-dest`/`--link-dest` rows move ❌ → ⚠️ (relative-DIR
    resolution base and over-limit MISS refusal): matrix now
    **116 ✅ / 13 ⚠️ / 28 ❌ = 157**.

15. **No-wire parity track 5b** on `feat/parity-2.28` (`PROTOCOL_VERSION` stays
    `2.28.0` by project decision): `-y`/`--fuzzy` reclassified ❌ → ⚠️. A probe
    against real rsync 3.4.1 (pinned `-B8192`, repeated-content 64 KiB corpus)
    showed the name heuristic is already rsync's (`util1.c fuzzy_distance` /
    `find_filename_suffix` + the exact size+mtime pass) and the output is always
    byte-exact; the only residual is candidate ELIGIBILITY, because FastSync's
    `delta_should_attempt` gate caps the size ratio at 10× and requires both
    files ≥ 16 KiB while rsync will reuse a basis from 0.25× to 10000× and below
    16 KiB. The choice is observable only as `--stats` bandwidth counters. Added
    differential case `fuzzy_basis` (same-suffix sibling, one name edit,
    identical content, block size pinned) asserting tree **and** normalized
    `--stats` parity where the choices coincide, plus `TestFuzzy` pinning the
    window boundary on both sides (>10× and <16 KiB siblings declined by
    FastSync while rsync uses them, both trees byte-identical). Matrix now
    **116 ✅ / 14 ⚠️ / 27 ❌ = 157**.

16. **Lockstep delete-default track 6** on `feat/parity-2.28` (`PROTOCOL_VERSION`
    stays `2.28.0`): plain `--delete` now defaults to rsync's delete-during
    (`--del`) timing, normalized on the client onto the existing `delete_during`
    wire bool. The old late whole-tree commit is opt-in via `--delete-after` or
    the FastSync-only long `--delete-commit` (identical `delete_after` timing).
    `-d/--dirs` still falls back to the end commit, `--delay-updates` still
    deletes before publication, and `--files-from`/`-R` scope is unchanged. The
    `STATUS_DELETE_PLAN` frame gained a one-int `apply` flag so the per-run
    config block (including `--delete-missing-args` exact paths) is always
    transmitted, on a config-only carrier when the scope allows no directory
    plan — fixing a latent bug with a file-only `--files-from` list. Differential
    cases `delete`/`delete_commit`/`filter_protect_after` plus the extended
    `test_delete_timing_parity.py` (plain `--delete` mid-abort removes reached
    extras, `--delete-commit` defers) pass; full `-m "not setpriv"` suite,
    clang-format and cppcheck clean. Matrix unchanged at
    **116 ✅ / 14 ⚠️ / 27 ❌ = 157** (the `--delete`/`--delete-during` rows stay
    ⚠️ for the abort boundary; `--delete-after` stays ✅).

17. **Audit cycle** on `fix/audit-cycle` (from `dev` @ `0fbb9de`;
    `PROTOCOL_VERSION` stays `2.28.0`): a security/correctness pass over the
    parity-2.29 baseline. It raises the decompression ceiling to the 256 MiB
    protocol whole-file bound (`-z` on 100–256 MiB files now works), paces the
    plaintext-TCP `--sendfile` path with `--bwlimit`, confines the `--temp-dir`
    scratch dir by the fd's real path (symlink escape refused), masks
    client-controlled setuid/setgid/sticky bits when super activities are not
    permitted, sets the daemon umask to `022`, makes `--partial-dir` imply
    `--partial`, rejects the unsupported filter modifiers (`x`/`e`/`n`/`w`),
    enforces `MAX_FILTER_RULES` client-side, rejects unknown wire `Status`
    values, and hardens credentials/signal handling (with the accompanying
    refactors and docs). No row changes classification, so the matrix stays
    **120 ✅ / 10 ⚠️ / 27 ❌ = 157**. This docs pass is on `fix/audit-docs2`.

## Next steps
1. **Open and merge the audit-cycle PR** (`fix/audit-cycle`, including this
   `fix/audit-docs2` docs pass) into `dev` once reviewed. `dev` is the default
   branch; all PRs target `dev`, never `main` directly.
2. **Remaining deferred items:**
   - **Large structural refactors:** delete-engine consolidation
     (`delete_extras_fd`/`manifest_delete_extras`/the delete-plan path),
     god-function splits, and translation-unit splits.
   - **`--progress`/`--info` receiver→sender event channel:** the root `./`
     line, ancestor-directory suppression, receiver-side `skip`/`backup` echo,
     and symlink/empty-dir quick-check feedback.
   - **`--delete-before` phase-0 keep-set** (rsync fixes the file list before
     the data pass; FastSync keeps its pre-scan snapshot race).
   - ~~**>256 MiB single-file streaming** (B4, the general whole-file limit).~~
     Closed by #318: the whole-file payload, the basis read/verify and the fuzzy
     basis are streamed through bounded buffers (lz4/append remain buffered).
   - **Wire native-size framing:** lengths are native `size_t` and the protocol
     assumes homogeneous word size/endianness — document or move to fixed-width
     framing.
   - **SCRAM-like daemon auth channel binding:** no TLS channel binding today
     (and it is not RFC 5802).
   - Still-open security nits: the pre-auth config/daemon-auth handshake has no
     aggregate wall-clock deadline (per-message timeout only — slowloris holds
     connection slots); the per-source registry fails open when the shared table
     is full (per-module/global caps and host ACLs still apply).
3. **Out of scope / intentional:** pull (remote source) mode is **not** planned —
   FastSync is push-only; see `RSYNC_COMPAT.md#direction`.

## Key facts / commands
- CI image: `gitea.tap-tap.win/taptap/fastsync-ci:v11` (alias `fastsync-ci:local`).
- Build/test: `cmake -B build -S . -DSTRICT_WARNINGS=ON && cmake --build build -j$(nproc) && ./build/tests`
  then `python3 -m pytest tests/integration/ -n 4 --dist=load -m "not setpriv"`.
- Dev shell: `nix-shell` (provides clang-format, cppcheck, pytest-xdist, openssh,
  rsync, iproute2, valgrind, lcov; does not build on entry).
- Gitea API token: supplied out-of-band via the `TOKEN` environment variable; it is
  intentionally **not** recorded in this file.
- CI polling: `GET /api/v1/repos/TapTap/FastSync/actions/runs?limit=N`, match `head_sha`,
  then `/actions/runs/<id>/jobs`.