TapTap/FastSync
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity
Files
main
FastSync/HANDOFF.md
T

21 KiB
Raw Permalink Blame History

FastSync — Session Handoff (2026-09-21)

Current status

  • Release v2.28.0 is tagged and merged to main: tag v2.28.0 points at ee6523a, and the PR #304 merge commit b4d54504 is on main.
  • dev is at 0fbb9de — the merge of parity cycle 2.29 (PR #305). The old 558782d (incremental-check flake fix) is an ancestor.
  • PROTOCOL_VERSION = "2.28.0" (src/shared/config.h); CMake project(FastFileTransfer VERSION 2.28.0).
  • Parity cycle 2.29 is merged to dev (PR #305), no wire change. It closed the scanner-order, delete-timing, relative-basis and fuzzy-eligibility residuals and improved the --info/--stats/--debug partials. Parity matrix: 120 ✅ / 10 ⚠️ / 27 ❌ = 157. Remaining ⚠️ rows: --info, --debug, --msgs2stderr, --stats, --progress, --delete-before, the three basis-dir options, and -y/--fuzzy.
  • Audit cycle complete on branch fix/audit-cycle (branched from dev @ 0fbb9de), integration PR to dev pending. No wire change (PROTOCOL_VERSION stays 2.28.0). It lands the receiver/client security and correctness fixes — --temp-dir symlink-escape confinement, special-bit masking under a super-off policy, daemon umask(022), the -z decompression ceiling raised to the 256 MiB whole-file bound, --bwlimit pacing the plaintext --sendfile path, --partial-dir implying --partial, rejection of unsupported filter modifiers (x/e/n/w), client-side MAX_FILTER_RULES enforcement, unknown wire Status rejection, and the accompanying refactors/docs. The parity matrix is unchanged at 120 ✅ / 10 ⚠️ / 27 ❌ = 157; this docs pass (worktree fix/audit-docs2) corrects the RSYNC_COMPAT.md summary tally to match the rows.

What landed this session

  1. Wave 8 (refactors): Config X-macro wire table; single-owner authorized_root; daemon per-module/per-host caps + cross-process auth lockout (daemon_limits.[ch]); Data charge returns to its owning ProtocolSession.

  2. Wave 9 (protocol 2.21.0): optional STATUS_ERROR_DETAIL rejection reasons; server-contacting --dry-run (STATUS_DRY_RUN_TRANSFER, receiver mutates nothing).

  3. Security wave: ran 5 parallel audits (wire parsing; daemon/transport/TLS/auth; receiver confinement; client/CLI/SSH; crypto/memory/limits). Fixed all HIGH and the confirmed MEDIUMs:

    • SSH -o ProxyCommand=… argument injection (RCE) — reject leading -, insert --.
    • Truncated zstd frame infinite CPU loop (remote DoS).
    • FIFO receiver opens lacked O_NONBLOCK (indefinite hang).
    • --inplace could write a FIFO/device (bypass of --write-devices gate).
    • --force not gated by server --allow-delete.
    • Privileged standalone server defaulted super activities on; added --allow-super (never honored with --stdio).
    • --dry-run content/hash oracle on read only/basis files removed.
    • Empty hosts allow/deny/auth users now rejected.
    • TLS: AEAD-only 1.2 + server preference, TOCTOU-safe key load, IP-SAN verify, CN-truncation guard. Glob backtracking bounded; line reads bounded; ACL xattrs gated on --acls; decompression/chunk memory charged; pre-auth basis_count NULL-deref fixed.

  4. Tooling: benchmark accuracy (data mix, verification, percentiles, tc, build-bench/, --warm mode); shell.nix full toolchain and no build-on-entry; docs state push-only / remote-source unsupported.

  5. Preserve-attribute split (protocol 2.22.0) landed on feat/preserve-attr-split: per-attribute -p/-t/-o/-g + --no-* negations, -a = -rlptgoD, and the 2.21.0 → 2.22.0 wire bump.

  6. Rsync-parity wave (protocol 2.23.0) on feat/rsync-parity: rsync short options/clustering/attached values (-r/-b/-L/-B, -av, -aAX, -B1000, -essh, -MOPT), -c checksum quick-check, --checksum-choice/--compress-choice validation and seed randomization, rsync timeout/max-alloc defaults, temp-dir confinement + EXDEV fallback, ownership/mapping parity (numeric-ids modifier, map ranges/*/empty-FROM, --chown+map conflicts, fake-super resolved-owner record), verbatim symlink storage with rsync --safe-links/--munge-links, socket recreation under --specials, --chmod 3.4.1 semantics, and delete scoping + --max-delete partial/exit-25. Wire: appended delete-manifest synchronized-directory section and STATUS_DELETE_LIMIT.

  7. Parity-completion wave (protocol 2.24.0 → 2.26.0) on feat/parity-completion: per-directory delete plans (STATUS_DELETE_PLAN) for --delete-during/--delete-delay; receiver STATUS_STATS counters feeding --stats/--progress and --out-format %b/%c/%C, plus -n --delete lines; lz4/zlib/zlibx compression and md4/sha1/none checksums with auto negotiation (default xxh128/zstd); general -R/--no-implied-dirs/-d; the full filter grammar (merge/dir-merge/hide/show/protect/risk/clear + modifiers) and corrected -F/-FF; receiver-side --chown/map TO-name resolution; absolute basis dirs + --link-dest relink; receiver-side --ignore-existing short-circuit; --preallocate over --sparse via fallocate(2); --iconv=./-/--no-iconv; lone -h help; aliases --ignore-non-existing/--protect-args/--msgs2stderr; and the full --info/--debug vocabulary. RSYNC_COMPAT.md reclassifies the matrix to 106 ✅ / 27 ⚠️ / 23 ❌; the later rsync-parity-stats pass (fix/parity-stats) moves it to 107 ✅ / 25 ⚠️ / 24 ❌ (see item 8).

  8. rsync-parity-stats pass on fix/parity-stats (no wire change, PROTOCOL_VERSION stays 2.26.0): --delete-delay now reports only entries it actually removes, while the --max-delete budget is charged at plan/snapshot time (planned, via defer_add) to bound the deferred list (a refilled deferred directory that survives ENOTEMPTY is not reported but still consumes budget); --stats gained the (reg/dir/link/special) Number of files breakdown and now counts only regular files actually stored for Number of regular files transferred/transferred size/literal data (up-to-date re-runs report 0); Total file size includes symlink target lengths; --progress prints the leading ./ root line and counts it in to-chk so a single-file transfer matches rsync; and %C uses the selected transfer checksum with checksum_digest_file supporting md4/sha1/none, byte-identical to rsync for every algorithm. --out-format reclassified ❌ (%b/delta-%c are protocol-specific). Differential + regression tests added; full suite + ASan + clang-format + cppcheck clean.

  9. Option-parity wave (protocol 2.26.0 → 2.27.0, on fix/parity-options): --bwlimit now ports rsync 3.4.1's units/quantization and paces like its leaky bucket; --ignore-errors reproduces rsync's default (an I/O error skips deletion unless the flag is set; the readable tree still transfers and the run exits 23) across every delete timing; the --info categories with a FastSync event (name/flist/del/remove/nonreg/progress) emit rsync's line format, with real-run deleting/*deleting lines carried over the new trailing config bool report_deletes (golden wire updated by tests/test_config.c). Two residuals were reclassified divergent: -M over daemon/TCP (no argv channel in FastSync's binary config handshake; rsync-daemon differential pins the rsync behavior) and receiver-side protect/risk re-derivation for destination-only entries (would need a receiver filter engine; differential pins the divergence — reversed by track 4a below, which adds that engine). The options pass stands at 110 ✅ / 21 ⚠️ / 26 ❌. New tests/integration/test_option_parity.py holds the rsync differentials (bwlimit parse+rate, info lines, real-setpriv --ignore-errors, rsync-daemon -M, filter-protect pin).

  10. rsync-parity-fs pass on fix/parity-fs (no wire change of its own; integrated on top of the 2.27.0 options wave): recursive transfers now recreate empty source directories (and -m/--prune-empty-dirs still suppresses them), a directory entry replaces a blocking destination regular file, and -R --no-implied-dirs --files-from places a listed file under a missing implied parent with default attributes instead of refusing (real rsync 3.4.1 parity, differential-tested). --iconv now reproduces rsync's push direction (destination charset = the spec's REMOTE half; a server --iconv overrides), and -T/--temp-dir relative semantics are confirmed identical while the absolute-path confinement is a deliberate divergence. The basis-dir options, --delay-updates and --dry-run were reclassified to ❌ after a differential test reproduced each exact residual (basis content verification, fixed staging-name collision, and dry-run would-delete over-report). --fuzzy was also reclassified to ❌ (deterministic heuristic with a 10× size window, not rsync's matcher), but its residual is the candidate-selection heuristic itself: the final tree is byte-exact by design, so it is pinned by the TestFuzzy threshold suite rather than a byte-level rsync differential. (Track 5b later found the name heuristic is rsync's own and moved the row ❌ → ⚠️, leaving only the narrower delta size window; see entry 15.) The parity-review pass then moved --delete-delay to ⚠️ (the plan-time --max-delete charge and non-recursive deferred removal differ from rsync when a snapshotted entry fails removal). Differential-gate allowlist entries min_size/empty_dirs_recursive/dirs_plain were removed. The integrated stats+options+fs branch stands at 111 ✅ / 13 ⚠️ / 33 ❌ = 157; full suite + ASan + clang-format + cppcheck clean.

  11. No-wire parity track 1 on feat/parity-2.28 (no protocol change): -n --delete now sends the same filter-excluded + size-pruned protected prefixes and synchronized-directory scope as a real run (dry-run would-delete matches rsync for source-derived protections; the destination-only exclude residual was later closed by track 4a, readdir ordering remains); --delete-delay now charges --max-delete on actual removals and re-scans a queued directory at commit to remove content created after the plan, with an independent deferred-list cap (only partial-delete ordering remains); and --info=name2 emits NAME is uptodate plus the leading ./ root name line for --info=name (only the root-line trigger condition and receiver-side skip wording remain). Matrix now 111 ✅ / 14 ⚠️ / 32 ❌ = 157; differential + unit tests added in test_features.py, test_option_parity.py, the unit test tests/test_delete_plan.c, test_delete_delay_budget_parity.py, test_delete_timing_parity.py.

  12. No-wire parity track 2b on feat/parity-2.28 (no protocol change): --progress/-P/--info=progress (when not --quiet) now run an opt-in paths-only metadata pre-count (no file reads/hashing) that supplies rsync's full file-list total for the to-chk denominator and the directory names, and emits per-directory/symlink/special name lines, in both the sequential and --threads paths. --delete-during/--delete-delay reuse their keep-set pre-scan instead of a second walk; non-progress runs are unaffected. Differential tests (progress/progress_threads over a new multidir corpus) match rsync's name set and to-chk denominator on a fresh transfer, and the single-file byte-identical test still passes; emission order (rsync's sorted depth-first vs FastSync's readdir/BFS stream) plus re-run over-naming (unconditional ./, ancestor dirs named with a transferred child, and no quick-check for symlinks/empty dirs) remain the caveats, so the row stays ⚠️ and the matrix is unchanged at 111 ✅ / 14 ⚠️ / 32 ❌ = 157.

  13. Wire parity track 4a on feat/parity-2.28 (PROTOCOL_VERSION stays 2.28.0): the receiver now has a delete-time filter engine. The sender compiles its root-level selection rules exactly as the scanner does (filter_base_build) and streams them as one bounded, self-describing config-frame block (action, sides, anchored, dir-only, negate, owner, pattern; bounded rule count and pattern bytes, unknown action/sides is a protocol error). The receiver reconstructs protect_rules and applies them first-match-wins to each extraneous destination path in every delete timing (the whole-tree commit walker, the --delete-during/--delete-delay per-directory plans, and the -n would-delete enumeration), so a P *.log rule protects a destination-only extra.log like rsync (with risk cancelling); the sender-derived protected-prefix behavior is preserved when no rules are sent and --delete-excluded semantics are unchanged. Per-directory merge (:/.) receiver re-derivation remains the residual. TestFilterProtect (real + dry-run) plus differential cases filter_protect, filter_protect_during, filter_protect_delay added and the --filter=RULE row moves ❌ → ✅: matrix now 115 ✅ / 11 ⚠️ / 31 ❌ = 157; unit tests, the three named integration files, clang-format and cppcheck clean.

  14. Wire parity track 5a on feat/parity-2.28 (PROTOCOL_VERSION stays 2.28.0 by project decision): the three basis-dir options now default to rsync's metadata quick-check (equal size + equal mtime, or size alone under --size-only; -I disables matching) instead of FastSync's historical xxHash64 content equality, so a same-size/different-content basis is trusted exactly as rsync trusts it. A new FastSync-only, long-only --verify-basis flag restores the strict whole-file content equality; its bool is appended to the basis block of the config frame (golden wire frame 882 → 886 bytes). --verify-basis streams the confined basis descriptor to hash it, and a basis hit is no longer capped at the 256 MiB whole-file payload bound: --copy-dest streams the basis through a bounded buffer and --link-dest's copy fallback streams from the basis, so an over-limit hit materializes (a basis MISS still falls back to the normal transfer and keeps its own bound). A --copy-dest hit re-applies the SOURCE attributes (the sender transmits the source metadata with the basis check frame), matching rsync's "copy then fix attributes"; a --link-dest success keeps the shared inode's attributes (writing through it would mutate the basis). Differential cases copy_dest and verify_basis added; test_basis_dir_size_only_content_residual converted to a passing parity assertion; TestBasisDestDirs updated for the new default + --verify-basis; unit tests cover the quick-check/verify decision and the same-size/different-content handshake. The --compare-dest/--copy-dest/--link-dest rows move ❌ → ⚠️ (relative-DIR resolution base and over-limit MISS refusal): matrix now 116 ✅ / 13 ⚠️ / 28 ❌ = 157.

  15. No-wire parity track 5b on feat/parity-2.28 (PROTOCOL_VERSION stays 2.28.0 by project decision): -y/--fuzzy reclassified ❌ → ⚠️. A probe against real rsync 3.4.1 (pinned -B8192, repeated-content 64 KiB corpus) showed the name heuristic is already rsync's (util1.c fuzzy_distance / find_filename_suffix + the exact size+mtime pass) and the output is always byte-exact; the only residual is candidate ELIGIBILITY, because FastSync's delta_should_attempt gate caps the size ratio at 10× and requires both files ≥ 16 KiB while rsync will reuse a basis from 0.25× to 10000× and below 16 KiB. The choice is observable only as --stats bandwidth counters. Added differential case fuzzy_basis (same-suffix sibling, one name edit, identical content, block size pinned) asserting tree and normalized --stats parity where the choices coincide, plus TestFuzzy pinning the window boundary on both sides (>10× and <16 KiB siblings declined by FastSync while rsync uses them, both trees byte-identical). Matrix now 116 ✅ / 14 ⚠️ / 27 ❌ = 157.

  16. Lockstep delete-default track 6 on feat/parity-2.28 (PROTOCOL_VERSION stays 2.28.0): plain --delete now defaults to rsync's delete-during (--del) timing, normalized on the client onto the existing delete_during wire bool. The old late whole-tree commit is opt-in via --delete-after or the FastSync-only long --delete-commit (identical delete_after timing). -d/--dirs still falls back to the end commit, --delay-updates still deletes before publication, and --files-from/-R scope is unchanged. The STATUS_DELETE_PLAN frame gained a one-int apply flag so the per-run config block (including --delete-missing-args exact paths) is always transmitted, on a config-only carrier when the scope allows no directory plan — fixing a latent bug with a file-only --files-from list. Differential cases delete/delete_commit/filter_protect_after plus the extended test_delete_timing_parity.py (plain --delete mid-abort removes reached extras, --delete-commit defers) pass; full -m "not setpriv" suite, clang-format and cppcheck clean. Matrix unchanged at 116 ✅ / 14 ⚠️ / 27 ❌ = 157 (the --delete/--delete-during rows stay ⚠️ for the abort boundary; --delete-after stays ✅).

  17. Audit cycle on fix/audit-cycle (from dev @ 0fbb9de; PROTOCOL_VERSION stays 2.28.0): a security/correctness pass over the parity-2.29 baseline. It raises the decompression ceiling to the 256 MiB protocol whole-file bound (-z on 100–256 MiB files now works), paces the plaintext-TCP --sendfile path with --bwlimit, confines the --temp-dir scratch dir by the fd's real path (symlink escape refused), masks client-controlled setuid/setgid/sticky bits when super activities are not permitted, sets the daemon umask to 022, makes --partial-dir imply --partial, rejects the unsupported filter modifiers (x/e/n/w), enforces MAX_FILTER_RULES client-side, rejects unknown wire Status values, and hardens credentials/signal handling (with the accompanying refactors and docs). No row changes classification, so the matrix stays 120 ✅ / 10 ⚠️ / 27 ❌ = 157. This docs pass is on fix/audit-docs2.

Next steps

  1. Open and merge the audit-cycle PR (fix/audit-cycle, including this fix/audit-docs2 docs pass) into dev once reviewed. dev is the default branch; all PRs target dev, never main directly.
  2. Remaining deferred items:
    • Large structural refactors: delete-engine consolidation (delete_extras_fd/manifest_delete_extras/the delete-plan path), god-function splits, and translation-unit splits.
    • --progress/--info receiver→sender event channel: the root ./ line, ancestor-directory suppression, receiver-side skip/backup echo, and symlink/empty-dir quick-check feedback.
    • --delete-before phase-0 keep-set (rsync fixes the file list before the data pass; FastSync keeps its pre-scan snapshot race).
    • >256 MiB single-file streaming (B4, the general whole-file limit).
    • Wire native-size framing: lengths are native size_t and the protocol assumes homogeneous word size/endianness — document or move to fixed-width framing.
    • SCRAM-like daemon auth channel binding: no TLS channel binding today (and it is not RFC 5802).
    • Still-open security nits: the pre-auth config/daemon-auth handshake has no aggregate wall-clock deadline (per-message timeout only — slowloris holds connection slots); the per-source registry fails open when the shared table is full (per-module/global caps and host ACLs still apply).
  3. Out of scope / intentional: pull (remote source) mode is not planned — FastSync is push-only; see RSYNC_COMPAT.md#direction.

Key facts / commands

  • CI image: gitea.tap-tap.win/taptap/fastsync-ci:v11 (alias fastsync-ci:local).
  • Build/test: cmake -B build -S . -DSTRICT_WARNINGS=ON && cmake --build build -j$(nproc) && ./build/tests then python3 -m pytest tests/integration/ -n 4 --dist=load -m "not setpriv".
  • Dev shell: nix-shell (provides clang-format, cppcheck, pytest-xdist, openssh, rsync, iproute2, valgrind, lcov; does not build on entry).
  • Gitea API token: supplied out-of-band via the TOKEN environment variable; it is intentionally not recorded in this file.
  • CI polling: GET /api/v1/repos/TapTap/FastSync/actions/runs?limit=N, match head_sha, then /actions/runs/<id>/jobs.
Reference in New Issue View Git Blame Copy Permalink