diff --git a/RSYNC_COMPAT.md b/RSYNC_COMPAT.md index 71ff82c..a6b9c78 100644 --- a/RSYNC_COMPAT.md +++ b/RSYNC_COMPAT.md @@ -6,11 +6,11 @@ This document maps rsync's full feature set to FastSync's current implementation | Status | Count | Description | |--------|-------|-------------| -| ✅ Implemented | 126 | Feature works end-to-end | +| ✅ Implemented | 129 | Feature works end-to-end | | 🔀 Alt Arg | 3 | Functionality exists but under different flag/semantics | | ⚠️ Partial | 10 | Flag parsed/stored but behavior incomplete | | 🔄 Compatibility No-op | 3 | Flag is accepted for CLI compatibility but has no effect | -| ❌ Not Implemented | 5 | Flag not recognized or no behavior | +| ❌ Not Implemented | 2 | Flag not recognized or no behavior | | **Total** | **147** | | --- @@ -661,9 +661,9 @@ now transmits targets (the prior behavior was broken/partial); its status moved | Flag | Rsync Description | FastSync Status | Notes | |------|-------------------|-----------------|-------| -| `--write-batch=FILE` | Write batched update to file | ❌ Not Implemented | | -| `--only-write-batch=FILE` | Write batch without updating dest | ❌ Not Implemented | | -| `--read-batch=FILE` | Read batched update from file | ❌ Not Implemented | | +| `--write-batch=FILE` | Write batched update to file | ✅ Implemented | Phase-6 residual-batch (client-only): runs the normal live transfer AND additionally emits a self-contained single-file batch of the whole source tree. The batch is a magic/format-version header followed by length-prefixed `chunk_serialize` blobs (full file images), replayable byte-identically by `--read-batch` on another machine with no source/server. `--write-batch` drives the single-threaded transfer path (the `-m` path consumes the config before the separate batch scan pass). See the Phase-6 batch note below | +| `--only-write-batch=FILE` | Write batch without updating dest | ✅ Implemented | Phase-6 residual-batch: emits the self-contained batch FILE only — NO destination update, NO server connection. Requires a source (scans it and serializes the full tree to FILE). Same single-file format as `--write-batch`, so the file is re-appliable via `--read-batch=FILE DEST`. See the Phase-6 batch note below | +| `--read-batch=FILE` | Read batched update from file | ✅ Implemented | Phase-6 residual-batch: applies a previously written batch FILE locally to the destination. NO source and NO server — positional args are the destination only. Reads the magic/version header, then length-prefixed records, `chunk_deserialize`, and applies each via the confined `file_save_to_disk_full` path (same O_NOFOLLOW / `..`-rejection / root-confinement as the network receiver, so an attacker-controlled batch cannot escape the destination root). Malformed/truncated/oversized/traversal records are rejected cleanly. See the Phase-6 batch note below | ## 17. Advanced @@ -778,7 +778,7 @@ These are the hardest compatibility items because they require durable formats o | Features | Effort | Implementation plan | |----------|--------|--------------------| -| `--write-batch=FILE`; `--only-write-batch=FILE`; `--read-batch=FILE` | XL | Specify a versioned batch format, persist all required metadata, and test replay, corruption, and partial application. | +| `--write-batch=FILE`; `--only-write-batch=FILE`; `--read-batch=FILE` | XL | ✅ Implemented (see the Batch Operations table and Phase-6 batch note below): a versioned self-contained single-file residual-batch format, persisted via the existing chunk codec, with replay, corruption, and partial-application safety tests | | `--protocol=NUM` | XL | ✅ Implemented (see the Advanced table and Phase-6 protocol note below): protocol-version forcing without weakening current validation; FastSync's single lockstep wire format means only the current `PROTOCOL_VERSION` is accepted, and everything else is rejected up-front | | `--iconv=CONVERT_SPEC` | L | ✅ Implemented (see the Advanced table and Phase-6 iconv notes below): filename charset conversion at the wire boundary with expansion/overflow safety and invalid-sequence test coverage | | `--stop-after=MINS`; `--stop-at=TIME` | M | ✅ Implemented (see the Advanced table and Phase-6 stop notes below): deadline propagation and safe early stop with --delete safety | @@ -792,6 +792,8 @@ These are the hardest compatibility items because they require durable formats o **Phase-1/2 selection-and-update status correction (docs):** `-I/--ignore-times`, `--size-only`, `-@/--modify-window`, `--existing`, `--ignore-existing`, `-u/--update`, `-W/--whole-file`, and `--compress-threads` were previously listed as not-implemented in this document but are in fact fully implemented and tested on `dev`. This pass corrects the matrix to match the code. The realistic model of these is that FastSync is a *sender-driven* whole-tree copy, so the size+mtime quick-check and all three receiver-policy skips (`--existing`, `--ignore-existing`, `-u`) are evaluated against the **destination** on the receiver side, and their booleans cross the wire in the config frame. `-I`/`--size-only`/`--modify-window` modify the `--incremental` per-file `STATUS_CHECK` handshake's match predicate (`-I` disables the mtime leg and forces transfer; `--size-only` drops only the mtime leg; `--modify-window` adds tolerance to `metadata_mtime_matches`); they require `--incremental` (or a basis dir) to have a handshake to affect, mirroring how they only matter where a quick-check exists in rsync. `--existing`/`--ignore-existing`/`-u` are receiver write-time policies (skipping the write / newer-destination guard) applied across the regular-file, `--delay-updates`-staged, hardlink-sibling, and special/device paths; `-u` implies `-M` metadata and uses a second-then-nanosecond strict `>` newer check; both correctly influence `--remove-source-files` (a skipped source is not removed). `-W/--whole-file` disables block-level delta (opt-in via `--delta`), folded into the wire `use_delta` so no protocol bump was needed, and makes `--fuzzy` inert; `--append`/`--append-verify` are rejected with `-W`. `--compress-threads=NUM` (1..64, client-only, never crosses the wire) sizes the zstd compression worker pool. No code was changed by this correction; the implementation had landed in earlier merge waves (feat/ignore-times, feat/ignore-existing via the newer `file_to_disk_secure_no_replace`/`linkat EEXIST` path, feat/size-only, feat/modify-window, feat/whole-file, feat/update, compression-threads). +**Phase 6, Wave D (batch) shipping note (no PROTOCOL_VERSION change):** FastSync batch mode is a **client-only, self-contained "residual batch"**: a single file `MAGIC "FSTRESBATCH" + format version 1 + metadata flag`, followed by length-prefixed `chunk_serialize` blobs that store full file images (regular files, dirs, symlinks, specials). It is NOT a raw capture of the live wire, because FastSync's protocol is per-file interactive (`STATUS_CHECK`/`STATUS_DELTA_SIGNATURE`/`STATUS_APPEND` handshake), so a raw sender-stream tee is not deterministically replayable against an arbitrary destination. Storing full residuals via the existing, fuzz-tested chunk codec makes `--read-batch` replay byte-identically by construction. `--write-batch=FILE` runs the normal live transfer AND emits the batch from a separate deterministic scan pass; `--only-write-batch=FILE` emits the batch only (no destination, no server); `--read-batch=FILE DEST` applies it locally (no source, no server; DEST is the only positional arg). Because batch is a local driver concern, it never crosses the wire: no new config-frame field and no `PROTOCOL_VERSION` bump (mirroring `--stop-after`/`--protocol`/`--compress-threads`). The READ side is hardened against untrusted/attacker-controlled batch files: magic+version validated before any record, per-record length bounds checked before allocation (64 MB cap), clean-EOF-after-prefix and truncated/oversized records rejected, and every applied path goes through the same confined `file_save_to_disk_full` machinery as the network receiver (O_NOFOLLOW fd-walk, `..`-rejection, root confinement — a malicious `../` or absolute/symlink path cannot escape the destination root; this was security-reviewed and valgrind/ASan-clean). Divergences from rsync: (1) the batch carries the FULL residual (complete file images) rather than rsync's update-only delta stream — always byte-correct but larger; (2) per-file data is capped at the chunk codec's ~64 MB (`BATCH_MAX_RECORD`), so very large files may be refused by the batch writer with a clean error (never a corrupt/truncated batch); (3) hard-links and xattr/ACL blocks are not represented by `chunk_serialize`, so `-H`/`-X`/`-A` are out of scope for batch; (4) there is no companion `.sh`/`.rsync_argvs` — the batch is invoked directly (`fastsync --read-batch=FILE DEST`, `--only-write-batch=FILE SOURCE`); (5) `--write-batch` drives the single-threaded transfer path. Integration/`-M` note: metadata is captured in the batch when `-M` is used and persisted in the header so it applies consistently regardless of the reading process's own `-M`. + ### Recommended Delivery Order 1. Resolve short-option conflicts (`-m`, `-M`, `-T`, `-f`, `-s`) and define the compatibility contract.