feat(delete): default --delete to rsync delete-during; add --delete-commit

- plain --delete with no timing flag now selects delete-during (progressive
  deletion, matching rsync and avoiding the full old+new tree peak)
- new long-only FastSync --delete-commit restores the old atomic behavior
  (delete only after the whole transfer succeeds); timing-identical to
  --delete-after, implemented via the same wire bool
- timing flags are mutually exclusive; --delete-commit conflicts with other
  timings; --delete-before/--delete-during rows reworded per Phase-0 probes
- CHANGELOG migration note; tally unchanged 116/14/27
This commit is contained in:
2026-09-19 16:29:48 +02:00
parent 711b7e50b3
commit 36fd0774e8
17 changed files with 281 additions and 55 deletions
+31
View File
@@ -4,6 +4,37 @@ All notable changes to FastSync are documented here. Versions match
`PROTOCOL_VERSION` (printed by `fastsync --version`); the client and server must `PROTOCOL_VERSION` (printed by `fastsync --version`); the client and server must
run the same version because the handshake is strict. run the same version because the handshake is strict.
## [Unreleased]
### Changed
- **`--delete` now defaults to delete-during (rsync `--del`) timing.** With no
explicit timing flag, a plain `--delete` removes each directory's extras as
that directory is processed instead of committing one whole-tree deletion only
after the entire transfer succeeds. This matches rsync, frees destination
space progressively, and avoids the whole-old+new-tree peak that could
`ENOSPC` a tight destination. The client maps the default onto the existing
`delete_during` wire boolean, so `PROTOCOL_VERSION` stays `2.28.0`.
- Added the FastSync-only long option **`--delete-commit`** (implies
`--delete`): it selects the old late whole-tree commit and is timing-identical
to `--delete-after` (the same `delete_after` wire boolean). Explicit timing
flags always win over the default, at most one timing flag may be given, and a
timing flag combined with `--no-delete` is still rejected.
- The per-directory `STATUS_DELETE_PLAN` frame gained a one-int `apply` flag:
the one-shot per-run config block (protected prefixes, size-pruned mirrors,
`--delete-missing-args` exact paths) is now always transmitted first on a
config-only carrier (`apply=false`), fixing a latent bug where a
`--delete-missing-args` run whose `--files-from` list synchronized no directory
never sent its exact deletions.
### Migration
- Scripts that relied on plain `--delete` deleting nothing until the transfer
fully succeeded must pass **`--delete-commit`** (or `--delete-after`) to keep
that behavior. Plain `--delete` now removes reached directories' extras during
the transfer, exactly like rsync's default; on a completed run the final tree
is unchanged.
## [2.26.0] - 2026-09-17 ## [2.26.0] - 2026-09-17
### Added ### Added
+18
View File
@@ -174,6 +174,24 @@
FastSync while rsync uses them, both trees byte-identical). Matrix now FastSync while rsync uses them, both trees byte-identical). Matrix now
**116 ✅ / 14 ⚠️ / 27 ❌ = 157**. **116 ✅ / 14 ⚠️ / 27 ❌ = 157**.
16. **Lockstep delete-default track 6** on `feat/parity-2.28` (`PROTOCOL_VERSION`
stays `2.28.0`): plain `--delete` now defaults to rsync's delete-during
(`--del`) timing, normalized on the client onto the existing `delete_during`
wire bool. The old late whole-tree commit is opt-in via `--delete-after` or
the FastSync-only long `--delete-commit` (identical `delete_after` timing).
`-d/--dirs` still falls back to the end commit, `--delay-updates` still
deletes before publication, and `--files-from`/`-R` scope is unchanged. The
`STATUS_DELETE_PLAN` frame gained a one-int `apply` flag so the per-run
config block (including `--delete-missing-args` exact paths) is always
transmitted, on a config-only carrier when the scope allows no directory
plan — fixing a latent bug with a file-only `--files-from` list. Differential
cases `delete`/`delete_commit`/`filter_protect_after` plus the extended
`test_delete_timing_parity.py` (plain `--delete` mid-abort removes reached
extras, `--delete-commit` defers) pass; full `-m "not setpriv"` suite,
clang-format and cppcheck clean. Matrix unchanged at
**116 ✅ / 14 ⚠️ / 27 ❌ = 157** (the `--delete`/`--delete-during` rows stay
⚠️ for the abort boundary; `--delete-after` stays ✅).
## Next steps ## Next steps
1. **Merge PR #284** (`dev` -> `main`) once reviewed (protected branch). 1. **Merge PR #284** (`dev` -> `main`) once reviewed (protected branch).
2. **Deferred security items** (documented, not implemented): 2. **Deferred security items** (documented, not implemented):
+18 -10
View File
@@ -53,6 +53,8 @@ matrix is **111 ✅ / 13 ⚠️ / 33 ❌ = 157**.
**Parity track 5b (no-wire, on `feat/parity-2.28`; `PROTOCOL_VERSION` stays 2.28.0 by project decision).** `-y`/`--fuzzy` is reclassified ❌ → ⚠️: the receiver-side similar-file basis is an internal bandwidth optimization (the config `fuzzy` bool over the existing receiver-driven delta handshake, unchanged since 2.9.0), and the transferred tree is byte-exact by design regardless of which basis — or no basis — is chosen. A probe against real rsync 3.4.1 showed the remaining difference is not the name rule (FastSync already ports `util1.c fuzzy_distance`/`find_filename_suffix` plus the exact size+mtime pass) but candidate ELIGIBILITY: rsync will pick a fuzzy basis whose size ratio to the source is unrestricted (empirically from 0.25× to 10000×, and for files as small as 300 B), while FastSync's `delta_should_attempt` gate caps the ratio at 10× and requires both files ≥ 16 KiB, so an out-of-window sibling is declined and the file is sent whole. The choice is observable only as bandwidth (`Matched data`/`Literal data`/`Total transferred file size` in `--stats`); the destination tree and exit code are identical either way. A new differential case (`fuzzy_basis`: same-suffix sibling one name-edit away, content identical, block size pinned to 8192) asserts tree **and** normalized `--stats` parity where the two tools' choices coincide; `TestFuzzy` pins the window boundary on both sides (a >10× and a <16 KiB sibling are declined by FastSync while rsync uses them, both trees byte-identical). No wire field changed. The matrix is now **116 ✅ / 14 ⚠️ / 27 ❌ = 157**. **Parity track 5b (no-wire, on `feat/parity-2.28`; `PROTOCOL_VERSION` stays 2.28.0 by project decision).** `-y`/`--fuzzy` is reclassified ❌ → ⚠️: the receiver-side similar-file basis is an internal bandwidth optimization (the config `fuzzy` bool over the existing receiver-driven delta handshake, unchanged since 2.9.0), and the transferred tree is byte-exact by design regardless of which basis — or no basis — is chosen. A probe against real rsync 3.4.1 showed the remaining difference is not the name rule (FastSync already ports `util1.c fuzzy_distance`/`find_filename_suffix` plus the exact size+mtime pass) but candidate ELIGIBILITY: rsync will pick a fuzzy basis whose size ratio to the source is unrestricted (empirically from 0.25× to 10000×, and for files as small as 300 B), while FastSync's `delta_should_attempt` gate caps the ratio at 10× and requires both files ≥ 16 KiB, so an out-of-window sibling is declined and the file is sent whole. The choice is observable only as bandwidth (`Matched data`/`Literal data`/`Total transferred file size` in `--stats`); the destination tree and exit code are identical either way. A new differential case (`fuzzy_basis`: same-suffix sibling one name-edit away, content identical, block size pinned to 8192) asserts tree **and** normalized `--stats` parity where the two tools' choices coincide; `TestFuzzy` pins the window boundary on both sides (a >10× and a <16 KiB sibling are declined by FastSync while rsync uses them, both trees byte-identical). No wire field changed. The matrix is now **116 ✅ / 14 ⚠️ / 27 ❌ = 157**.
**Lockstep track 6 (delete default; `PROTOCOL_VERSION` stays 2.28.0).** Plain `--delete` with no explicit timing flag now defaults to rsync's delete-during (`--del`) timing: the client normalizes it onto the existing `delete_during` wire bool in `cli_finalize_config`, so no config-frame field was added, and a tight destination no longer has to hold the whole old+new tree at once (the old atomic commit could hit `ENOSPC`). The old late whole-tree commit is opt-in via `--delete-after` or the FastSync-only long spelling `--delete-commit`, which selects the identical `delete_after` timing (documented equivalence). Precedence is unchanged and order-independent: each timing flag implies `--delete`, at most one timing flag may be given, and a timing flag with `--no-delete` is rejected. `-d/--dirs` still falls back to the end commit; `--delay-updates` still deletes genuine extras before publication (the per-directory skip list protects the staging dir); `--files-from`/`-R` scope is unchanged. The per-directory `STATUS_DELETE_PLAN` frame gained a one-int `apply` flag (still 2.28.0): the one-shot per-run config block (protected prefixes, size-pruned mirrors, `--delete-missing-args` exact paths) is now always sent first on a config-only carrier with `apply=false`, fixing a latent bug where a `--delete-missing-args` run whose `--files-from` list synchronized no directory never transmitted its exact deletions. Differential evidence: `delete` (plain, vs rsync's default), `delete_commit` (FastSync `--delete-commit` vs rsync `--delete-after`), and `filter_protect_after` (whole-tree protect) cases; `TestDeleteTimingFinalStateParity` compares plain `--delete`/`--delete-commit` against rsync on completed runs, and `TestDeleteTimingFailure` proves plain `--delete` removes reached extras on a mid-transfer abort while `--delete-commit` removes nothing. The matrix is unchanged at **116 ✅ / 14 ⚠️ / 27 ❌ = 157** (the `--delete`/`--delete-during` rows stay ⚠️ for the abort boundary; `--delete-after` stays ✅).
**Parity completion wave (protocol 2.23.0 → 2.26.0).** This wave closed the **Parity completion wave (protocol 2.23.0 → 2.26.0).** This wave closed the
remaining gaps the rsync-parity wave left open (delete timing, wire counters and remaining gaps the rsync-parity wave left open (delete timing, wire counters and
output, codec breadth, general `-R`/`-d`, the full filter grammar, receiver-side output, codec breadth, general `-R`/`-d`, the full filter grammar, receiver-side
@@ -167,11 +169,11 @@ Every one of those has an entry below with its remaining caveats.
| Flag | Rsync Description | FastSync Status | Notes | | Flag | Rsync Description | FastSync Status | Notes |
|------|-------------------|-----------------|-------| |------|-------------------|-----------------|-------|
| `--delete` | Delete extraneous files from dest | ⚠️ Caveat | `use_delete` config field. Deletion is always derived from the transmitted keep-set manifest of the paths the sender sent/keeps (never from unchecked input), runs through the symlink-safe walker bounded by `MAX_SERVER_DELETE_COUNT`, and skips the `.fastsync-stage` staging dir under `--delay-updates`. FastSync's default timing when no timing flag is given is **delete-after** (extras are removed only once the whole transfer succeeded) — intentionally NOT rsync's `--del`/delete-during default, to preserve FastSync's commit-style safety. By default the destination mirror of a path the source scan pruned (filter/exclude/size rules) is **protected** from deletion — matching rsync, which does not delete excluded files under `--delete`; `--delete-excluded` opts back into deleting them (see below). Deletion is scoped to the **synchronized directories** sent in the manifest (protocol 2.23.0), so a `--files-from` subset no longer deletes untransmitted paths outside the listed directory subtrees. The walk is bounded: a client `--max-delete=NUM` (or the 100000-entry server bound) makes it **partial** — entries up to the bound are removed, the rest are skipped, and the client exits **25** (`RERR_PARTIAL`), matching rsync, rather than failing the transfer. Extraneous destination symlinks are unlinked by name (never followed); a directory still holding a kept/protected entry is left behind rather than failing | | `--delete` | Delete extraneous files from dest | ⚠️ Caveat | `use_delete` config field. Deletion is always derived from the keep-set the sender actually transmitted (the per-directory `STATUS_DELETE_PLAN` set by default, or the whole-tree manifest for the late timings — never from unchecked input), runs through the symlink-safe walker bounded by `MAX_SERVER_DELETE_COUNT`, and skips the `.fastsync-stage` staging dir under `--delay-updates`. **Lockstep track 6 (protocol 2.28.0): plain `--delete` with no explicit timing flag now defaults to `--delete-during`**, exactly like rsync's `--del` (the client normalizes it to the existing `delete_during` wire bool; no new wire field). This frees destination space progressively during the transfer and avoids the whole-old+new-tree peak that could `ENOSPC` a tight destination. The old late whole-tree commit is opt-in via `--delete-after` or the FastSync-only long spelling `--delete-commit`. **Caveat (shared with `--delete-during`):** the exact mid-transfer abort boundary can differ from rsync's generator (rsync removes all extras ahead of its throttled sender; FastSync removes only the directories it has reached), `-d/--dirs` falls back to the end-of-transfer commit, and the surviving-set ordering under a partial `--max-delete` can differ — on a completed run the trees agree. By default the destination mirror of a path the source scan pruned (filter/exclude/size rules) is **protected** from deletion — matching rsync, which does not delete excluded files under `--delete`; `--delete-excluded` opts back into deleting them (see below). Deletion is scoped to the **synchronized directories** sent on the wire (protocol 2.23.0), so a `--files-from` subset no longer deletes untransmitted paths outside the listed directory subtrees. The walk is bounded: a client `--max-delete=NUM` (or the 100000-entry server bound) makes it **partial** — entries up to the bound are removed, the rest are skipped, and the client exits **25** (`RERR_PARTIAL`), matching rsync, rather than failing the transfer. Extraneous destination symlinks are unlinked by name (never followed); a directory still holding a kept/protected entry is left behind rather than failing |
| `--delete-before` | Delete before transfer | ⚠️ Caveat | Implies `--delete`. The sender runs a full source pre-scan (paths only) and transmits the keep-set manifest BEFORE any file data; the receiver validates it, removes every destination entry not listed (bounded walk, staging-dir skip, protected prefixes honored), then acks `STATUS_OK`. The sender only starts streaming after the deletion committed, or aborts if the receiver reported a deletion error. By definition the deletions already happened when a later transfer phase fails — rsync's delete-before is destructive the same way; a subsequent failure does not restore the removed files. Divergence: the keep-set is the pre-scan snapshot, so a file that appears on the source between the pre-scan and the data pass is still transferred but was not protected from deletion | | `--delete-before` | Delete before transfer | ⚠️ Caveat | Implies `--delete`. The sender runs a full source pre-scan (paths only) and transmits the keep-set manifest BEFORE any file data; the receiver validates it, removes every destination entry not listed (bounded walk, staging-dir skip, protected prefixes honored), then acks `STATUS_OK`. The sender only starts streaming after the deletion committed, or aborts if the receiver reported a deletion error. By definition the deletions already happened when a later transfer phase fails — rsync's delete-before is destructive the same way; a subsequent failure does not restore the removed files. **Phase-0 divergence (sharpened):** rsync builds the full file list first, so a source file created after that scan is NOT transferred and its destination extra is deleted; FastSync's single-threaded data pass re-scans the source, so the late file IS transferred (a safe superset), while FastSync `--threads` pipelines the scan and matches rsync |
| `--del`, `--delete-during` | Delete during transfer | ⚠️ Caveat | Both spellings accepted; imply `--delete`. **Protocol 2.24.0 implements per-directory delete plans:** as the sender finishes each source directory it streams a `STATUS_DELETE_PLAN` for that directory and the receiver removes that directory's extras before applying the next directory's data, so a mid-transfer failure has already removed the extras of the directories reached (verified with a byte-slicing proxy). **Remaining divergence:** the exact abort boundary and the progressive ordering of removals versus rsync's generator can differ, and `-d`/`--dirs` (no descent) falls back to the end-of-transfer commit. `-R` plans are scoped to the transferred prefix subtree | | `--del`, `--delete-during` | Delete during transfer | ⚠️ Caveat | Both spellings accepted; imply `--delete`, and since lockstep track 6 this is also the default timing of a plain `--delete`. **Protocol 2.24.0 implements per-directory delete plans:** as the sender reaches each source directory it streams a `STATUS_DELETE_PLAN` for that directory and the receiver removes that directory's extras (verified with a byte-slicing proxy). The one-shot per-run config block (protected prefixes, size-pruned mirrors, `--delete-missing-args` exact paths) rides a dedicated config-only carrier frame with an `apply=false` flag, so it reaches the receiver even when the scope allows no directory plan at all (a `--files-from` list of bare files synchronizes no directory). **Phase-0 divergence (sharpened):** on a mid-transfer abort rsync's generator (which runs ahead of its throttled sender) has already removed ALL the extras it planned, whereas FastSync has removed only the directories it actually reached; on a completed run both agree. Also `-d`/`--dirs` (no descent) falls back to the end-of-transfer whole-tree commit, and the surviving-set ordering under a partial `--max-delete` can differ. `-R` plans are scoped to the transferred prefix subtree |
| `--delete-delay` | Find deletions during, delete after | ⚠️ Caveat | Implies `--delete`. **Protocol 2.24.0 implements rsync's delete-delay timing:** the sender records each directory's delete plan while scanning and the receiver commits those removals only after the whole transfer succeeds (per plan), so an extra created in the destination after its directory's plan survives while `--delete-after` re-scans and removes it, and a failed transfer removes nothing. The **reported** deleted count advances only on an actual removal. **Fixed (no-wire):** the `--max-delete` budget is now charged on ACTUAL removals (an unlink/rmdir that succeeded), not at plan/snapshot time, and a queued directory is re-scanned at commit and removed recursively (content created after the plan included), matching rsync: a snapshotted entry that fails or is skipped consumes no budget, so a later extra rsync would delete is still deleted. The deferred snapshot list keeps an independent hard cap (`DELETE_PLAN_SERVER_LIMIT`) so it cannot grow without bound now that the budget is no longer charged while scanning. A `--max-delete=2` partial delete reports exactly 2 and exits 25 in both tools, and the refilled-directory differential (late content removed, directory removed, budget shared) now matches rsync 3.4.1 on both sides (`test_delete_delay_budget_parity.py`, `test_delete_timing_parity.py`). Unit tests cover recursive removal, actual-removal charging, and the bounded deferred list. **Caveat:** the exact ordering of which extras are removed first under a partial `--max-delete` can still differ from rsync's generator (the survivor set is compared by count, not identity) | | `--delete-delay` | Find deletions during, delete after | ⚠️ Caveat | Implies `--delete`. **Protocol 2.24.0 implements rsync's delete-delay timing:** the sender records each directory's delete plan while scanning and the receiver commits those removals only after the whole transfer succeeds (per plan), so an extra created in the destination after its directory's plan survives while `--delete-after` re-scans and removes it, and a failed transfer removes nothing. The **reported** deleted count advances only on an actual removal. **Fixed (no-wire):** the `--max-delete` budget is now charged on ACTUAL removals (an unlink/rmdir that succeeded), not at plan/snapshot time, and a queued directory is re-scanned at commit and removed recursively (content created after the plan included), matching rsync: a snapshotted entry that fails or is skipped consumes no budget, so a later extra rsync would delete is still deleted. The deferred snapshot list keeps an independent hard cap (`DELETE_PLAN_SERVER_LIMIT`) so it cannot grow without bound now that the budget is no longer charged while scanning. A `--max-delete=2` partial delete reports exactly 2 and exits 25 in both tools, and the refilled-directory differential (late content removed, directory removed, budget shared) now matches rsync 3.4.1 on both sides (`test_delete_delay_budget_parity.py`, `test_delete_timing_parity.py`). Unit tests cover recursive removal, actual-removal charging, and the bounded deferred list. **Caveat:** the exact ordering of which extras are removed first under a partial `--max-delete` can still differ from rsync's generator (the survivor set is compared by count, not identity) |
| `--delete-after` | Delete after transfer | ✅ Parity | Implies `--delete`. The delete-after timing is also what plain `--delete` does: the keep-set manifest closes the data stream and the receiver commits the bounded deletion only after the terminal `STATUS_FINISHED` proves the whole transfer (every data frame received and stored) succeeded. A failed or aborted transfer removes nothing | | `--delete-after` | Delete after transfer | ✅ Parity | Implies `--delete`. Selects the late whole-tree commit: the keep-set manifest closes the data stream and the receiver commits the bounded deletion only after the terminal `STATUS_FINISHED` proves the whole transfer (every data frame received and stored) succeeded. A failed or aborted transfer removes nothing. Since lockstep track 6 a plain `--delete` defaults to delete-during (rsync's `--del`); `--delete-after` — or the FastSync-only `--delete-commit` spelling, which selects the identical timing — is the explicit way to keep the old commit-style behavior |
| `--delete-excluded` | Also delete excluded files | ✅ Parity | `delete_excluded` config field. Under `--delete` FastSync protects (rsync's default) the destination mirror of paths the sender's source scan pruned by the user-selection rules — the `--filter`/`-F`/`-C` layer and the legacy `--exclude`/`--include` layer. The sender transmits those concrete pruned paths as **protected prefixes** in the delete-manifest frame (see the Phase-3 notes below); the walker never descends into or removes them. `--delete-excluded` opts back in: the sender sends an empty protected list, so the excluded destination mirrors become ordinary extras and are removed. **`--max-size`/`--min-size` pruned mirrors are a separate, always-on protection** (protocol 2.23.0, rsync parity): size-pruned source mirrors survive `--delete` even with `--delete-excluded`. Track 4a (protocol 2.28.0) additionally re-applies the received `protect`/`risk` rules on the receiver, so a destination-only entry matching an exclude rule is protected (or left at risk) exactly like rsync; the remaining sender-derived `--delete-excluded` behavior (an unqualified rule becomes sender-only, so its source mirror and matching destination-only extras are deleted) is unchanged | | `--delete-excluded` | Also delete excluded files | ✅ Parity | `delete_excluded` config field. Under `--delete` FastSync protects (rsync's default) the destination mirror of paths the sender's source scan pruned by the user-selection rules — the `--filter`/`-F`/`-C` layer and the legacy `--exclude`/`--include` layer. The sender transmits those concrete pruned paths as **protected prefixes** in the delete-manifest frame (see the Phase-3 notes below); the walker never descends into or removes them. `--delete-excluded` opts back in: the sender sends an empty protected list, so the excluded destination mirrors become ordinary extras and are removed. **`--max-size`/`--min-size` pruned mirrors are a separate, always-on protection** (protocol 2.23.0, rsync parity): size-pruned source mirrors survive `--delete` even with `--delete-excluded`. Track 4a (protocol 2.28.0) additionally re-applies the received `protect`/`risk` rules on the receiver, so a destination-only entry matching an exclude rule is protected (or left at risk) exactly like rsync; the remaining sender-derived `--delete-excluded` behavior (an unqualified rule becomes sender-only, so its source mirror and matching destination-only extras are deleted) is unchanged |
| `--max-delete=NUM` | Max files to delete | ✅ Parity | `max_delete` config field (default -1 = no client limit; 0 = delete nothing). **Protocol 2.23.0 matches rsync's partial semantics:** the receiver deletes up to NUM entries (regular files, symlinks and empty directories; each directory removal counts as one) and then **stops deleting, skips the rest, and reports the run as partial**. The client prints a "deletions stopped due to `--max-delete` limit" message and exits **25** (rsync's `RERR_PARTIAL`), not a hard failure — the transfer itself succeeded. NUM only applies together with `--delete` (it is inert otherwise, matching rsync). A client NUM below the server hard bound `MAX_SERVER_DELETE_COUNT` (100000) replaces it; a NUM above it never raises that cap. Deleting an entire destination with no limit is still bounded by the server's 100000-entry ceiling. `--delete-missing-args` exact-path deletions and the ordinary extras walk draw from the same budget, matching rsync | | `--max-delete=NUM` | Max files to delete | ✅ Parity | `max_delete` config field (default -1 = no client limit; 0 = delete nothing). **Protocol 2.23.0 matches rsync's partial semantics:** the receiver deletes up to NUM entries (regular files, symlinks and empty directories; each directory removal counts as one) and then **stops deleting, skips the rest, and reports the run as partial**. The client prints a "deletions stopped due to `--max-delete` limit" message and exits **25** (rsync's `RERR_PARTIAL`), not a hard failure — the transfer itself succeeded. NUM only applies together with `--delete` (it is inert otherwise, matching rsync). A client NUM below the server hard bound `MAX_SERVER_DELETE_COUNT` (100000) replaces it; a NUM above it never raises that cap. Deleting an entire destination with no limit is still bounded by the server's 100000-entry ceiling. `--delete-missing-args` exact-path deletions and the ordinary extras walk draw from the same budget, matching rsync |
| `--ignore-errors` | Delete even with I/O errors | ✅ Parity | Sender-side, client-only config field. Matches rsync's semantics exactly: an unreadable source subdirectory is always skipped so the readable tree transfers (the transfer root itself stays fatal), and the run reports rsync's partial-transfer exit **23**. Deletion policy follows rsync: by default an I/O error suppresses deletion (`IO error encountered -- skipping file deletion`), while `--ignore-errors` lets the deletion commit. The decision applies to every timing (`--delete`, `--delete-before`, `--delete-during`, `--delete-delay`, `--delete-after`) in both the sequential and `--threads` send paths. Differential-tested against rsync 3.4.1 with both tools run as an unprivileged user (mode-000 source directory); the reference build's root-only gate still excludes the EACCES differential, but the setpriv differential test exercises it. The piece that stays FastSync-specific is documented under the recursive-empty-directory residual: FastSync never emits an unreadable (or empty) directory entry, so that mirror is an extra that a run with `--ignore-errors` removes, where rsync emits the directory and keeps its mirror | | `--ignore-errors` | Delete even with I/O errors | ✅ Parity | Sender-side, client-only config field. Matches rsync's semantics exactly: an unreadable source subdirectory is always skipped so the readable tree transfers (the transfer root itself stays fatal), and the run reports rsync's partial-transfer exit **23**. Deletion policy follows rsync: by default an I/O error suppresses deletion (`IO error encountered -- skipping file deletion`), while `--ignore-errors` lets the deletion commit. The decision applies to every timing (`--delete`, `--delete-before`, `--delete-during`, `--delete-delay`, `--delete-after`) in both the sequential and `--threads` send paths. Differential-tested against rsync 3.4.1 with both tools run as an unprivileged user (mode-000 source directory); the reference build's root-only gate still excludes the EACCES differential, but the setpriv differential test exercises it. The piece that stays FastSync-specific is documented under the recursive-empty-directory residual: FastSync never emits an unreadable (or empty) directory entry, so that mirror is an extra that a run with `--ignore-errors` removes, where rsync emits the directory and keeps its mirror |
@@ -186,8 +188,8 @@ The `STATUS_MANIFEST` frame is count-delimited and position-independent: the
receiver commits the deletion either when the manifest arrives (early modes: receiver commits the deletion either when the manifest arrives (early modes:
`--delete-before`/`--delete-during`, which additionally acknowledge with `--delete-before`/`--delete-during`, which additionally acknowledge with
`STATUS_OK` before data flows) or after the terminal `STATUS_FINISHED` proves `STATUS_OK` before data flows) or after the terminal `STATUS_FINISHED` proves
the whole transfer succeeded (commit modes: plain `--delete`/`--delete-after`/ the whole transfer succeeded (commit modes: `--delete-after`/`--delete-delay`;
`--delete-delay`). Timing is chosen purely from the config, so server policy plain `--delete` joined the per-directory plans in track 6). Timing is chosen purely from the config, so server policy
(`--allow-delete` off) still disables deletion without deadlocking the early (`--allow-delete` off) still disables deletion without deadlocking the early
manifest ack. `--delete-delay` and `--delete-during` are each implemented as manifest ack. `--delete-delay` and `--delete-during` are each implemented as
the closest safe approximation their engine mode allows; the divergences are the closest safe approximation their engine mode allows; the divergences are
@@ -269,7 +271,10 @@ Flag-conflict policy: unlike rsync's last-one-wins behaviour, every deletion
timing flag implies `--delete`, and combining a timing flag with `--no-delete` timing flag implies `--delete`, and combining a timing flag with `--no-delete`
(in either argument order) — or more than one timing flag — is rejected as a (in either argument order) — or more than one timing flag — is rejected as a
configuration error rather than silently resolved. Note the check is configuration error rather than silently resolved. Note the check is
order-independent because it runs over the fully parsed config. The deletion order-independent because it runs over the fully parsed config. The
FastSync-only `--delete-commit` is a late-timing spelling (it maps onto
`--delete-after`), so it conflicts with any different timing flag exactly like
`--delete-after` does. The deletion
POLICY flags (`--delete-excluded`, `--max-delete`, `--ignore-errors`, `--force`) POLICY flags (`--delete-excluded`, `--max-delete`, `--ignore-errors`, `--force`)
do NOT imply `--delete`; without `--delete` they are inert (matching rsync). do NOT imply `--delete`; without `--delete` they are inert (matching rsync).
@@ -1169,9 +1174,11 @@ These remain after the wave; the individual rows carry the precise wording.
but `%b`/`%c` count FastSync wire bytes (protocol-specific, hence ❌). but `%b`/`%c` count FastSync wire bytes (protocol-specific, hence ❌).
- **`-n --delete`** ordering can differ from rsync's delete-during walk (the - **`-n --delete`** ordering can differ from rsync's delete-during walk (the
differential compares the sorted would-delete set). differential compares the sorted would-delete set).
- **Delete timing:** the default `--delete` remains delete-after rather than - **Delete timing:** plain `--delete` now defaults to rsync's delete-during
rsync's delete-during; per-directory plans have generator-order/abort-boundary (lockstep track 6); the residual is the mid-transfer abort boundary, where
differences; `--delete-before` keeps its pre-scan snapshot race; and while rsync's generator removes all extras ahead of its throttled sender while
FastSync removes only the reached directories (completed runs agree).
`--delete-before` keeps its pre-scan snapshot race; and while
base-rule `protect`/`risk` rules are now re-applied on the receiver (track 4a), base-rule `protect`/`risk` rules are now re-applied on the receiver (track 4a),
per-directory merge (`.rsync-filter`) protection is still sender-derived, so a per-directory merge (`.rsync-filter`) protection is still sender-derived, so a
destination-only entry matching only a per-directory rule is not re-derived. destination-only entry matching only a per-directory rule is not re-derived.
@@ -1265,5 +1272,6 @@ Ranked by user demand, implementation complexity, and interoperability impact (_
| `--fastsync-server-path` | Path to fastsync-server binary | | `--fastsync-server-path` | Path to fastsync-server binary |
| `--server-host` / `--server-port` | Direct TCP connection | | `--server-host` / `--server-port` | Direct TCP connection |
| `--verify-basis` | FastSync-only (long form, not in rsync): require a `--compare-dest`/`--copy-dest`/`--link-dest` basis hit to match the source by whole-file digest instead of trusting rsync's size+mtime (or `--size-only`) quick-check. Off by default (the default matches rsync). Crosses the wire (protocol 2.28.0) | | `--verify-basis` | FastSync-only (long form, not in rsync): require a `--compare-dest`/`--copy-dest`/`--link-dest` basis hit to match the source by whole-file digest instead of trusting rsync's size+mtime (or `--size-only`) quick-check. Off by default (the default matches rsync). Crosses the wire (protocol 2.28.0) |
| `--delete-commit` | FastSync-only (long form, not in rsync): restore the late whole-tree delete commit — the keep-set manifest is committed only after the entire transfer succeeded. Identical timing to `--delete-after`, and implemented as the same `delete_after` wire bool (no new field); it exists because plain `--delete` now defaults to `--delete-during` (lockstep track 6). Implies `--delete` and conflicts with any different timing flag |
| Incremental sync | Skip unchanged files (size+mtime) | | Incremental sync | Skip unchanged files (size+mtime) |
| Delta transfer | Block-level delta for changed files | | Delta transfer | Block-level delta for changed files |
+21 -1
View File
@@ -997,6 +997,12 @@ static const OptionEntry OPTION_TABLE[] = {
{"--delete-during", "--del", OPT_FLAG, offsetof(Config, delete_during)}, {"--delete-during", "--del", OPT_FLAG, offsetof(Config, delete_during)},
{"--delete-delay", NULL, OPT_FLAG, offsetof(Config, delete_delay)}, {"--delete-delay", NULL, OPT_FLAG, offsetof(Config, delete_delay)},
{"--delete-after", NULL, OPT_FLAG, offsetof(Config, delete_after)}, {"--delete-after", NULL, OPT_FLAG, offsetof(Config, delete_after)},
/* FastSync-only long spelling of the late whole-tree commit, which selects
the same timing as rsync's --delete-after in FastSync (the whole-tree
keep-set manifest is committed only after the entire transfer succeeded).
Plain --delete now defaults to delete-during, so this restores the old
FastSync behavior; it maps onto the same delete_after wire field. */
{"--delete-commit", NULL, OPT_FLAG, offsetof(Config, delete_after)},
{"--delete-excluded", NULL, OPT_FLAG, offsetof(Config, delete_excluded)}, {"--delete-excluded", NULL, OPT_FLAG, offsetof(Config, delete_excluded)},
{"--max-delete", NULL, OPT_SIGNED_INT, offsetof(Config, max_delete)}, {"--max-delete", NULL, OPT_SIGNED_INT, offsetof(Config, max_delete)},
{"--ignore-errors", NULL, OPT_FLAG, offsetof(Config, ignore_errors)}, {"--ignore-errors", NULL, OPT_FLAG, offsetof(Config, ignore_errors)},
@@ -1513,7 +1519,9 @@ static bool cli_handle_table_option(CliParseCtx* ctx) {
if (entry->offset == offsetof(Config, per_dir_filter) && config->per_dir_filter_count < INT_MAX) if (entry->offset == offsetof(Config, per_dir_filter) && config->per_dir_filter_count < INT_MAX)
config->per_dir_filter_count++; config->per_dir_filter_count++;
/* A delete-timing flag selects when --delete removes extras, so it /* A delete-timing flag selects when --delete removes extras, so it
implies --delete exactly like the rsync options do. */ implies --delete exactly like the rsync options do. --delete-commit (the
FastSync-only late-commit spelling) is mapped onto delete_after and so is
covered here too. */
if (entry->offset == offsetof(Config, delete_before) || if (entry->offset == offsetof(Config, delete_before) ||
entry->offset == offsetof(Config, delete_during) || entry->offset == offsetof(Config, delete_during) ||
entry->offset == offsetof(Config, delete_delay) || entry->offset == offsetof(Config, delete_delay) ||
@@ -2567,6 +2575,18 @@ static bool cli_handle_outbuf_option(CliParseCtx* ctx) {
* -1 on error. */ * -1 on error. */
static int cli_finalize_config(Config* config, bool verbose, bool no_delta, bool no_incremental) { static int cli_finalize_config(Config* config, bool verbose, bool no_delta, bool no_incremental) {
set_log_level(config->quiet ? LOG_LEVEL_ERROR : (verbose ? LOG_LEVEL_DEBUG : LOG_LEVEL_WARNING)); set_log_level(config->quiet ? LOG_LEVEL_ERROR : (verbose ? LOG_LEVEL_DEBUG : LOG_LEVEL_WARNING));
/* rsync's plain --delete defaults to delete-during (--del): each directory's
extras are removed as that directory is processed, so space is freed
progressively and a tight destination never has to hold the whole old+new
tree at once. The late whole-tree commit FastSync historically used is
still selected explicitly by --delete-after or by the FastSync-only long
spelling --delete-commit (an exact alias for --delete-after). Resolve the
default on the client, before validation and before the config crosses the
wire, so exactly one timing flag is ever set; an explicit timing (including
--delete-commit) always wins. */
if (config->use_delete && !config->delete_before && !config->delete_during &&
!config->delete_delay && !config->delete_after)
config->delete_during = true;
if (config->compress_choice) { if (config->compress_choice) {
int algo = compression_algo_from_name(config->compress_choice); int algo = compression_algo_from_name(config->compress_choice);
if (algo >= 0) { if (algo >= 0) {
+2 -1
View File
@@ -3287,7 +3287,8 @@ int send_files(Config* config) {
prepared.options.synced_dirs = synced_dirs; prepared.options.synced_dirs = synced_dirs;
} }
} }
/* The late-timing modes (plain --delete / --delete-after) build the manifest /* The late-timing modes (--delete-after/--delete-commit and a plain --delete
that fell back from per-dir mode because of -d/--dirs) build the manifest
while streaming and send it after the last data frame. --delete-before while streaming and send it after the last data frame. --delete-before
sends a whole-tree keep-set up front; --delete-during/--delete-delay build a sends a whole-tree keep-set up front; --delete-during/--delete-delay build a
per-directory plan set up front (paths only) and stream the plans alongside per-directory plan set up front (paths only) and stream the plans alongside
+6 -4
View File
@@ -62,8 +62,7 @@ void print_usage(void) {
printf(" NOTE: the FastSync batch format is NOT interoperable with rsync's batch\n"); printf(" NOTE: the FastSync batch format is NOT interoperable with rsync's batch\n");
printf(" files (different container format); do not mix the two tools.\n"); printf(" files (different container format); do not mix the two tools.\n");
printf(" --delete Delete files on receiver not in source\n"); printf(" --delete Delete files on receiver not in source\n");
printf(" (default timing: delete only after the whole\n"); printf(" (default timing: delete-during, like rsync --del)\n");
printf(" transfer has succeeded)\n");
printf(" --delete-before Delete extras before the transfer starts\n"); printf(" --delete-before Delete extras before the transfer starts\n");
printf(" (implies --delete)\n"); printf(" (implies --delete)\n");
printf(" --delete-during Delete a directory's extras as that directory is\n"); printf(" --delete-during Delete a directory's extras as that directory is\n");
@@ -72,7 +71,9 @@ void print_usage(void) {
printf(" --delete-delay Record the extras during the scan but remove them\n"); printf(" --delete-delay Record the extras during the scan but remove them\n");
printf(" only after a successful transfer (implies --delete)\n"); printf(" only after a successful transfer (implies --delete)\n");
printf(" --delete-after Delete only after the whole transfer succeeded\n"); printf(" --delete-after Delete only after the whole transfer succeeded\n");
printf(" (the default --delete timing; implies --delete)\n"); printf(" (implies --delete)\n");
printf(" --delete-commit FastSync-only: restore the late whole-tree commit\n");
printf(" (identical to --delete-after; implies --delete)\n");
printf(" --delete-excluded Also delete destination files that were excluded on\n"); printf(" --delete-excluded Also delete destination files that were excluded on\n");
printf(" the source (default protects them, matching rsync)\n"); printf(" the source (default protects them, matching rsync)\n");
printf(" --max-delete=NUM Delete at most NUM destination entries per run; if the\n"); printf(" --max-delete=NUM Delete at most NUM destination entries per run; if the\n");
@@ -93,7 +94,8 @@ void print_usage(void) {
printf(" -m, --prune-empty-dirs Do not create empty directories (a recursive transfer\n"); printf(" -m, --prune-empty-dirs Do not create empty directories (a recursive transfer\n");
printf(" otherwise recreates them, like rsync)\n"); printf(" otherwise recreates them, like rsync)\n");
printf(" Note: each timing flag implies --delete. Combining a timing flag with\n"); printf(" Note: each timing flag implies --delete. Combining a timing flag with\n");
printf(" --no-delete (in either order) is rejected as a config error.\n"); printf(" --no-delete (in either order) is rejected as a config error, as is more\n");
printf(" than one timing flag.\n");
printf(" --ignore-existing Skip files that already exist on receiver\n"); printf(" --ignore-existing Skip files that already exist on receiver\n");
printf(" --delay-updates Put updated files into place only at the end of transfer\n"); printf(" --delay-updates Put updated files into place only at the end of transfer\n");
printf(" --dirs, -d, --old-dirs, --old-d Transfer the named directory entries without\n"); printf(" --dirs, -d, --old-dirs, --old-d Transfer the named directory entries without\n");
+6 -4
View File
@@ -305,14 +305,16 @@ int receiver_process(Config* config, int file_descriptor, const ReceiverSink* si
/* Runs the whole receive loop. The delete manifest may legitimately arrive /* Runs the whole receive loop. The delete manifest may legitimately arrive
either FIRST (--delete-before / --delete-during: the sender transmits the either FIRST (--delete-before / --delete-during: the sender transmits the
validated keep-set before any file data) or LAST (plain --delete / validated keep-set before any file data) or LAST (--delete-after /
--delete-after / --delete-delay: the manifest closes the data stream). In --delete-commit / --delete-delay: the manifest closes the data stream). In
the early modes the receiver deletes as soon as the manifest has been read the early modes the receiver deletes as soon as the manifest has been read
and acknowledges with STATUS_OK so the sender only starts streaming once the and acknowledges with STATUS_OK so the sender only starts streaming once the
deletion has committed (or failed); in the late modes the manifest is held deletion has committed (or failed); in the late modes the manifest is held
and the deletion is committed only after the terminal STATUS_FINISHED proves and the deletion is committed only after the terminal STATUS_FINISHED proves
the whole transfer succeeded. See receiver_process_pending() for how the -m the whole transfer succeeded. A plain --delete defaults to the per-directory
receiver defers that commit until its disk writer has drained. */ delete-during plan mode (no manifest at all). See
receiver_process_pending() for how the -m receiver defers that commit until
its disk writer has drained. */
int receiver_process_pending(Config* config, int file_descriptor, const ReceiverSink* sink, int receiver_process_pending(Config* config, int file_descriptor, const ReceiverSink* sink,
DeleteManifest** pending_manifest, DeletePlanSession** pending_plans) { DeleteManifest** pending_manifest, DeletePlanSession** pending_plans) {
Status status; Status status;
+13 -6
View File
@@ -642,10 +642,14 @@ typedef struct Config {
source directory is streamed in directory order, and the receiver removes source directory is streamed in directory order, and the receiver removes
each directory's extras when its plan arrives (during) or snapshots them each directory's extras when its plan arrives (during) or snapshots them
and removes them only after a successful transfer (delay). delete_after and removes them only after a successful transfer (delay). delete_after
(and plain --delete) keep the whole-tree commit mode: extras are removed keeps the whole-tree commit mode: extras are removed from a fresh
from a fresh end-of-transfer destination scan only after the whole transfer end-of-transfer destination scan only after the whole transfer succeeded.
succeeded. See config_delete_timing_early()/config_delete_timing_per_dir() A plain --delete with no explicit timing flag defaults to delete_during on
below. */ the client (cli_finalize_config), matching rsync's --del default; the old
late-commit behavior is selected explicitly by --delete-after or the
FastSync-only long spelling --delete-commit (an exact alias for
--delete-after, mapped onto the same wire field). See
config_delete_timing_early()/config_delete_timing_per_dir() below. */
/* partial_dir */ /* partial_dir */
// PR #174: Partial transfer resumption // PR #174: Partial transfer resumption
/* suffix */ /* suffix */
@@ -1168,8 +1172,11 @@ bool config_delete_timing_early(const Config* config);
* commits them only after a fully-successful transfer (delay). */ * commits them only after a fully-successful transfer (delay). */
bool config_delete_timing_per_dir(const Config* config); bool config_delete_timing_per_dir(const Config* config);
/* Delete-timing sanity: with deletion enabled at most one timing flag may be /* Delete-timing sanity: with deletion enabled at most one timing flag may be
* set (none = the default delete-after commit timing); without deletion no * set; without deletion no timing flag may be set (each timing flag implies
* timing flag may be set (each timing flag implies --delete). */ * --delete). A plain --delete is normalized to delete_during by
* cli_finalize_config on the client, so a transmitted use_delete config always
* carries exactly one timing; the zero-timing case remains valid only for a
* config that has not been through the CLI. */
bool config_has_valid_delete_timing(const Config* config); bool config_has_valid_delete_timing(const Config* config);
/* Single source of truth for the cross-field ("combination") invariants a /* Single source of truth for the cross-field ("combination") invariants a
+38 -1
View File
@@ -331,6 +331,8 @@ static int send_plan_node(int fd, DeletePlanSender* sender, PlanNode* node) {
return -1; return -1;
sender->config_sent = true; sender->config_sent = true;
} }
if (!send_int(fd, 1)) /* apply = true */
return -1;
if (!send_wire_str(fd, node->dir)) if (!send_wire_str(fd, node->dir))
return -1; return -1;
if (send_str_section(fd, node->dirs) != 0 || send_str_section(fd, node->files) != 0) if (send_str_section(fd, node->dirs) != 0 || send_str_section(fd, node->files) != 0)
@@ -339,6 +341,29 @@ static int send_plan_node(int fd, DeletePlanSender* sender, PlanNode* node) {
return 0; return 0;
} }
/* Transmit the one-shot per-run config block (protected prefixes, size-pruned
* mirrors, --delete-missing-args exact paths) on its own carrier frame, with
* apply=false so the receiver consumes the config but walks nothing. This is
* how the config still reaches the receiver when the scope allows no directory
* plan at all (a --files-from list of bare files synchronizes no directory):
* without it, the missing-args exact deletions would be lost. Idempotent. */
static int send_config_only(int fd, DeletePlanSender* sender) {
if (!sender || sender->config_sent)
return 0;
if (!send_status(fd, STATUS_DELETE_PLAN) || !send_int(fd, 1))
return -1;
if (send_str_section(fd, sender->protected_prefixes) != 0 ||
send_str_section(fd, sender->size_skipped) != 0 ||
send_str_section(fd, sender->missing_args) != 0)
return -1;
sender->config_sent = true;
if (!send_int(fd, 0)) /* apply = false */
return -1;
if (!send_wire_str(fd, ".") || !send_int(fd, 0) || !send_int(fd, 0))
return -1;
return 0;
}
static int send_prefix_plan(int fd, DeletePlanSender* sender, const char* dir) { static int send_prefix_plan(int fd, DeletePlanSender* sender, const char* dir) {
PlanNode* node = plan_find(sender, dir); PlanNode* node = plan_find(sender, dir);
if (!node || node->sent) if (!node || node->sent)
@@ -354,6 +379,10 @@ int delete_plan_send_root(int fd, DeletePlanSender* sender) {
const char* root = sender->walk_root ? sender->walk_root : "."; const char* root = sender->walk_root ? sender->walk_root : ".";
if (!plan_ensure(sender, root)) if (!plan_ensure(sender, root))
return -1; return -1;
/* Put the config block on the wire first, on its own carrier frame, so the
receiver always sees it even when the scope permits no directory plan. */
if (send_config_only(fd, sender) != 0)
return -1;
return send_prefix_plan(fd, sender, root); return send_prefix_plan(fd, sender, root);
} }
@@ -850,6 +879,14 @@ int delete_plan_session_receive(DeletePlanSession* session, const Config* config
} }
session->config_seen = true; session->config_seen = true;
} }
/* apply=false is the config-only carrier frame: the receiver consumes the
config (and the missing-args exact deletions) but must not walk any
directory. Every real plan carries apply=true. */
int apply;
if (!receive_int(fd, &apply) || (apply != 0 && apply != 1)) {
send_status(fd, STATUS_ERROR);
return -1;
}
char* dir = receive_wire_str(fd); char* dir = receive_wire_str(fd);
ArrayList* dirs = array_list_create(free); ArrayList* dirs = array_list_create(free);
ArrayList* files = array_list_create(free); ArrayList* files = array_list_create(free);
@@ -867,7 +904,7 @@ int delete_plan_session_receive(DeletePlanSession* session, const Config* config
if (!session->dry_run && enabled) { if (!session->dry_run && enabled) {
if (!session->defer && !apply_missing(session, config)) if (!session->defer && !apply_missing(session, config))
ok = false; ok = false;
if (ok && !apply_plan_dir(session, config, dir, dirs, files)) if (ok && apply && !apply_plan_dir(session, config, dir, dirs, files))
ok = false; ok = false;
} }
free(dir); free(dir);
+5 -2
View File
@@ -48,11 +48,14 @@ void delete_plan_sender_finalize(DeletePlanSender* sender, const ArrayList* sync
Directory keep entries do not count, so an I/O error that hid every file Directory keep entries do not count, so an I/O error that hid every file
still refuses to delete. */ still refuses to delete. */
bool delete_plan_sender_empty(const DeletePlanSender* sender); bool delete_plan_sender_empty(const DeletePlanSender* sender);
/* Attach the global config sections advertised on the first plan frame. */ /* Attach the global config sections advertised on the first plan frame. The
* block is always transmitted by delete_plan_send_root(), on a config-only
* carrier frame when the scope allows no directory plan. */
void delete_plan_sender_set_config(DeletePlanSender* sender, const ArrayList* protected_prefixes, void delete_plan_sender_set_config(DeletePlanSender* sender, const ArrayList* protected_prefixes,
const ArrayList* size_skipped, const ArrayList* missing_args); const ArrayList* size_skipped, const ArrayList* missing_args);
/* Send the root plan (even before any data, so root extras are handled like /* Send the root plan (even before any data, so root extras are handled like
* rsync's first generator directory). Returns -1 on I/O error. */ * rsync's first generator directory), after transmitting the per-run config
* block on its own carrier frame. Returns -1 on I/O error. */
int delete_plan_send_root(int fd, DeletePlanSender* sender); int delete_plan_send_root(int fd, DeletePlanSender* sender);
/* Send the plans for every ancestor of `path` (root-first) and, when is_dir, /* Send the plans for every ancestor of `path` (root-first) and, when is_dir,
* for `path` itself; already-sent plans are skipped. */ * for `path` itself; already-sent plans are skipped. */
+3 -1
View File
@@ -191,7 +191,9 @@ enum NET_STATUS {
* (--delete-delay). Payload: an int32 has_config flag (1 on the first plan * (--delete-delay). Payload: an int32 has_config flag (1 on the first plan
* of the run, 0 afterwards); when set, the three global config sections * of the run, 0 afterwards); when set, the three global config sections
* (protected-prefix count+paths, size-skipped count+paths, missing-args * (protected-prefix count+paths, size-skipped count+paths, missing-args
* count+paths); then the destination-relative directory path wire string * count+paths); then an int32 apply flag (1 for a real plan, 0 for a
* config-only carrier frame that must not walk a directory); then the
* destination-relative directory path wire string
* ("." for the receive root); then the child-directory count + names and the * ("." for the receive root); then the child-directory count + names and the
* child-file count + names that must be kept. Appended after * child-file count + names that must be kept. Appended after
* STATUS_DEST_INFO so no existing status is renumbered. */ * STATUS_DEST_INFO so no existing status is renumbered. */
+45 -18
View File
@@ -217,7 +217,21 @@ class _SlicingProxy:
class TestDeleteTimingFinalStateParity: class TestDeleteTimingFinalStateParity:
"""On a successful transfer the per-directory timings match rsync's result.""" """On a successful transfer the per-directory timings match rsync's result.
Plain ``--delete`` has no rsync-incompatible spelling: it defaults to
delete-during on both tools, so it is compared against rsync's own default.
``--delete-commit`` is FastSync-only and selects the late whole-tree commit,
which is rsync's ``--delete-after`` timing.
"""
# (fastsync flag, rsync flag)
PAIRS = [
("--delete", "--delete"),
("--delete-during", "--delete-during"),
("--delete-delay", "--delete-delay"),
("--delete-commit", "--delete-after"),
]
def _run_fastsync(self, tag, timing): def _run_fastsync(self, tag, timing):
source, dest, received = _seed_pair(tag) source, dest, received = _seed_pair(tag)
@@ -226,12 +240,12 @@ class TestDeleteTimingFinalStateParity:
result, _ = run_client(source, dest, flags=[timing], port=server.port) result, _ = run_client(source, dest, flags=[timing], port=server.port)
return result, received return result, received
@pytest.mark.parametrize("timing", ["--delete-during", "--delete-delay"]) @pytest.mark.parametrize("fs_timing,rs_timing", PAIRS)
@requires_rsync @requires_rsync
def test_success_final_state_matches_rsync(self, timing): def test_success_final_state_matches_rsync(self, fs_timing, rs_timing):
# Worker-safe names: xdist may run both parametrizations concurrently, so # Worker-safe names: xdist may run the parametrizations concurrently, so
# the timing is part of every fixture path. # the flags are part of every fixture path.
label = timing.lstrip("-") label = f"{fs_timing.lstrip('-')}_vs_{rs_timing.lstrip('-')}"
# Build the rsync fixture from the same seed so both sides start equal. # Build the rsync fixture from the same seed so both sides start equal.
source, dest, received = _seed_pair(f"parity_rsync_{label}") source, dest, received = _seed_pair(f"parity_rsync_{label}")
source2 = source source2 = source
@@ -240,17 +254,18 @@ class TestDeleteTimingFinalStateParity:
# rsync mirrors src/ into dst/; seed the same extra. # rsync mirrors src/ into dst/; seed the same extra.
_write(os.path.join(rsync_dst, "d", "old_extra"), b"stale extra\n") _write(os.path.join(rsync_dst, "d", "old_extra"), b"stale extra\n")
rsync_result = _rsync(["-a", timing, source2 + "/", rsync_dst + "/"]) rsync_result = _rsync(["-a", rs_timing, source2 + "/", rsync_dst + "/"])
assert rsync_result.returncode == 0, rsync_result.stderr assert rsync_result.returncode == 0, rsync_result.stderr
rsync_tree = _tree(rsync_dst) rsync_tree = _tree(rsync_dst)
with ServerManager() as server: with ServerManager() as server:
server.start(extra_args=["--allow-delete"]) server.start(extra_args=["--allow-delete"])
result, _ = run_client(source, dest, flags=[timing], port=server.port) result, _ = run_client(source, dest, flags=[fs_timing], port=server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300] assert result.returncode == 0, (result.stderr or result.stdout)[:300]
fastsync_tree = _tree(received) fastsync_tree = _tree(received)
assert fastsync_tree == rsync_tree, ( assert fastsync_tree == rsync_tree, (
f"{timing}: fastsync tree {fastsync_tree} != rsync tree {rsync_tree}" f"{fs_timing} vs rsync {rs_timing}: fastsync tree {fastsync_tree} != "
f"rsync tree {rsync_tree}"
) )
@@ -292,7 +307,14 @@ class TestDeleteTimingTypeConflictParity:
class TestDeleteTimingFailure: class TestDeleteTimingFailure:
"""A mid-transfer failure distinguishes during from delay.""" """A mid-transfer failure distinguishes the during timings from the late
commit timings.
Plain ``--delete`` must behave like ``--delete-during`` (the rsync default),
removing the extras of the directories already reached; ``--delete-commit``
must behave like ``--delete-after`` and remove nothing until the transfer
has fully succeeded.
"""
@pytest.mark.parametrize("mt", [False, True]) @pytest.mark.parametrize("mt", [False, True])
def test_during_removes_delay_preserves_on_failure(self, mt): def test_during_removes_delay_preserves_on_failure(self, mt):
@@ -301,8 +323,12 @@ class TestDeleteTimingFailure:
assert os.path.exists(extra) assert os.path.exists(extra)
with ServerManager() as server: with ServerManager() as server:
server.start(extra_args=["--allow-delete"]) server.start(extra_args=["--allow-delete"])
for timing, expect_removed in (("--delete-during", True), for timing, expect_removed in (
("--delete-delay", False)): ("--delete-during", True),
("--delete", True),
("--delete-delay", False),
("--delete-commit", False),
("--delete-after", False)):
# Re-seed the extra before each run. # Re-seed the extra before each run.
_write(extra, b"stale extra\n") _write(extra, b"stale extra\n")
proxy = _SlicingProxy(server.port, forward_limit=MID_TRANSFER_BYTES, throttle=PROXY_THROTTLE) proxy = _SlicingProxy(server.port, forward_limit=MID_TRANSFER_BYTES, throttle=PROXY_THROTTLE)
@@ -449,12 +475,13 @@ class TestDeleteDelayVsAfterSnapshot:
class TestDeleteAfterThreadsKeepSet: class TestDeleteAfterThreadsKeepSet:
"""Regression: -m/--threads with the default delete-after timing (plain """Regression: -j/--threads must still transmit the delete keep-set in every
--delete) must still transmit the keep-set manifest and remove destination timing. PipelineContextSender.delete_suppressed was left uninitialized, so a
extras. PipelineContextSender.delete_suppressed was left uninitialized, so a garbage true silently skipped the late keep-set manifest under --threads.
garbage true silently skipped the manifest under --threads.""" Plain --delete now uses the per-directory plans, while --delete-commit /
--delete-after keep exercising the late whole-tree manifest."""
@pytest.mark.parametrize("delete_flag", ["--delete", "--delete-after"]) @pytest.mark.parametrize("delete_flag", ["--delete", "--delete-commit", "--delete-after"])
def test_threads_delete_after_sends_keep_set(self, delete_flag): def test_threads_delete_after_sends_keep_set(self, delete_flag):
source, dest, received = _seed_pair("mtkeep") source, dest, received = _seed_pair("mtkeep")
extra = os.path.join(received, "d", "old_extra") extra = os.path.join(received, "d", "old_extra")
@@ -465,7 +492,7 @@ class TestDeleteAfterThreadsKeepSet:
port=server.port) port=server.port)
assert result.returncode == 0, (result.stderr or result.stdout)[:300] assert result.returncode == 0, (result.stderr or result.stdout)[:300]
assert not os.path.exists(extra), ( assert not os.path.exists(extra), (
f"{delete_flag} --threads did not remove an extra: keep-set manifest was suppressed" f"{delete_flag} --threads did not remove an extra: delete keep-set was suppressed"
) )
class TestDeleteDelayMaxDeleteRefilledDir: class TestDeleteDelayMaxDeleteRefilledDir:
@@ -241,6 +241,9 @@ _CASES = [
server_args=DELETE, ref="--delete-delay"), server_args=DELETE, ref="--delete-delay"),
H.Case("delete_after", "basic", ["-a", "--delete-after"], seed=seed_extras, H.Case("delete_after", "basic", ["-a", "--delete-after"], seed=seed_extras,
server_args=DELETE, ref="--delete-after"), server_args=DELETE, ref="--delete-after"),
H.Case("delete_commit", "basic", ["-a", "--delete-after"], seed=seed_extras,
fastsync_flags=["-a", "--delete-commit"], server_args=DELETE,
ref="FastSync-only --delete-commit == rsync --delete-after"),
H.Case("delete_excluded", "filters", H.Case("delete_excluded", "filters",
["-a", "--delete", "--delete-excluded", "--exclude=*.log"], ["-a", "--delete", "--delete-excluded", "--exclude=*.log"],
seed=seed_delete_excluded, server_args=DELETE, ref="--delete-excluded"), seed=seed_delete_excluded, server_args=DELETE, ref="--delete-excluded"),
@@ -255,7 +258,7 @@ _CASES = [
H.Case("filter_protect", "filters", H.Case("filter_protect", "filters",
["-a", "--delete", "--filter=P *.log"], ["-a", "--delete", "--filter=P *.log"],
seed=seed_filter_protect, server_args=DELETE, ci=True, seed=seed_filter_protect, server_args=DELETE, ci=True,
ref="--filter P/--protect receiver-side delete protection"), ref="--filter P/--protect receiver-side delete protection (default during)"),
H.Case("filter_protect_during", "filters", H.Case("filter_protect_during", "filters",
["-a", "--delete-during", "--filter=P *.log"], ["-a", "--delete-during", "--filter=P *.log"],
seed=seed_filter_protect, server_args=DELETE, ci=True, seed=seed_filter_protect, server_args=DELETE, ci=True,
@@ -264,6 +267,10 @@ _CASES = [
["-a", "--delete-delay", "--filter=P *.log"], ["-a", "--delete-delay", "--filter=P *.log"],
seed=seed_filter_protect, server_args=DELETE, ci=True, seed=seed_filter_protect, server_args=DELETE, ci=True,
ref="--filter P/--protect under --delete-delay"), ref="--filter P/--protect under --delete-delay"),
H.Case("filter_protect_after", "filters",
["-a", "--delete-after", "--filter=P *.log"],
seed=seed_filter_protect, server_args=DELETE, ci=True,
ref="--filter P/--protect under the whole-tree --delete-after commit"),
# --- relative / dirs -------------------------------------------------- # --- relative / dirs --------------------------------------------------
H.Case("relative_general", "basic", ["-a", "-R"], layout=H.MIRROR_ABS, H.Case("relative_general", "basic", ["-a", "-R"], layout=H.MIRROR_ABS,
+7 -6
View File
@@ -3982,15 +3982,16 @@ class TestDeleteTiming:
assert _read_file(os.path.join(received, "sub", "deep.txt")) == b"deeply nested file\n", \ assert _read_file(os.path.join(received, "sub", "deep.txt")) == b"deeply nested file\n", \
f"{flag}: nested file was not written after the early deletion" f"{flag}: nested file was not written after the early deletion"
@pytest.mark.parametrize("flag", ["--delete", "--delete-after"]) @pytest.mark.parametrize("flag", ["--delete-commit", "--delete-after"])
@pytest.mark.parametrize("mt", [False, True]) @pytest.mark.parametrize("mt", [False, True])
def test_late_flags_commit_only_after_success(self, flag, mt): def test_late_flags_commit_only_after_success(self, flag, mt):
"""Plain --delete/--delete-after defer deletion until the whole transfer """--delete-commit/--delete-after defer deletion until the whole transfer
succeeds: a mid-transfer write failure must leave every extra in place succeeds: a mid-transfer write failure must leave every extra in place
(commit-style safety). The -m receiver must also keep the extras: the (commit-style safety). Plain --delete no longer defers (it defaults to
deferred keep-set is committed by the server only after the disk-writer delete-during), so only the explicitly late timings are exercised here.
thread has finished, and a failing writer means the manifest is freed, The --threads receiver must also keep the extras: the deferred keep-set is
never applied.""" committed by the server only after the disk-writer thread has finished,
and a failing writer means the manifest is freed, never applied."""
source = self._seed("late") source = self._seed("late")
dest = os.path.join(TEST_DATA_DIR, "deltiming_late_dst") dest = os.path.join(TEST_DATA_DIR, "deltiming_late_dst")
clean_dir(dest) clean_dir(dest)
+58
View File
@@ -1150,6 +1150,63 @@ static void test_parse_args_delete_timing_flags() {
config_delete(cfg); config_delete(cfg);
} }
/* Plain --delete with no explicit timing defaults to delete-during, matching
* rsync's --del default (progressive deletion). --delete-commit is the
* FastSync-only long spelling that selects rsync's --delete-after timing (the
* late whole-tree commit), and an explicit timing always wins over the default.
*/
static void test_parse_args_delete_default_timing_and_commit() {
Config* cfg = config_create();
char* argv[] = {"fastsync", "--delete", "/src", "/dst"};
int positional_args[2];
int positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->use_delete);
EXPECT_TRUE(cfg->delete_during);
EXPECT_FALSE(cfg->delete_before);
EXPECT_FALSE(cfg->delete_delay);
EXPECT_FALSE(cfg->delete_after);
cfg->send_directory = str_dup("/src");
cfg->receive_root_directory = str_dup("/dst");
EXPECT_TRUE(validate_config(cfg));
config_delete(cfg);
/* --delete-commit selects the late whole-tree commit (delete_after) and
implies --delete. */
cfg = config_create();
char* argv_commit[] = {"fastsync", "--delete-commit", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 4, argv_commit, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->use_delete);
EXPECT_TRUE(cfg->delete_after);
EXPECT_FALSE(cfg->delete_before);
EXPECT_FALSE(cfg->delete_during);
EXPECT_FALSE(cfg->delete_delay);
cfg->send_directory = str_dup("/src");
cfg->receive_root_directory = str_dup("/dst");
EXPECT_TRUE(validate_config(cfg));
config_delete(cfg);
/* An explicit --delete-after alongside plain --delete keeps the late timing:
the default never overwrites an explicit timing. */
cfg = config_create();
char* argv_after[] = {"fastsync", "--delete", "--delete-after", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv_after, positional_args, &positional_count), 0);
EXPECT_TRUE(cfg->use_delete);
EXPECT_TRUE(cfg->delete_after);
EXPECT_FALSE(cfg->delete_during);
config_delete(cfg);
/* --delete-commit conflicts with a different timing. */
cfg = config_create();
char* argv_conflict[] = {"fastsync", "--delete-commit", "--delete-during", "/src", "/dst"};
positional_count = 0;
EXPECT_EQ_INT(parse_args(cfg, 5, argv_conflict, positional_args, &positional_count), 0);
EXPECT_FALSE(validate_config(cfg));
config_delete(cfg);
}
/* Two different delete-timing flags on one command line are a conflict, not a /* Two different delete-timing flags on one command line are a conflict, not a
* silent last-one-wins choice. */ * silent last-one-wins choice. */
static void test_parse_args_delete_timing_conflict_rejected() { static void test_parse_args_delete_timing_conflict_rejected() {
@@ -4758,6 +4815,7 @@ void test_client_cli() {
test_parse_args_relative_no_implied_mkpath(); test_parse_args_relative_no_implied_mkpath();
test_parse_args_delete_during_alias(); test_parse_args_delete_during_alias();
test_parse_args_delete_timing_flags(); test_parse_args_delete_timing_flags();
test_parse_args_delete_default_timing_and_commit();
test_parse_args_delete_timing_conflict_rejected(); test_parse_args_delete_timing_conflict_rejected();
test_parse_args_delete_timing_without_delete_rejected(); test_parse_args_delete_timing_without_delete_rejected();
test_parse_args_rejects_unimplemented_options(); test_parse_args_rejects_unimplemented_options();
+1
View File
@@ -17,6 +17,7 @@
* caller/receiver entry point) describing `dir` with no kept children. */ * caller/receiver entry point) describing `dir` with no kept children. */
static void send_plan_frame(int fd, const char* dir) { static void send_plan_frame(int fd, const char* dir) {
EXPECT_TRUE(send_int(fd, 0)); /* has_config */ EXPECT_TRUE(send_int(fd, 0)); /* has_config */
EXPECT_TRUE(send_int(fd, 1)); /* apply: a real plan */
EXPECT_TRUE(send_wire_str(fd, dir)); EXPECT_TRUE(send_wire_str(fd, dir));
EXPECT_TRUE(send_int(fd, 0)); /* kept child dirs */ EXPECT_TRUE(send_int(fd, 0)); /* kept child dirs */
EXPECT_TRUE(send_int(fd, 0)); /* kept child files */ EXPECT_TRUE(send_int(fd, 0)); /* kept child files */
+1
View File
@@ -1291,6 +1291,7 @@ static void test_dry_run_delete_plan_commit_does_not_delete() {
EXPECT_TRUE(send_int(p[1], 0)); /* size-skipped prefixes */ EXPECT_TRUE(send_int(p[1], 0)); /* size-skipped prefixes */
EXPECT_TRUE(send_int(p[1], 1)); /* missing-args exact deletions */ EXPECT_TRUE(send_int(p[1], 1)); /* missing-args exact deletions */
EXPECT_TRUE(send_str(p[1], "victim.txt")); EXPECT_TRUE(send_str(p[1], "victim.txt"));
EXPECT_TRUE(send_int(p[1], 1)); /* apply: a real plan */
EXPECT_TRUE(send_str(p[1], ".")); /* receive root plan */ EXPECT_TRUE(send_str(p[1], ".")); /* receive root plan */
EXPECT_TRUE(send_int(p[1], 0)); /* kept child directories */ EXPECT_TRUE(send_int(p[1], 0)); /* kept child directories */
EXPECT_TRUE(send_int(p[1], 0)); /* kept child files */ EXPECT_TRUE(send_int(p[1], 0)); /* kept child files */