Compare commits
10
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
cd7b96d0bb | ||
|
|
f6f49d536e | ||
|
|
38d304103c | ||
|
|
4b09213b88 | ||
|
|
36fd0774e8 | ||
|
|
711b7e50b3 | ||
|
|
67076bf218 | ||
|
|
8ec8cb7203 | ||
|
|
82959395fb | ||
|
|
c9f94ea46e |
@@ -4,6 +4,37 @@ All notable changes to FastSync are documented here. Versions match
|
|||||||
`PROTOCOL_VERSION` (printed by `fastsync --version`); the client and server must
|
`PROTOCOL_VERSION` (printed by `fastsync --version`); the client and server must
|
||||||
run the same version because the handshake is strict.
|
run the same version because the handshake is strict.
|
||||||
|
|
||||||
|
## [Unreleased]
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- **`--delete` now defaults to delete-during (rsync `--del`) timing.** With no
|
||||||
|
explicit timing flag, a plain `--delete` removes each directory's extras as
|
||||||
|
that directory is processed instead of committing one whole-tree deletion only
|
||||||
|
after the entire transfer succeeds. This matches rsync, frees destination
|
||||||
|
space progressively, and avoids the whole-old+new-tree peak that could
|
||||||
|
`ENOSPC` a tight destination. The client maps the default onto the existing
|
||||||
|
`delete_during` wire boolean, so `PROTOCOL_VERSION` stays `2.28.0`.
|
||||||
|
- Added the FastSync-only long option **`--delete-commit`** (implies
|
||||||
|
`--delete`): it selects the old late whole-tree commit and is timing-identical
|
||||||
|
to `--delete-after` (the same `delete_after` wire boolean). Explicit timing
|
||||||
|
flags always win over the default, at most one timing flag may be given, and a
|
||||||
|
timing flag combined with `--no-delete` is still rejected.
|
||||||
|
- The per-directory `STATUS_DELETE_PLAN` frame gained a one-int `apply` flag:
|
||||||
|
the one-shot per-run config block (protected prefixes, size-pruned mirrors,
|
||||||
|
`--delete-missing-args` exact paths) is now always transmitted first on a
|
||||||
|
config-only carrier (`apply=false`), fixing a latent bug where a
|
||||||
|
`--delete-missing-args` run whose `--files-from` list synchronized no directory
|
||||||
|
never sent its exact deletions.
|
||||||
|
|
||||||
|
### Migration
|
||||||
|
|
||||||
|
- Scripts that relied on plain `--delete` deleting nothing until the transfer
|
||||||
|
fully succeeded must pass **`--delete-commit`** (or `--delete-after`) to keep
|
||||||
|
that behavior. Plain `--delete` now removes reached directories' extras during
|
||||||
|
the transfer, exactly like rsync's default; on a completed run the final tree
|
||||||
|
is unchanged.
|
||||||
|
|
||||||
## [2.26.0] - 2026-09-17
|
## [2.26.0] - 2026-09-17
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
|
|||||||
+87
-5
@@ -52,9 +52,10 @@
|
|||||||
`tests/test_config.c`). Two residuals were reclassified **divergent**: `-M`
|
`tests/test_config.c`). Two residuals were reclassified **divergent**: `-M`
|
||||||
over daemon/TCP (no argv channel in FastSync's binary config handshake;
|
over daemon/TCP (no argv channel in FastSync's binary config handshake;
|
||||||
rsync-daemon differential pins the rsync behavior) and receiver-side
|
rsync-daemon differential pins the rsync behavior) and receiver-side
|
||||||
`protect`/`risk` re-derivation for destination-only entries (would need a
|
`protect`/`risk` re-derivation for destination-only entries (would need a
|
||||||
receiver filter engine; differential pins the divergence). The options pass
|
receiver filter engine; differential pins the divergence — **reversed by
|
||||||
stands at **110 ✅ / 21 ⚠️ / 26 ❌**. New `tests/integration/test_option_parity.py`
|
track 4a below**, which adds that engine). The options pass
|
||||||
|
stands at **110 ✅ / 21 ⚠️ / 26 ❌**. New `tests/integration/test_option_parity.py`
|
||||||
holds the rsync differentials (bwlimit parse+rate, info lines, real-setpriv
|
holds the rsync differentials (bwlimit parse+rate, info lines, real-setpriv
|
||||||
`--ignore-errors`, rsync-daemon `-M`, filter-protect pin).
|
`--ignore-errors`, rsync-daemon `-M`, filter-protect pin).
|
||||||
|
|
||||||
@@ -74,7 +75,8 @@
|
|||||||
heuristic with a 10× size window, not rsync's matcher), but its residual is the
|
heuristic with a 10× size window, not rsync's matcher), but its residual is the
|
||||||
candidate-selection heuristic itself: the final tree is byte-exact by design, so
|
candidate-selection heuristic itself: the final tree is byte-exact by design, so
|
||||||
it is pinned by the `TestFuzzy` threshold suite rather than a byte-level rsync
|
it is pinned by the `TestFuzzy` threshold suite rather than a byte-level rsync
|
||||||
differential. The parity-review pass then moved `--delete-delay` to ⚠️ (the
|
differential. (Track 5b later found the name heuristic is rsync's own and moved
|
||||||
|
the row ❌ → ⚠️, leaving only the narrower delta size window; see entry 15.) The parity-review pass then moved `--delete-delay` to ⚠️ (the
|
||||||
plan-time `--max-delete` charge and non-recursive deferred removal differ from
|
plan-time `--max-delete` charge and non-recursive deferred removal differ from
|
||||||
rsync when a snapshotted entry fails removal). Differential-gate allowlist
|
rsync when a snapshotted entry fails removal). Differential-gate allowlist
|
||||||
entries `min_size`/`empty_dirs_recursive`/`dirs_plain` were removed. The
|
entries `min_size`/`empty_dirs_recursive`/`dirs_plain` were removed. The
|
||||||
@@ -85,7 +87,8 @@
|
|||||||
`-n --delete` now sends the same filter-excluded + size-pruned protected
|
`-n --delete` now sends the same filter-excluded + size-pruned protected
|
||||||
prefixes and synchronized-directory scope as a real run (dry-run would-delete
|
prefixes and synchronized-directory scope as a real run (dry-run would-delete
|
||||||
matches rsync for source-derived protections; the destination-only exclude
|
matches rsync for source-derived protections; the destination-only exclude
|
||||||
residual and readdir ordering remain); `--delete-delay` now charges
|
residual was later closed by track 4a, readdir ordering remains);
|
||||||
|
`--delete-delay` now charges
|
||||||
`--max-delete` on actual removals and re-scans a queued directory at commit
|
`--max-delete` on actual removals and re-scans a queued directory at commit
|
||||||
to remove content created after the plan, with an independent deferred-list
|
to remove content created after the plan, with an independent deferred-list
|
||||||
cap (only partial-delete ordering remains); and `--info=name2` emits `NAME is
|
cap (only partial-delete ordering remains); and `--info=name2` emits `NAME is
|
||||||
@@ -110,6 +113,85 @@
|
|||||||
caveats, so the row stays ⚠️ and the matrix is unchanged at
|
caveats, so the row stays ⚠️ and the matrix is unchanged at
|
||||||
**111 ✅ / 14 ⚠️ / 32 ❌ = 157**.
|
**111 ✅ / 14 ⚠️ / 32 ❌ = 157**.
|
||||||
|
|
||||||
|
13. **Wire parity track 4a** on `feat/parity-2.28` (`PROTOCOL_VERSION` stays
|
||||||
|
`2.28.0`): the receiver now has a delete-time filter engine. The sender
|
||||||
|
compiles its root-level selection rules exactly as the scanner does
|
||||||
|
(`filter_base_build`) and streams them as one bounded, self-describing
|
||||||
|
config-frame block (action, sides, anchored, dir-only, negate, owner,
|
||||||
|
pattern; bounded rule count and pattern bytes, unknown action/sides is a
|
||||||
|
protocol error). The receiver reconstructs `protect_rules` and applies them
|
||||||
|
first-match-wins to each extraneous destination path in every delete timing
|
||||||
|
(the whole-tree commit walker, the `--delete-during`/`--delete-delay`
|
||||||
|
per-directory plans, and the `-n` would-delete enumeration), so a
|
||||||
|
`P *.log` rule protects a destination-only `extra.log` like rsync (with
|
||||||
|
`risk` cancelling); the sender-derived protected-prefix behavior is
|
||||||
|
preserved when no rules are sent and `--delete-excluded` semantics are
|
||||||
|
unchanged. Per-directory merge (`:`/`.`) receiver re-derivation remains the
|
||||||
|
residual. `TestFilterProtect` (real + dry-run) plus differential cases
|
||||||
|
`filter_protect`, `filter_protect_during`, `filter_protect_delay` added and
|
||||||
|
the `--filter=RULE` row moves ❌ → ✅: matrix now
|
||||||
|
**115 ✅ / 11 ⚠️ / 31 ❌ = 157**; unit tests, the three named integration
|
||||||
|
files, clang-format and cppcheck clean.
|
||||||
|
|
||||||
|
14. **Wire parity track 5a** on `feat/parity-2.28` (`PROTOCOL_VERSION` stays
|
||||||
|
`2.28.0` by project decision): the three basis-dir options now default to
|
||||||
|
rsync's metadata quick-check (equal size + equal mtime, or size alone under
|
||||||
|
`--size-only`; `-I` disables matching) instead of FastSync's historical
|
||||||
|
xxHash64 content equality, so a same-size/different-content basis is trusted
|
||||||
|
exactly as rsync trusts it. A new FastSync-only, long-only `--verify-basis`
|
||||||
|
flag restores the strict whole-file content equality; its bool is appended to
|
||||||
|
the basis block of the config frame (golden wire frame 882 → 886 bytes).
|
||||||
|
`--verify-basis` streams the confined basis descriptor to hash it, and a
|
||||||
|
basis hit is no longer capped at the 256 MiB whole-file payload bound:
|
||||||
|
`--copy-dest` streams the basis through a bounded buffer and `--link-dest`'s
|
||||||
|
copy fallback streams from the basis, so an over-limit hit materializes (a
|
||||||
|
basis MISS still falls back to the normal transfer and keeps its own bound).
|
||||||
|
A `--copy-dest` hit re-applies the SOURCE attributes (the sender transmits
|
||||||
|
the source metadata with the basis check frame), matching rsync's
|
||||||
|
"copy then fix attributes"; a `--link-dest` success keeps the shared inode's
|
||||||
|
attributes (writing through it would mutate the basis). Differential cases
|
||||||
|
`copy_dest` and `verify_basis` added; `test_basis_dir_size_only_content_residual`
|
||||||
|
converted to a passing parity assertion; `TestBasisDestDirs` updated for the
|
||||||
|
new default + `--verify-basis`; unit tests cover the quick-check/verify
|
||||||
|
decision and the same-size/different-content handshake. The
|
||||||
|
`--compare-dest`/`--copy-dest`/`--link-dest` rows move ❌ → ⚠️ (relative-DIR
|
||||||
|
resolution base and over-limit MISS refusal): matrix now
|
||||||
|
**116 ✅ / 13 ⚠️ / 28 ❌ = 157**.
|
||||||
|
|
||||||
|
15. **No-wire parity track 5b** on `feat/parity-2.28` (`PROTOCOL_VERSION` stays
|
||||||
|
`2.28.0` by project decision): `-y`/`--fuzzy` reclassified ❌ → ⚠️. A probe
|
||||||
|
against real rsync 3.4.1 (pinned `-B8192`, repeated-content 64 KiB corpus)
|
||||||
|
showed the name heuristic is already rsync's (`util1.c fuzzy_distance` /
|
||||||
|
`find_filename_suffix` + the exact size+mtime pass) and the output is always
|
||||||
|
byte-exact; the only residual is candidate ELIGIBILITY, because FastSync's
|
||||||
|
`delta_should_attempt` gate caps the size ratio at 10× and requires both
|
||||||
|
files ≥ 16 KiB while rsync will reuse a basis from 0.25× to 10000× and below
|
||||||
|
16 KiB. The choice is observable only as `--stats` bandwidth counters. Added
|
||||||
|
differential case `fuzzy_basis` (same-suffix sibling, one name edit,
|
||||||
|
identical content, block size pinned) asserting tree **and** normalized
|
||||||
|
`--stats` parity where the choices coincide, plus `TestFuzzy` pinning the
|
||||||
|
window boundary on both sides (>10× and <16 KiB siblings declined by
|
||||||
|
FastSync while rsync uses them, both trees byte-identical). Matrix now
|
||||||
|
**116 ✅ / 14 ⚠️ / 27 ❌ = 157**.
|
||||||
|
|
||||||
|
16. **Lockstep delete-default track 6** on `feat/parity-2.28` (`PROTOCOL_VERSION`
|
||||||
|
stays `2.28.0`): plain `--delete` now defaults to rsync's delete-during
|
||||||
|
(`--del`) timing, normalized on the client onto the existing `delete_during`
|
||||||
|
wire bool. The old late whole-tree commit is opt-in via `--delete-after` or
|
||||||
|
the FastSync-only long `--delete-commit` (identical `delete_after` timing).
|
||||||
|
`-d/--dirs` still falls back to the end commit, `--delay-updates` still
|
||||||
|
deletes before publication, and `--files-from`/`-R` scope is unchanged. The
|
||||||
|
`STATUS_DELETE_PLAN` frame gained a one-int `apply` flag so the per-run
|
||||||
|
config block (including `--delete-missing-args` exact paths) is always
|
||||||
|
transmitted, on a config-only carrier when the scope allows no directory
|
||||||
|
plan — fixing a latent bug with a file-only `--files-from` list. Differential
|
||||||
|
cases `delete`/`delete_commit`/`filter_protect_after` plus the extended
|
||||||
|
`test_delete_timing_parity.py` (plain `--delete` mid-abort removes reached
|
||||||
|
extras, `--delete-commit` defers) pass; full `-m "not setpriv"` suite,
|
||||||
|
clang-format and cppcheck clean. Matrix unchanged at
|
||||||
|
**116 ✅ / 14 ⚠️ / 27 ❌ = 157** (the `--delete`/`--delete-during` rows stay
|
||||||
|
⚠️ for the abort boundary; `--delete-after` stays ✅).
|
||||||
|
|
||||||
## Next steps
|
## Next steps
|
||||||
1. **Merge PR #284** (`dev` -> `main`) once reviewed (protected branch).
|
1. **Merge PR #284** (`dev` -> `main`) once reviewed (protected branch).
|
||||||
2. **Deferred security items** (documented, not implemented):
|
2. **Deferred security items** (documented, not implemented):
|
||||||
|
|||||||
@@ -113,9 +113,11 @@ matrix is classified as parity, caveat, or divergent in
|
|||||||
(`-B1000`, `-essh`, `-MOPT`, `--opt=value`) are accepted, matching rsync.
|
(`-B1000`, `-essh`, `-MOPT`, `--opt=value`) are accepted, matching rsync.
|
||||||
- `-r`, `-b`, `-L`, and `-B` are parsed with the rsync short names.
|
- `-r`, `-b`, `-L`, and `-B` are parsed with the rsync short names.
|
||||||
- `--stats` prints the counters FastSync can observe plus the receiver-only
|
- `--stats` prints the counters FastSync can observe plus the receiver-only
|
||||||
counters (`Matched data`, deleted files) reported over the wire; rsync's
|
counters reported over the wire (`Matched data`, deleted files, and the
|
||||||
per-type `Number of files` breakdown is not reproduced. `--progress` prints
|
created/literal counters); `Number of files` and `Number of created files`
|
||||||
rsync-style per-file blocks (without rsync's leading `./` line).
|
carry rsync's per-type breakdown. `--progress` prints rsync-style per-file
|
||||||
|
blocks including the leading `./` line, and (when progress is requested) a
|
||||||
|
paths-only pre-count supplies rsync's `to-chk` denominator.
|
||||||
- Codecs match rsync 3.4.1: `zstd`/`lz4`/`zlib`/`zlibx` compression and
|
- Codecs match rsync 3.4.1: `zstd`/`lz4`/`zlib`/`zlibx` compression and
|
||||||
`xxh128`/`xxh3`/`xxh64`/`md5`/`md4`/`sha1`/`none` checksums. `auto` honors
|
`xxh128`/`xxh3`/`xxh64`/`md5`/`md4`/`sha1`/`none` checksums. `auto` honors
|
||||||
`RSYNC_COMPRESS_LIST`/`RSYNC_CHECKSUM_LIST` and otherwise follows rsync's
|
`RSYNC_COMPRESS_LIST`/`RSYNC_CHECKSUM_LIST` and otherwise follows rsync's
|
||||||
@@ -205,11 +207,13 @@ This produces `./build/client` and `./build/server`. `compile_commands.json` is
|
|||||||
| `--compare-dest <dir>` | Extra comparison basis: unchanged files are not transferred (requires/implies `--incremental`) |
|
| `--compare-dest <dir>` | Extra comparison basis: unchanged files are not transferred (requires/implies `--incremental`) |
|
||||||
| `--copy-dest <dir>` | Like `--compare-dest`, but copies the unchanged file from DIR into the destination |
|
| `--copy-dest <dir>` | Like `--compare-dest`, but copies the unchanged file from DIR into the destination |
|
||||||
| `--link-dest <dir>` | Like `--copy-dest`, but hard-links the unchanged file from DIR (repeatable; earlier DIRs win) |
|
| `--link-dest <dir>` | Like `--copy-dest`, but hard-links the unchanged file from DIR (repeatable; earlier DIRs win) |
|
||||||
| `--delete` | Delete files on receiver not present in source (default timing: delete-after, i.e. only after the whole transfer succeeded). Scoped to the synchronized directories, so `--files-from` subsets are safe |
|
| `--verify-basis` | FastSync-only: require a basis hit (`--compare-dest`/`--copy-dest`/`--link-dest`) to match the source by whole-file digest instead of trusting the size+mtime quick-check (default matches rsync) |
|
||||||
|
| `--delete` | Delete files on receiver not present in source (default timing: delete-during, matching rsync, so destination space is freed progressively). Scoped to the synchronized directories, so `--files-from` subsets are safe |
|
||||||
| `--delete-before` | Delete extras before the transfer starts (implies `--delete`) |
|
| `--delete-before` | Delete extras before the transfer starts (implies `--delete`) |
|
||||||
| `--delete-during`, `--del` | Delete extras once the keep-set is known, before data is applied (implies `--delete`) |
|
| `--delete-during`, `--del` | Delete extras once the keep-set is known, before data is applied (implies `--delete`) |
|
||||||
| `--delete-delay` | Delete extras only after a successful transfer (implies `--delete`) |
|
| `--delete-delay` | Delete extras only after a successful transfer (implies `--delete`) |
|
||||||
| `--delete-after` | Explicit delete-after timing (implies `--delete`) |
|
| `--delete-after` | Explicit delete-after timing (implies `--delete`) |
|
||||||
|
| `--delete-commit` | FastSync-only: keep the pre-2.28 atomic timing — delete only after the whole transfer succeeded (identical timing to `--delete-after`) |
|
||||||
| `--delete-excluded` | Also delete filter-excluded destination mirrors (size-pruned mirrors stay protected) |
|
| `--delete-excluded` | Also delete filter-excluded destination mirrors (size-pruned mirrors stay protected) |
|
||||||
| `--max-delete <n>` | Delete at most n destination entries; the rest are skipped and the run exits 25 (partial), matching rsync |
|
| `--max-delete <n>` | Delete at most n destination entries; the rest are skipped and the run exits 25 (partial), matching rsync |
|
||||||
| `--delay-updates` | Put updated files into place only at the end of the transfer (`--force` is honored at publication) |
|
| `--delay-updates` | Put updated files into place only at the end of the transfer (`--force` is honored at publication) |
|
||||||
@@ -567,6 +571,7 @@ remote SSH argv is already built injection-safe.
|
|||||||
| `--compare-dest <dir>` | Extra comparison basis: unchanged files are not transferred (requires/implies `--incremental`). |
|
| `--compare-dest <dir>` | Extra comparison basis: unchanged files are not transferred (requires/implies `--incremental`). |
|
||||||
| `--copy-dest <dir>` | Like `--compare-dest`, but copies the unchanged file from DIR into the destination. |
|
| `--copy-dest <dir>` | Like `--compare-dest`, but copies the unchanged file from DIR into the destination. |
|
||||||
| `--link-dest <dir>` | Like `--copy-dest`, but hard-links the unchanged file from DIR (repeatable; earlier DIRs win). |
|
| `--link-dest <dir>` | Like `--copy-dest`, but hard-links the unchanged file from DIR (repeatable; earlier DIRs win). |
|
||||||
|
| `--verify-basis` | FastSync-only: require a basis hit to match the source by whole-file digest instead of trusting the size+mtime quick-check (default matches rsync). |
|
||||||
| `--preallocate` | Allocate destination file space up front (fail-fast on a full disk). |
|
| `--preallocate` | Allocate destination file space up front (fail-fast on a full disk). |
|
||||||
| `--append` | Resume a shorter destination by appending only its tail (prefix not verified; requires `--incremental`). |
|
| `--append` | Resume a shorter destination by appending only its tail (prefix not verified; requires `--incremental`). |
|
||||||
| `--append-verify` | Like `--append`, but verifies the retained prefix checksum first (falls back to a full transfer on mismatch). |
|
| `--append-verify` | Like `--append`, but verifies the retained prefix checksum first (falls back to a full transfer on mismatch). |
|
||||||
@@ -574,6 +579,7 @@ remote SSH argv is already built injection-safe.
|
|||||||
| `--delete-before` | Delete extras before the transfer starts (implies `--delete`). |
|
| `--delete-before` | Delete extras before the transfer starts (implies `--delete`). |
|
||||||
| `--delete-during`, `--del` | Delete extras once the keep-set manifest is known, before data is applied (implies `--delete`; early mode, same engine behaviour as `--delete-before`). |
|
| `--delete-during`, `--del` | Delete extras once the keep-set manifest is known, before data is applied (implies `--delete`; early mode, same engine behaviour as `--delete-before`). |
|
||||||
| `--delete-delay` | Delete extras only after a successful transfer (implies `--delete`; commit mode, same behaviour as `--delete-after`). |
|
| `--delete-delay` | Delete extras only after a successful transfer (implies `--delete`; commit mode, same behaviour as `--delete-after`). |
|
||||||
|
| `--delete-commit` | FastSync-only: atomic delete-after timing (only after the whole transfer succeeded). |
|
||||||
| `--delete-after` | Explicit delete-after timing: delete only after the transfer succeeded (implies `--delete`). |
|
| `--delete-after` | Explicit delete-after timing: delete only after the transfer succeeded (implies `--delete`). |
|
||||||
| `--delete-excluded` | Also delete filter-excluded destination mirrors (size-pruned mirrors stay protected). |
|
| `--delete-excluded` | Also delete filter-excluded destination mirrors (size-pruned mirrors stay protected). |
|
||||||
| `--max-delete <n>` | Delete at most n destination entries; the rest are skipped and the run exits 25 (partial), matching rsync. |
|
| `--max-delete <n>` | Delete at most n destination entries; the rest are skipped and the run exits 25 (partial), matching rsync. |
|
||||||
|
|||||||
+64
-35
File diff suppressed because one or more lines are too long
+27
-1
@@ -997,6 +997,12 @@ static const OptionEntry OPTION_TABLE[] = {
|
|||||||
{"--delete-during", "--del", OPT_FLAG, offsetof(Config, delete_during)},
|
{"--delete-during", "--del", OPT_FLAG, offsetof(Config, delete_during)},
|
||||||
{"--delete-delay", NULL, OPT_FLAG, offsetof(Config, delete_delay)},
|
{"--delete-delay", NULL, OPT_FLAG, offsetof(Config, delete_delay)},
|
||||||
{"--delete-after", NULL, OPT_FLAG, offsetof(Config, delete_after)},
|
{"--delete-after", NULL, OPT_FLAG, offsetof(Config, delete_after)},
|
||||||
|
/* FastSync-only long spelling of the late whole-tree commit, which selects
|
||||||
|
the same timing as rsync's --delete-after in FastSync (the whole-tree
|
||||||
|
keep-set manifest is committed only after the entire transfer succeeded).
|
||||||
|
Plain --delete now defaults to delete-during, so this restores the old
|
||||||
|
FastSync behavior; it maps onto the same delete_after wire field. */
|
||||||
|
{"--delete-commit", NULL, OPT_FLAG, offsetof(Config, delete_after)},
|
||||||
{"--delete-excluded", NULL, OPT_FLAG, offsetof(Config, delete_excluded)},
|
{"--delete-excluded", NULL, OPT_FLAG, offsetof(Config, delete_excluded)},
|
||||||
{"--max-delete", NULL, OPT_SIGNED_INT, offsetof(Config, max_delete)},
|
{"--max-delete", NULL, OPT_SIGNED_INT, offsetof(Config, max_delete)},
|
||||||
{"--ignore-errors", NULL, OPT_FLAG, offsetof(Config, ignore_errors)},
|
{"--ignore-errors", NULL, OPT_FLAG, offsetof(Config, ignore_errors)},
|
||||||
@@ -1054,6 +1060,11 @@ static const OptionEntry OPTION_TABLE[] = {
|
|||||||
* --remote-option is parsed. --trust-sender is a local receiver policy and
|
* --remote-option is parsed. --trust-sender is a local receiver policy and
|
||||||
* never travels to the remote peer. */
|
* never travels to the remote peer. */
|
||||||
{"--trust-sender", NULL, OPT_FLAG, offsetof(Config, trust_sender)},
|
{"--trust-sender", NULL, OPT_FLAG, offsetof(Config, trust_sender)},
|
||||||
|
/* FastSync-only (not an rsync option): require a basis-hit's content to
|
||||||
|
* match the source by whole-file digest instead of trusting rsync's
|
||||||
|
* size+mtime quick-check. Long-only; crosses the wire so the receiver
|
||||||
|
* performs the extra read/hash. */
|
||||||
|
{"--verify-basis", NULL, OPT_FLAG, offsetof(Config, verify_basis)},
|
||||||
};
|
};
|
||||||
|
|
||||||
/* Only boolean options with no required argument are safe to negate. */
|
/* Only boolean options with no required argument are safe to negate. */
|
||||||
@@ -1096,6 +1107,7 @@ static const NegatableOption NEGATABLE_OPTIONS[] = {
|
|||||||
{"xattrs", "X", offsetof(Config, preserve_xattrs)},
|
{"xattrs", "X", offsetof(Config, preserve_xattrs)},
|
||||||
{"acls", "A", offsetof(Config, preserve_acls)},
|
{"acls", "A", offsetof(Config, preserve_acls)},
|
||||||
{"fake-super", NULL, offsetof(Config, fake_super)},
|
{"fake-super", NULL, offsetof(Config, fake_super)},
|
||||||
|
{"verify-basis", NULL, offsetof(Config, verify_basis)},
|
||||||
};
|
};
|
||||||
|
|
||||||
static bool opt_is(const char* arg, const char* name, const char* alias) {
|
static bool opt_is(const char* arg, const char* name, const char* alias) {
|
||||||
@@ -1507,7 +1519,9 @@ static bool cli_handle_table_option(CliParseCtx* ctx) {
|
|||||||
if (entry->offset == offsetof(Config, per_dir_filter) && config->per_dir_filter_count < INT_MAX)
|
if (entry->offset == offsetof(Config, per_dir_filter) && config->per_dir_filter_count < INT_MAX)
|
||||||
config->per_dir_filter_count++;
|
config->per_dir_filter_count++;
|
||||||
/* A delete-timing flag selects when --delete removes extras, so it
|
/* A delete-timing flag selects when --delete removes extras, so it
|
||||||
implies --delete exactly like the rsync options do. */
|
implies --delete exactly like the rsync options do. --delete-commit (the
|
||||||
|
FastSync-only late-commit spelling) is mapped onto delete_after and so is
|
||||||
|
covered here too. */
|
||||||
if (entry->offset == offsetof(Config, delete_before) ||
|
if (entry->offset == offsetof(Config, delete_before) ||
|
||||||
entry->offset == offsetof(Config, delete_during) ||
|
entry->offset == offsetof(Config, delete_during) ||
|
||||||
entry->offset == offsetof(Config, delete_delay) ||
|
entry->offset == offsetof(Config, delete_delay) ||
|
||||||
@@ -2561,6 +2575,18 @@ static bool cli_handle_outbuf_option(CliParseCtx* ctx) {
|
|||||||
* -1 on error. */
|
* -1 on error. */
|
||||||
static int cli_finalize_config(Config* config, bool verbose, bool no_delta, bool no_incremental) {
|
static int cli_finalize_config(Config* config, bool verbose, bool no_delta, bool no_incremental) {
|
||||||
set_log_level(config->quiet ? LOG_LEVEL_ERROR : (verbose ? LOG_LEVEL_DEBUG : LOG_LEVEL_WARNING));
|
set_log_level(config->quiet ? LOG_LEVEL_ERROR : (verbose ? LOG_LEVEL_DEBUG : LOG_LEVEL_WARNING));
|
||||||
|
/* rsync's plain --delete defaults to delete-during (--del): each directory's
|
||||||
|
extras are removed as that directory is processed, so space is freed
|
||||||
|
progressively and a tight destination never has to hold the whole old+new
|
||||||
|
tree at once. The late whole-tree commit FastSync historically used is
|
||||||
|
still selected explicitly by --delete-after or by the FastSync-only long
|
||||||
|
spelling --delete-commit (an exact alias for --delete-after). Resolve the
|
||||||
|
default on the client, before validation and before the config crosses the
|
||||||
|
wire, so exactly one timing flag is ever set; an explicit timing (including
|
||||||
|
--delete-commit) always wins. */
|
||||||
|
if (config->use_delete && !config->delete_before && !config->delete_during &&
|
||||||
|
!config->delete_delay && !config->delete_after)
|
||||||
|
config->delete_during = true;
|
||||||
if (config->compress_choice) {
|
if (config->compress_choice) {
|
||||||
int algo = compression_algo_from_name(config->compress_choice);
|
int algo = compression_algo_from_name(config->compress_choice);
|
||||||
if (algo >= 0) {
|
if (algo >= 0) {
|
||||||
|
|||||||
+18
-69
@@ -1047,53 +1047,6 @@ static bool files_from_list_check(const Config* config, ArrayList* missing_dest,
|
|||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Basis directories are honored by the receiver's per-file incremental check,
|
|
||||||
which (like every whole-file payload path in FastSync) is bounded by
|
|
||||||
MAX_RECEIVE_WHOLE_FILE_SIZE. rsync would apply basis dirs to files of any
|
|
||||||
size; FastSync cannot, so when basis dirs are requested this preflight scan
|
|
||||||
refuses the run up front with a clear diagnostic instead of letting the
|
|
||||||
receiver abort the whole transfer mid-stream with no client explanation.
|
|
||||||
Returns true when the tree can be transferred. */
|
|
||||||
static bool basis_oversize_preflight(const Config* config) {
|
|
||||||
PreparedScanner prepared;
|
|
||||||
if (!prepare_scanner(config, 0, &prepared))
|
|
||||||
return false;
|
|
||||||
DirectoryScanner* scanner =
|
|
||||||
directory_scanner_create_with_options(config->send_directory, &prepared.options);
|
|
||||||
if (!scanner) {
|
|
||||||
prepared_scanner_destroy(&prepared);
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
bool ok = true;
|
|
||||||
Chunk* chunk;
|
|
||||||
while ((chunk = directory_scanner_next(scanner)) != NULL) {
|
|
||||||
for (int i = 0; i < chunk->element_count; i++) {
|
|
||||||
File* f = chunk->items[i];
|
|
||||||
if (f == NULL || f->is_dir || f->data == NULL || f->data->size <= MAX_RECEIVE_WHOLE_FILE_SIZE)
|
|
||||||
continue;
|
|
||||||
char* escaped = output_escape(file_wire_path(f), config->eight_bit_output);
|
|
||||||
log_message(LOG_LEVEL_ERROR,
|
|
||||||
"%s is %llu bytes, larger than the %llu-byte whole-file transfer limit; "
|
|
||||||
"--compare-dest/--copy-dest/--link-dest cannot sync files above this limit",
|
|
||||||
escaped ? escaped : "<allocation failed>", (unsigned long long)f->data->size,
|
|
||||||
(unsigned long long)MAX_RECEIVE_WHOLE_FILE_SIZE);
|
|
||||||
free(escaped);
|
|
||||||
ok = false;
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
chunk_destroy(chunk);
|
|
||||||
if (!ok)
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
if (directory_scanner_failed(scanner) || directory_scanner_had_io_error(scanner))
|
|
||||||
ok = false;
|
|
||||||
/* The scanner borrows prepared.options' base_filters/hardlinks pointers, so
|
|
||||||
prepared must outlive the scanner. */
|
|
||||||
directory_scanner_destroy(scanner);
|
|
||||||
prepared_scanner_destroy(&prepared);
|
|
||||||
return ok;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Read the daemon's MOTD frame and, unless --no-motd, display it on stdout.
|
/* Read the daemon's MOTD frame and, unless --no-motd, display it on stdout.
|
||||||
*
|
*
|
||||||
* The daemon sends the MOTD as the first thing after the config-frame STATUS_OK
|
* The daemon sends the MOTD as the first thing after the config-frame STATUS_OK
|
||||||
@@ -1939,11 +1892,13 @@ static int incremental_check(Client* client, File* file, const Config* config,
|
|||||||
return -1;
|
return -1;
|
||||||
if (!send_n_data(client->file_descriptor, &mtime_nsec, sizeof(mtime_nsec)))
|
if (!send_n_data(client->file_descriptor, &mtime_nsec, sizeof(mtime_nsec)))
|
||||||
return -1;
|
return -1;
|
||||||
/* With alternate basis directories the receiver must be able to verify the
|
/* The whole-file digest (negotiated --checksum-choice algorithm and
|
||||||
* content of every candidate basis file, so the sender supplies its whole-file
|
* --checksum-seed) is only needed when it drives a decision: --checksum's
|
||||||
* digest (computed with the negotiated --checksum-choice algorithm and
|
* per-file quick check, or a --verify-basis content equality. Under the
|
||||||
* --checksum-seed) for every file even when --checksum was not requested. */
|
* default metadata quick-check the receiver never reads it, so the sender
|
||||||
if (config->checksum || config_has_basis(config)) {
|
* skips the full-file read/hash exactly as rsync does for a plain
|
||||||
|
* --link-dest run. */
|
||||||
|
if (config->checksum || config->verify_basis) {
|
||||||
uint8_t digest[CHECKSUM_MAX_DIGEST_LEN];
|
uint8_t digest[CHECKSUM_MAX_DIGEST_LEN];
|
||||||
size_t digest_len = 0;
|
size_t digest_len = 0;
|
||||||
if (!file_checksum(file, (ChecksumAlgo)config->checksum_algo, config->checksum_seed, digest,
|
if (!file_checksum(file, (ChecksumAlgo)config->checksum_algo, config->checksum_seed, digest,
|
||||||
@@ -1954,6 +1909,15 @@ static int incremental_check(Client* client, File* file, const Config* config,
|
|||||||
!send_n_data(client->file_descriptor, digest, wire_len))
|
!send_n_data(client->file_descriptor, digest, wire_len))
|
||||||
return -1;
|
return -1;
|
||||||
}
|
}
|
||||||
|
/* Basis directories: the receiver materializes a hit from the basis without a
|
||||||
|
* data frame, so it would otherwise only have the basis inode's metadata.
|
||||||
|
* Transmit the SOURCE metadata with the check (rsync's copy-then-fix) so a
|
||||||
|
* --copy-dest hit / --link-dest copy fallback applies the source's
|
||||||
|
* attributes. Symmetric with incremental_check_receive_request. */
|
||||||
|
if (config_has_basis(config) && config->use_metadata) {
|
||||||
|
if (!metadata_send(client->file_descriptor, file->metadata))
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
Status s;
|
Status s;
|
||||||
if (!receive_status(client->file_descriptor, &s))
|
if (!receive_status(client->file_descriptor, &s))
|
||||||
return -1;
|
return -1;
|
||||||
@@ -2193,12 +2157,6 @@ static int send_dry_run_remote(Config* config) {
|
|||||||
}
|
}
|
||||||
if (missing_args)
|
if (missing_args)
|
||||||
array_list_delete(missing_args);
|
array_list_delete(missing_args);
|
||||||
/* Alternate basis dirs force the whole-file per-file check on the real
|
|
||||||
receiver; refuse an oversize source up front exactly as send_files does so
|
|
||||||
dry-run reports the same clear diagnostic instead of aborting mid-stream. */
|
|
||||||
if (config_has_basis(config) && !basis_oversize_preflight(config))
|
|
||||||
return 1;
|
|
||||||
|
|
||||||
/* A live session may follow, so arm graceful abort handling. */
|
/* A live session may follow, so arm graceful abort handling. */
|
||||||
client_set_abort_armed(true);
|
client_set_abort_armed(true);
|
||||||
Client* client = connect_transfer_client(config);
|
Client* client = connect_transfer_client(config);
|
||||||
@@ -3240,11 +3198,6 @@ int send_files(Config* config) {
|
|||||||
array_list_delete(missing_args);
|
array_list_delete(missing_args);
|
||||||
return 1;
|
return 1;
|
||||||
}
|
}
|
||||||
if (config_has_basis(config) && !basis_oversize_preflight(config)) {
|
|
||||||
if (missing_args)
|
|
||||||
array_list_delete(missing_args);
|
|
||||||
return 1;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* From here on a server session may be live, so Ctrl-C/SIGTERM should set the
|
/* From here on a server session may be live, so Ctrl-C/SIGTERM should set the
|
||||||
abort flag (and be forwarded as STATUS_ABORT) instead of terminating. */
|
abort flag (and be forwarded as STATUS_ABORT) instead of terminating. */
|
||||||
@@ -3334,7 +3287,8 @@ int send_files(Config* config) {
|
|||||||
prepared.options.synced_dirs = synced_dirs;
|
prepared.options.synced_dirs = synced_dirs;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
/* The late-timing modes (plain --delete / --delete-after) build the manifest
|
/* The late-timing modes (--delete-after/--delete-commit and a plain --delete
|
||||||
|
that fell back from per-dir mode because of -d/--dirs) build the manifest
|
||||||
while streaming and send it after the last data frame. --delete-before
|
while streaming and send it after the last data frame. --delete-before
|
||||||
sends a whole-tree keep-set up front; --delete-during/--delete-delay build a
|
sends a whole-tree keep-set up front; --delete-during/--delete-delay build a
|
||||||
per-directory plan set up front (paths only) and stream the plans alongside
|
per-directory plan set up front (paths only) and stream the plans alongside
|
||||||
@@ -3682,11 +3636,6 @@ int send_files_multithreaded(Config** config_ptr) {
|
|||||||
array_list_delete(missing_args);
|
array_list_delete(missing_args);
|
||||||
return 1;
|
return 1;
|
||||||
}
|
}
|
||||||
if (config_has_basis(config) && !basis_oversize_preflight(config)) {
|
|
||||||
if (missing_args)
|
|
||||||
array_list_delete(missing_args);
|
|
||||||
return 1;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Armed only once a session may go live (see send_files). */
|
/* Armed only once a session may go live (see send_files). */
|
||||||
client_set_abort_armed(true);
|
client_set_abort_armed(true);
|
||||||
|
|||||||
+9
-4
@@ -62,8 +62,7 @@ void print_usage(void) {
|
|||||||
printf(" NOTE: the FastSync batch format is NOT interoperable with rsync's batch\n");
|
printf(" NOTE: the FastSync batch format is NOT interoperable with rsync's batch\n");
|
||||||
printf(" files (different container format); do not mix the two tools.\n");
|
printf(" files (different container format); do not mix the two tools.\n");
|
||||||
printf(" --delete Delete files on receiver not in source\n");
|
printf(" --delete Delete files on receiver not in source\n");
|
||||||
printf(" (default timing: delete only after the whole\n");
|
printf(" (default timing: delete-during, like rsync --del)\n");
|
||||||
printf(" transfer has succeeded)\n");
|
|
||||||
printf(" --delete-before Delete extras before the transfer starts\n");
|
printf(" --delete-before Delete extras before the transfer starts\n");
|
||||||
printf(" (implies --delete)\n");
|
printf(" (implies --delete)\n");
|
||||||
printf(" --delete-during Delete a directory's extras as that directory is\n");
|
printf(" --delete-during Delete a directory's extras as that directory is\n");
|
||||||
@@ -72,7 +71,9 @@ void print_usage(void) {
|
|||||||
printf(" --delete-delay Record the extras during the scan but remove them\n");
|
printf(" --delete-delay Record the extras during the scan but remove them\n");
|
||||||
printf(" only after a successful transfer (implies --delete)\n");
|
printf(" only after a successful transfer (implies --delete)\n");
|
||||||
printf(" --delete-after Delete only after the whole transfer succeeded\n");
|
printf(" --delete-after Delete only after the whole transfer succeeded\n");
|
||||||
printf(" (the default --delete timing; implies --delete)\n");
|
printf(" (implies --delete)\n");
|
||||||
|
printf(" --delete-commit FastSync-only: restore the late whole-tree commit\n");
|
||||||
|
printf(" (identical to --delete-after; implies --delete)\n");
|
||||||
printf(" --delete-excluded Also delete destination files that were excluded on\n");
|
printf(" --delete-excluded Also delete destination files that were excluded on\n");
|
||||||
printf(" the source (default protects them, matching rsync)\n");
|
printf(" the source (default protects them, matching rsync)\n");
|
||||||
printf(" --max-delete=NUM Delete at most NUM destination entries per run; if the\n");
|
printf(" --max-delete=NUM Delete at most NUM destination entries per run; if the\n");
|
||||||
@@ -93,7 +94,8 @@ void print_usage(void) {
|
|||||||
printf(" -m, --prune-empty-dirs Do not create empty directories (a recursive transfer\n");
|
printf(" -m, --prune-empty-dirs Do not create empty directories (a recursive transfer\n");
|
||||||
printf(" otherwise recreates them, like rsync)\n");
|
printf(" otherwise recreates them, like rsync)\n");
|
||||||
printf(" Note: each timing flag implies --delete. Combining a timing flag with\n");
|
printf(" Note: each timing flag implies --delete. Combining a timing flag with\n");
|
||||||
printf(" --no-delete (in either order) is rejected as a config error.\n");
|
printf(" --no-delete (in either order) is rejected as a config error, as is more\n");
|
||||||
|
printf(" than one timing flag.\n");
|
||||||
printf(" --ignore-existing Skip files that already exist on receiver\n");
|
printf(" --ignore-existing Skip files that already exist on receiver\n");
|
||||||
printf(" --delay-updates Put updated files into place only at the end of transfer\n");
|
printf(" --delay-updates Put updated files into place only at the end of transfer\n");
|
||||||
printf(" --dirs, -d, --old-dirs, --old-d Transfer the named directory entries without\n");
|
printf(" --dirs, -d, --old-dirs, --old-d Transfer the named directory entries without\n");
|
||||||
@@ -138,6 +140,9 @@ void print_usage(void) {
|
|||||||
printf(" into the destination instead of transferring its data\n");
|
printf(" into the destination instead of transferring its data\n");
|
||||||
printf(" --link-dest <dir> Like --copy-dest, but hard-links the unchanged file from DIR\n");
|
printf(" --link-dest <dir> Like --copy-dest, but hard-links the unchanged file from DIR\n");
|
||||||
printf(" into the destination (repeatable; earlier DIRs win)\n");
|
printf(" into the destination (repeatable; earlier DIRs win)\n");
|
||||||
|
printf(" --verify-basis FastSync-only: require a basis hit's content to match the\n");
|
||||||
|
printf(" source by whole-file digest instead of trusting rsync's\n");
|
||||||
|
printf(" size+mtime (or --size-only) quick-check\n");
|
||||||
printf(" --checksum-choice, --cc <alg> Whole-file checksum algorithm for --incremental/\n");
|
printf(" --checksum-choice, --cc <alg> Whole-file checksum algorithm for --incremental/\n");
|
||||||
printf(" --checksum compares. Accepted: xxh128 (default), xxh3, xxh64\n");
|
printf(" --checksum compares. Accepted: xxh128 (default), xxh3, xxh64\n");
|
||||||
printf(" (aka xxhash), md5, md4, sha1, or none. A two-name\n");
|
printf(" (aka xxhash), md5, md4, sha1, or none. A two-name\n");
|
||||||
|
|||||||
@@ -305,14 +305,16 @@ int receiver_process(Config* config, int file_descriptor, const ReceiverSink* si
|
|||||||
|
|
||||||
/* Runs the whole receive loop. The delete manifest may legitimately arrive
|
/* Runs the whole receive loop. The delete manifest may legitimately arrive
|
||||||
either FIRST (--delete-before / --delete-during: the sender transmits the
|
either FIRST (--delete-before / --delete-during: the sender transmits the
|
||||||
validated keep-set before any file data) or LAST (plain --delete /
|
validated keep-set before any file data) or LAST (--delete-after /
|
||||||
--delete-after / --delete-delay: the manifest closes the data stream). In
|
--delete-commit / --delete-delay: the manifest closes the data stream). In
|
||||||
the early modes the receiver deletes as soon as the manifest has been read
|
the early modes the receiver deletes as soon as the manifest has been read
|
||||||
and acknowledges with STATUS_OK so the sender only starts streaming once the
|
and acknowledges with STATUS_OK so the sender only starts streaming once the
|
||||||
deletion has committed (or failed); in the late modes the manifest is held
|
deletion has committed (or failed); in the late modes the manifest is held
|
||||||
and the deletion is committed only after the terminal STATUS_FINISHED proves
|
and the deletion is committed only after the terminal STATUS_FINISHED proves
|
||||||
the whole transfer succeeded. See receiver_process_pending() for how the -m
|
the whole transfer succeeded. A plain --delete defaults to the per-directory
|
||||||
receiver defers that commit until its disk writer has drained. */
|
delete-during plan mode (no manifest at all). See
|
||||||
|
receiver_process_pending() for how the -m receiver defers that commit until
|
||||||
|
its disk writer has drained. */
|
||||||
int receiver_process_pending(Config* config, int file_descriptor, const ReceiverSink* sink,
|
int receiver_process_pending(Config* config, int file_descriptor, const ReceiverSink* sink,
|
||||||
DeleteManifest** pending_manifest, DeletePlanSession** pending_plans) {
|
DeleteManifest** pending_manifest, DeletePlanSession** pending_plans) {
|
||||||
Status status;
|
Status status;
|
||||||
|
|||||||
+13
-11
@@ -224,23 +224,31 @@ bool checksum_digest_file(ChecksumAlgo algo, uint64_t seed, const char* path, ui
|
|||||||
if (fd < 0)
|
if (fd < 0)
|
||||||
return false;
|
return false;
|
||||||
|
|
||||||
|
bool ok = checksum_digest_fd(algo, seed, fd, out, out_capacity, out_len);
|
||||||
|
close(fd);
|
||||||
|
return ok;
|
||||||
|
}
|
||||||
|
|
||||||
|
bool checksum_digest_fd(ChecksumAlgo algo, uint64_t seed, int fd, uint8_t* out, size_t out_capacity,
|
||||||
|
size_t* out_len) {
|
||||||
|
if (fd < 0 || !out || !out_len || out_capacity < CHECKSUM_MAX_DIGEST_LEN)
|
||||||
|
return false;
|
||||||
|
|
||||||
if (algo == CHECKSUM_ALGO_NONE) {
|
if (algo == CHECKSUM_ALGO_NONE) {
|
||||||
/* No checksum requested: nothing to read; an empty digest succeeds. */
|
/* No checksum requested: nothing to read; an empty digest succeeds. */
|
||||||
close(fd);
|
|
||||||
*out_len = 0;
|
*out_len = 0;
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
uint8_t buffer[64 * 1024];
|
uint8_t buffer[64 * 1024];
|
||||||
bool ok = false;
|
bool ok = false;
|
||||||
|
lseek(fd, 0, SEEK_SET);
|
||||||
|
|
||||||
if (algo == CHECKSUM_ALGO_MD5 || algo == CHECKSUM_ALGO_SHA1) {
|
if (algo == CHECKSUM_ALGO_MD5 || algo == CHECKSUM_ALGO_SHA1) {
|
||||||
const EVP_MD* md = algo == CHECKSUM_ALGO_MD5 ? EVP_md5() : EVP_sha1();
|
const EVP_MD* md = algo == CHECKSUM_ALGO_MD5 ? EVP_md5() : EVP_sha1();
|
||||||
EVP_MD_CTX* ctx = EVP_MD_CTX_new();
|
EVP_MD_CTX* ctx = EVP_MD_CTX_new();
|
||||||
if (!ctx) {
|
if (!ctx)
|
||||||
close(fd);
|
|
||||||
return false;
|
return false;
|
||||||
}
|
|
||||||
unsigned int digest_len = 0;
|
unsigned int digest_len = 0;
|
||||||
if (EVP_DigestInit_ex(ctx, md, NULL) == 1) {
|
if (EVP_DigestInit_ex(ctx, md, NULL) == 1) {
|
||||||
ok = true;
|
ok = true;
|
||||||
@@ -259,7 +267,6 @@ bool checksum_digest_file(ChecksumAlgo algo, uint64_t seed, const char* path, ui
|
|||||||
ok = false;
|
ok = false;
|
||||||
}
|
}
|
||||||
EVP_MD_CTX_free(ctx);
|
EVP_MD_CTX_free(ctx);
|
||||||
close(fd);
|
|
||||||
return ok;
|
return ok;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -276,7 +283,6 @@ bool checksum_digest_file(ChecksumAlgo algo, uint64_t seed, const char* path, ui
|
|||||||
md4_final(&ctx, out);
|
md4_final(&ctx, out);
|
||||||
*out_len = 16;
|
*out_len = 16;
|
||||||
}
|
}
|
||||||
close(fd);
|
|
||||||
return ok;
|
return ok;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -286,16 +292,13 @@ bool checksum_digest_file(ChecksumAlgo algo, uint64_t seed, const char* path, ui
|
|||||||
XXH64_reset(&xxh64, seed);
|
XXH64_reset(&xxh64, seed);
|
||||||
} else if (algo == CHECKSUM_ALGO_XXH3 || algo == CHECKSUM_ALGO_XXH128) {
|
} else if (algo == CHECKSUM_ALGO_XXH3 || algo == CHECKSUM_ALGO_XXH128) {
|
||||||
xxh3 = XXH3_createState();
|
xxh3 = XXH3_createState();
|
||||||
if (!xxh3) {
|
if (!xxh3)
|
||||||
close(fd);
|
|
||||||
return false;
|
return false;
|
||||||
}
|
|
||||||
if (algo == CHECKSUM_ALGO_XXH3)
|
if (algo == CHECKSUM_ALGO_XXH3)
|
||||||
XXH3_64bits_reset_withSeed(xxh3, seed);
|
XXH3_64bits_reset_withSeed(xxh3, seed);
|
||||||
else
|
else
|
||||||
XXH3_128bits_reset_withSeed(xxh3, seed);
|
XXH3_128bits_reset_withSeed(xxh3, seed);
|
||||||
} else {
|
} else {
|
||||||
close(fd);
|
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -329,7 +332,6 @@ bool checksum_digest_file(ChecksumAlgo algo, uint64_t seed, const char* path, ui
|
|||||||
}
|
}
|
||||||
if (xxh3)
|
if (xxh3)
|
||||||
XXH3_freeState(xxh3);
|
XXH3_freeState(xxh3);
|
||||||
close(fd);
|
|
||||||
return ok;
|
return ok;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -51,6 +51,13 @@ bool checksum_digest(ChecksumAlgo algo, uint64_t seed, const void* data, size_t
|
|||||||
bool checksum_digest_file(ChecksumAlgo algo, uint64_t seed, const char* path, uint8_t* out,
|
bool checksum_digest_file(ChecksumAlgo algo, uint64_t seed, const char* path, uint8_t* out,
|
||||||
size_t out_capacity, size_t* out_len);
|
size_t out_capacity, size_t* out_len);
|
||||||
|
|
||||||
|
/* Descriptor form of the streaming digest: rewinds `fd` to the start and hashes
|
||||||
|
* to EOF without closing it. Used by the --verify-basis path to hash an
|
||||||
|
* already-open, root-confined basis descriptor. Same contract as
|
||||||
|
* checksum_digest_file. */
|
||||||
|
bool checksum_digest_fd(ChecksumAlgo algo, uint64_t seed, int fd, uint8_t* out, size_t out_capacity,
|
||||||
|
size_t* out_len);
|
||||||
|
|
||||||
/* Resolve a --checksum-choice string (case-insensitive) to an algorithm id.
|
/* Resolve a --checksum-choice string (case-insensitive) to an algorithm id.
|
||||||
* Accepts "xxh64"/"xxhash", "xxh3", "xxh128", "md5", "md4", "sha1", "none".
|
* Accepts "xxh64"/"xxhash", "xxh3", "xxh128", "md5", "md4", "sha1", "none".
|
||||||
* "auto" is not an algorithm here; the caller resolves it to the negotiated
|
* "auto" is not an algorithm here; the caller resolves it to the negotiated
|
||||||
|
|||||||
+129
-2
@@ -791,6 +791,8 @@ void config_delete(Config* config) {
|
|||||||
if (config->filters) {
|
if (config->filters) {
|
||||||
array_list_delete(config->filters);
|
array_list_delete(config->filters);
|
||||||
}
|
}
|
||||||
|
filter_rule_list_free(config->protect_rules);
|
||||||
|
config->protect_rules = NULL;
|
||||||
/* A --delay-updates staging tree is transient receiver state: remove any
|
/* A --delay-updates staging tree is transient receiver state: remove any
|
||||||
leftovers on every exit path (success already emptied it). */
|
leftovers on every exit path (success already emptied it). */
|
||||||
if (config->delay_context)
|
if (config->delay_context)
|
||||||
@@ -1026,6 +1028,124 @@ static bool receive_basis_entries(int fd, Config* c, ConfigStringBudget* budget)
|
|||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Receiver-side delete-protection rules (protocol 2.28.0). The sender compiles
|
||||||
|
* its command-line selection rules exactly as the scanner does and streams the
|
||||||
|
* result as one bounded, self-describing block (count + per-rule records); the
|
||||||
|
* receiver reconstructs a FilterRuleList for the --delete extras walk. owner
|
||||||
|
* and pattern are charged through the shared ConfigStringBudget and the block
|
||||||
|
* additionally enforces MAX_FILTER_RULES / MAX_FILTER_BYTES. */
|
||||||
|
static bool send_protect_entries(int fd, const Config* c) {
|
||||||
|
int count = c->filters ? c->filters->size : 0;
|
||||||
|
const char** texts = NULL;
|
||||||
|
if (count > 0) {
|
||||||
|
texts = malloc((size_t)count * sizeof(char*));
|
||||||
|
if (!texts)
|
||||||
|
return false;
|
||||||
|
for (int i = 0; i < count; i++)
|
||||||
|
texts[i] = (const char*)c->filters->items[i];
|
||||||
|
}
|
||||||
|
char err[160];
|
||||||
|
FilterRuleList* rules =
|
||||||
|
filter_base_build(texts, count, c->cvs_exclude, c->delete_excluded, err, sizeof(err));
|
||||||
|
free(texts);
|
||||||
|
if (!rules) {
|
||||||
|
log_message(LOG_LEVEL_ERROR, "invalid filter rule: %s", err);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
bool ok = send_int(fd, rules->count);
|
||||||
|
for (int i = 0; ok && i < rules->count; i++) {
|
||||||
|
const FilterRule* r = rules->items[i];
|
||||||
|
/* Mirror the receiver's limit so the peer never receives a rule it will
|
||||||
|
reject as a protocol error. */
|
||||||
|
if (r->pattern && strlen(r->pattern) > MAX_PROTECT_PATTERN_LEN) {
|
||||||
|
log_message(LOG_LEVEL_ERROR, "filter pattern exceeds %d bytes", MAX_PROTECT_PATTERN_LEN);
|
||||||
|
filter_rule_list_free(rules);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
ok = send_int(fd, (int)r->action) && send_int(fd, (int)r->sides) &&
|
||||||
|
send_int(fd, r->anchored ? 1 : 0) && send_int(fd, r->dir_only ? 1 : 0) &&
|
||||||
|
send_int(fd, r->negate ? 1 : 0) && send_str(fd, r->owner ? r->owner : "") &&
|
||||||
|
send_str(fd, r->pattern ? r->pattern : "");
|
||||||
|
}
|
||||||
|
filter_rule_list_free(rules);
|
||||||
|
return ok;
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool receive_protect_entries(int fd, Config* c, ConfigStringBudget* budget) {
|
||||||
|
int count;
|
||||||
|
if (!receive_int(fd, &count))
|
||||||
|
return false;
|
||||||
|
if (count < 0 || count > MAX_FILTER_RULES)
|
||||||
|
return false;
|
||||||
|
if (count == 0)
|
||||||
|
return true;
|
||||||
|
FilterRuleList* list = filter_rule_list_create();
|
||||||
|
if (!list)
|
||||||
|
return false;
|
||||||
|
size_t pattern_bytes = 0;
|
||||||
|
for (int i = 0; i < count; i++) {
|
||||||
|
int action;
|
||||||
|
int sides;
|
||||||
|
bool anchored;
|
||||||
|
bool dir_only;
|
||||||
|
bool negate;
|
||||||
|
if (!receive_int(fd, &action) ||
|
||||||
|
(action != FILTER_ACTION_EXCLUDE && action != FILTER_ACTION_INCLUDE) ||
|
||||||
|
!receive_int(fd, &sides) || sides < (int)FILTER_SIDE_SENDER ||
|
||||||
|
sides > (int)(FILTER_SIDE_SENDER | FILTER_SIDE_RECEIVER) ||
|
||||||
|
!receive_wire_bool(fd, &anchored) || !receive_wire_bool(fd, &dir_only) ||
|
||||||
|
!receive_wire_bool(fd, &negate))
|
||||||
|
goto fail;
|
||||||
|
char* owner = config_receive_str(fd, budget);
|
||||||
|
if (!owner)
|
||||||
|
goto fail;
|
||||||
|
char* pattern = config_receive_str(fd, budget);
|
||||||
|
if (!pattern || pattern[0] == '\0') {
|
||||||
|
free(owner);
|
||||||
|
free(pattern);
|
||||||
|
goto fail;
|
||||||
|
}
|
||||||
|
/* A pattern too long to be evaluated by glob_match against a PATH_MAX path
|
||||||
|
would silently fail to match and leave a protect rule inert (fail-open:
|
||||||
|
the entry is then deleted). Reject it up front as a protocol error
|
||||||
|
rather than accept a rule that can never shield anything. */
|
||||||
|
if (strlen(pattern) > MAX_PROTECT_PATTERN_LEN) {
|
||||||
|
free(owner);
|
||||||
|
free(pattern);
|
||||||
|
goto fail;
|
||||||
|
}
|
||||||
|
size_t bytes = strlen(owner) + strlen(pattern);
|
||||||
|
if (bytes > MAX_FILTER_BYTES - pattern_bytes) {
|
||||||
|
free(owner);
|
||||||
|
free(pattern);
|
||||||
|
goto fail;
|
||||||
|
}
|
||||||
|
pattern_bytes += bytes;
|
||||||
|
FilterRule* rule = calloc(1, sizeof(FilterRule));
|
||||||
|
if (!rule) {
|
||||||
|
free(owner);
|
||||||
|
free(pattern);
|
||||||
|
goto fail;
|
||||||
|
}
|
||||||
|
rule->action = (FilterAction)action;
|
||||||
|
rule->sides = (unsigned)sides;
|
||||||
|
rule->anchored = anchored;
|
||||||
|
rule->dir_only = dir_only;
|
||||||
|
rule->negate = negate;
|
||||||
|
rule->owner = owner;
|
||||||
|
rule->pattern = pattern;
|
||||||
|
if (!filter_rule_list_add(list, rule)) {
|
||||||
|
filter_rule_free(rule);
|
||||||
|
goto fail;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
c->protect_rules = list;
|
||||||
|
return true;
|
||||||
|
fail:
|
||||||
|
filter_rule_list_free(list);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
static bool send_identity_entries(int fd, const IdentityMap* map, int count) {
|
static bool send_identity_entries(int fd, const IdentityMap* map, int count) {
|
||||||
for (int i = 0; i < count; i++) {
|
for (int i = 0; i < count; i++) {
|
||||||
if (!send_int(fd, map[i].from) || !send_int(fd, map[i].from_hi) || !send_int(fd, map[i].to) ||
|
if (!send_int(fd, map[i].from) || !send_int(fd, map[i].from_hi) || !send_int(fd, map[i].to) ||
|
||||||
@@ -1153,6 +1273,9 @@ fail:
|
|||||||
#define CONFIG_RECV_BLOCK_IDMAP(name) \
|
#define CONFIG_RECV_BLOCK_IDMAP(name) \
|
||||||
receive_identity_entries(fd, budget, c->name##_count, &c->name)
|
receive_identity_entries(fd, budget, c->name##_count, &c->name)
|
||||||
|
|
||||||
|
#define CONFIG_SEND_BLOCK_PROTECT_RULES(name) send_protect_entries(fd, c)
|
||||||
|
#define CONFIG_RECV_BLOCK_PROTECT_RULES(name) receive_protect_entries(fd, c, budget)
|
||||||
|
|
||||||
/* One table entry, applied in sequence. XSEND/XRECV are statement macros so
|
/* One table entry, applied in sequence. XSEND/XRECV are statement macros so
|
||||||
* consecutive entries read as a plain sequence of assignments. */
|
* consecutive entries read as a plain sequence of assignments. */
|
||||||
#define XSEND(name, ctype, def, kind) ok = ok && (CONFIG_SEND_##kind(name));
|
#define XSEND(name, ctype, def, kind) ok = ok && (CONFIG_SEND_##kind(name));
|
||||||
@@ -1190,6 +1313,7 @@ CONFIG_DEFINE_SEND(send_privilege_options, CONFIG_WIRE_PRIVILEGE_FIELDS)
|
|||||||
CONFIG_DEFINE_SEND(send_copy_as_options, CONFIG_WIRE_COPY_AS_FIELDS)
|
CONFIG_DEFINE_SEND(send_copy_as_options, CONFIG_WIRE_COPY_AS_FIELDS)
|
||||||
CONFIG_DEFINE_SEND(send_output_options, CONFIG_WIRE_OUTPUT_FIELDS)
|
CONFIG_DEFINE_SEND(send_output_options, CONFIG_WIRE_OUTPUT_FIELDS)
|
||||||
CONFIG_DEFINE_SEND(send_codec_options, CONFIG_WIRE_CODEC_FIELDS)
|
CONFIG_DEFINE_SEND(send_codec_options, CONFIG_WIRE_CODEC_FIELDS)
|
||||||
|
CONFIG_DEFINE_SEND(send_protect_options, CONFIG_WIRE_PROTECT_FIELDS)
|
||||||
|
|
||||||
CONFIG_DEFINE_RECV(receive_core_fields, CONFIG_WIRE_CORE_FIELDS)
|
CONFIG_DEFINE_RECV(receive_core_fields, CONFIG_WIRE_CORE_FIELDS)
|
||||||
CONFIG_DEFINE_RECV(receive_delta_fields, CONFIG_WIRE_DELTA_FIELDS)
|
CONFIG_DEFINE_RECV(receive_delta_fields, CONFIG_WIRE_DELTA_FIELDS)
|
||||||
@@ -1210,6 +1334,7 @@ CONFIG_DEFINE_RECV(receive_privilege_options, CONFIG_WIRE_PRIVILEGE_FIELDS)
|
|||||||
CONFIG_DEFINE_RECV(receive_copy_as_options, CONFIG_WIRE_COPY_AS_FIELDS)
|
CONFIG_DEFINE_RECV(receive_copy_as_options, CONFIG_WIRE_COPY_AS_FIELDS)
|
||||||
CONFIG_DEFINE_RECV(receive_output_options, CONFIG_WIRE_OUTPUT_FIELDS)
|
CONFIG_DEFINE_RECV(receive_output_options, CONFIG_WIRE_OUTPUT_FIELDS)
|
||||||
CONFIG_DEFINE_RECV(receive_codec_options, CONFIG_WIRE_CODEC_FIELDS)
|
CONFIG_DEFINE_RECV(receive_codec_options, CONFIG_WIRE_CODEC_FIELDS)
|
||||||
|
CONFIG_DEFINE_RECV(receive_protect_options, CONFIG_WIRE_PROTECT_FIELDS)
|
||||||
|
|
||||||
#undef XSEND
|
#undef XSEND
|
||||||
#undef XRECV
|
#undef XRECV
|
||||||
@@ -1328,7 +1453,8 @@ bool config_send_wire_block(int file_descriptor, const Config* config) {
|
|||||||
send_privilege_options(file_descriptor, config) &&
|
send_privilege_options(file_descriptor, config) &&
|
||||||
send_copy_as_options(file_descriptor, config) &&
|
send_copy_as_options(file_descriptor, config) &&
|
||||||
send_output_options(file_descriptor, config) &&
|
send_output_options(file_descriptor, config) &&
|
||||||
send_codec_options(file_descriptor, config);
|
send_codec_options(file_descriptor, config) &&
|
||||||
|
send_protect_options(file_descriptor, config);
|
||||||
}
|
}
|
||||||
|
|
||||||
bool config_send(int file_descriptor, const Config* config) {
|
bool config_send(int file_descriptor, const Config* config) {
|
||||||
@@ -1400,7 +1526,8 @@ Config* config_receive_with_validate(int file_descriptor, ConfigValidateFunc val
|
|||||||
!receive_privilege_options(file_descriptor, config, &budget) ||
|
!receive_privilege_options(file_descriptor, config, &budget) ||
|
||||||
!receive_copy_as_options(file_descriptor, config, &budget) ||
|
!receive_copy_as_options(file_descriptor, config, &budget) ||
|
||||||
!receive_output_options(file_descriptor, config, &budget) ||
|
!receive_output_options(file_descriptor, config, &budget) ||
|
||||||
!receive_codec_options(file_descriptor, config, &budget))
|
!receive_codec_options(file_descriptor, config, &budget) ||
|
||||||
|
!receive_protect_options(file_descriptor, config, &budget))
|
||||||
goto error;
|
goto error;
|
||||||
/* Validate/normalize the negotiated codec. compress_choice is the human
|
/* Validate/normalize the negotiated codec. compress_choice is the human
|
||||||
* spelling (NULL or "" when -z was not given); compression_algo is the
|
* spelling (NULL or "" when -z was not given); compression_algo is the
|
||||||
|
|||||||
+60
-8
@@ -4,6 +4,7 @@
|
|||||||
#include "array_list.h"
|
#include "array_list.h"
|
||||||
#include "checksum.h"
|
#include "checksum.h"
|
||||||
#include "compression.h"
|
#include "compression.h"
|
||||||
|
#include "filter.h"
|
||||||
#include <stdbool.h>
|
#include <stdbool.h>
|
||||||
#include <stdint.h>
|
#include <stdint.h>
|
||||||
#include <stdio.h>
|
#include <stdio.h>
|
||||||
@@ -197,9 +198,18 @@ typedef enum SuperMode { SUPER_MODE_AUTO = 0, SUPER_MODE_ON = 1, SUPER_MODE_OFF
|
|||||||
X(skip_compress_count, int, 0, INT_SKIPCOUNT) \
|
X(skip_compress_count, int, 0, INT_SKIPCOUNT) \
|
||||||
X(skip_compress_suffixes, char**, NULL, BLOCK_SKIP_SUFFIXES)
|
X(skip_compress_suffixes, char**, NULL, BLOCK_SKIP_SUFFIXES)
|
||||||
|
|
||||||
|
/* FastSync-only --verify-basis (protocol 2.28.0, no version bump by project
|
||||||
|
* decision): restores the stricter content equality on a basis hit. By
|
||||||
|
* default a basis hit is accepted on rsync's metadata quick-check alone (equal
|
||||||
|
* size plus equal mtime, or size alone under --size-only); with this flag the
|
||||||
|
* receiver ALSO requires the basis bytes' whole-file digest (the negotiated
|
||||||
|
* --checksum-choice algorithm) to equal the sender's, exactly FastSync's
|
||||||
|
* historical behavior. It is a receiver policy and crosses the wire so the
|
||||||
|
* receiver knows whether to read and hash the basis content. */
|
||||||
#define CONFIG_WIRE_BASIS_FIELDS(X) \
|
#define CONFIG_WIRE_BASIS_FIELDS(X) \
|
||||||
X(basis_count, int, 0, INT_BASISCOUNT) \
|
X(basis_count, int, 0, INT_BASISCOUNT) \
|
||||||
X(basis_dirs, BasisDest*, NULL, BLOCK_BASIS)
|
X(basis_dirs, BasisDest*, NULL, BLOCK_BASIS) \
|
||||||
|
X(verify_basis, bool, false, BOOL)
|
||||||
|
|
||||||
#define CONFIG_WIRE_FUZZY_FIELDS(X) X(fuzzy, bool, false, BOOL)
|
#define CONFIG_WIRE_FUZZY_FIELDS(X) X(fuzzy, bool, false, BOOL)
|
||||||
|
|
||||||
@@ -293,6 +303,20 @@ typedef enum SuperMode { SUPER_MODE_AUTO = 0, SUPER_MODE_ON = 1, SUPER_MODE_OFF
|
|||||||
#define CONFIG_WIRE_CODEC_FIELDS(X) \
|
#define CONFIG_WIRE_CODEC_FIELDS(X) \
|
||||||
X(compression_algo, int, COMPRESSION_ALGO_ZSTD, INT_COMPRESSION_ALGO)
|
X(compression_algo, int, COMPRESSION_ALGO_ZSTD, INT_COMPRESSION_ALGO)
|
||||||
|
|
||||||
|
/* Receiver-side delete-protection filter rules (protocol 2.28.0). The sender
|
||||||
|
* compiles its root-level selection rules exactly as the scanner does
|
||||||
|
* (filter_base_build over --filter/-f/--exclude/--include/-C) and streams them
|
||||||
|
* as one self-describing, bounded block (count followed by per-rule records).
|
||||||
|
* The receiver reconstructs `protect_rules` and evaluates them against
|
||||||
|
* DESTINATION-ONLY entries during the --delete extras walk, so a
|
||||||
|
* `protect`/`P` rule protects an extra that never appeared on the sender
|
||||||
|
* (rsync re-derives deletion protection from the filter list; FastSync
|
||||||
|
* historically derived it only from the source scan). `protect_rules` is NULL
|
||||||
|
* on the sender and is owned/freed by the receiver Config. Bounded by
|
||||||
|
* MAX_FILTER_RULES and MAX_FILTER_BYTES; an unknown action/sides is a protocol
|
||||||
|
* error. */
|
||||||
|
#define CONFIG_WIRE_PROTECT_FIELDS(X) X(protect_rules, FilterRuleList*, NULL, BLOCK_PROTECT_RULES)
|
||||||
|
|
||||||
/* All serialized fields, in exact wire order. Concatenating the per-segment
|
/* All serialized fields, in exact wire order. Concatenating the per-segment
|
||||||
* lists here is what keeps the declaration order = the wire order. */
|
* lists here is what keeps the declaration order = the wire order. */
|
||||||
#define CONFIG_WIRE_FIELDS(X) \
|
#define CONFIG_WIRE_FIELDS(X) \
|
||||||
@@ -315,7 +339,8 @@ typedef enum SuperMode { SUPER_MODE_AUTO = 0, SUPER_MODE_ON = 1, SUPER_MODE_OFF
|
|||||||
CONFIG_WIRE_PRIVILEGE_FIELDS(X) \
|
CONFIG_WIRE_PRIVILEGE_FIELDS(X) \
|
||||||
CONFIG_WIRE_COPY_AS_FIELDS(X) \
|
CONFIG_WIRE_COPY_AS_FIELDS(X) \
|
||||||
CONFIG_WIRE_OUTPUT_FIELDS(X) \
|
CONFIG_WIRE_OUTPUT_FIELDS(X) \
|
||||||
CONFIG_WIRE_CODEC_FIELDS(X)
|
CONFIG_WIRE_CODEC_FIELDS(X) \
|
||||||
|
CONFIG_WIRE_PROTECT_FIELDS(X)
|
||||||
|
|
||||||
typedef struct Config {
|
typedef struct Config {
|
||||||
/* -j/--threads=N: number of parallel scanner worker threads for the -m
|
/* -j/--threads=N: number of parallel scanner worker threads for the -m
|
||||||
@@ -617,10 +642,14 @@ typedef struct Config {
|
|||||||
source directory is streamed in directory order, and the receiver removes
|
source directory is streamed in directory order, and the receiver removes
|
||||||
each directory's extras when its plan arrives (during) or snapshots them
|
each directory's extras when its plan arrives (during) or snapshots them
|
||||||
and removes them only after a successful transfer (delay). delete_after
|
and removes them only after a successful transfer (delay). delete_after
|
||||||
(and plain --delete) keep the whole-tree commit mode: extras are removed
|
keeps the whole-tree commit mode: extras are removed from a fresh
|
||||||
from a fresh end-of-transfer destination scan only after the whole transfer
|
end-of-transfer destination scan only after the whole transfer succeeded.
|
||||||
succeeded. See config_delete_timing_early()/config_delete_timing_per_dir()
|
A plain --delete with no explicit timing flag defaults to delete_during on
|
||||||
below. */
|
the client (cli_finalize_config), matching rsync's --del default; the old
|
||||||
|
late-commit behavior is selected explicitly by --delete-after or the
|
||||||
|
FastSync-only long spelling --delete-commit (an exact alias for
|
||||||
|
--delete-after, mapped onto the same wire field). See
|
||||||
|
config_delete_timing_early()/config_delete_timing_per_dir() below. */
|
||||||
/* partial_dir */
|
/* partial_dir */
|
||||||
// PR #174: Partial transfer resumption
|
// PR #174: Partial transfer resumption
|
||||||
/* suffix */
|
/* suffix */
|
||||||
@@ -1025,11 +1054,31 @@ typedef struct Config {
|
|||||||
* same-version handshake (config_receive rejects a mismatched version before
|
* same-version handshake (config_receive rejects a mismatched version before
|
||||||
* parsing anything else) keeps mixed deployments from ever reaching that
|
* parsing anything else) keeps mixed deployments from ever reaching that
|
||||||
* state. */
|
* state. */
|
||||||
|
/* (9) Receiver-side delete protection (still protocol 2.28.0): the config frame
|
||||||
|
* gains one trailing self-describing block carrying the sender's compiled base
|
||||||
|
* filter rules so the receiver can protect DESTINATION-ONLY entries from
|
||||||
|
* --delete with `protect`/`risk` rules (rsync parity). The block appends after
|
||||||
|
* compression_algo; see CONFIG_WIRE_PROTECT_FIELDS. */
|
||||||
#define PROTOCOL_VERSION "2.28.0"
|
#define PROTOCOL_VERSION "2.28.0"
|
||||||
#define DEFAULT_CHUNK_SIZE (10 * 1024 * 1024)
|
#define DEFAULT_CHUNK_SIZE (10 * 1024 * 1024)
|
||||||
/* Upper bound on total basis-dir entries (rsync caps --link-dest at 20). */
|
/* Upper bound on total basis-dir entries (rsync caps --link-dest at 20). */
|
||||||
#define MAX_BASIS_DIRS 64
|
#define MAX_BASIS_DIRS 64
|
||||||
|
|
||||||
|
/* Bounds on the received receiver-side delete-protection rule block. The rule
|
||||||
|
* count and the aggregate pattern+owner bytes are each capped so a hostile
|
||||||
|
* peer cannot pin unbounded pre-auth memory; both are validated strictly on
|
||||||
|
* receive (alongside the per-string ConfigStringBudget). */
|
||||||
|
/* A peer may supply protect rules; cap the list so a crafted config cannot make
|
||||||
|
* the receiver's delete walk evaluate an unbounded number of glob patterns per
|
||||||
|
* destination entry (glob_match is O(pattern x path)). 1024 is far above any
|
||||||
|
* legitimate selection. */
|
||||||
|
#define MAX_FILTER_RULES 1024
|
||||||
|
#define MAX_FILTER_BYTES (256 * 1024)
|
||||||
|
/* glob_match's DP is capped at 64 Mi work units; a pattern longer than this
|
||||||
|
* could exceed the cap against a PATH_MAX path and silently stop matching,
|
||||||
|
* leaving a protect rule inert. Reject such a rule at receive time. */
|
||||||
|
#define MAX_PROTECT_PATTERN_LEN 8192
|
||||||
|
|
||||||
/* Upper bound on the number of --skip-compress suffixes accepted from the wire.
|
/* Upper bound on the number of --skip-compress suffixes accepted from the wire.
|
||||||
* Each suffix is an independent wire string (up to MAX_STRING_SIZE = 64 KiB), so
|
* Each suffix is an independent wire string (up to MAX_STRING_SIZE = 64 KiB), so
|
||||||
* without this a hostile pre-auth client could otherwise retain
|
* without this a hostile pre-auth client could otherwise retain
|
||||||
@@ -1131,8 +1180,11 @@ bool config_delete_timing_early(const Config* config);
|
|||||||
* commits them only after a fully-successful transfer (delay). */
|
* commits them only after a fully-successful transfer (delay). */
|
||||||
bool config_delete_timing_per_dir(const Config* config);
|
bool config_delete_timing_per_dir(const Config* config);
|
||||||
/* Delete-timing sanity: with deletion enabled at most one timing flag may be
|
/* Delete-timing sanity: with deletion enabled at most one timing flag may be
|
||||||
* set (none = the default delete-after commit timing); without deletion no
|
* set; without deletion no timing flag may be set (each timing flag implies
|
||||||
* timing flag may be set (each timing flag implies --delete). */
|
* --delete). A plain --delete is normalized to delete_during by
|
||||||
|
* cli_finalize_config on the client, so a transmitted use_delete config always
|
||||||
|
* carries exactly one timing; the zero-timing case remains valid only for a
|
||||||
|
* config that has not been through the CLI. */
|
||||||
bool config_has_valid_delete_timing(const Config* config);
|
bool config_has_valid_delete_timing(const Config* config);
|
||||||
|
|
||||||
/* Single source of truth for the cross-field ("combination") invariants a
|
/* Single source of truth for the cross-field ("combination") invariants a
|
||||||
|
|||||||
@@ -331,6 +331,8 @@ static int send_plan_node(int fd, DeletePlanSender* sender, PlanNode* node) {
|
|||||||
return -1;
|
return -1;
|
||||||
sender->config_sent = true;
|
sender->config_sent = true;
|
||||||
}
|
}
|
||||||
|
if (!send_int(fd, 1)) /* apply = true */
|
||||||
|
return -1;
|
||||||
if (!send_wire_str(fd, node->dir))
|
if (!send_wire_str(fd, node->dir))
|
||||||
return -1;
|
return -1;
|
||||||
if (send_str_section(fd, node->dirs) != 0 || send_str_section(fd, node->files) != 0)
|
if (send_str_section(fd, node->dirs) != 0 || send_str_section(fd, node->files) != 0)
|
||||||
@@ -339,6 +341,29 @@ static int send_plan_node(int fd, DeletePlanSender* sender, PlanNode* node) {
|
|||||||
return 0;
|
return 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Transmit the one-shot per-run config block (protected prefixes, size-pruned
|
||||||
|
* mirrors, --delete-missing-args exact paths) on its own carrier frame, with
|
||||||
|
* apply=false so the receiver consumes the config but walks nothing. This is
|
||||||
|
* how the config still reaches the receiver when the scope allows no directory
|
||||||
|
* plan at all (a --files-from list of bare files synchronizes no directory):
|
||||||
|
* without it, the missing-args exact deletions would be lost. Idempotent. */
|
||||||
|
static int send_config_only(int fd, DeletePlanSender* sender) {
|
||||||
|
if (!sender || sender->config_sent)
|
||||||
|
return 0;
|
||||||
|
if (!send_status(fd, STATUS_DELETE_PLAN) || !send_int(fd, 1))
|
||||||
|
return -1;
|
||||||
|
if (send_str_section(fd, sender->protected_prefixes) != 0 ||
|
||||||
|
send_str_section(fd, sender->size_skipped) != 0 ||
|
||||||
|
send_str_section(fd, sender->missing_args) != 0)
|
||||||
|
return -1;
|
||||||
|
sender->config_sent = true;
|
||||||
|
if (!send_int(fd, 0)) /* apply = false */
|
||||||
|
return -1;
|
||||||
|
if (!send_wire_str(fd, ".") || !send_int(fd, 0) || !send_int(fd, 0))
|
||||||
|
return -1;
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
static int send_prefix_plan(int fd, DeletePlanSender* sender, const char* dir) {
|
static int send_prefix_plan(int fd, DeletePlanSender* sender, const char* dir) {
|
||||||
PlanNode* node = plan_find(sender, dir);
|
PlanNode* node = plan_find(sender, dir);
|
||||||
if (!node || node->sent)
|
if (!node || node->sent)
|
||||||
@@ -354,6 +379,10 @@ int delete_plan_send_root(int fd, DeletePlanSender* sender) {
|
|||||||
const char* root = sender->walk_root ? sender->walk_root : ".";
|
const char* root = sender->walk_root ? sender->walk_root : ".";
|
||||||
if (!plan_ensure(sender, root))
|
if (!plan_ensure(sender, root))
|
||||||
return -1;
|
return -1;
|
||||||
|
/* Put the config block on the wire first, on its own carrier frame, so the
|
||||||
|
receiver always sees it even when the scope permits no directory plan. */
|
||||||
|
if (send_config_only(fd, sender) != 0)
|
||||||
|
return -1;
|
||||||
return send_prefix_plan(fd, sender, root);
|
return send_prefix_plan(fd, sender, root);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -546,12 +575,18 @@ static int open_plan_dir(const Config* config, const char* dir) {
|
|||||||
typedef struct PlanSkips {
|
typedef struct PlanSkips {
|
||||||
DeleteSkipEntry* entries;
|
DeleteSkipEntry* entries;
|
||||||
int count;
|
int count;
|
||||||
|
/* Receiver-side delete-protection rules received on the config frame (NULL
|
||||||
|
when the sender sent none). Evaluated per extra so a protect/risk rule is
|
||||||
|
honored under --delete-during/--delete-delay exactly like the whole-tree
|
||||||
|
commit walker. */
|
||||||
|
const FilterRuleList* protect_rules;
|
||||||
} PlanSkips;
|
} PlanSkips;
|
||||||
|
|
||||||
static bool build_plan_skips(const Config* config, const DeletePlanSession* session,
|
static bool build_plan_skips(const Config* config, const DeletePlanSession* session,
|
||||||
PlanSkips* out) {
|
PlanSkips* out) {
|
||||||
out->entries = NULL;
|
out->entries = NULL;
|
||||||
out->count = 0;
|
out->count = 0;
|
||||||
|
out->protect_rules = config->protect_rules;
|
||||||
int count = (config->delay_updates ? 1 : 0) + config->basis_count +
|
int count = (config->delay_updates ? 1 : 0) + config->basis_count +
|
||||||
session->protected_prefixes->size + session->size_skipped->size;
|
session->protected_prefixes->size + session->size_skipped->size;
|
||||||
if (count == 0)
|
if (count == 0)
|
||||||
@@ -733,6 +768,10 @@ static bool process_children(int dirfd, const char* dir_rel, const ArrayList* ke
|
|||||||
bool is_dir = S_ISDIR(st.st_mode);
|
bool is_dir = S_ISDIR(st.st_mode);
|
||||||
bool in_keep_dirs = is_dir && list_contains_str(keep_dirs, entry->d_name);
|
bool in_keep_dirs = is_dir && list_contains_str(keep_dirs, entry->d_name);
|
||||||
bool in_keep_files = !is_dir && list_contains_str(keep_files, entry->d_name);
|
bool in_keep_files = !is_dir && list_contains_str(keep_files, entry->d_name);
|
||||||
|
bool rule_protected =
|
||||||
|
skips->protect_rules &&
|
||||||
|
filter_rules_apply_side(skips->protect_rules, child_rel, entry->d_name, is_dir,
|
||||||
|
FILTER_SIDE_RECEIVER) == FILTER_ACTION_PROTECT;
|
||||||
if (in_keep_dirs) {
|
if (in_keep_dirs) {
|
||||||
local_survives = true;
|
local_survives = true;
|
||||||
} else if (keep_dirs && !is_dir && list_contains_str(keep_dirs, entry->d_name)) {
|
} else if (keep_dirs && !is_dir && list_contains_str(keep_dirs, entry->d_name)) {
|
||||||
@@ -750,11 +789,18 @@ static bool process_children(int dirfd, const char* dir_rel, const ArrayList* ke
|
|||||||
else if (!removed)
|
else if (!removed)
|
||||||
local_survives = true;
|
local_survives = true;
|
||||||
} else if (is_dir) {
|
} else if (is_dir) {
|
||||||
bool removed = false;
|
if (rule_protected) {
|
||||||
if (!process_extra_dir(dirfd, entry->d_name, child_rel, force_now, skips, session, &removed))
|
|
||||||
operation_ok = false;
|
|
||||||
else if (!removed)
|
|
||||||
local_survives = true;
|
local_survives = true;
|
||||||
|
} else {
|
||||||
|
bool removed = false;
|
||||||
|
if (!process_extra_dir(dirfd, entry->d_name, child_rel, force_now, skips, session,
|
||||||
|
&removed))
|
||||||
|
operation_ok = false;
|
||||||
|
else if (!removed)
|
||||||
|
local_survives = true;
|
||||||
|
}
|
||||||
|
} else if (rule_protected) {
|
||||||
|
local_survives = true;
|
||||||
} else {
|
} else {
|
||||||
if (!process_extra_file(dirfd, entry->d_name, child_rel, force_now, session))
|
if (!process_extra_file(dirfd, entry->d_name, child_rel, force_now, session))
|
||||||
operation_ok = false;
|
operation_ok = false;
|
||||||
@@ -833,6 +879,14 @@ int delete_plan_session_receive(DeletePlanSession* session, const Config* config
|
|||||||
}
|
}
|
||||||
session->config_seen = true;
|
session->config_seen = true;
|
||||||
}
|
}
|
||||||
|
/* apply=false is the config-only carrier frame: the receiver consumes the
|
||||||
|
config (and the missing-args exact deletions) but must not walk any
|
||||||
|
directory. Every real plan carries apply=true. */
|
||||||
|
int apply;
|
||||||
|
if (!receive_int(fd, &apply) || (apply != 0 && apply != 1)) {
|
||||||
|
send_status(fd, STATUS_ERROR);
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
char* dir = receive_wire_str(fd);
|
char* dir = receive_wire_str(fd);
|
||||||
ArrayList* dirs = array_list_create(free);
|
ArrayList* dirs = array_list_create(free);
|
||||||
ArrayList* files = array_list_create(free);
|
ArrayList* files = array_list_create(free);
|
||||||
@@ -850,7 +904,7 @@ int delete_plan_session_receive(DeletePlanSession* session, const Config* config
|
|||||||
if (!session->dry_run && enabled) {
|
if (!session->dry_run && enabled) {
|
||||||
if (!session->defer && !apply_missing(session, config))
|
if (!session->defer && !apply_missing(session, config))
|
||||||
ok = false;
|
ok = false;
|
||||||
if (ok && !apply_plan_dir(session, config, dir, dirs, files))
|
if (ok && apply && !apply_plan_dir(session, config, dir, dirs, files))
|
||||||
ok = false;
|
ok = false;
|
||||||
}
|
}
|
||||||
free(dir);
|
free(dir);
|
||||||
|
|||||||
@@ -48,11 +48,14 @@ void delete_plan_sender_finalize(DeletePlanSender* sender, const ArrayList* sync
|
|||||||
Directory keep entries do not count, so an I/O error that hid every file
|
Directory keep entries do not count, so an I/O error that hid every file
|
||||||
still refuses to delete. */
|
still refuses to delete. */
|
||||||
bool delete_plan_sender_empty(const DeletePlanSender* sender);
|
bool delete_plan_sender_empty(const DeletePlanSender* sender);
|
||||||
/* Attach the global config sections advertised on the first plan frame. */
|
/* Attach the global config sections advertised on the first plan frame. The
|
||||||
|
* block is always transmitted by delete_plan_send_root(), on a config-only
|
||||||
|
* carrier frame when the scope allows no directory plan. */
|
||||||
void delete_plan_sender_set_config(DeletePlanSender* sender, const ArrayList* protected_prefixes,
|
void delete_plan_sender_set_config(DeletePlanSender* sender, const ArrayList* protected_prefixes,
|
||||||
const ArrayList* size_skipped, const ArrayList* missing_args);
|
const ArrayList* size_skipped, const ArrayList* missing_args);
|
||||||
/* Send the root plan (even before any data, so root extras are handled like
|
/* Send the root plan (even before any data, so root extras are handled like
|
||||||
* rsync's first generator directory). Returns -1 on I/O error. */
|
* rsync's first generator directory), after transmitting the per-run config
|
||||||
|
* block on its own carrier frame. Returns -1 on I/O error. */
|
||||||
int delete_plan_send_root(int fd, DeletePlanSender* sender);
|
int delete_plan_send_root(int fd, DeletePlanSender* sender);
|
||||||
/* Send the plans for every ancestor of `path` (root-first) and, when is_dir,
|
/* Send the plans for every ancestor of `path` (root-first) and, when is_dir,
|
||||||
* for `path` itself; already-sent plans are skipped. */
|
* for `path` itself; already-sent plans are skipped. */
|
||||||
|
|||||||
+209
-1
@@ -15,6 +15,7 @@
|
|||||||
#include <unistd.h>
|
#include <unistd.h>
|
||||||
|
|
||||||
#include "data.h"
|
#include "data.h"
|
||||||
|
#include "checksum.h"
|
||||||
#include "delta.h"
|
#include "delta.h"
|
||||||
#include "file.h"
|
#include "file.h"
|
||||||
#include "file_store.h"
|
#include "file_store.h"
|
||||||
@@ -24,6 +25,13 @@
|
|||||||
#include "utils.h"
|
#include "utils.h"
|
||||||
#include "protocol.h"
|
#include "protocol.h"
|
||||||
#include "xattr.h"
|
#include "xattr.h"
|
||||||
|
#include <fcntl.h>
|
||||||
|
#include <unistd.h>
|
||||||
|
|
||||||
|
/* Files larger than this are not loaded whole for transfer (the sender streams
|
||||||
|
* them); a whole-file digest is computed from the path instead. Kept in sync
|
||||||
|
* with the sender's streaming threshold. */
|
||||||
|
#define STREAM_THRESHOLD (64ULL * 1024 * 1024)
|
||||||
|
|
||||||
static bool write_all(int fd, const void* data, unsigned long long size) {
|
static bool write_all(int fd, const void* data, unsigned long long size) {
|
||||||
const unsigned char* p = data;
|
const unsigned char* p = data;
|
||||||
@@ -39,6 +47,31 @@ static bool write_all(int fd, const void* data, unsigned long long size) {
|
|||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Streaming copy of an open source descriptor into the just-created destination
|
||||||
|
`fd` (already at offset 0). Used by the --copy-dest basis install so a basis
|
||||||
|
larger than any in-memory whole-file bound still materializes without
|
||||||
|
buffering the entire file. `expected_size` is the caller-verified basis
|
||||||
|
size; the copy must produce exactly that many bytes (a short source is a hard
|
||||||
|
error, never a silently truncated destination). The final ftruncate drops
|
||||||
|
any residual tail a raced-in longer source might have left. */
|
||||||
|
static bool copy_fd_all(int dst_fd, int src_fd, unsigned long long expected_size) {
|
||||||
|
unsigned char buf[1 << 20];
|
||||||
|
unsigned long long done = 0;
|
||||||
|
while (done < expected_size) {
|
||||||
|
unsigned long long remaining = expected_size - done;
|
||||||
|
size_t want = remaining < sizeof(buf) ? (size_t)remaining : sizeof(buf);
|
||||||
|
ssize_t n = read(src_fd, buf, want);
|
||||||
|
if (n < 0 && errno == EINTR)
|
||||||
|
continue;
|
||||||
|
if (n <= 0)
|
||||||
|
return false;
|
||||||
|
if (!write_all(dst_fd, buf, (unsigned long long)n))
|
||||||
|
return false;
|
||||||
|
done += (unsigned long long)n;
|
||||||
|
}
|
||||||
|
return ftruncate(dst_fd, (off_t)expected_size) == 0;
|
||||||
|
}
|
||||||
|
|
||||||
/* Preallocate `size` bytes on `fd` before any data is written (--preallocate).
|
/* Preallocate `size` bytes on `fd` before any data is written (--preallocate).
|
||||||
* fallocate(2) reserves real disk blocks, so an out-of-space condition
|
* fallocate(2) reserves real disk blocks, so an out-of-space condition
|
||||||
* (ENOSPC/EDQUOT) surfaces up front instead of partway through a transfer;
|
* (ENOSPC/EDQUOT) surfaces up front instead of partway through a transfer;
|
||||||
@@ -140,6 +173,12 @@ bool file_checksum(File* file, ChecksumAlgo algo, uint64_t seed, uint8_t* out, s
|
|||||||
if (file->data->size == 0) {
|
if (file->data->size == 0) {
|
||||||
return checksum_digest(algo, seed, "", 0, out, out_capacity, out_len);
|
return checksum_digest(algo, seed, "", 0, out, out_capacity, out_len);
|
||||||
}
|
}
|
||||||
|
/* A streamed source (data not loaded) may exceed any in-memory whole-file
|
||||||
|
bound; hash it from the file path in bounded buffers instead of forcing a
|
||||||
|
full load. This is the same digest the receiver recomputes on the basis. */
|
||||||
|
if (!file->data->data && file->path && file->data->size > STREAM_THRESHOLD &&
|
||||||
|
checksum_digest_file(algo, seed, file->path, out, out_capacity, out_len))
|
||||||
|
return true;
|
||||||
if (!file->data->data && !file_load_data(file))
|
if (!file->data->data && !file_load_data(file))
|
||||||
return false;
|
return false;
|
||||||
return checksum_digest(algo, seed, file->data->data, file->data->size, out, out_capacity,
|
return checksum_digest(algo, seed, file->data->data, file->data->size, out, out_capacity,
|
||||||
@@ -176,6 +215,7 @@ File* file_create(const char* path) {
|
|||||||
file->is_dir = false;
|
file->is_dir = false;
|
||||||
file->dir_time_only = false;
|
file->dir_time_only = false;
|
||||||
file->basis_link = NULL;
|
file->basis_link = NULL;
|
||||||
|
file->basis_copy = NULL;
|
||||||
file->link_group = 0;
|
file->link_group = 0;
|
||||||
file->link_first = false;
|
file->link_first = false;
|
||||||
file->hardlink_target = NULL;
|
file->hardlink_target = NULL;
|
||||||
@@ -205,6 +245,8 @@ void file_destroy(void* item) {
|
|||||||
file->send_path = NULL;
|
file->send_path = NULL;
|
||||||
free(file->basis_link);
|
free(file->basis_link);
|
||||||
file->basis_link = NULL;
|
file->basis_link = NULL;
|
||||||
|
free(file->basis_copy);
|
||||||
|
file->basis_copy = NULL;
|
||||||
free(file->hardlink_target);
|
free(file->hardlink_target);
|
||||||
file->hardlink_target = NULL;
|
file->hardlink_target = NULL;
|
||||||
free(file->symlink_target);
|
free(file->symlink_target);
|
||||||
@@ -1512,6 +1554,161 @@ bool file_to_disk_secure_attrs_counted(const char* path, const void* data,
|
|||||||
* basis). Likewise `xattrs`/`fake_super` are applied only on the copy
|
* basis). Likewise `xattrs`/`fake_super` are applied only on the copy
|
||||||
* fallback, so a fallback copy preserves the per-file attributes instead of
|
* fallback, so a fallback copy preserves the per-file attributes instead of
|
||||||
* silently dropping them. */
|
* silently dropping them. */
|
||||||
|
/* Streaming --copy-dest basis install: atomically materialize `path` from the
|
||||||
|
* bytes of `basis_path` without holding the file in memory, so a basis larger
|
||||||
|
* than any whole-file bound still works. Mirrors the ordinary secure store
|
||||||
|
* path (confined parent walk, temp + rename, --update/--ignore-existing/
|
||||||
|
* --preallocate/--temp-dir) but sources the data from the basis descriptor
|
||||||
|
* rather than a caller buffer, and applies the SOURCE metadata (rsync copies
|
||||||
|
* then fixes attributes). A hard-link install that falls back to a byte copy
|
||||||
|
* also routes through here when the caller supplies the basis path. */
|
||||||
|
static bool file_copy_basis_stream_impl(const char* path, const char* basis_path,
|
||||||
|
unsigned long long expected_size, bool preallocate,
|
||||||
|
const FileMetadata* metadata, FileAttrPolicy policy,
|
||||||
|
bool update, bool no_replace, bool use_fsync,
|
||||||
|
const FileXattrList* xattrs, bool fake_super,
|
||||||
|
const char* temp_dir, unsigned* dirs_created,
|
||||||
|
const char* count_floor) {
|
||||||
|
if (!path || !basis_path)
|
||||||
|
return false;
|
||||||
|
char* leaf = NULL;
|
||||||
|
int dirfd = file_open_secure_parent_counted(path, &leaf, true, dirs_created, count_floor);
|
||||||
|
if (dirfd < 0)
|
||||||
|
return false;
|
||||||
|
|
||||||
|
char* basis_leaf = NULL;
|
||||||
|
int basis_dirfd = file_open_secure_parent(basis_path, &basis_leaf, false);
|
||||||
|
int src_fd = -1;
|
||||||
|
if (basis_dirfd >= 0 && basis_leaf != NULL) {
|
||||||
|
/* O_NONBLOCK rejects a raced-in FIFO without blocking; the S_ISREG gate
|
||||||
|
below is the real type check. */
|
||||||
|
src_fd = openat(basis_dirfd, basis_leaf, O_RDONLY | O_CLOEXEC | O_NOFOLLOW | O_NONBLOCK);
|
||||||
|
struct stat src_st;
|
||||||
|
if (src_fd >= 0 && (fstat(src_fd, &src_st) != 0 || !S_ISREG(src_st.st_mode))) {
|
||||||
|
close(src_fd);
|
||||||
|
src_fd = -1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (basis_dirfd >= 0)
|
||||||
|
close(basis_dirfd);
|
||||||
|
free(basis_leaf);
|
||||||
|
if (src_fd < 0) {
|
||||||
|
close(dirfd);
|
||||||
|
free(leaf);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
struct stat destination_stat;
|
||||||
|
bool destination_is_regular = fstatat(dirfd, leaf, &destination_stat, AT_SYMLINK_NOFOLLOW) == 0 &&
|
||||||
|
S_ISREG(destination_stat.st_mode);
|
||||||
|
if (update && metadata && destination_is_regular && stat_is_newer(&destination_stat, metadata)) {
|
||||||
|
close(src_fd);
|
||||||
|
close(dirfd);
|
||||||
|
free(leaf);
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
if (no_replace && file_path_exists_secure(path)) {
|
||||||
|
close(src_fd);
|
||||||
|
close(dirfd);
|
||||||
|
free(leaf);
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
int scratch_dirfd = -1;
|
||||||
|
if (temp_dir) {
|
||||||
|
scratch_dirfd = file_open_temp_dir(temp_dir);
|
||||||
|
if (scratch_dirfd < 0) {
|
||||||
|
int saved_errno = errno;
|
||||||
|
log_message(LOG_LEVEL_ERROR,
|
||||||
|
"--temp-dir '%s' could not be opened (rsync requires it to already exist): %s",
|
||||||
|
temp_dir, strerror(saved_errno));
|
||||||
|
close(src_fd);
|
||||||
|
close(dirfd);
|
||||||
|
free(leaf);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
int target_dirfd = scratch_dirfd >= 0 ? scratch_dirfd : dirfd;
|
||||||
|
int tmp_size = snprintf(NULL, 0, ".%s.tmp.%ld.%llu", leaf, (long)getpid(), ~0ULL);
|
||||||
|
char* tmp = NULL;
|
||||||
|
bool ok = false;
|
||||||
|
if (tmp_size >= 0)
|
||||||
|
tmp = malloc((size_t)tmp_size + 1);
|
||||||
|
if (tmp) {
|
||||||
|
for (unsigned int i = 0; i < 100 && !ok; ++i) {
|
||||||
|
if (scratch_dirfd >= 0)
|
||||||
|
snprintf(tmp, (size_t)tmp_size + 1, ".%s.tmp.%ld.%llu", leaf, (long)getpid(),
|
||||||
|
next_temp_sequence());
|
||||||
|
else
|
||||||
|
snprintf(tmp, (size_t)tmp_size + 1, ".%s.tmp.%ld.%u", leaf, (long)getpid(), i);
|
||||||
|
int fd =
|
||||||
|
openat(target_dirfd, tmp, O_WRONLY | O_CREAT | O_EXCL | O_CLOEXEC | O_NOFOLLOW, 0600);
|
||||||
|
if (fd < 0) {
|
||||||
|
if (errno != EEXIST)
|
||||||
|
break;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
bool wrote = true;
|
||||||
|
if (preallocate && expected_size > 0 && preallocate_fd(fd, expected_size) != 0)
|
||||||
|
wrote = false;
|
||||||
|
if (wrote)
|
||||||
|
wrote = copy_fd_all(fd, src_fd, expected_size);
|
||||||
|
if (wrote && metadata) {
|
||||||
|
if (!policy.perms &&
|
||||||
|
fchmod(fd, file_mode_base(metadata, destination_is_regular,
|
||||||
|
destination_is_regular ? destination_stat.st_mode & 0777
|
||||||
|
: 0)) != 0)
|
||||||
|
wrote = false;
|
||||||
|
if (wrote)
|
||||||
|
wrote = file_restore_metadata_fd(fd, metadata, policy);
|
||||||
|
} else if (wrote && fchmod(fd, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH) != 0) {
|
||||||
|
wrote = false;
|
||||||
|
}
|
||||||
|
if (wrote)
|
||||||
|
restore_extra_fd(fd, metadata, xattrs, fake_super, policy);
|
||||||
|
if (wrote && use_fsync)
|
||||||
|
wrote = fsync(fd) == 0;
|
||||||
|
if (close(fd) != 0)
|
||||||
|
wrote = false;
|
||||||
|
if (wrote && renameat(target_dirfd, tmp, dirfd, leaf) != 0)
|
||||||
|
wrote = false;
|
||||||
|
if (!wrote)
|
||||||
|
unlinkat(target_dirfd, tmp, 0);
|
||||||
|
ok = wrote;
|
||||||
|
}
|
||||||
|
free(tmp);
|
||||||
|
}
|
||||||
|
if (!ok && scratch_dirfd >= 0) {
|
||||||
|
/* Retry once with no scratch dir (rsync's EXDEV fallback). */
|
||||||
|
close(scratch_dirfd);
|
||||||
|
close(src_fd);
|
||||||
|
close(dirfd);
|
||||||
|
free(leaf);
|
||||||
|
return file_copy_basis_stream_impl(path, basis_path, expected_size, preallocate, metadata,
|
||||||
|
policy, update, no_replace, use_fsync, xattrs, fake_super,
|
||||||
|
NULL, dirs_created, count_floor);
|
||||||
|
}
|
||||||
|
if (scratch_dirfd >= 0)
|
||||||
|
close(scratch_dirfd);
|
||||||
|
close(src_fd);
|
||||||
|
close(dirfd);
|
||||||
|
free(leaf);
|
||||||
|
return ok;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* --copy-dest basis install (streaming). Applies the source metadata and the
|
||||||
|
per-file xattrs / --fake-super record. */
|
||||||
|
bool file_copy_basis_stream_attrs(const char* path, const char* basis_path,
|
||||||
|
unsigned long long expected_size, bool preallocate,
|
||||||
|
const FileMetadata* metadata, FileAttrPolicy policy, bool update,
|
||||||
|
bool use_fsync, const FileXattrList* xattrs, bool fake_super,
|
||||||
|
const char* temp_dir) {
|
||||||
|
return file_copy_basis_stream_impl(path, basis_path, expected_size, preallocate, metadata, policy,
|
||||||
|
update, false, use_fsync, xattrs, fake_super, temp_dir, NULL,
|
||||||
|
NULL);
|
||||||
|
}
|
||||||
|
|
||||||
static bool file_to_disk_secure_link_impl(const char* path, const char* basis_path,
|
static bool file_to_disk_secure_link_impl(const char* path, const char* basis_path,
|
||||||
const void* data, unsigned long long data_size,
|
const void* data, unsigned long long data_size,
|
||||||
bool preallocate, const FileMetadata* metadata,
|
bool preallocate, const FileMetadata* metadata,
|
||||||
@@ -1521,6 +1718,10 @@ static bool file_to_disk_secure_link_impl(const char* path, const char* basis_pa
|
|||||||
const char* count_floor) {
|
const char* count_floor) {
|
||||||
if (!path || !basis_path)
|
if (!path || !basis_path)
|
||||||
return false;
|
return false;
|
||||||
|
/* The caller-supplied buffer is no longer used: the copy fallback streams
|
||||||
|
from the basis path (which may hold an over-limit file). Kept in the
|
||||||
|
signature for the existing API. */
|
||||||
|
(void)data;
|
||||||
char* leaf = NULL;
|
char* leaf = NULL;
|
||||||
int dirfd = file_open_secure_parent_counted(path, &leaf, true, dirs_created, count_floor);
|
int dirfd = file_open_secure_parent_counted(path, &leaf, true, dirs_created, count_floor);
|
||||||
if (dirfd < 0)
|
if (dirfd < 0)
|
||||||
@@ -1604,7 +1805,14 @@ static bool file_to_disk_secure_link_impl(const char* path, const char* basis_pa
|
|||||||
close(dirfd);
|
close(dirfd);
|
||||||
free(leaf);
|
free(leaf);
|
||||||
/* The basis file could not be linked in (missing, cross-device, refused
|
/* The basis file could not be linked in (missing, cross-device, refused
|
||||||
by the filesystem). Write a byte-identical local copy instead. */
|
by the filesystem). Stream a byte-identical local copy from the basis
|
||||||
|
itself (never the possibly-absent caller buffer) so an over-limit basis
|
||||||
|
still materializes. When the basis path is not a readable regular file
|
||||||
|
(e.g. a directory raced in), fall back to the caller-supplied bytes. */
|
||||||
|
if (file_copy_basis_stream_impl(path, basis_path, data_size, preallocate, metadata, policy,
|
||||||
|
false, false, use_fsync, xattrs, fake_super, temp_dir,
|
||||||
|
dirs_created, count_floor))
|
||||||
|
return true;
|
||||||
return file_to_disk_secure_attrs_counted(
|
return file_to_disk_secure_attrs_counted(
|
||||||
path, data, data_size, false, false, preallocate, metadata, policy, false, false, use_fsync,
|
path, data, data_size, false, false, preallocate, metadata, policy, false, false, use_fsync,
|
||||||
xattrs, fake_super, false, temp_dir, dirs_created, count_floor);
|
xattrs, fake_super, false, temp_dir, dirs_created, count_floor);
|
||||||
|
|||||||
@@ -163,6 +163,16 @@ bool file_to_disk_secure_link_attrs(const char* path, const char* basis_path, co
|
|||||||
const FileMetadata* metadata, FileAttrPolicy policy,
|
const FileMetadata* metadata, FileAttrPolicy policy,
|
||||||
bool use_fsync, const FileXattrList* xattrs, bool fake_super,
|
bool use_fsync, const FileXattrList* xattrs, bool fake_super,
|
||||||
const char* temp_dir);
|
const char* temp_dir);
|
||||||
|
/* Streaming --copy-dest install: atomically materialize `path` by copying the
|
||||||
|
* bytes of `basis_path` through a bounded buffer (no whole-file buffering, so
|
||||||
|
* an arbitrarily large basis works), applying the SOURCE metadata and the
|
||||||
|
* per-file xattrs / --fake-super record. `update` honors a newer destination;
|
||||||
|
* a --temp-dir scratch location falls back to a direct write on EXDEV. */
|
||||||
|
bool file_copy_basis_stream_attrs(const char* path, const char* basis_path,
|
||||||
|
unsigned long long expected_size, bool preallocate,
|
||||||
|
const FileMetadata* metadata, FileAttrPolicy policy, bool update,
|
||||||
|
bool use_fsync, const FileXattrList* xattrs, bool fake_super,
|
||||||
|
const char* temp_dir);
|
||||||
/* Protocol 2.28.0 receiver-stat variants: like the two above but additionally
|
/* Protocol 2.28.0 receiver-stat variants: like the two above but additionally
|
||||||
* report through `dirs_created` (when non-NULL) how many parent directories the
|
* report through `dirs_created` (when non-NULL) how many parent directories the
|
||||||
* confined secure walk had to create that lie strictly below `count_floor` (a
|
* confined secure walk had to create that lie strictly below `count_floor` (a
|
||||||
|
|||||||
+156
-101
@@ -99,6 +99,12 @@ static FileSaveResult file_stage_delayed_update(const char* root_directory,
|
|||||||
if (file->basis_link) {
|
if (file->basis_link) {
|
||||||
ok = file_to_disk_secure_link(staged_path, file->basis_link, file->data->data, file->data->size,
|
ok = file_to_disk_secure_link(staged_path, file->basis_link, file->data->data, file->data->size,
|
||||||
config->preallocate, metadata, policy, config->use_fsync, NULL);
|
config->preallocate, metadata, policy, config->use_fsync, NULL);
|
||||||
|
} else if (file->basis_copy) {
|
||||||
|
/* --copy-dest basis hit: stream the basis into the staging tree (bounded
|
||||||
|
buffers, so an over-limit basis still stages). */
|
||||||
|
ok = file_copy_basis_stream_attrs(staged_path, file->basis_copy, file->data->size,
|
||||||
|
config->preallocate, metadata, policy, config->update,
|
||||||
|
config->use_fsync, file->xattrs, config->fake_super, NULL);
|
||||||
} else {
|
} else {
|
||||||
ok =
|
ok =
|
||||||
file_to_disk_secure_attrs(staged_path, file->data->data, file->data->size, false, sparse,
|
file_to_disk_secure_attrs(staged_path, file->data->data, file->data->size, false, sparse,
|
||||||
@@ -652,7 +658,8 @@ FileSaveResult file_save_to_disk_full_ex(const char* root_directory, const File*
|
|||||||
char* destination_path = NULL;
|
char* destination_path = NULL;
|
||||||
char *backup_path = NULL, *parent_copy = NULL;
|
char *backup_path = NULL, *parent_copy = NULL;
|
||||||
|
|
||||||
if (!file || !file->path || !file->data || (file->data->size != 0 && !file->data->data) ||
|
if (!file || !file->path || !file->data ||
|
||||||
|
(file->data->size != 0 && !file->data->data && !file->basis_link && !file->basis_copy) ||
|
||||||
(!file_get_trust_sender() && has_path_traversal(file->path)) ||
|
(!file_get_trust_sender() && has_path_traversal(file->path)) ||
|
||||||
(backup_enabled &&
|
(backup_enabled &&
|
||||||
(!backup_suffix || backup_suffix[0] == '\0' || strchr(backup_suffix, '/') != NULL ||
|
(!backup_suffix || backup_suffix[0] == '\0' || strchr(backup_suffix, '/') != NULL ||
|
||||||
@@ -975,6 +982,13 @@ FileSaveResult file_save_to_disk_full_ex(const char* root_directory, const File*
|
|||||||
disk_path, file->basis_link, file->data->data, file->data->size, config->preallocate,
|
disk_path, file->basis_link, file->data->data, file->data->size, config->preallocate,
|
||||||
metadata, policy, config->use_fsync, file->xattrs, config->fake_super, confined_temp,
|
metadata, policy, config->use_fsync, file->xattrs, config->fake_super, confined_temp,
|
||||||
created_dirs, count_floor);
|
created_dirs, count_floor);
|
||||||
|
} else if (config && file->basis_copy) {
|
||||||
|
/* --copy-dest: stream the basis bytes through a bounded buffer so a basis
|
||||||
|
larger than any whole-file bound still materializes. The source
|
||||||
|
metadata was transmitted with the check frame. */
|
||||||
|
ok = file_copy_basis_stream_attrs(
|
||||||
|
disk_path, file->basis_copy, file->data->size, config->preallocate, metadata, policy,
|
||||||
|
config->update, config->use_fsync, file->xattrs, config->fake_super, confined_temp);
|
||||||
} else {
|
} else {
|
||||||
/* The plain no-replace / update / with-fsync engines, plus per-file xattr
|
/* The plain no-replace / update / with-fsync engines, plus per-file xattr
|
||||||
(-X/-A) and --fake-super application on the written fd. */
|
(-X/-A) and --fake-super application on the written fd. */
|
||||||
@@ -1024,6 +1038,13 @@ void receiver_stats_note_saved(ReceiverStats* stats, const File* file, bool crea
|
|||||||
unsigned created_dirs) {
|
unsigned created_dirs) {
|
||||||
if (!stats || !file)
|
if (!stats || !file)
|
||||||
return;
|
return;
|
||||||
|
/* A basis-dir hit (--link-dest/--copy-dest) materializes bytes the sender
|
||||||
|
* never transferred. rsync reports no literal data and no created entry for
|
||||||
|
* such a file, and does not count the parent directories it creates only to
|
||||||
|
* hold it, so exclude the whole entry from the receiver tallies. */
|
||||||
|
bool basis_sourced = file->basis_link != NULL || file->basis_copy != NULL;
|
||||||
|
if (basis_sourced)
|
||||||
|
return;
|
||||||
bool is_sibling = file->link_group != 0 && !file->link_first;
|
bool is_sibling = file->link_group != 0 && !file->link_first;
|
||||||
if (!file->is_dir && !file->is_symlink && !file->is_special && !is_sibling) {
|
if (!file->is_dir && !file->is_symlink && !file->is_special && !is_sibling) {
|
||||||
unsigned long long literal = file->literal_bytes;
|
unsigned long long literal = file->literal_bytes;
|
||||||
@@ -1298,16 +1319,17 @@ static File* receive_delta_file(int fd, const Config* config, const char* check_
|
|||||||
|
|
||||||
/* ---- Alternate basis directories (--compare-dest / --copy-dest / --link-dest) ----
|
/* ---- Alternate basis directories (--compare-dest / --copy-dest / --link-dest) ----
|
||||||
* The receiver consults the ordered basis-dir list only when the destination
|
* The receiver consults the ordered basis-dir list only when the destination
|
||||||
* entry is NOT already up to date. An "exact match" requires an equal size,
|
* entry is NOT already up to date. By default an "exact match" is rsync's
|
||||||
* an equal mtime (unless --size-only), and an equal content xxHash64, so a
|
* metadata quick-check: an equal size and an equal mtime (unless --size-only).
|
||||||
* hard link / local copy is only ever made from byte-identical content. */
|
* The FastSync-only --verify-basis additionally requires an equal whole-file
|
||||||
|
* content digest, so a hard link / local copy is only then made from
|
||||||
|
* byte-verified content. */
|
||||||
|
|
||||||
typedef struct BasisMatch {
|
typedef struct BasisMatch {
|
||||||
bool hit;
|
bool hit;
|
||||||
BasisDestType type;
|
BasisDestType type;
|
||||||
char* basis_path; /* owned absolute path of the matched basis file */
|
char* basis_path; /* owned absolute path of the matched basis file */
|
||||||
struct stat st; /* fstat() of the matched basis file */
|
struct stat st; /* fstat() of the matched basis file */
|
||||||
Data* content; /* owned basis bytes (or empty Data), NULL when not loaded */
|
|
||||||
} BasisMatch;
|
} BasisMatch;
|
||||||
|
|
||||||
static void basis_match_free(BasisMatch* match) {
|
static void basis_match_free(BasisMatch* match) {
|
||||||
@@ -1315,8 +1337,6 @@ static void basis_match_free(BasisMatch* match) {
|
|||||||
return;
|
return;
|
||||||
free(match->basis_path);
|
free(match->basis_path);
|
||||||
match->basis_path = NULL;
|
match->basis_path = NULL;
|
||||||
data_destroy(match->content);
|
|
||||||
match->content = NULL;
|
|
||||||
match->hit = false;
|
match->hit = false;
|
||||||
match->type = BASIS_DEST_NONE;
|
match->type = BASIS_DEST_NONE;
|
||||||
}
|
}
|
||||||
@@ -1348,32 +1368,10 @@ static bool basis_open_regular(const char* path, unsigned long long expected_siz
|
|||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Read the whole remaining content of an open descriptor. A zero-length file
|
|
||||||
yields an empty Data (data pointer NULL). */
|
|
||||||
static Data* basis_read_content(int fd, unsigned long long size) {
|
|
||||||
if (size == 0)
|
|
||||||
return data_create_reserve(0);
|
|
||||||
if (size > MAX_RECEIVE_WHOLE_FILE_SIZE || size > SIZE_MAX)
|
|
||||||
return NULL;
|
|
||||||
void* buf = protocol_alloc((size_t)size);
|
|
||||||
if (!buf)
|
|
||||||
return NULL;
|
|
||||||
size_t got = 0;
|
|
||||||
while (got < (size_t)size) {
|
|
||||||
ssize_t n = read(fd, (char*)buf + got, (size_t)size - got);
|
|
||||||
if (n <= 0) {
|
|
||||||
free(buf);
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
got += (size_t)n;
|
|
||||||
}
|
|
||||||
return data_create(buf, (size_t)size);
|
|
||||||
}
|
|
||||||
|
|
||||||
/* --ignore-times forces every file to be updated, so no basis hit is ever
|
/* --ignore-times forces every file to be updated, so no basis hit is ever
|
||||||
declared (matching rsync, where -I prevents link-dest from linking). */
|
declared (matching rsync, where -I prevents link-dest from linking). */
|
||||||
static bool basis_quick_matches(const Config* config, const struct stat* st, time_t check_mtime,
|
bool file_basis_quick_match(const Config* config, const struct stat* st, time_t check_mtime,
|
||||||
long check_mtime_nsec) {
|
long check_mtime_nsec) {
|
||||||
if (config->size_only)
|
if (config->size_only)
|
||||||
return true;
|
return true;
|
||||||
long mtime_nsec = 0;
|
long mtime_nsec = 0;
|
||||||
@@ -1384,28 +1382,39 @@ static bool basis_quick_matches(const Config* config, const struct stat* st, tim
|
|||||||
config->modify_window);
|
config->modify_window);
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Search the basis-dir list in command-line order and return the first exact
|
/* True when a basis hit must be confirmed by a whole-file content digest
|
||||||
match. When load_content is true the matched bytes are kept in out->content
|
(--verify-basis). False is the rsync-parity default: the metadata
|
||||||
so the caller can materialize the file without re-reading it.
|
quick-check alone decides a hit. */
|
||||||
|
bool file_basis_content_required(const Config* config) {
|
||||||
|
return config != NULL && config->verify_basis;
|
||||||
|
}
|
||||||
|
|
||||||
An exact match ALSO requires the basis bytes' digest to equal the source's,
|
/* Search the basis-dir list in command-line order and return the first match.
|
||||||
so `hash_content` gates the content read/hash itself. A server-contacting
|
By default (no --verify-basis) rsync's metadata quick-check is sufficient:
|
||||||
--dry-run passes hash_content=false: no basis file may be read or hashed
|
basis_open_regular has already required an equal size, and
|
||||||
(that would be a 1-bit content oracle against a client-supplied digest), so a
|
file_basis_quick_match applies rsync's mtime (or --size-only) rule.
|
||||||
metadata-only pass can never confirm a hit and declines it. The real path
|
--verify-basis additionally requires the basis bytes' whole-file digest to
|
||||||
always passes hash_content=true, keeping its behavior byte-for-byte. */
|
equal the sender's, restoring FastSync's historical content equality; that
|
||||||
|
digest is computed by streaming the open basis descriptor, so an arbitrarily
|
||||||
|
large basis is verified without buffering it. A copy/link install re-reads
|
||||||
|
the basis from its path in bounded buffers, so no content buffer is kept.
|
||||||
|
|
||||||
|
`hash_content` gates content READS under --verify-basis: a server-contacting
|
||||||
|
--dry-run passes false because hashing a basis against a client-supplied
|
||||||
|
digest would be a 1-bit content oracle. Without --verify-basis a dry-run can
|
||||||
|
still confirm the metadata-only hit without reading any basis bytes, matching
|
||||||
|
rsync's read-only quick-check. */
|
||||||
static bool basis_match_find(const Config* config, const char* check_path,
|
static bool basis_match_find(const Config* config, const char* check_path,
|
||||||
unsigned long long check_size, time_t check_mtime,
|
unsigned long long check_size, time_t check_mtime,
|
||||||
long check_mtime_nsec, const uint8_t* check_digest,
|
long check_mtime_nsec, const uint8_t* check_digest,
|
||||||
size_t check_digest_len, bool load_content, bool hash_content,
|
size_t check_digest_len, bool hash_content, BasisMatch* out) {
|
||||||
BasisMatch* out) {
|
|
||||||
memset(out, 0, sizeof(*out));
|
memset(out, 0, sizeof(*out));
|
||||||
if (!config || !config_has_basis(config) || config->ignore_times)
|
if (!config || !config_has_basis(config) || config->ignore_times)
|
||||||
return false;
|
return false;
|
||||||
/* Dry-run: never read/hash basis content. A hit cannot be decided from
|
/* --verify-basis needs the basis content; a content-blind (dry-run) pass can
|
||||||
metadata alone, so report no match (the caller treats it as would-transfer)
|
never confirm it and must not read the file, so decline without touching
|
||||||
without touching the file's contents. */
|
the basis bytes. */
|
||||||
if (!hash_content)
|
if (file_basis_content_required(config) && !hash_content)
|
||||||
return false;
|
return false;
|
||||||
for (int i = 0; i < config->basis_count; i++) {
|
for (int i = 0; i < config->basis_count; i++) {
|
||||||
const BasisDest* entry = &config->basis_dirs[i];
|
const BasisDest* entry = &config->basis_dirs[i];
|
||||||
@@ -1424,29 +1433,26 @@ static bool basis_match_find(const Config* config, const char* check_path,
|
|||||||
int fd;
|
int fd;
|
||||||
struct stat st;
|
struct stat st;
|
||||||
if (basis_open_regular(candidate, check_size, &fd, &st)) {
|
if (basis_open_regular(candidate, check_size, &fd, &st)) {
|
||||||
if (basis_quick_matches(config, &st, check_mtime, check_mtime_nsec)) {
|
if (file_basis_quick_match(config, &st, check_mtime, check_mtime_nsec)) {
|
||||||
Data* content = basis_read_content(fd, check_size);
|
bool hit = true;
|
||||||
if (content) {
|
if (file_basis_content_required(config)) {
|
||||||
uint8_t basis_digest[CHECKSUM_MAX_DIGEST_LEN];
|
uint8_t basis_digest[CHECKSUM_MAX_DIGEST_LEN];
|
||||||
size_t basis_len = 0;
|
size_t basis_len = 0;
|
||||||
bool hashed = checksum_digest((ChecksumAlgo)config->checksum_algo, config->checksum_seed,
|
bool hashed =
|
||||||
content->data, content->size, basis_digest,
|
checksum_digest_fd((ChecksumAlgo)config->checksum_algo, config->checksum_seed, fd,
|
||||||
sizeof(basis_digest), &basis_len);
|
basis_digest, sizeof(basis_digest), &basis_len);
|
||||||
if (hashed && basis_len == check_digest_len && check_digest_len > 0 &&
|
hit = hashed && basis_len == check_digest_len && check_digest_len > 0 &&
|
||||||
memcmp(basis_digest, check_digest, check_digest_len) == 0) {
|
memcmp(basis_digest, check_digest, check_digest_len) == 0;
|
||||||
out->hit = true;
|
}
|
||||||
out->type = entry->type;
|
if (hit) {
|
||||||
out->basis_path = candidate;
|
out->hit = true;
|
||||||
candidate = NULL; /* ownership transferred to out */
|
out->type = entry->type;
|
||||||
out->st = st;
|
out->basis_path = candidate;
|
||||||
out->content = load_content ? content : NULL;
|
candidate = NULL; /* ownership transferred to out */
|
||||||
if (!load_content)
|
out->st = st;
|
||||||
data_destroy(content);
|
close(fd);
|
||||||
close(fd);
|
return true;
|
||||||
return true;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
data_destroy(content);
|
|
||||||
}
|
}
|
||||||
close(fd);
|
close(fd);
|
||||||
}
|
}
|
||||||
@@ -1871,6 +1877,12 @@ typedef struct {
|
|||||||
long long check_mtime_nsec;
|
long long check_mtime_nsec;
|
||||||
uint8_t check_digest[CHECKSUM_MAX_DIGEST_LEN];
|
uint8_t check_digest[CHECKSUM_MAX_DIGEST_LEN];
|
||||||
size_t check_digest_len;
|
size_t check_digest_len;
|
||||||
|
/* Source metadata carried alongside the check frame whenever a basis dir is
|
||||||
|
configured (rsync keeps the whole file list; FastSync's sender-driven
|
||||||
|
incremental path otherwise never transmits metadata for a SKIPPED file).
|
||||||
|
A basis materialization applies these SOURCE attributes instead of the
|
||||||
|
basis inode's, matching rsync's "copy then fix attributes". */
|
||||||
|
FileMetadata* source_metadata;
|
||||||
bool dest_exists; /* any destination entry exists (lstat succeeded) */
|
bool dest_exists; /* any destination entry exists (lstat succeeded) */
|
||||||
bool has_old_file;
|
bool has_old_file;
|
||||||
int old_fd;
|
int old_fd;
|
||||||
@@ -1903,6 +1915,8 @@ static void incremental_check_state_cleanup(IncrementalCheckState* state) {
|
|||||||
if (state->old_fd >= 0)
|
if (state->old_fd >= 0)
|
||||||
close(state->old_fd);
|
close(state->old_fd);
|
||||||
state->old_fd = -1;
|
state->old_fd = -1;
|
||||||
|
file_metadata_destroy(state->source_metadata);
|
||||||
|
state->source_metadata = NULL;
|
||||||
free(state->full_path);
|
free(state->full_path);
|
||||||
state->full_path = NULL;
|
state->full_path = NULL;
|
||||||
free(state->check_path);
|
free(state->check_path);
|
||||||
@@ -1927,7 +1941,7 @@ static IncrementalCheckOutcome incremental_check_receive_request(IncrementalChec
|
|||||||
send_error_detail(fd, "invalid check mtime nanoseconds");
|
send_error_detail(fd, "invalid check mtime nanoseconds");
|
||||||
return INCREMENTAL_ERROR;
|
return INCREMENTAL_ERROR;
|
||||||
}
|
}
|
||||||
if ((config->checksum || config_has_basis(config))) {
|
if ((config->checksum || config->verify_basis)) {
|
||||||
uint8_t wire_len;
|
uint8_t wire_len;
|
||||||
if (!receive_n_data(fd, &wire_len, sizeof(wire_len)) || wire_len == 0 ||
|
if (!receive_n_data(fd, &wire_len, sizeof(wire_len)) || wire_len == 0 ||
|
||||||
wire_len > CHECKSUM_MAX_DIGEST_LEN ||
|
wire_len > CHECKSUM_MAX_DIGEST_LEN ||
|
||||||
@@ -1939,8 +1953,24 @@ static IncrementalCheckOutcome incremental_check_receive_request(IncrementalChec
|
|||||||
if (!receive_n_data(fd, state->check_digest, state->check_digest_len))
|
if (!receive_n_data(fd, state->check_digest, state->check_digest_len))
|
||||||
return INCREMENTAL_ERROR;
|
return INCREMENTAL_ERROR;
|
||||||
}
|
}
|
||||||
|
/* The sender transmits the source metadata with every basis-configured check
|
||||||
|
so a basis hit can be materialized with the SOURCE's attributes (rsync
|
||||||
|
copies/copies-then-fixes; the receiver would otherwise only have the basis
|
||||||
|
inode's stat). The block is symmetric and consumed unconditionally here,
|
||||||
|
whether or not this file ends up as a basis hit. */
|
||||||
|
if (config_has_basis(config) && config->use_metadata) {
|
||||||
|
int meta_ok = 1;
|
||||||
|
state->source_metadata = metadata_receive(fd, &meta_ok);
|
||||||
|
if (!meta_ok)
|
||||||
|
return INCREMENTAL_ERROR;
|
||||||
|
}
|
||||||
|
|
||||||
if (state->check_size > MAX_RECEIVE_WHOLE_FILE_SIZE) {
|
/* A basis-configured run may materialize a file larger than the whole-file
|
||||||
|
payload bound: a basis hit is streamed from the basis path (bounded
|
||||||
|
buffers), so the check size is not itself an allocation. Every other
|
||||||
|
path (delta/append/full) still applies MAX_RECEIVE_WHOLE_FILE_SIZE, and a
|
||||||
|
miss simply falls through to the normal transfer with its own bound. */
|
||||||
|
if (!config_has_basis(config) && state->check_size > MAX_RECEIVE_WHOLE_FILE_SIZE) {
|
||||||
send_error_detail(fd, "check size exceeds receiver limit");
|
send_error_detail(fd, "check size exceeds receiver limit");
|
||||||
return INCREMENTAL_ERROR;
|
return INCREMENTAL_ERROR;
|
||||||
}
|
}
|
||||||
@@ -2030,6 +2060,20 @@ incremental_check_ignore_existing(const IncrementalCheckState* state) {
|
|||||||
return INCREMENTAL_SKIP;
|
return INCREMENTAL_SKIP;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Metadata for a materialized basis hit: prefer the SOURCE metadata the sender
|
||||||
|
transmitted with the check frame (rsync copies then fixes the destination to
|
||||||
|
the source's attributes); fall back to the basis inode's own stat when
|
||||||
|
metadata was not negotiated. Consumes state->source_metadata on success. */
|
||||||
|
static FileMetadata* basis_take_metadata(IncrementalCheckState* state,
|
||||||
|
const struct stat* basis_st) {
|
||||||
|
if (state->source_metadata) {
|
||||||
|
FileMetadata* meta = state->source_metadata;
|
||||||
|
state->source_metadata = NULL;
|
||||||
|
return meta;
|
||||||
|
}
|
||||||
|
return file_metadata_create(NULL, basis_st, false, false);
|
||||||
|
}
|
||||||
|
|
||||||
/* --link-dest relink of an already up-to-date destination. rsync hard-links a
|
/* --link-dest relink of an already up-to-date destination. rsync hard-links a
|
||||||
destination entry to a matching basis even when the entry is already correct,
|
destination entry to a matching basis even when the entry is already correct,
|
||||||
so a run over an existing tree still maximizes sharing with the basis. Only a
|
so a run over an existing tree still maximizes sharing with the basis. Only a
|
||||||
@@ -2047,7 +2091,7 @@ static IncrementalCheckOutcome incremental_check_link_dest_relink(IncrementalChe
|
|||||||
BasisMatch basis;
|
BasisMatch basis;
|
||||||
basis_match_find(config, state->check_path, state->check_size, (time_t)state->check_mtime,
|
basis_match_find(config, state->check_path, state->check_size, (time_t)state->check_mtime,
|
||||||
(long)state->check_mtime_nsec, state->check_digest, state->check_digest_len,
|
(long)state->check_mtime_nsec, state->check_digest, state->check_digest_len,
|
||||||
true, true, &basis);
|
true, &basis);
|
||||||
/* Only a link-dest hit relinks; a copy-dest/compare-dest hit (or a miss) lets
|
/* Only a link-dest hit relinks; a copy-dest/compare-dest hit (or a miss) lets
|
||||||
the up-to-date check below keep the existing destination. */
|
the up-to-date check below keep the existing destination. */
|
||||||
if (!basis.hit || basis.type != BASIS_DEST_LINK) {
|
if (!basis.hit || basis.type != BASIS_DEST_LINK) {
|
||||||
@@ -2060,11 +2104,16 @@ static IncrementalCheckOutcome incremental_check_link_dest_relink(IncrementalChe
|
|||||||
return INCREMENTAL_CONTINUE;
|
return INCREMENTAL_CONTINUE;
|
||||||
}
|
}
|
||||||
File* materialized = file_create(state->check_path);
|
File* materialized = file_create(state->check_path);
|
||||||
if (materialized && basis.content) {
|
if (materialized) {
|
||||||
data_destroy(materialized->data);
|
data_destroy(materialized->data);
|
||||||
materialized->data = basis.content;
|
materialized->data = data_create_reserve((size_t)state->check_size);
|
||||||
basis.content = NULL;
|
if (!materialized->data) {
|
||||||
materialized->metadata = file_metadata_create(NULL, &basis.st, false, false);
|
file_destroy(materialized);
|
||||||
|
materialized = NULL;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (materialized) {
|
||||||
|
materialized->metadata = basis_take_metadata(state, &basis.st);
|
||||||
materialized->skip = true;
|
materialized->skip = true;
|
||||||
materialized->basis_link = basis.basis_path;
|
materialized->basis_link = basis.basis_path;
|
||||||
basis.basis_path = NULL;
|
basis.basis_path = NULL;
|
||||||
@@ -2072,9 +2121,6 @@ static IncrementalCheckOutcome incremental_check_link_dest_relink(IncrementalChe
|
|||||||
file_destroy(materialized);
|
file_destroy(materialized);
|
||||||
materialized = NULL;
|
materialized = NULL;
|
||||||
}
|
}
|
||||||
} else {
|
|
||||||
file_destroy(materialized);
|
|
||||||
materialized = NULL;
|
|
||||||
}
|
}
|
||||||
if (materialized) {
|
if (materialized) {
|
||||||
if (!send_status(state->fd, STATUS_OK)) {
|
if (!send_status(state->fd, STATUS_OK)) {
|
||||||
@@ -2175,12 +2221,13 @@ static IncrementalCheckOutcome incremental_check_quick_skip(IncrementalCheckStat
|
|||||||
materialize nothing (no basis link/copy, no append/delta/full transfer) and
|
materialize nothing (no basis link/copy, no append/delta/full transfer) and
|
||||||
the sender must send no data, so answer STATUS_DRY_RUN_TRANSFER and stop.
|
the sender must send no data, so answer STATUS_DRY_RUN_TRANSFER and stop.
|
||||||
|
|
||||||
The basis lookup is deliberately content-blind: a real run would only accept
|
The basis lookup is content-blind: under the default metadata quick-check a
|
||||||
a --compare-dest exact hit after hashing the basis file and comparing it with
|
hit needs no basis bytes and is honored here just as in a real run; under
|
||||||
the client-supplied digest, which in a dry-run is a 1-bit content oracle.
|
--verify-basis a real run hashes the basis against the client-supplied digest,
|
||||||
Under dry_run no basis bytes may be read, so an otherwise-matching entry is
|
which in a dry-run is a 1-bit content oracle, so no basis bytes may be read
|
||||||
treated as would-transfer instead of a skip. Everything read here (the
|
and an otherwise-matching entry is reported as would-transfer. Everything
|
||||||
destination file's metadata, basis candidates' metadata) is read-only. */
|
read here (the destination file's metadata, basis candidates' metadata) is
|
||||||
|
read-only. */
|
||||||
static IncrementalCheckOutcome incremental_check_dry_run_shortcut(IncrementalCheckState* state,
|
static IncrementalCheckOutcome incremental_check_dry_run_shortcut(IncrementalCheckState* state,
|
||||||
bool* skipped,
|
bool* skipped,
|
||||||
bool* would_transfer) {
|
bool* would_transfer) {
|
||||||
@@ -2191,12 +2238,14 @@ static IncrementalCheckOutcome incremental_check_dry_run_shortcut(IncrementalChe
|
|||||||
bool skip_via_compare = false;
|
bool skip_via_compare = false;
|
||||||
if (config_has_basis(config) && !config->ignore_times) {
|
if (config_has_basis(config) && !config->ignore_times) {
|
||||||
BasisMatch basis;
|
BasisMatch basis;
|
||||||
/* hash_content=false: a dry-run must not read or hash the basis file. No
|
/* hash_content=false: a dry-run must not read or hash the basis file, so
|
||||||
content comparison is possible, so no compare-dest hit can be confirmed
|
under --verify-basis no compare-dest hit can be confirmed and an
|
||||||
and an otherwise-matching file is reported as would-transfer. */
|
otherwise-matching file is reported as would-transfer. Without
|
||||||
|
--verify-basis the metadata quick-check confirms it without touching any
|
||||||
|
basis bytes. */
|
||||||
basis_match_find(config, state->check_path, state->check_size, (time_t)state->check_mtime,
|
basis_match_find(config, state->check_path, state->check_size, (time_t)state->check_mtime,
|
||||||
(long)state->check_mtime_nsec, state->check_digest, state->check_digest_len,
|
(long)state->check_mtime_nsec, state->check_digest, state->check_digest_len,
|
||||||
false, false, &basis);
|
false, &basis);
|
||||||
if (basis.hit && basis.type == BASIS_DEST_COMPARE && !state->has_old_file)
|
if (basis.hit && basis.type == BASIS_DEST_COMPARE && !state->has_old_file)
|
||||||
skip_via_compare = true;
|
skip_via_compare = true;
|
||||||
basis_match_free(&basis);
|
basis_match_free(&basis);
|
||||||
@@ -2224,7 +2273,7 @@ static IncrementalCheckOutcome incremental_check_try_basis(IncrementalCheckState
|
|||||||
BasisMatch basis;
|
BasisMatch basis;
|
||||||
basis_match_find(config, state->check_path, state->check_size, (time_t)state->check_mtime,
|
basis_match_find(config, state->check_path, state->check_size, (time_t)state->check_mtime,
|
||||||
(long)state->check_mtime_nsec, state->check_digest, state->check_digest_len,
|
(long)state->check_mtime_nsec, state->check_digest, state->check_digest_len,
|
||||||
true, true, &basis);
|
true, &basis);
|
||||||
if (basis.hit) {
|
if (basis.hit) {
|
||||||
if (basis.type == BASIS_DEST_COMPARE) {
|
if (basis.type == BASIS_DEST_COMPARE) {
|
||||||
basis_match_free(&basis);
|
basis_match_free(&basis);
|
||||||
@@ -2234,24 +2283,30 @@ static IncrementalCheckOutcome incremental_check_try_basis(IncrementalCheckState
|
|||||||
return INCREMENTAL_SKIP;
|
return INCREMENTAL_SKIP;
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
|
/* Copy/link installs source their bytes from the basis PATH at install
|
||||||
|
time (bounded buffers), so no whole-file content buffer is needed here
|
||||||
|
even for an over-limit basis. */
|
||||||
File* materialized = file_create(state->check_path);
|
File* materialized = file_create(state->check_path);
|
||||||
if (materialized && basis.content) {
|
if (materialized) {
|
||||||
data_destroy(materialized->data);
|
data_destroy(materialized->data);
|
||||||
materialized->data = basis.content;
|
materialized->data = data_create_reserve((size_t)state->check_size);
|
||||||
basis.content = NULL;
|
if (!materialized->data) {
|
||||||
materialized->metadata = file_metadata_create(NULL, &basis.st, false, false);
|
file_destroy(materialized);
|
||||||
materialized->skip = true; /* receiver must not ack this as a data file */
|
materialized = NULL;
|
||||||
if (basis.type == BASIS_DEST_LINK) {
|
|
||||||
materialized->basis_link = basis.basis_path;
|
|
||||||
basis.basis_path = NULL;
|
|
||||||
}
|
}
|
||||||
|
}
|
||||||
|
if (materialized) {
|
||||||
|
materialized->metadata = basis_take_metadata(state, &basis.st);
|
||||||
|
materialized->skip = true; /* receiver must not ack this as a data file */
|
||||||
|
if (basis.type == BASIS_DEST_LINK)
|
||||||
|
materialized->basis_link = basis.basis_path;
|
||||||
|
else
|
||||||
|
materialized->basis_copy = basis.basis_path;
|
||||||
|
basis.basis_path = NULL;
|
||||||
if (!materialized->metadata) {
|
if (!materialized->metadata) {
|
||||||
file_destroy(materialized);
|
file_destroy(materialized);
|
||||||
materialized = NULL;
|
materialized = NULL;
|
||||||
}
|
}
|
||||||
} else {
|
|
||||||
file_destroy(materialized);
|
|
||||||
materialized = NULL;
|
|
||||||
}
|
}
|
||||||
if (materialized) {
|
if (materialized) {
|
||||||
if (!send_status(fd, STATUS_OK)) {
|
if (!send_status(fd, STATUS_OK)) {
|
||||||
@@ -3328,7 +3383,7 @@ static bool delete_extras_budgeted_observed(const Config* config, DeleteManifest
|
|||||||
size_t skipped = 0;
|
size_t skipped = 0;
|
||||||
DeleteWalkResult result = delete_extras_limited_observed(
|
DeleteWalkResult result = delete_extras_limited_observed(
|
||||||
config->receive_root_directory, manifest->keeps, manifest->dirs, remaining, skips, used,
|
config->receive_root_directory, manifest->keeps, manifest->dirs, remaining, skips, used,
|
||||||
&deleted, &skipped, observer, observer_context);
|
config->protect_rules, &deleted, &skipped, observer, observer_context);
|
||||||
if (owned_prefixes) {
|
if (owned_prefixes) {
|
||||||
for (int i = 0; i < config->basis_count; i++)
|
for (int i = 0; i < config->basis_count; i++)
|
||||||
free(owned_prefixes[i]);
|
free(owned_prefixes[i]);
|
||||||
@@ -3509,7 +3564,7 @@ static bool delete_missing_args_budgeted_observed(const Config* config, DeleteMa
|
|||||||
PrefixedDeleteObserver nested = {observer, observer_context, rel};
|
PrefixedDeleteObserver nested = {observer, observer_context, rel};
|
||||||
DeleteWalkResult walk =
|
DeleteWalkResult walk =
|
||||||
no_keeps ? delete_extras_limited_observed(full, no_keeps, NULL, remaining, NULL, 0,
|
no_keeps ? delete_extras_limited_observed(full, no_keeps, NULL, remaining, NULL, 0,
|
||||||
&contents_deleted, &contents_skipped,
|
NULL, &contents_deleted, &contents_skipped,
|
||||||
observer ? prefixed_delete_observer : NULL,
|
observer ? prefixed_delete_observer : NULL,
|
||||||
observer ? &nested : NULL)
|
observer ? &nested : NULL)
|
||||||
: DELETE_WALK_ERROR;
|
: DELETE_WALK_ERROR;
|
||||||
@@ -3620,7 +3675,7 @@ bool manifest_would_delete_list(const Config* config, DeleteManifest* manifest,
|
|||||||
used = idx;
|
used = idx;
|
||||||
}
|
}
|
||||||
bool ok = delete_extras_list(config->receive_root_directory, manifest->keeps, manifest->dirs,
|
bool ok = delete_extras_list(config->receive_root_directory, manifest->keeps, manifest->dirs,
|
||||||
skips, used, out, count_out);
|
skips, used, config->protect_rules, out, count_out);
|
||||||
if (owned_prefixes) {
|
if (owned_prefixes) {
|
||||||
for (int i = 0; i < config->basis_count; i++)
|
for (int i = 0; i < config->basis_count; i++)
|
||||||
free(owned_prefixes[i]);
|
free(owned_prefixes[i]);
|
||||||
|
|||||||
@@ -24,6 +24,16 @@ File* file_receive_hardlink(int file_descriptor);
|
|||||||
File* file_receive_symlink(int file_descriptor, const Config* config);
|
File* file_receive_symlink(int file_descriptor, const Config* config);
|
||||||
File* file_receive_special(int file_descriptor);
|
File* file_receive_special(int file_descriptor);
|
||||||
bool file_special_rdev_valid(int32_t major, int32_t minor, mode_t mode);
|
bool file_special_rdev_valid(int32_t major, int32_t minor, mode_t mode);
|
||||||
|
/* Testable basis quick-check / verification policy. file_basis_quick_match is
|
||||||
|
* rsync's metadata quick-check for a basis candidate (equal size is required
|
||||||
|
* separately by the caller; this adds the --size-only / mtime / --modify-window
|
||||||
|
* leg). file_basis_content_required reports whether a hit must ALSO be
|
||||||
|
* confirmed by a whole-file content digest (--verify-basis; false is the
|
||||||
|
* default rsync-parity behavior). */
|
||||||
|
bool file_basis_quick_match(const Config* config, const struct stat* st, time_t check_mtime,
|
||||||
|
long check_mtime_nsec);
|
||||||
|
bool file_basis_content_required(const Config* config);
|
||||||
|
|
||||||
File* receive_incremental_check(int fd, const Config* config, bool* skipped);
|
File* receive_incremental_check(int fd, const Config* config, bool* skipped);
|
||||||
/* Extended variant used by the receiver. `would_transfer` (may be NULL) is set
|
/* Extended variant used by the receiver. `would_transfer` (may be NULL) is set
|
||||||
* true only on the server-contacting --dry-run path when the file is not up to
|
* true only on the server-contacting --dry-run path when the file is not up to
|
||||||
|
|||||||
@@ -57,6 +57,11 @@ typedef struct {
|
|||||||
* equals the incoming file, and `data` is kept as the cross-filesystem
|
* equals the incoming file, and `data` is kept as the cross-filesystem
|
||||||
* fallback (a local copy) if the hard link cannot be created. */
|
* fallback (a local copy) if the hard link cannot be created. */
|
||||||
char* basis_link;
|
char* basis_link;
|
||||||
|
/* Receiver-only, --copy-dest: when set (and basis_link is NULL), stream the
|
||||||
|
* basis file's bytes into the destination instead of `data`/`data->size`.
|
||||||
|
* This lets a basis larger than any whole-file bound materialize without
|
||||||
|
* buffering it; the source metadata on `metadata` is applied afterwards. */
|
||||||
|
char* basis_copy;
|
||||||
/* --hard-links (-H), sender + receiver wire state. link_group is a run-local
|
/* --hard-links (-H), sender + receiver wire state. link_group is a run-local
|
||||||
* id shared by every member of one source inode (0 = not part of a group).
|
* id shared by every member of one source inode (0 = not part of a group).
|
||||||
* The FIRST member (link_first == true) carries its data on the wire and is
|
* The FIRST member (link_first == true) carries its data on the wire and is
|
||||||
|
|||||||
+1
-1
@@ -53,7 +53,7 @@ typedef struct {
|
|||||||
char* pattern; /* cleaned glob pattern (no leading '/', no trailing '/') */
|
char* pattern; /* cleaned glob pattern (no leading '/', no trailing '/') */
|
||||||
} FilterRule;
|
} FilterRule;
|
||||||
|
|
||||||
typedef struct {
|
typedef struct FilterRuleList {
|
||||||
FilterRule** items; /* owned array of rule pointers */
|
FilterRule** items; /* owned array of rule pointers */
|
||||||
int count;
|
int count;
|
||||||
int capacity;
|
int capacity;
|
||||||
|
|||||||
@@ -191,7 +191,9 @@ enum NET_STATUS {
|
|||||||
* (--delete-delay). Payload: an int32 has_config flag (1 on the first plan
|
* (--delete-delay). Payload: an int32 has_config flag (1 on the first plan
|
||||||
* of the run, 0 afterwards); when set, the three global config sections
|
* of the run, 0 afterwards); when set, the three global config sections
|
||||||
* (protected-prefix count+paths, size-skipped count+paths, missing-args
|
* (protected-prefix count+paths, size-skipped count+paths, missing-args
|
||||||
* count+paths); then the destination-relative directory path wire string
|
* count+paths); then an int32 apply flag (1 for a real plan, 0 for a
|
||||||
|
* config-only carrier frame that must not walk a directory); then the
|
||||||
|
* destination-relative directory path wire string
|
||||||
* ("." for the receive root); then the child-directory count + names and the
|
* ("." for the receive root); then the child-directory count + names and the
|
||||||
* child-file count + names that must be kept. Appended after
|
* child-file count + names that must be kept. Appended after
|
||||||
* STATUS_DEST_INFO so no existing status is renumbered. */
|
* STATUS_DEST_INFO so no existing status is renumbered. */
|
||||||
|
|||||||
+40
-14
@@ -682,7 +682,8 @@ static bool is_synced_dir(const PathIndex* dirs, const char* rel) {
|
|||||||
like any other non-directory extra (never followed). */
|
like any other non-directory extra (never followed). */
|
||||||
static bool delete_extras_fd(int dirfd, const char* rel_path, const PathIndex* keep,
|
static bool delete_extras_fd(int dirfd, const char* rel_path, const PathIndex* keep,
|
||||||
const PathIndex* dirs, DeleteBudget* budget,
|
const PathIndex* dirs, DeleteBudget* budget,
|
||||||
const DeleteSkipEntry* skips, int skip_count, bool parent_deletable,
|
const DeleteSkipEntry* skips, int skip_count,
|
||||||
|
const FilterRuleList* protect_rules, bool parent_deletable,
|
||||||
bool* all_removed, DeletePathObserver observer,
|
bool* all_removed, DeletePathObserver observer,
|
||||||
void* observer_context) {
|
void* observer_context) {
|
||||||
/* openat(dirfd, ".") opens an independent file description: a dup() would
|
/* openat(dirfd, ".") opens an independent file description: a dup() would
|
||||||
@@ -729,12 +730,23 @@ static bool delete_extras_fd(int dirfd, const char* rel_path, const PathIndex* k
|
|||||||
free(child_rel);
|
free(child_rel);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
if (S_ISDIR(st.st_mode)) {
|
bool is_dir = S_ISDIR(st.st_mode);
|
||||||
|
if (protect_rules && filter_rules_apply_side(protect_rules, child_rel, entry->d_name, is_dir,
|
||||||
|
FILTER_SIDE_RECEIVER) == FILTER_ACTION_PROTECT) {
|
||||||
|
/* A first-match protect rule shields the extra; for a directory the whole
|
||||||
|
subtree is shielded (rsync prunes an excluded directory), so do not
|
||||||
|
descend. */
|
||||||
|
local_survives = true;
|
||||||
|
free(child_rel);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (is_dir) {
|
||||||
int childfd = openat(dirfd, entry->d_name, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
|
int childfd = openat(dirfd, entry->d_name, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
|
||||||
bool child_all_removed = false;
|
bool child_all_removed = false;
|
||||||
if (childfd >= 0) {
|
if (childfd >= 0) {
|
||||||
if (!delete_extras_fd(childfd, child_rel, keep, dirs, budget, skips, skip_count, deletable,
|
if (!delete_extras_fd(childfd, child_rel, keep, dirs, budget, skips, skip_count,
|
||||||
&child_all_removed, observer, observer_context))
|
protect_rules, deletable, &child_all_removed, observer,
|
||||||
|
observer_context))
|
||||||
operation_ok = false;
|
operation_ok = false;
|
||||||
close(childfd);
|
close(childfd);
|
||||||
} else if (errno != ENOENT) {
|
} else if (errno != ENOENT) {
|
||||||
@@ -802,7 +814,8 @@ static bool delete_extras_fd(int dirfd, const char* rel_path, const PathIndex* k
|
|||||||
reportable children (depth-first), matching the delete pass's ordering. */
|
reportable children (depth-first), matching the delete pass's ordering. */
|
||||||
static bool list_extras_fd(int dirfd, const char* rel_path, const PathIndex* keep,
|
static bool list_extras_fd(int dirfd, const char* rel_path, const PathIndex* keep,
|
||||||
const PathIndex* dirs, ArrayList* out, size_t* recorded,
|
const PathIndex* dirs, ArrayList* out, size_t* recorded,
|
||||||
const DeleteSkipEntry* skips, int skip_count, bool parent_deletable,
|
const DeleteSkipEntry* skips, int skip_count,
|
||||||
|
const FilterRuleList* protect_rules, bool parent_deletable,
|
||||||
bool* all_removed) {
|
bool* all_removed) {
|
||||||
int scanfd = openat(dirfd, ".", O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
|
int scanfd = openat(dirfd, ".", O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
|
||||||
if (scanfd < 0)
|
if (scanfd < 0)
|
||||||
@@ -836,12 +849,21 @@ static bool list_extras_fd(int dirfd, const char* rel_path, const PathIndex* kee
|
|||||||
free(child_rel);
|
free(child_rel);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
if (S_ISDIR(st.st_mode)) {
|
bool is_dir = S_ISDIR(st.st_mode);
|
||||||
|
if (protect_rules && filter_rules_apply_side(protect_rules, child_rel, entry->d_name, is_dir,
|
||||||
|
FILTER_SIDE_RECEIVER) == FILTER_ACTION_PROTECT) {
|
||||||
|
/* Mirror the delete walk: a protected entry is never reported as a
|
||||||
|
would-delete and a protected directory's subtree is not enumerated. */
|
||||||
|
local_survives = true;
|
||||||
|
free(child_rel);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (is_dir) {
|
||||||
int childfd = openat(dirfd, entry->d_name, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
|
int childfd = openat(dirfd, entry->d_name, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
|
||||||
bool child_all_removed = false;
|
bool child_all_removed = false;
|
||||||
if (childfd >= 0) {
|
if (childfd >= 0) {
|
||||||
if (!list_extras_fd(childfd, child_rel, keep, dirs, out, recorded, skips, skip_count,
|
if (!list_extras_fd(childfd, child_rel, keep, dirs, out, recorded, skips, skip_count,
|
||||||
deletable, &child_all_removed))
|
protect_rules, deletable, &child_all_removed))
|
||||||
operation_ok = false;
|
operation_ok = false;
|
||||||
close(childfd);
|
close(childfd);
|
||||||
} else if (errno != ENOENT) {
|
} else if (errno != ENOENT) {
|
||||||
@@ -892,7 +914,7 @@ static bool list_extras_fd(int dirfd, const char* rel_path, const PathIndex* kee
|
|||||||
|
|
||||||
bool delete_extras_list(const char* dest_root, const ArrayList* manifest,
|
bool delete_extras_list(const char* dest_root, const ArrayList* manifest,
|
||||||
const ArrayList* synced_dirs, const DeleteSkipEntry* skips, int skip_count,
|
const ArrayList* synced_dirs, const DeleteSkipEntry* skips, int skip_count,
|
||||||
ArrayList* out, size_t* count_out) {
|
const FilterRuleList* protect_rules, ArrayList* out, size_t* count_out) {
|
||||||
if (count_out)
|
if (count_out)
|
||||||
*count_out = 0;
|
*count_out = 0;
|
||||||
if (!manifest || !out)
|
if (!manifest || !out)
|
||||||
@@ -928,7 +950,7 @@ bool delete_extras_list(const char* dest_root, const ArrayList* manifest,
|
|||||||
bool all_removed = false;
|
bool all_removed = false;
|
||||||
size_t recorded = 0;
|
size_t recorded = 0;
|
||||||
bool ok = list_extras_fd(rootfd, "", &keep, have_dirs ? &dirs : NULL, out, &recorded, skips,
|
bool ok = list_extras_fd(rootfd, "", &keep, have_dirs ? &dirs : NULL, out, &recorded, skips,
|
||||||
skip_count, false, &all_removed);
|
skip_count, protect_rules, false, &all_removed);
|
||||||
if (close(rootfd) != 0)
|
if (close(rootfd) != 0)
|
||||||
ok = false;
|
ok = false;
|
||||||
path_index_free(&keep);
|
path_index_free(&keep);
|
||||||
@@ -942,6 +964,7 @@ bool delete_extras_list(const char* dest_root, const ArrayList* manifest,
|
|||||||
DeleteWalkResult delete_extras_limited_observed(const char* dest_root, const ArrayList* manifest,
|
DeleteWalkResult delete_extras_limited_observed(const char* dest_root, const ArrayList* manifest,
|
||||||
const ArrayList* synced_dirs, size_t max_delete,
|
const ArrayList* synced_dirs, size_t max_delete,
|
||||||
const DeleteSkipEntry* skips, int skip_count,
|
const DeleteSkipEntry* skips, int skip_count,
|
||||||
|
const FilterRuleList* protect_rules,
|
||||||
size_t* deleted_out, size_t* skipped_out,
|
size_t* deleted_out, size_t* skipped_out,
|
||||||
DeletePathObserver observer,
|
DeletePathObserver observer,
|
||||||
void* observer_context) {
|
void* observer_context) {
|
||||||
@@ -984,8 +1007,9 @@ DeleteWalkResult delete_extras_limited_observed(const char* dest_root, const Arr
|
|||||||
}
|
}
|
||||||
DeleteBudget budget = {.max_delete = max_delete, .deleted = 0, .skipped = 0, .limit_hit = false};
|
DeleteBudget budget = {.max_delete = max_delete, .deleted = 0, .skipped = 0, .limit_hit = false};
|
||||||
bool all_removed = false;
|
bool all_removed = false;
|
||||||
bool ok = delete_extras_fd(rootfd, "", &keep, have_dirs ? &dirs : NULL, &budget, skips,
|
bool ok =
|
||||||
skip_count, false, &all_removed, observer, observer_context);
|
delete_extras_fd(rootfd, "", &keep, have_dirs ? &dirs : NULL, &budget, skips, skip_count,
|
||||||
|
protect_rules, false, &all_removed, observer, observer_context);
|
||||||
if (close(rootfd) != 0)
|
if (close(rootfd) != 0)
|
||||||
ok = false;
|
ok = false;
|
||||||
path_index_free(&keep);
|
path_index_free(&keep);
|
||||||
@@ -1003,13 +1027,15 @@ DeleteWalkResult delete_extras_limited_observed(const char* dest_root, const Arr
|
|||||||
DeleteWalkResult delete_extras_limited(const char* dest_root, const ArrayList* manifest,
|
DeleteWalkResult delete_extras_limited(const char* dest_root, const ArrayList* manifest,
|
||||||
const ArrayList* synced_dirs, size_t max_delete,
|
const ArrayList* synced_dirs, size_t max_delete,
|
||||||
const DeleteSkipEntry* skips, int skip_count,
|
const DeleteSkipEntry* skips, int skip_count,
|
||||||
size_t* deleted_out, size_t* skipped_out) {
|
const FilterRuleList* protect_rules, size_t* deleted_out,
|
||||||
|
size_t* skipped_out) {
|
||||||
return delete_extras_limited_observed(dest_root, manifest, synced_dirs, max_delete, skips,
|
return delete_extras_limited_observed(dest_root, manifest, synced_dirs, max_delete, skips,
|
||||||
skip_count, deleted_out, skipped_out, NULL, NULL);
|
skip_count, protect_rules, deleted_out, skipped_out, NULL,
|
||||||
|
NULL);
|
||||||
}
|
}
|
||||||
|
|
||||||
bool delete_extras(const char* dest_root, const ArrayList* manifest) {
|
bool delete_extras(const char* dest_root, const ArrayList* manifest) {
|
||||||
return delete_extras_limited(dest_root, manifest, NULL, SIZE_MAX, NULL, 0, NULL, NULL) ==
|
return delete_extras_limited(dest_root, manifest, NULL, SIZE_MAX, NULL, 0, NULL, NULL, NULL) ==
|
||||||
DELETE_WALK_OK;
|
DELETE_WALK_OK;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
+10
-3
@@ -2,6 +2,7 @@
|
|||||||
#define UTILS_H
|
#define UTILS_H
|
||||||
|
|
||||||
#include "array_list.h"
|
#include "array_list.h"
|
||||||
|
#include "filter.h"
|
||||||
#include <stddef.h>
|
#include <stddef.h>
|
||||||
#include <stdbool.h>
|
#include <stdbool.h>
|
||||||
#include <stdio.h>
|
#include <stdio.h>
|
||||||
@@ -148,7 +149,8 @@ bool path_under_skip_prefix(const char* child_rel, bool at_root, const DeleteSki
|
|||||||
DeleteWalkResult delete_extras_limited(const char* dest_root, const ArrayList* manifest,
|
DeleteWalkResult delete_extras_limited(const char* dest_root, const ArrayList* manifest,
|
||||||
const ArrayList* synced_dirs, size_t max_delete,
|
const ArrayList* synced_dirs, size_t max_delete,
|
||||||
const DeleteSkipEntry* skips, int skip_count,
|
const DeleteSkipEntry* skips, int skip_count,
|
||||||
size_t* deleted_out, size_t* skipped_out);
|
const FilterRuleList* protect_rules, size_t* deleted_out,
|
||||||
|
size_t* skipped_out);
|
||||||
|
|
||||||
/* Optional per-deletion observer: called for each destination-relative path
|
/* Optional per-deletion observer: called for each destination-relative path
|
||||||
actually removed (a file, symlink, or directory), in removal order, so the
|
actually removed (a file, symlink, or directory), in removal order, so the
|
||||||
@@ -156,10 +158,15 @@ DeleteWalkResult delete_extras_limited(const char* dest_root, const ArrayList* m
|
|||||||
typedef void (*DeletePathObserver)(void* context, const char* rel_path);
|
typedef void (*DeletePathObserver)(void* context, const char* rel_path);
|
||||||
|
|
||||||
/* `delete_extras_limited_observed` is delete_extras_limited with an optional
|
/* `delete_extras_limited_observed` is delete_extras_limited with an optional
|
||||||
observer; the observer is invoked only for entries truly removed. */
|
* observer; the observer is invoked only for entries truly removed. When
|
||||||
|
* `protect_rules` is non-NULL its receiver-side verdict is evaluated for every
|
||||||
|
* candidate extra: a first-match PROTECT leaves the entry (and, for a
|
||||||
|
* directory, its whole subtree) in place, while RISK/NONE fall through to the
|
||||||
|
* ordinary skip-prefix/keep-set logic. */
|
||||||
DeleteWalkResult delete_extras_limited_observed(const char* dest_root, const ArrayList* manifest,
|
DeleteWalkResult delete_extras_limited_observed(const char* dest_root, const ArrayList* manifest,
|
||||||
const ArrayList* synced_dirs, size_t max_delete,
|
const ArrayList* synced_dirs, size_t max_delete,
|
||||||
const DeleteSkipEntry* skips, int skip_count,
|
const DeleteSkipEntry* skips, int skip_count,
|
||||||
|
const FilterRuleList* protect_rules,
|
||||||
size_t* deleted_out, size_t* skipped_out,
|
size_t* deleted_out, size_t* skipped_out,
|
||||||
DeletePathObserver observer,
|
DeletePathObserver observer,
|
||||||
void* observer_context);
|
void* observer_context);
|
||||||
@@ -170,7 +177,7 @@ DeleteWalkResult delete_extras_limited_observed(const char* dest_root, const Arr
|
|||||||
strings appended to `out` and receives their count in *count_out. */
|
strings appended to `out` and receives their count in *count_out. */
|
||||||
bool delete_extras_list(const char* dest_root, const ArrayList* manifest,
|
bool delete_extras_list(const char* dest_root, const ArrayList* manifest,
|
||||||
const ArrayList* synced_dirs, const DeleteSkipEntry* skips, int skip_count,
|
const ArrayList* synced_dirs, const DeleteSkipEntry* skips, int skip_count,
|
||||||
ArrayList* out, size_t* count_out);
|
const FilterRuleList* protect_rules, ArrayList* out, size_t* count_out);
|
||||||
bool delete_extras(const char* dest_root, const ArrayList* manifest);
|
bool delete_extras(const char* dest_root, const ArrayList* manifest);
|
||||||
/* Open the existing destination directory at `dest_root`, confined to the
|
/* Open the existing destination directory at `dest_root`, confined to the
|
||||||
authorized root with an O_NOFOLLOW component walk (the same confinement the
|
authorized root with an O_NOFOLLOW component walk (the same confinement the
|
||||||
|
|||||||
@@ -119,6 +119,13 @@ static void build_canonical_frame(void) {
|
|||||||
cfg->usermap[0].to = MAP_TO;
|
cfg->usermap[0].to = MAP_TO;
|
||||||
cfg->usermap[0].to_name = NULL;
|
cfg->usermap[0].to_name = NULL;
|
||||||
}
|
}
|
||||||
|
/* Force a non-empty receiver delete-protection block so the fuzzer mutates
|
||||||
|
* its rule count, action/sides codes and pattern strings. */
|
||||||
|
cfg->filters = array_list_create(free);
|
||||||
|
if (cfg->filters) {
|
||||||
|
array_list_add(cfg->filters, str_dup("P *.log"));
|
||||||
|
array_list_add(cfg->filters, str_dup("+r keep/**"));
|
||||||
|
}
|
||||||
if (!cfg->send_directory || !cfg->receive_root_directory || !cfg->usermap) {
|
if (!cfg->send_directory || !cfg->receive_root_directory || !cfg->usermap) {
|
||||||
config_delete(cfg);
|
config_delete(cfg);
|
||||||
return;
|
return;
|
||||||
|
|||||||
@@ -229,6 +229,18 @@ def corpus_iconv(root: str) -> None:
|
|||||||
os.utime(full, (_SRC_MTIME, _SRC_MTIME))
|
os.utime(full, (_SRC_MTIME, _SRC_MTIME))
|
||||||
|
|
||||||
|
|
||||||
|
# Payload for the --fuzzy basis corpus: large enough for the delta engine's
|
||||||
|
# 16 KiB minimum and with repeated content so a coinciding basis yields a
|
||||||
|
# non-zero (and identical) Matched data count in both tools.
|
||||||
|
FUZZY_PAYLOAD = (b"the quick brown fox jumps over the lazy dog\n" * 2000)[:65536]
|
||||||
|
|
||||||
|
|
||||||
|
def corpus_fuzzy(root: str) -> None:
|
||||||
|
"""A named regular file; the `fuzzy` seed adds the similar-suffix sibling."""
|
||||||
|
clean_dir(root)
|
||||||
|
_write(os.path.join(root, "report_v2.txt"), FUZZY_PAYLOAD)
|
||||||
|
|
||||||
|
|
||||||
CORPORA: Dict[str, Callable[[str], None]] = {
|
CORPORA: Dict[str, Callable[[str], None]] = {
|
||||||
"basic": corpus_basic,
|
"basic": corpus_basic,
|
||||||
"unicode": corpus_unicode,
|
"unicode": corpus_unicode,
|
||||||
@@ -240,6 +252,7 @@ CORPORA: Dict[str, Callable[[str], None]] = {
|
|||||||
"multidir": corpus_multidir,
|
"multidir": corpus_multidir,
|
||||||
"relative": corpus_relative,
|
"relative": corpus_relative,
|
||||||
"iconv": corpus_iconv,
|
"iconv": corpus_iconv,
|
||||||
|
"fuzzy": corpus_fuzzy,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -9,6 +9,7 @@ directory when the shared test server is launched), so every scratch tree lives
|
|||||||
under ``TEST_DATA_DIR`` rather than pytest's ``tmp_path``.
|
under ``TEST_DATA_DIR`` rather than pytest's ``tmp_path``.
|
||||||
"""
|
"""
|
||||||
import os
|
import os
|
||||||
|
import random
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
import sys
|
import sys
|
||||||
@@ -68,6 +69,84 @@ def _tree_bytes(root):
|
|||||||
return out
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
_CC_DELTA_T0 = 1_600_000_000
|
||||||
|
_CC_DELTA_T1 = 1_600_000_100
|
||||||
|
|
||||||
|
_DELTA_STATS_KEYS = (
|
||||||
|
"Number of created files",
|
||||||
|
"Number of regular files transferred",
|
||||||
|
"Total transferred file size",
|
||||||
|
"Literal data",
|
||||||
|
"Matched data",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _pin_tree(root, mtime):
|
||||||
|
for dirpath, dirnames, filenames in os.walk(root):
|
||||||
|
for name in dirnames + filenames:
|
||||||
|
path = os.path.join(dirpath, name)
|
||||||
|
if not os.path.islink(path):
|
||||||
|
os.utime(path, (mtime, mtime))
|
||||||
|
os.utime(root, (mtime, mtime))
|
||||||
|
|
||||||
|
|
||||||
|
def _make_delta_basis(src, size=512 * 1024):
|
||||||
|
"""Build a source and a matching pre-modification basis tree.
|
||||||
|
|
||||||
|
The source's ``big.bin`` is then modified in a few disjoint places and given
|
||||||
|
a newer mtime so both tools take the delta path. Returns the basis dir.
|
||||||
|
"""
|
||||||
|
clean_dir(src)
|
||||||
|
original = random.Random(20240101).randbytes(size)
|
||||||
|
with open(os.path.join(src, "big.bin"), "wb") as fh:
|
||||||
|
fh.write(original)
|
||||||
|
with open(os.path.join(src, "small.txt"), "wb") as fh:
|
||||||
|
fh.write(b"hello world\n")
|
||||||
|
_pin_tree(src, _CC_DELTA_T0)
|
||||||
|
|
||||||
|
basis = src.rstrip("/") + "_basis"
|
||||||
|
clean_dir(basis)
|
||||||
|
shutil.copy2(os.path.join(src, "big.bin"), os.path.join(basis, "big.bin"))
|
||||||
|
shutil.copy2(os.path.join(src, "small.txt"), os.path.join(basis, "small.txt"))
|
||||||
|
_pin_tree(basis, _CC_DELTA_T0)
|
||||||
|
|
||||||
|
modified = bytearray(original)
|
||||||
|
for off in (0, size // 3, 2 * size // 3, size - 64):
|
||||||
|
for i in range(32):
|
||||||
|
modified[off + i] ^= 0x5A
|
||||||
|
with open(os.path.join(src, "big.bin"), "wb") as fh:
|
||||||
|
fh.write(bytes(modified))
|
||||||
|
os.utime(os.path.join(src, "big.bin"), (_CC_DELTA_T1, _CC_DELTA_T1))
|
||||||
|
return basis
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_from_basis(basis, target):
|
||||||
|
clean_dir(target)
|
||||||
|
for name in os.listdir(basis):
|
||||||
|
shutil.copy2(os.path.join(basis, name), os.path.join(target, name))
|
||||||
|
|
||||||
|
|
||||||
|
def _delta_stats(text):
|
||||||
|
found = {}
|
||||||
|
for line in text.splitlines():
|
||||||
|
for key in _DELTA_STATS_KEYS:
|
||||||
|
if line.startswith(key + ":"):
|
||||||
|
found[key] = line.split(":", 1)[1].strip()
|
||||||
|
return found
|
||||||
|
|
||||||
|
|
||||||
|
def _big_bin_outfmt(text):
|
||||||
|
"""The ``(c, C)`` pair from the ``big.bin`` out-format line (`%c|%C %n`)."""
|
||||||
|
for line in text.splitlines():
|
||||||
|
stripped = line.strip()
|
||||||
|
if "|" not in stripped or not stripped.endswith("big.bin"):
|
||||||
|
continue
|
||||||
|
c_field, rest = stripped.split("|", 1)
|
||||||
|
fields = rest.split()
|
||||||
|
return c_field.strip(), (fields[0] if fields else "")
|
||||||
|
return None, None
|
||||||
|
|
||||||
|
|
||||||
class TestCodecChoiceMatrix:
|
class TestCodecChoiceMatrix:
|
||||||
"""The CLI accept/reject set and exit codes must match rsync 3.4.1."""
|
"""The CLI accept/reject set and exit codes must match rsync 3.4.1."""
|
||||||
|
|
||||||
@@ -191,6 +270,76 @@ class TestCodecTransferDifferential:
|
|||||||
assert _tree_bytes(received) == _tree_bytes(rsync_dst)
|
assert _tree_bytes(received) == _tree_bytes(rsync_dst)
|
||||||
|
|
||||||
|
|
||||||
|
class TestChecksumChoiceDeltaSurface:
|
||||||
|
"""--checksum-choice does not move the delta-transfer parity surface.
|
||||||
|
|
||||||
|
FastSync's delta BLOCK strong checksum is a fixed xxHash32, so the
|
||||||
|
negotiated algorithm only selects the whole-file comparison digest (and the
|
||||||
|
``%C`` transfer digest). A pre-seeded delta transfer must therefore land
|
||||||
|
byte-identical bytes and report the same counters for every choice, while
|
||||||
|
``%C`` -- the one token that tracks the choice -- stays byte-identical to
|
||||||
|
rsync. This pins the Track-3b reclassification in RSYNC_COMPAT.md.
|
||||||
|
"""
|
||||||
|
|
||||||
|
CHOICES = ["xxh64", "xxh128", "xxh3", "md5", "md4", "sha1",
|
||||||
|
"xxh64,sha1", "sha1,xxh64"]
|
||||||
|
|
||||||
|
@requires_rsync
|
||||||
|
@pytest.mark.ci
|
||||||
|
def test_delta_surface_invariant_to_checksum_choice(self, shared_server):
|
||||||
|
src = _scratch("ccdelta_src")
|
||||||
|
basis = _make_delta_basis(src)
|
||||||
|
source_bytes = _tree_bytes(src)
|
||||||
|
|
||||||
|
rsync_c, fastsync_c, digests = {}, {}, {}
|
||||||
|
rsync_stats, fastsync_stats = {}, {}
|
||||||
|
for choice in self.CHOICES:
|
||||||
|
tag = choice.replace(",", "_")
|
||||||
|
rsync_dst = _scratch(f"ccdelta_rs_{tag}")
|
||||||
|
fs_dst = _scratch(f"ccdelta_fs_{tag}")
|
||||||
|
fs_root = get_dest_received_dir(fs_dst, src)
|
||||||
|
_seed_from_basis(basis, rsync_dst)
|
||||||
|
_seed_from_basis(basis, fs_root)
|
||||||
|
|
||||||
|
# Pin the block size on both ends so the literal/matched split is
|
||||||
|
# comparable (rsync's adaptive default would otherwise differ from
|
||||||
|
# FastSync's 8192-byte default).
|
||||||
|
rsync_result = _rsync(["-a", "--no-whole-file", "-B8192", "--stats",
|
||||||
|
"--out-format=%c|%C %n", f"--cc={choice}",
|
||||||
|
src + "/", rsync_dst + "/"])
|
||||||
|
assert rsync_result.returncode == 0, rsync_result.stderr
|
||||||
|
result, _ = run_client(
|
||||||
|
src, fs_dst,
|
||||||
|
flags=["-a", "--incremental", "--delta", "-B8192", "--stats",
|
||||||
|
"--out-format=%c|%C %n", f"--cc={choice}"],
|
||||||
|
port=shared_server.port)
|
||||||
|
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
|
||||||
|
|
||||||
|
assert _tree_bytes(rsync_dst) == source_bytes, choice
|
||||||
|
assert _tree_bytes(fs_root) == source_bytes, choice
|
||||||
|
assert _delta_stats(rsync_result.stdout) == _delta_stats(result.stdout), choice
|
||||||
|
|
||||||
|
rs_c, rs_C = _big_bin_outfmt(rsync_result.stdout)
|
||||||
|
fs_c, fs_C = _big_bin_outfmt(result.stdout)
|
||||||
|
assert rs_C == fs_C, f"{choice}: %C rsync={rs_C!r} fastsync={fs_C!r}"
|
||||||
|
rsync_c[choice] = rs_c
|
||||||
|
fastsync_c[choice] = fs_c
|
||||||
|
digests[choice] = fs_C
|
||||||
|
rsync_stats[choice] = _delta_stats(rsync_result.stdout)
|
||||||
|
fastsync_stats[choice] = _delta_stats(result.stdout)
|
||||||
|
|
||||||
|
# The choice is only observable in %C, and it is effective (the digests
|
||||||
|
# are not all the same algorithm's output).
|
||||||
|
assert len(set(digests.values())) > 1, digests
|
||||||
|
# The compared --stats counters are invariant across choices in each tool
|
||||||
|
# (and were asserted equal cross-tool inside the loop).
|
||||||
|
assert len({tuple(sorted(s.items())) for s in rsync_stats.values()}) == 1, rsync_stats
|
||||||
|
assert len({tuple(sorted(s.items())) for s in fastsync_stats.values()}) == 1, fastsync_stats
|
||||||
|
# The block-checksum token (%c) is invariant across choices in each tool.
|
||||||
|
assert len(set(rsync_c.values())) == 1, rsync_c
|
||||||
|
assert len(set(fastsync_c.values())) == 1, fastsync_c
|
||||||
|
|
||||||
|
|
||||||
class TestCodecNegotiationFallback:
|
class TestCodecNegotiationFallback:
|
||||||
"""FastSync's auto negotiation and deterministic fallback order."""
|
"""FastSync's auto negotiation and deterministic fallback order."""
|
||||||
|
|
||||||
|
|||||||
@@ -217,7 +217,21 @@ class _SlicingProxy:
|
|||||||
|
|
||||||
|
|
||||||
class TestDeleteTimingFinalStateParity:
|
class TestDeleteTimingFinalStateParity:
|
||||||
"""On a successful transfer the per-directory timings match rsync's result."""
|
"""On a successful transfer the per-directory timings match rsync's result.
|
||||||
|
|
||||||
|
Plain ``--delete`` has no rsync-incompatible spelling: it defaults to
|
||||||
|
delete-during on both tools, so it is compared against rsync's own default.
|
||||||
|
``--delete-commit`` is FastSync-only and selects the late whole-tree commit,
|
||||||
|
which is rsync's ``--delete-after`` timing.
|
||||||
|
"""
|
||||||
|
|
||||||
|
# (fastsync flag, rsync flag)
|
||||||
|
PAIRS = [
|
||||||
|
("--delete", "--delete"),
|
||||||
|
("--delete-during", "--delete-during"),
|
||||||
|
("--delete-delay", "--delete-delay"),
|
||||||
|
("--delete-commit", "--delete-after"),
|
||||||
|
]
|
||||||
|
|
||||||
def _run_fastsync(self, tag, timing):
|
def _run_fastsync(self, tag, timing):
|
||||||
source, dest, received = _seed_pair(tag)
|
source, dest, received = _seed_pair(tag)
|
||||||
@@ -226,12 +240,12 @@ class TestDeleteTimingFinalStateParity:
|
|||||||
result, _ = run_client(source, dest, flags=[timing], port=server.port)
|
result, _ = run_client(source, dest, flags=[timing], port=server.port)
|
||||||
return result, received
|
return result, received
|
||||||
|
|
||||||
@pytest.mark.parametrize("timing", ["--delete-during", "--delete-delay"])
|
@pytest.mark.parametrize("fs_timing,rs_timing", PAIRS)
|
||||||
@requires_rsync
|
@requires_rsync
|
||||||
def test_success_final_state_matches_rsync(self, timing):
|
def test_success_final_state_matches_rsync(self, fs_timing, rs_timing):
|
||||||
# Worker-safe names: xdist may run both parametrizations concurrently, so
|
# Worker-safe names: xdist may run the parametrizations concurrently, so
|
||||||
# the timing is part of every fixture path.
|
# the flags are part of every fixture path.
|
||||||
label = timing.lstrip("-")
|
label = f"{fs_timing.lstrip('-')}_vs_{rs_timing.lstrip('-')}"
|
||||||
# Build the rsync fixture from the same seed so both sides start equal.
|
# Build the rsync fixture from the same seed so both sides start equal.
|
||||||
source, dest, received = _seed_pair(f"parity_rsync_{label}")
|
source, dest, received = _seed_pair(f"parity_rsync_{label}")
|
||||||
source2 = source
|
source2 = source
|
||||||
@@ -240,17 +254,18 @@ class TestDeleteTimingFinalStateParity:
|
|||||||
# rsync mirrors src/ into dst/; seed the same extra.
|
# rsync mirrors src/ into dst/; seed the same extra.
|
||||||
_write(os.path.join(rsync_dst, "d", "old_extra"), b"stale extra\n")
|
_write(os.path.join(rsync_dst, "d", "old_extra"), b"stale extra\n")
|
||||||
|
|
||||||
rsync_result = _rsync(["-a", timing, source2 + "/", rsync_dst + "/"])
|
rsync_result = _rsync(["-a", rs_timing, source2 + "/", rsync_dst + "/"])
|
||||||
assert rsync_result.returncode == 0, rsync_result.stderr
|
assert rsync_result.returncode == 0, rsync_result.stderr
|
||||||
rsync_tree = _tree(rsync_dst)
|
rsync_tree = _tree(rsync_dst)
|
||||||
|
|
||||||
with ServerManager() as server:
|
with ServerManager() as server:
|
||||||
server.start(extra_args=["--allow-delete"])
|
server.start(extra_args=["--allow-delete"])
|
||||||
result, _ = run_client(source, dest, flags=[timing], port=server.port)
|
result, _ = run_client(source, dest, flags=[fs_timing], port=server.port)
|
||||||
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
|
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
|
||||||
fastsync_tree = _tree(received)
|
fastsync_tree = _tree(received)
|
||||||
assert fastsync_tree == rsync_tree, (
|
assert fastsync_tree == rsync_tree, (
|
||||||
f"{timing}: fastsync tree {fastsync_tree} != rsync tree {rsync_tree}"
|
f"{fs_timing} vs rsync {rs_timing}: fastsync tree {fastsync_tree} != "
|
||||||
|
f"rsync tree {rsync_tree}"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -292,7 +307,14 @@ class TestDeleteTimingTypeConflictParity:
|
|||||||
|
|
||||||
|
|
||||||
class TestDeleteTimingFailure:
|
class TestDeleteTimingFailure:
|
||||||
"""A mid-transfer failure distinguishes during from delay."""
|
"""A mid-transfer failure distinguishes the during timings from the late
|
||||||
|
commit timings.
|
||||||
|
|
||||||
|
Plain ``--delete`` must behave like ``--delete-during`` (the rsync default),
|
||||||
|
removing the extras of the directories already reached; ``--delete-commit``
|
||||||
|
must behave like ``--delete-after`` and remove nothing until the transfer
|
||||||
|
has fully succeeded.
|
||||||
|
"""
|
||||||
|
|
||||||
@pytest.mark.parametrize("mt", [False, True])
|
@pytest.mark.parametrize("mt", [False, True])
|
||||||
def test_during_removes_delay_preserves_on_failure(self, mt):
|
def test_during_removes_delay_preserves_on_failure(self, mt):
|
||||||
@@ -301,8 +323,12 @@ class TestDeleteTimingFailure:
|
|||||||
assert os.path.exists(extra)
|
assert os.path.exists(extra)
|
||||||
with ServerManager() as server:
|
with ServerManager() as server:
|
||||||
server.start(extra_args=["--allow-delete"])
|
server.start(extra_args=["--allow-delete"])
|
||||||
for timing, expect_removed in (("--delete-during", True),
|
for timing, expect_removed in (
|
||||||
("--delete-delay", False)):
|
("--delete-during", True),
|
||||||
|
("--delete", True),
|
||||||
|
("--delete-delay", False),
|
||||||
|
("--delete-commit", False),
|
||||||
|
("--delete-after", False)):
|
||||||
# Re-seed the extra before each run.
|
# Re-seed the extra before each run.
|
||||||
_write(extra, b"stale extra\n")
|
_write(extra, b"stale extra\n")
|
||||||
proxy = _SlicingProxy(server.port, forward_limit=MID_TRANSFER_BYTES, throttle=PROXY_THROTTLE)
|
proxy = _SlicingProxy(server.port, forward_limit=MID_TRANSFER_BYTES, throttle=PROXY_THROTTLE)
|
||||||
@@ -449,12 +475,13 @@ class TestDeleteDelayVsAfterSnapshot:
|
|||||||
|
|
||||||
|
|
||||||
class TestDeleteAfterThreadsKeepSet:
|
class TestDeleteAfterThreadsKeepSet:
|
||||||
"""Regression: -m/--threads with the default delete-after timing (plain
|
"""Regression: -j/--threads must still transmit the delete keep-set in every
|
||||||
--delete) must still transmit the keep-set manifest and remove destination
|
timing. PipelineContextSender.delete_suppressed was left uninitialized, so a
|
||||||
extras. PipelineContextSender.delete_suppressed was left uninitialized, so a
|
garbage true silently skipped the late keep-set manifest under --threads.
|
||||||
garbage true silently skipped the manifest under --threads."""
|
Plain --delete now uses the per-directory plans, while --delete-commit /
|
||||||
|
--delete-after keep exercising the late whole-tree manifest."""
|
||||||
|
|
||||||
@pytest.mark.parametrize("delete_flag", ["--delete", "--delete-after"])
|
@pytest.mark.parametrize("delete_flag", ["--delete", "--delete-commit", "--delete-after"])
|
||||||
def test_threads_delete_after_sends_keep_set(self, delete_flag):
|
def test_threads_delete_after_sends_keep_set(self, delete_flag):
|
||||||
source, dest, received = _seed_pair("mtkeep")
|
source, dest, received = _seed_pair("mtkeep")
|
||||||
extra = os.path.join(received, "d", "old_extra")
|
extra = os.path.join(received, "d", "old_extra")
|
||||||
@@ -465,7 +492,7 @@ class TestDeleteAfterThreadsKeepSet:
|
|||||||
port=server.port)
|
port=server.port)
|
||||||
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
|
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
|
||||||
assert not os.path.exists(extra), (
|
assert not os.path.exists(extra), (
|
||||||
f"{delete_flag} --threads did not remove an extra: keep-set manifest was suppressed"
|
f"{delete_flag} --threads did not remove an extra: delete keep-set was suppressed"
|
||||||
)
|
)
|
||||||
|
|
||||||
class TestDeleteDelayMaxDeleteRefilledDir:
|
class TestDeleteDelayMaxDeleteRefilledDir:
|
||||||
|
|||||||
@@ -28,6 +28,7 @@ from common import ( # noqa: E402
|
|||||||
ServerManager,
|
ServerManager,
|
||||||
TEST_DATA_DIR,
|
TEST_DATA_DIR,
|
||||||
clean_dir,
|
clean_dir,
|
||||||
|
get_dest_received_dir,
|
||||||
)
|
)
|
||||||
from parity_caveats import ASPECTS, caveat_for # noqa: E402
|
from parity_caveats import ASPECTS, caveat_for # noqa: E402
|
||||||
import parity_harness as H # noqa: E402
|
import parity_harness as H # noqa: E402
|
||||||
@@ -116,12 +117,32 @@ def seed_delete_excluded(_src, rroot, froot):
|
|||||||
_mk(os.path.join(root, "keep.txt"), b"keep\n", _OLD_MTIME)
|
_mk(os.path.join(root, "keep.txt"), b"keep\n", _OLD_MTIME)
|
||||||
|
|
||||||
|
|
||||||
|
def seed_filter_protect(_src, rroot, froot):
|
||||||
|
"""Destination-only entries, including nested ones, for the receiver-side
|
||||||
|
`protect` rule: the `.log` extras must survive --delete, the rest go."""
|
||||||
|
for root in (rroot, froot):
|
||||||
|
_mk(os.path.join(root, "extra.log"), b"dest-only log\n", _OLD_MTIME)
|
||||||
|
_mk(os.path.join(root, "other.txt"), b"dest-only other\n", _OLD_MTIME)
|
||||||
|
_mk(os.path.join(root, "sub", "extra2.log"), b"nested dest-only log\n", _OLD_MTIME)
|
||||||
|
_mk(os.path.join(root, "sub", "other2.txt"), b"nested dest-only other\n", _OLD_MTIME)
|
||||||
|
|
||||||
|
|
||||||
def seed_max_delete(_src, rroot, froot):
|
def seed_max_delete(_src, rroot, froot):
|
||||||
for root in (rroot, froot):
|
for root in (rroot, froot):
|
||||||
_mk(os.path.join(root, "extra1.txt"), b"e1\n", _OLD_MTIME)
|
_mk(os.path.join(root, "extra1.txt"), b"e1\n", _OLD_MTIME)
|
||||||
_mk(os.path.join(root, "extra2.txt"), b"e2\n", _OLD_MTIME)
|
_mk(os.path.join(root, "extra2.txt"), b"e2\n", _OLD_MTIME)
|
||||||
|
|
||||||
|
|
||||||
|
def fuzzy_basis_seed(_src, rroot, froot):
|
||||||
|
"""Seed a same-suffix sibling whose name is one edit from the source and
|
||||||
|
whose content matches it, with a DIFFERENT mtime so rsync's exact
|
||||||
|
size+mtime pass cannot fire: both tools must select it via the
|
||||||
|
name-distance pass. Where the two tools' basis choices coincide the
|
||||||
|
block-level results are identical when the block size is pinned."""
|
||||||
|
for root in (rroot, froot):
|
||||||
|
_mk(os.path.join(root, "report_v1.txt"), H.FUZZY_PAYLOAD, _OLD_MTIME)
|
||||||
|
|
||||||
|
|
||||||
def max_delete_count_check(_src, rroot, froot, _rs, _fs):
|
def max_delete_count_check(_src, rroot, froot, _rs, _fs):
|
||||||
"""The exact survivor set is order-dependent; the count must still match."""
|
"""The exact survivor set is order-dependent; the count must still match."""
|
||||||
r = H.snapshot(rroot)
|
r = H.snapshot(rroot)
|
||||||
@@ -197,6 +218,18 @@ _CASES = [
|
|||||||
H.Case("chmod", "basic", ["-a", "--chmod=Fu+rwx"], compare_modes=True,
|
H.Case("chmod", "basic", ["-a", "--chmod=Fu+rwx"], compare_modes=True,
|
||||||
ci=True, ref="--chmod"),
|
ci=True, ref="--chmod"),
|
||||||
|
|
||||||
|
# --- delta / similar-file basis (--fuzzy) -----------------------------
|
||||||
|
# Basis choices coincide here (same-suffix sibling, name distance one edit,
|
||||||
|
# content identical); with the block size pinned both tools report the same
|
||||||
|
# Matched/Literal/transferred counters. The residual (FastSync's narrower
|
||||||
|
# delta size window) is covered by TestFuzzy in test_parity_quickwins.py.
|
||||||
|
H.Case("fuzzy_basis", "fuzzy",
|
||||||
|
["-a", "--no-whole-file", "--fuzzy", "--stats", "-B8192"],
|
||||||
|
fastsync_flags=["-a", "--incremental", "--delta", "--fuzzy",
|
||||||
|
"--stats", "--delta-block=8192"],
|
||||||
|
seed=fuzzy_basis_seed, stdout=H.STDOUT_STATS, ci=True,
|
||||||
|
ref="-y/--fuzzy similar-file basis"),
|
||||||
|
|
||||||
# --- deletion ---------------------------------------------------------
|
# --- deletion ---------------------------------------------------------
|
||||||
H.Case("delete", "basic", ["-a", "--delete"], seed=seed_extras,
|
H.Case("delete", "basic", ["-a", "--delete"], seed=seed_extras,
|
||||||
server_args=DELETE, ci=True, ref="--delete"),
|
server_args=DELETE, ci=True, ref="--delete"),
|
||||||
@@ -208,13 +241,36 @@ _CASES = [
|
|||||||
server_args=DELETE, ref="--delete-delay"),
|
server_args=DELETE, ref="--delete-delay"),
|
||||||
H.Case("delete_after", "basic", ["-a", "--delete-after"], seed=seed_extras,
|
H.Case("delete_after", "basic", ["-a", "--delete-after"], seed=seed_extras,
|
||||||
server_args=DELETE, ref="--delete-after"),
|
server_args=DELETE, ref="--delete-after"),
|
||||||
|
H.Case("delete_commit", "basic", ["-a", "--delete-after"], seed=seed_extras,
|
||||||
|
fastsync_flags=["-a", "--delete-commit"], server_args=DELETE,
|
||||||
|
ref="FastSync-only --delete-commit == rsync --delete-after"),
|
||||||
H.Case("delete_excluded", "filters",
|
H.Case("delete_excluded", "filters",
|
||||||
["-a", "--delete", "--delete-excluded", "--exclude=*.log"],
|
["-a", "--delete", "--delete-excluded", "--exclude=*.log"],
|
||||||
seed=seed_delete_excluded, server_args=DELETE, ref="--delete-excluded"),
|
seed=seed_delete_excluded, server_args=DELETE, ref="--delete-excluded"),
|
||||||
|
H.Case("exclude_protect_dest_only", "filters",
|
||||||
|
["-a", "--delete", "--exclude=*.log"],
|
||||||
|
seed=seed_delete_excluded, server_args=DELETE, ci=True,
|
||||||
|
ref="--delete protects a destination-only excluded entry like rsync"),
|
||||||
H.Case("max_delete", "basic", ["-a", "--delete", "--max-delete=1"],
|
H.Case("max_delete", "basic", ["-a", "--delete", "--max-delete=1"],
|
||||||
seed=seed_max_delete, server_args=DELETE,
|
seed=seed_max_delete, server_args=DELETE,
|
||||||
extra_check=max_delete_count_check, compare_tree=False,
|
extra_check=max_delete_count_check, compare_tree=False,
|
||||||
ref="--max-delete"),
|
ref="--max-delete"),
|
||||||
|
H.Case("filter_protect", "filters",
|
||||||
|
["-a", "--delete", "--filter=P *.log"],
|
||||||
|
seed=seed_filter_protect, server_args=DELETE, ci=True,
|
||||||
|
ref="--filter P/--protect receiver-side delete protection (default during)"),
|
||||||
|
H.Case("filter_protect_during", "filters",
|
||||||
|
["-a", "--delete-during", "--filter=P *.log"],
|
||||||
|
seed=seed_filter_protect, server_args=DELETE, ci=True,
|
||||||
|
ref="--filter P/--protect under --delete-during"),
|
||||||
|
H.Case("filter_protect_delay", "filters",
|
||||||
|
["-a", "--delete-delay", "--filter=P *.log"],
|
||||||
|
seed=seed_filter_protect, server_args=DELETE, ci=True,
|
||||||
|
ref="--filter P/--protect under --delete-delay"),
|
||||||
|
H.Case("filter_protect_after", "filters",
|
||||||
|
["-a", "--delete-after", "--filter=P *.log"],
|
||||||
|
seed=seed_filter_protect, server_args=DELETE, ci=True,
|
||||||
|
ref="--filter P/--protect under the whole-tree --delete-after commit"),
|
||||||
|
|
||||||
# --- relative / dirs --------------------------------------------------
|
# --- relative / dirs --------------------------------------------------
|
||||||
H.Case("relative_general", "basic", ["-a", "-R"], layout=H.MIRROR_ABS,
|
H.Case("relative_general", "basic", ["-a", "-R"], layout=H.MIRROR_ABS,
|
||||||
@@ -324,7 +380,11 @@ def _result_aspects(result):
|
|||||||
_STANDALONE_REFS = {
|
_STANDALONE_REFS = {
|
||||||
"incremental_modified": "-i/--itemize-changes + incremental second run",
|
"incremental_modified": "-i/--itemize-changes + incremental second run",
|
||||||
"compare_dest": "--compare-dest",
|
"compare_dest": "--compare-dest",
|
||||||
|
"copy_dest": "--copy-dest",
|
||||||
"link_dest": "--link-dest",
|
"link_dest": "--link-dest",
|
||||||
|
"link_dest_stats": "--link-dest + --stats",
|
||||||
|
"verify_basis": "--verify-basis (FastSync-only)",
|
||||||
|
"verify_basis_default": "--verify-basis (default quick-check vs rsync)",
|
||||||
"added_and_deleted": "--delete across two runs",
|
"added_and_deleted": "--delete across two runs",
|
||||||
"added_and_deleted_seed": "--delete across two runs",
|
"added_and_deleted_seed": "--delete across two runs",
|
||||||
"one_file_system": "-x/--one-file-system",
|
"one_file_system": "-x/--one-file-system",
|
||||||
@@ -385,14 +445,17 @@ def test_compare_dest_skips_basis(parity_server_factory):
|
|||||||
fdst = os.path.join(TEST_DATA_DIR, "parity_cmpd_fdst")
|
fdst = os.path.join(TEST_DATA_DIR, "parity_cmpd_fdst")
|
||||||
clean_dir(src)
|
clean_dir(src)
|
||||||
_mk(os.path.join(src, "f.txt"), b"basis-content\n")
|
_mk(os.path.join(src, "f.txt"), b"basis-content\n")
|
||||||
|
_pin(os.path.join(src, "f.txt"), _OLD_MTIME)
|
||||||
server = parity_server_factory(SUPER)
|
server = parity_server_factory(SUPER)
|
||||||
rel = os.path.abspath(src).lstrip(os.sep)
|
rel = os.path.abspath(src).lstrip(os.sep)
|
||||||
|
|
||||||
# rsync resolves --compare-dest relative to the destination dir; FastSync
|
# rsync resolves --compare-dest relative to the destination dir; FastSync
|
||||||
# resolves it under the receive root and appends the mirrored source path.
|
# resolves it under the receive root and appends the mirrored source path.
|
||||||
|
# Both rely on rsync's size+mtime quick-check, so the basis mtime is pinned
|
||||||
|
# to the source's to keep the match deterministic across a second boundary.
|
||||||
def seed(_src, rroot, froot):
|
def seed(_src, rroot, froot):
|
||||||
_mk(os.path.join(rroot, "basis", "f.txt"), b"basis-content\n")
|
_mk(os.path.join(rroot, "basis", "f.txt"), b"basis-content\n", _OLD_MTIME)
|
||||||
_mk(os.path.join(fdst, "basis", rel, "f.txt"), b"basis-content\n")
|
_mk(os.path.join(fdst, "basis", rel, "f.txt"), b"basis-content\n", _OLD_MTIME)
|
||||||
|
|
||||||
def extra(_src, rroot, froot, _rs, _fs):
|
def extra(_src, rroot, froot, _rs, _fs):
|
||||||
out = []
|
out = []
|
||||||
@@ -420,12 +483,13 @@ def test_link_dest_hardlinks_basis(parity_server_factory):
|
|||||||
fdst = os.path.join(TEST_DATA_DIR, "parity_linkd_fdst")
|
fdst = os.path.join(TEST_DATA_DIR, "parity_linkd_fdst")
|
||||||
clean_dir(src)
|
clean_dir(src)
|
||||||
_mk(os.path.join(src, "f.txt"), b"link-basis-content\n")
|
_mk(os.path.join(src, "f.txt"), b"link-basis-content\n")
|
||||||
|
_pin(os.path.join(src, "f.txt"), _OLD_MTIME)
|
||||||
server = parity_server_factory(SUPER)
|
server = parity_server_factory(SUPER)
|
||||||
rel = os.path.abspath(src).lstrip(os.sep)
|
rel = os.path.abspath(src).lstrip(os.sep)
|
||||||
|
|
||||||
def seed(_src, rroot, froot):
|
def seed(_src, rroot, froot):
|
||||||
_mk(os.path.join(rroot, "basis", "f.txt"), b"link-basis-content\n")
|
_mk(os.path.join(rroot, "basis", "f.txt"), b"link-basis-content\n", _OLD_MTIME)
|
||||||
_mk(os.path.join(fdst, "basis", rel, "f.txt"), b"link-basis-content\n")
|
_mk(os.path.join(fdst, "basis", rel, "f.txt"), b"link-basis-content\n", _OLD_MTIME)
|
||||||
|
|
||||||
def extra(_src, rroot, froot, _rs, _fs):
|
def extra(_src, rroot, froot, _rs, _fs):
|
||||||
r_basis = os.stat(os.path.join(rroot, "basis", "f.txt")).st_ino
|
r_basis = os.stat(os.path.join(rroot, "basis", "f.txt")).st_ino
|
||||||
@@ -448,6 +512,134 @@ def test_link_dest_hardlinks_basis(parity_server_factory):
|
|||||||
_run_and_check(case_id, result)
|
_run_and_check(case_id, result)
|
||||||
|
|
||||||
|
|
||||||
|
@requires_rsync
|
||||||
|
@parity
|
||||||
|
def test_link_dest_stats_matches_rsync(parity_server_factory):
|
||||||
|
"""A basis hit must not be counted as created or literal data: rsync reports
|
||||||
|
zero for both, so FastSync's receiver tallies must too (regression for the
|
||||||
|
basis materialization over-report)."""
|
||||||
|
case_id = "link_dest_stats"
|
||||||
|
src = os.path.join(TEST_DATA_DIR, "parity_linkds_src")
|
||||||
|
rdst = os.path.join(TEST_DATA_DIR, "parity_linkds_rdst")
|
||||||
|
fdst = os.path.join(TEST_DATA_DIR, "parity_linkds_fdst")
|
||||||
|
clean_dir(src)
|
||||||
|
_mk(os.path.join(src, "f.txt"), b"link-basis-content\n")
|
||||||
|
_pin(os.path.join(src, "f.txt"), _OLD_MTIME)
|
||||||
|
server = parity_server_factory(SUPER)
|
||||||
|
rel = os.path.abspath(src).lstrip(os.sep)
|
||||||
|
|
||||||
|
def seed(_src, rroot, froot):
|
||||||
|
_mk(os.path.join(rroot, "basis", "f.txt"), b"link-basis-content\n", _OLD_MTIME)
|
||||||
|
_mk(os.path.join(fdst, "basis", rel, "f.txt"), b"link-basis-content\n", _OLD_MTIME)
|
||||||
|
|
||||||
|
result = H.run_differential(
|
||||||
|
src, rdst, fdst,
|
||||||
|
["-a", "--link-dest=basis", "--stats"],
|
||||||
|
["-a", f"--link-dest={os.path.join(fdst, 'basis')}", "--incremental", "--stats"],
|
||||||
|
server, seed=seed, ignore_paths=("basis",), stdout=H.STDOUT_STATS)
|
||||||
|
_run_and_check(case_id, result, ref="--link-dest + --stats")
|
||||||
|
|
||||||
|
|
||||||
|
@requires_rsync
|
||||||
|
@parity
|
||||||
|
def test_copy_dest_copies_basis(parity_server_factory):
|
||||||
|
"""--copy-dest: a basis match is materialized as an independent copy with the
|
||||||
|
source's attributes, matching rsync (copy then fix attributes)."""
|
||||||
|
case_id = "copy_dest"
|
||||||
|
src = os.path.join(TEST_DATA_DIR, "parity_copyd_src")
|
||||||
|
rdst = os.path.join(TEST_DATA_DIR, "parity_copyd_rdst")
|
||||||
|
fdst = os.path.join(TEST_DATA_DIR, "parity_copyd_fdst")
|
||||||
|
clean_dir(src)
|
||||||
|
_mk(os.path.join(src, "f.txt"), b"copy-basis-content\n")
|
||||||
|
_pin(os.path.join(src, "f.txt"), 1_600_000_000)
|
||||||
|
os.chmod(os.path.join(src, "f.txt"), 0o755)
|
||||||
|
server = parity_server_factory(SUPER)
|
||||||
|
rel = os.path.abspath(src).lstrip(os.sep)
|
||||||
|
|
||||||
|
def seed(_src, rroot, froot):
|
||||||
|
# Basis content matches the source; give the basis a different mode so a
|
||||||
|
# wrong "keep basis attributes" implementation is visible.
|
||||||
|
_mk(os.path.join(rroot, "basis", "f.txt"), b"copy-basis-content\n",
|
||||||
|
1_600_000_000)
|
||||||
|
os.chmod(os.path.join(rroot, "basis", "f.txt"), 0o644)
|
||||||
|
_mk(os.path.join(fdst, "basis", rel, "f.txt"), b"copy-basis-content\n",
|
||||||
|
1_600_000_000)
|
||||||
|
os.chmod(os.path.join(fdst, "basis", rel, "f.txt"), 0o644)
|
||||||
|
|
||||||
|
def extra(_src, rroot, froot, _rs, _fs):
|
||||||
|
out = []
|
||||||
|
bases = {"rsync": os.path.join(rroot, "basis", "f.txt"),
|
||||||
|
"fastsync": os.path.join(fdst, "basis", rel, "f.txt")}
|
||||||
|
for label, root in (("rsync", rroot), ("fastsync", froot)):
|
||||||
|
target = os.path.join(root, "f.txt")
|
||||||
|
if not os.path.exists(target):
|
||||||
|
out.append(f"{label}: f.txt missing")
|
||||||
|
continue
|
||||||
|
if os.stat(target).st_ino == os.stat(bases[label]).st_ino:
|
||||||
|
out.append(f"{label}: f.txt is hard-linked, not copied")
|
||||||
|
if (os.stat(target).st_mode & 0o777) != 0o755:
|
||||||
|
out.append(f"{label}: f.txt mode "
|
||||||
|
f"{oct(os.stat(target).st_mode & 0o777)} != 0o755")
|
||||||
|
return out
|
||||||
|
|
||||||
|
result = H.run_differential(
|
||||||
|
src, rdst, fdst,
|
||||||
|
["-a", "--copy-dest=basis"],
|
||||||
|
["-a", f"--copy-dest={os.path.join(fdst, 'basis')}", "--incremental"],
|
||||||
|
server, seed=seed, ignore_paths=("basis",), extra_check=extra,
|
||||||
|
compare_modes=True)
|
||||||
|
_run_and_check(case_id, result)
|
||||||
|
|
||||||
|
|
||||||
|
@requires_rsync
|
||||||
|
@parity
|
||||||
|
def test_verify_basis_restores_strict_content(parity_server_factory):
|
||||||
|
"""Default matches rsync's metadata quick-check; FastSync-only
|
||||||
|
`--verify-basis` restores strict content equality and transfers the source
|
||||||
|
when a same-size/different-content basis would otherwise be trusted."""
|
||||||
|
case_id = "verify_basis"
|
||||||
|
src = os.path.join(TEST_DATA_DIR, "parity_vbasis_src")
|
||||||
|
rdst = os.path.join(TEST_DATA_DIR, "parity_vbasis_rdst")
|
||||||
|
fdst = os.path.join(TEST_DATA_DIR, "parity_vbasis_fdst")
|
||||||
|
clean_dir(src)
|
||||||
|
_mk(os.path.join(src, "f.txt"), b"AAAA\n")
|
||||||
|
_pin(os.path.join(src, "f.txt"), _OLD_MTIME)
|
||||||
|
server = parity_server_factory(SUPER)
|
||||||
|
rel = os.path.abspath(src).lstrip(os.sep)
|
||||||
|
|
||||||
|
def seed(_src, rroot, froot):
|
||||||
|
# Same size and mtime as the source, different bytes: a metadata
|
||||||
|
# quick-check trusts it; --verify-basis must not.
|
||||||
|
for root, basis_rel in ((rroot, os.path.join("basis", "f.txt")),
|
||||||
|
(fdst, os.path.join("basis", rel, "f.txt"))):
|
||||||
|
_mk(os.path.join(root, basis_rel), b"BBBB\n", _OLD_MTIME)
|
||||||
|
|
||||||
|
# Default: both tools trust the basis (rsync's quick check), so the
|
||||||
|
# destination carries the basis bytes and the trees match.
|
||||||
|
result = H.run_differential(
|
||||||
|
src, rdst, fdst,
|
||||||
|
["-a", "--link-dest=basis"],
|
||||||
|
["-a", f"--link-dest={os.path.join(fdst, 'basis')}", "--incremental"],
|
||||||
|
server, seed=seed, ignore_paths=("basis",))
|
||||||
|
_run_and_check(case_id + "_default", result)
|
||||||
|
|
||||||
|
# --verify-basis (FastSync only): the digest mismatch rejects the basis and
|
||||||
|
# the source is transferred, so the destination is the source bytes. rsync
|
||||||
|
# has no such flag; assert the FastSync outcome directly against the source.
|
||||||
|
fdst2 = os.path.join(TEST_DATA_DIR, "parity_vbasis_fdst2")
|
||||||
|
clean_dir(fdst2)
|
||||||
|
for root, basis_rel in ((fdst2, os.path.join("basis", rel, "f.txt")),):
|
||||||
|
_mk(os.path.join(root, basis_rel), b"BBBB\n", _OLD_MTIME)
|
||||||
|
result, _ = H.run_fastsync(src, fdst2,
|
||||||
|
["-a", f"--link-dest={os.path.join(fdst2, 'basis')}",
|
||||||
|
"--incremental", "--verify-basis"], server.port)
|
||||||
|
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
|
||||||
|
target = os.path.join(get_dest_received_dir(fdst2, src), "f.txt")
|
||||||
|
with open(target, "rb") as fh:
|
||||||
|
assert fh.read() == b"AAAA\n", \
|
||||||
|
"--verify-basis must reject the same-size/different-content basis"
|
||||||
|
|
||||||
|
|
||||||
@requires_rsync
|
@requires_rsync
|
||||||
@parity
|
@parity
|
||||||
def test_added_and_deleted_between_runs(parity_server_factory):
|
def test_added_and_deleted_between_runs(parity_server_factory):
|
||||||
|
|||||||
@@ -589,11 +589,10 @@ class TestRemoteDryRun:
|
|||||||
|
|
||||||
Covers the three cases that a real run protects: the file being updated
|
Covers the three cases that a real run protects: the file being updated
|
||||||
(in the keep set), a filter-excluded source entry (protected prefix), and
|
(in the keep set), a filter-excluded source entry (protected prefix), and
|
||||||
a --max-size-pruned source entry (always-protected prefix). Only the
|
a --max-size-pruned source entry (always-protected prefix). Track 4a
|
||||||
genuine destination-only extras may appear. Residual: a destination-only
|
adds a fourth: a destination-only entry matching the exclude rule is
|
||||||
entry matching an exclude pattern is still removed (FastSync derives
|
re-derived on the receiver and also protected, so only the genuine
|
||||||
delete protection from the source scan, not a receiver filter engine);
|
destination-only `extra.txt` appears.
|
||||||
that divergence is pinned by TestOptionParity.
|
|
||||||
"""
|
"""
|
||||||
source = os.path.join(TEST_DATA_DIR, "dryrep_src")
|
source = os.path.join(TEST_DATA_DIR, "dryrep_src")
|
||||||
rdst = os.path.join(TEST_DATA_DIR, "dryrep_rdst")
|
rdst = os.path.join(TEST_DATA_DIR, "dryrep_rdst")
|
||||||
@@ -610,7 +609,8 @@ class TestRemoteDryRun:
|
|||||||
os.makedirs(received, exist_ok=True)
|
os.makedirs(received, exist_ok=True)
|
||||||
for root in (rdst, received):
|
for root in (rdst, received):
|
||||||
for name, data in (("a.txt", b"old\n"), ("keep.log", b"log\n"),
|
for name, data in (("a.txt", b"old\n"), ("keep.log", b"log\n"),
|
||||||
("big.bin", b"B" * 2000), ("extra.txt", b"extra\n")):
|
("big.bin", b"B" * 2000), ("extra.txt", b"extra\n"),
|
||||||
|
("stray.log", b"dest only\n")):
|
||||||
with open(os.path.join(root, name), "wb") as fh:
|
with open(os.path.join(root, name), "wb") as fh:
|
||||||
fh.write(data)
|
fh.write(data)
|
||||||
os.utime(os.path.join(root, name), (1_500_000_000, 1_500_000_000))
|
os.utime(os.path.join(root, name), (1_500_000_000, 1_500_000_000))
|
||||||
@@ -631,6 +631,8 @@ class TestRemoteDryRun:
|
|||||||
fs_del = sorted(l for l in (result.stdout or "").splitlines()
|
fs_del = sorted(l for l in (result.stdout or "").splitlines()
|
||||||
if l.startswith("*deleting"))
|
if l.startswith("*deleting"))
|
||||||
assert fs_del == rsync_del, f"rsync={rsync_del}\nfastsync={fs_del}"
|
assert fs_del == rsync_del, f"rsync={rsync_del}\nfastsync={fs_del}"
|
||||||
|
assert os.path.exists(os.path.join(received, "stray.log")), \
|
||||||
|
"destination-only exclude match must be protected in the dry-run report"
|
||||||
|
|
||||||
@pytest.mark.ci
|
@pytest.mark.ci
|
||||||
def test_remote_dry_run_quiet_is_silent(self, shared_server):
|
def test_remote_dry_run_quiet_is_silent(self, shared_server):
|
||||||
@@ -3980,15 +3982,16 @@ class TestDeleteTiming:
|
|||||||
assert _read_file(os.path.join(received, "sub", "deep.txt")) == b"deeply nested file\n", \
|
assert _read_file(os.path.join(received, "sub", "deep.txt")) == b"deeply nested file\n", \
|
||||||
f"{flag}: nested file was not written after the early deletion"
|
f"{flag}: nested file was not written after the early deletion"
|
||||||
|
|
||||||
@pytest.mark.parametrize("flag", ["--delete", "--delete-after"])
|
@pytest.mark.parametrize("flag", ["--delete-commit", "--delete-after"])
|
||||||
@pytest.mark.parametrize("mt", [False, True])
|
@pytest.mark.parametrize("mt", [False, True])
|
||||||
def test_late_flags_commit_only_after_success(self, flag, mt):
|
def test_late_flags_commit_only_after_success(self, flag, mt):
|
||||||
"""Plain --delete/--delete-after defer deletion until the whole transfer
|
"""--delete-commit/--delete-after defer deletion until the whole transfer
|
||||||
succeeds: a mid-transfer write failure must leave every extra in place
|
succeeds: a mid-transfer write failure must leave every extra in place
|
||||||
(commit-style safety). The -m receiver must also keep the extras: the
|
(commit-style safety). Plain --delete no longer defers (it defaults to
|
||||||
deferred keep-set is committed by the server only after the disk-writer
|
delete-during), so only the explicitly late timings are exercised here.
|
||||||
thread has finished, and a failing writer means the manifest is freed,
|
The --threads receiver must also keep the extras: the deferred keep-set is
|
||||||
never applied."""
|
committed by the server only after the disk-writer thread has finished,
|
||||||
|
and a failing writer means the manifest is freed, never applied."""
|
||||||
source = self._seed("late")
|
source = self._seed("late")
|
||||||
dest = os.path.join(TEST_DATA_DIR, "deltiming_late_dst")
|
dest = os.path.join(TEST_DATA_DIR, "deltiming_late_dst")
|
||||||
clean_dir(dest)
|
clean_dir(dest)
|
||||||
@@ -4693,11 +4696,11 @@ class TestDeletePolicy:
|
|||||||
finally:
|
finally:
|
||||||
os.chmod(source, 0o755)
|
os.chmod(source, 0o755)
|
||||||
|
|
||||||
def test_delete_excluded_protection_is_sender_derived(self):
|
def test_delete_protection_reapplied_on_receiver(self):
|
||||||
"""Plain --delete protects destination mirrors of files the SOURCE scan
|
"""Plain --delete protects destination mirrors of files the SOURCE scan
|
||||||
excluded, but a destination-only file that merely matches an exclude
|
excluded, and (track 4a) also protects a destination-only file matching
|
||||||
rule is still an extra and is removed (protection never re-applies rules
|
an exclude rule because the compiled rule set is re-applied on the
|
||||||
to the destination)."""
|
receiver, matching rsync."""
|
||||||
source = os.path.join(TEST_DATA_DIR, "senderderived_src")
|
source = os.path.join(TEST_DATA_DIR, "senderderived_src")
|
||||||
clean_dir(source)
|
clean_dir(source)
|
||||||
self._write(os.path.join(source, "keep.txt"), b"kept\n")
|
self._write(os.path.join(source, "keep.txt"), b"kept\n")
|
||||||
@@ -4717,8 +4720,8 @@ class TestDeletePolicy:
|
|||||||
f"delete sync failed: {(result.stderr or result.stdout)[:300]}"
|
f"delete sync failed: {(result.stderr or result.stdout)[:300]}"
|
||||||
assert os.path.exists(os.path.join(received, "secret.log")), \
|
assert os.path.exists(os.path.join(received, "secret.log")), \
|
||||||
"source-excluded mirror was deleted under plain --delete"
|
"source-excluded mirror was deleted under plain --delete"
|
||||||
assert not os.path.exists(os.path.join(received, "stray.log")), \
|
assert os.path.exists(os.path.join(received, "stray.log")), \
|
||||||
"destination-only file matching the exclude rule was left (should be deleted)"
|
"destination-only file matching the exclude rule must be protected like rsync"
|
||||||
|
|
||||||
|
|
||||||
def _pin_mtime(path, ts):
|
def _pin_mtime(path, ts):
|
||||||
@@ -4738,11 +4741,12 @@ class TestBasisDestDirs:
|
|||||||
STAGING = ".fastsync-stage"
|
STAGING = ".fastsync-stage"
|
||||||
TS = 1577836800 # 2020-01-01 00:00:00 UTC, used to pin matching mtimes
|
TS = 1577836800 # 2020-01-01 00:00:00 UTC, used to pin matching mtimes
|
||||||
|
|
||||||
# fixture files: source and basis share the mtime pin, so a basis "match"
|
# fixture files: source and basis share the mtime pin, so the DEFAULT
|
||||||
# is decided purely by content (xxHash). unchanged.txt is byte-identical;
|
# (rsync-parity) quick-check is a size+mtime match and trusts the basis even
|
||||||
# changed.txt is byte-DIFFERENT but has the SAME SIZE as the source (and
|
# when the body differs. unchanged.txt is byte-identical; changed.txt is
|
||||||
# the same pinned mtime), which is what forces the content-hash gate;
|
# byte-DIFFERENT but has the SAME SIZE as the source (and the same pinned
|
||||||
# added.txt does not exist in the basis at all.
|
# mtime), which is what the FastSync-only --verify-basis content gate
|
||||||
|
# rejects; added.txt does not exist in the basis at all.
|
||||||
UNCHANGED = "unchanged.txt"
|
UNCHANGED = "unchanged.txt"
|
||||||
CHANGED = "changed.txt"
|
CHANGED = "changed.txt"
|
||||||
ADDED = "added.txt"
|
ADDED = "added.txt"
|
||||||
@@ -4782,18 +4786,19 @@ class TestBasisDestDirs:
|
|||||||
}
|
}
|
||||||
|
|
||||||
def _basis_tree(self, prefix):
|
def _basis_tree(self, prefix):
|
||||||
# unchanged.txt is identical to the source; changed.txt has the SAME
|
# unchanged.txt is identical to the source; changed.txt has a DIFFERENT
|
||||||
# byte size and pinned mtime but a different body (equal size forces
|
# size (and body) so the size leg of the quick-check fails and it is
|
||||||
# the xxHash gate); added.txt is missing from the basis.
|
# transferred normally; added.txt is missing from the basis.
|
||||||
return {
|
return {
|
||||||
self.UNCHANGED: b"stable content v1\n",
|
self.UNCHANGED: b"stable content v1\n",
|
||||||
self.CHANGED: b"CHANGED CONTENT NOW\n",
|
self.CHANGED: b"CHANGED CONTENT NOW AND LONGER\n",
|
||||||
}
|
}
|
||||||
|
|
||||||
def test_same_size_different_content_is_not_a_basis_match(self, shared_server):
|
def test_same_size_different_content_default_trusts_quick_check(self, shared_server):
|
||||||
# Core safety property: equal size + pinned mtime but different content
|
# Default rsync-parity behavior: equal size + pinned mtime is a basis
|
||||||
# must NEVER be hard-linked or copied from the basis -- the xxHash gate
|
# match, so the basis body is materialized/linked without reading it.
|
||||||
# rejects it and the sender's data is transferred instead.
|
# This mirrors rsync 3.4.1's quick check (differential-tested in
|
||||||
|
# test_differential_parity.py::test_verify_basis_restores_strict_content).
|
||||||
for flag, basis_dir in (("--link-dest", "szlb"), ("--copy-dest", "szcp"),
|
for flag, basis_dir in (("--link-dest", "szlb"), ("--copy-dest", "szcp"),
|
||||||
("--compare-dest", "szcmp")):
|
("--compare-dest", "szcmp")):
|
||||||
source = self._make_source("basis_same_size_src",
|
source = self._make_source("basis_same_size_src",
|
||||||
@@ -4805,7 +4810,36 @@ class TestBasisDestDirs:
|
|||||||
result, _ = run_client(source, dest, flags=[f"{flag}={basis_dir}"],
|
result, _ = run_client(source, dest, flags=[f"{flag}={basis_dir}"],
|
||||||
port=shared_server.port)
|
port=shared_server.port)
|
||||||
assert result.returncode == 0, \
|
assert result.returncode == 0, \
|
||||||
f"{flag} same-size mismatch failed: {result.stderr[:300]}"
|
f"{flag} same-size quick-check failed: {result.stderr[:300]}"
|
||||||
|
received = get_dest_received_dir(dest, source)
|
||||||
|
dest_file = os.path.join(received, self.UNCHANGED)
|
||||||
|
if flag == "--compare-dest":
|
||||||
|
assert not os.path.exists(dest_file), \
|
||||||
|
f"{flag}: compare-dest must leave a matching file sparse"
|
||||||
|
else:
|
||||||
|
assert _read_file(dest_file) == b"SAME LENGTH BODY!", \
|
||||||
|
f"{flag}: default quick-check did not trust the basis body"
|
||||||
|
if flag == "--link-dest":
|
||||||
|
assert os.stat(dest_file).st_ino == os.stat(basis_file).st_ino, \
|
||||||
|
f"{flag}: basis was not hard-linked"
|
||||||
|
|
||||||
|
def test_verify_basis_rejects_same_size_different_content(self, shared_server):
|
||||||
|
# FastSync-only --verify-basis: the whole-file digest gate rejects the
|
||||||
|
# same-size/different-content basis, so the source data is transferred
|
||||||
|
# instead of the wrong basis bytes.
|
||||||
|
for flag, basis_dir in (("--link-dest", "vszlb"), ("--copy-dest", "vszcp"),
|
||||||
|
("--compare-dest", "vszcmp")):
|
||||||
|
source = self._make_source("basis_verify_src",
|
||||||
|
{self.UNCHANGED: b"same length body\n"})
|
||||||
|
dest = os.path.join(TEST_DATA_DIR, f"basis_verify_dst_{basis_dir}")
|
||||||
|
clean_dir(dest)
|
||||||
|
basis_file = self._seed_basis_file(dest, source, basis_dir, self.UNCHANGED,
|
||||||
|
b"SAME LENGTH BODY!")
|
||||||
|
result, _ = run_client(source, dest,
|
||||||
|
flags=[f"{flag}={basis_dir}", "--verify-basis"],
|
||||||
|
port=shared_server.port)
|
||||||
|
assert result.returncode == 0, \
|
||||||
|
f"{flag} --verify-basis failed: {result.stderr[:300]}"
|
||||||
received = get_dest_received_dir(dest, source)
|
received = get_dest_received_dir(dest, source)
|
||||||
dest_file = os.path.join(received, self.UNCHANGED)
|
dest_file = os.path.join(received, self.UNCHANGED)
|
||||||
assert _read_file(dest_file) == b"same length body\n", \
|
assert _read_file(dest_file) == b"same length body\n", \
|
||||||
@@ -4837,11 +4871,13 @@ class TestBasisDestDirs:
|
|||||||
self._source_tree("c")[self.ADDED], "added file not transferred"
|
self._source_tree("c")[self.ADDED], "added file not transferred"
|
||||||
|
|
||||||
@pytest.mark.ci
|
@pytest.mark.ci
|
||||||
def test_dry_run_compare_dest_does_not_read_basis(self, shared_server):
|
def test_dry_run_compare_dest_quick_check_does_not_read_basis(self, shared_server):
|
||||||
# A dry-run --compare-dest must never read/hash the basis file: doing so
|
# A dry-run --compare-dest must never read/hash the basis file. Under
|
||||||
# is a 1-bit content oracle against the client-supplied digest. Even a
|
# the default metadata quick-check a matching basis is reported as a
|
||||||
# byte-identical basis with a matching size+mtime is therefore reported
|
# skip (matching rsync) without reading it; nothing is created. Under
|
||||||
# as would-transfer, and nothing is created.
|
# --verify-basis, which would require hashing, the dry-run cannot
|
||||||
|
# confirm the hit (that would be a 1-bit content oracle) and reports
|
||||||
|
# would-transfer instead.
|
||||||
source = self._make_source("basis_dry_src", {self.UNCHANGED: b"stable content v1\n"})
|
source = self._make_source("basis_dry_src", {self.UNCHANGED: b"stable content v1\n"})
|
||||||
dest = os.path.join(TEST_DATA_DIR, "basis_dry_dst")
|
dest = os.path.join(TEST_DATA_DIR, "basis_dry_dst")
|
||||||
clean_dir(dest)
|
clean_dir(dest)
|
||||||
@@ -4852,11 +4888,30 @@ class TestBasisDestDirs:
|
|||||||
port=shared_server.port)
|
port=shared_server.port)
|
||||||
assert result.returncode == 0, \
|
assert result.returncode == 0, \
|
||||||
f"dry-run compare-dest failed: {result.stderr[:300]}"
|
f"dry-run compare-dest failed: {result.stderr[:300]}"
|
||||||
assert self.UNCHANGED in result.stdout, (
|
assert self.UNCHANGED not in result.stdout, (
|
||||||
"dry-run compare-dest silently skipped: receiver read the basis content"
|
"dry-run compare-dest did not honor the metadata quick-check "
|
||||||
|
"(reported would-transfer for a matching basis)"
|
||||||
)
|
)
|
||||||
assert _snapshot_tree(dest) == before, "dry-run compare-dest mutated the destination"
|
assert _snapshot_tree(dest) == before, "dry-run compare-dest mutated the destination"
|
||||||
|
|
||||||
|
# --verify-basis: the hit needs the basis content, which a dry-run must
|
||||||
|
# not read, so the file is reported as would-transfer.
|
||||||
|
dest2 = os.path.join(TEST_DATA_DIR, "basis_dry_verify_dst")
|
||||||
|
clean_dir(dest2)
|
||||||
|
self._seed_basis(dest2, source, "drybasis", {self.UNCHANGED: b"stable content v1\n"})
|
||||||
|
before2 = _snapshot_tree(dest2)
|
||||||
|
result, _ = run_client(source, dest2,
|
||||||
|
flags=["--compare-dest=drybasis", "--dry-run",
|
||||||
|
"--verify-basis"],
|
||||||
|
port=shared_server.port)
|
||||||
|
assert result.returncode == 0, \
|
||||||
|
f"dry-run --verify-basis compare-dest failed: {result.stderr[:300]}"
|
||||||
|
assert self.UNCHANGED in result.stdout, (
|
||||||
|
"dry-run --verify-basis must not read the basis to confirm a hit"
|
||||||
|
)
|
||||||
|
assert _snapshot_tree(dest2) == before2, \
|
||||||
|
"dry-run --verify-basis compare-dest mutated the destination"
|
||||||
|
|
||||||
def test_compare_dest_content_mismatch_forces_transfer(self, shared_server):
|
def test_compare_dest_content_mismatch_forces_transfer(self, shared_server):
|
||||||
# The basis holds a file with a DIFFERENT body: even though it shares
|
# The basis holds a file with a DIFFERENT body: even though it shares
|
||||||
# the mtime pin, the xxHash check fails and the data must be sent.
|
# the mtime pin, the xxHash check fails and the data must be sent.
|
||||||
@@ -5095,27 +5150,36 @@ class TestBasisDestDirs:
|
|||||||
assert os.stat(dest_file).st_ino != os.stat(basis_file).st_ino, \
|
assert os.stat(dest_file).st_ino != os.stat(basis_file).st_ino, \
|
||||||
"--ignore-times must not hard-link to a basis file"
|
"--ignore-times must not hard-link to a basis file"
|
||||||
|
|
||||||
def test_basis_refuses_file_above_whole_file_limit(self, shared_server):
|
def test_basis_handles_file_above_whole_file_limit(self, shared_server):
|
||||||
# Every whole-file payload path in FastSync (basis dirs included) is
|
# Track 5a: a basis hit streams the copy (and the --verify-basis digest
|
||||||
# bounded by MAX_RECEIVE_WHOLE_FILE_SIZE. rsync supports basis dirs for
|
# streams the basis), so a source larger than the whole-file payload
|
||||||
# arbitrary sizes; FastSync refuses such a run up front with a clear
|
# bound is supported for basis dirs exactly like rsync. A basis MISS
|
||||||
# diagnostic instead of letting the receiver abort the whole transfer
|
# still falls back to the normal transfer, which keeps its own bound.
|
||||||
# mid-stream with no client-side explanation.
|
|
||||||
source = self._make_source("basis_oversize_src", {"small.txt": b"ok\n"})
|
source = self._make_source("basis_oversize_src", {"small.txt": b"ok\n"})
|
||||||
big = os.path.join(source, "huge.bin")
|
big = os.path.join(source, "huge.bin")
|
||||||
with open(big, "wb") as fh:
|
with open(big, "wb") as fh:
|
||||||
os.ftruncate(fh.fileno(), 256 * 1024 * 1024 + 4096)
|
os.ftruncate(fh.fileno(), 256 * 1024 * 1024 + 4096)
|
||||||
dest = os.path.join(TEST_DATA_DIR, "basis_oversize_dst")
|
dest = os.path.join(TEST_DATA_DIR, "basis_oversize_dst")
|
||||||
clean_dir(dest)
|
clean_dir(dest)
|
||||||
result, _ = run_client(source, dest, flags=["--link-dest=nope"],
|
|
||||||
port=shared_server.port)
|
|
||||||
assert result.returncode != 0, \
|
|
||||||
"basis run with an over-limit file unexpectedly succeeded"
|
|
||||||
assert "larger than" in result.stderr, \
|
|
||||||
f"no clear over-limit diagnostic: {result.stderr[:300]}"
|
|
||||||
received = get_dest_received_dir(dest, source)
|
received = get_dest_received_dir(dest, source)
|
||||||
assert not os.path.exists(received), \
|
rel = os.path.relpath(received, dest)
|
||||||
"over-limit basis run transferred files before failing"
|
basis_big = os.path.join(dest, "ob", rel, "huge.bin")
|
||||||
|
os.makedirs(os.path.dirname(basis_big), exist_ok=True)
|
||||||
|
shutil.copyfile(big, basis_big)
|
||||||
|
os.utime(basis_big, (self.TS, self.TS))
|
||||||
|
os.utime(big, (self.TS, self.TS))
|
||||||
|
|
||||||
|
result, _ = run_client(source, dest,
|
||||||
|
flags=["--link-dest=ob", "--incremental"],
|
||||||
|
port=shared_server.port)
|
||||||
|
assert result.returncode == 0, \
|
||||||
|
f"over-limit basis run failed: {result.stderr[:300]}"
|
||||||
|
dest_big = os.path.join(received, "huge.bin")
|
||||||
|
assert os.path.exists(dest_big), "over-limit basis hit was not materialized"
|
||||||
|
assert os.path.getsize(dest_big) == 256 * 1024 * 1024 + 4096
|
||||||
|
assert os.stat(dest_big).st_ino == os.stat(basis_big).st_ino, \
|
||||||
|
"over-limit --link-dest did not hard-link to the basis"
|
||||||
|
assert _read_file(os.path.join(received, "small.txt")) == b"ok\n"
|
||||||
|
|
||||||
|
|
||||||
def _random_payloads(size=2 * 1024 * 1024, changed=64 * 1024, seed=1234):
|
def _random_payloads(size=2 * 1024 * 1024, changed=64 * 1024, seed=1234):
|
||||||
|
|||||||
@@ -478,14 +478,14 @@ class TestRemoteOptionDaemon:
|
|||||||
assert "remote-option" in (result.stderr + result.stdout)
|
assert "remote-option" in (result.stderr + result.stdout)
|
||||||
|
|
||||||
|
|
||||||
class TestFilterProtectDivergence:
|
class TestFilterProtect:
|
||||||
"""Documented residual: a protect rule that matches only a destination-only
|
"""Receiver-derived delete protection: a `protect`/`P` rule is compiled by
|
||||||
entry is not re-derived on the receiver (FastSync derives delete protection
|
the sender and sent on the config frame, so the receiver shields a
|
||||||
from the source scan), so rsync protects the extra but FastSync removes it."""
|
destination-only entry that never appeared on the sender, matching rsync."""
|
||||||
|
|
||||||
@requires_rsync
|
@requires_rsync
|
||||||
@pytest.mark.ci
|
@pytest.mark.ci
|
||||||
def test_protect_dest_only_divergence(self, shared_server):
|
def test_protect_dest_only_matches_rsync(self, shared_server):
|
||||||
source = os.path.join(TEST_DATA_DIR, "fpd_src")
|
source = os.path.join(TEST_DATA_DIR, "fpd_src")
|
||||||
dest = os.path.join(TEST_DATA_DIR, "fpd_dst")
|
dest = os.path.join(TEST_DATA_DIR, "fpd_dst")
|
||||||
rdst = os.path.join(TEST_DATA_DIR, "fpd_rdst")
|
rdst = os.path.join(TEST_DATA_DIR, "fpd_rdst")
|
||||||
@@ -498,6 +498,7 @@ class TestFilterProtectDivergence:
|
|||||||
rsync_result = _rsync(["-a", "--delete", "--filter=P *.log", source + "/", rdst + "/"])
|
rsync_result = _rsync(["-a", "--delete", "--filter=P *.log", source + "/", rdst + "/"])
|
||||||
assert rsync_result.returncode == 0, rsync_result.stderr
|
assert rsync_result.returncode == 0, rsync_result.stderr
|
||||||
assert os.path.exists(os.path.join(rdst, "extra.log")), "rsync did not protect extra.log"
|
assert os.path.exists(os.path.join(rdst, "extra.log")), "rsync did not protect extra.log"
|
||||||
|
assert not os.path.exists(os.path.join(rdst, "other.txt")), "rsync did not delete other.txt"
|
||||||
|
|
||||||
clean_dir(dest)
|
clean_dir(dest)
|
||||||
received = get_dest_received_dir(dest, source)
|
received = get_dest_received_dir(dest, source)
|
||||||
@@ -509,9 +510,29 @@ class TestFilterProtectDivergence:
|
|||||||
flags=["-a", "--delete", "--filter=P *.log"],
|
flags=["-a", "--delete", "--filter=P *.log"],
|
||||||
port=server.port)
|
port=server.port)
|
||||||
assert result.returncode == 0, (result.stderr or result.stdout)[:200]
|
assert result.returncode == 0, (result.stderr or result.stdout)[:200]
|
||||||
# Pin the known divergence: FastSync deletes the destination-only file.
|
assert os.path.exists(os.path.join(received, "extra.log")), (
|
||||||
assert not os.path.exists(os.path.join(received, "extra.log")), (
|
"FastSync must protect a destination-only P match like rsync")
|
||||||
"FastSync now protects destination-only P matches; the --filter row may be "
|
|
||||||
"upgradable to full parity"
|
|
||||||
)
|
|
||||||
assert not os.path.exists(os.path.join(received, "other.txt"))
|
assert not os.path.exists(os.path.join(received, "other.txt"))
|
||||||
|
|
||||||
|
@pytest.mark.ci
|
||||||
|
def test_protect_dest_only_dry_run_enumeration(self, shared_server):
|
||||||
|
source = os.path.join(TEST_DATA_DIR, "fpd_nd_src")
|
||||||
|
dest = os.path.join(TEST_DATA_DIR, "fpd_nd_dst")
|
||||||
|
clean_dir(source)
|
||||||
|
_write(os.path.join(source, "keep.txt"), b"keep\n")
|
||||||
|
received = get_dest_received_dir(dest, source)
|
||||||
|
clean_dir(received)
|
||||||
|
_write(os.path.join(received, "keep.txt"), b"keep\n")
|
||||||
|
_write(os.path.join(received, "extra.log"), b"extra\n")
|
||||||
|
_write(os.path.join(received, "other.txt"), b"other\n")
|
||||||
|
with ServerManager() as server:
|
||||||
|
server.start(extra_args=["--allow-delete"])
|
||||||
|
result, _ = run_client(source, dest,
|
||||||
|
flags=["-a", "-n", "--delete", "--out-format=%n",
|
||||||
|
"--filter=P *.log"],
|
||||||
|
port=server.port)
|
||||||
|
assert result.returncode == 0, (result.stderr or result.stdout)[:300]
|
||||||
|
assert "other.txt" in result.stdout, result.stdout
|
||||||
|
assert "extra.log" not in result.stdout, result.stdout
|
||||||
|
assert os.path.exists(os.path.join(received, "extra.log"))
|
||||||
|
assert os.path.exists(os.path.join(received, "other.txt"))
|
||||||
|
|||||||
@@ -719,13 +719,18 @@ class TestVerifyAndFlip:
|
|||||||
source = self._src("cmpd")
|
source = self._src("cmpd")
|
||||||
dest = self._dst("cmpd")
|
dest = self._dst("cmpd")
|
||||||
rdst = self._dst("cmpd_r")
|
rdst = self._dst("cmpd_r")
|
||||||
|
# Pin the mtime so rsync's size+mtime quick-check (and FastSync's
|
||||||
|
# default) matches deterministically across a second boundary.
|
||||||
|
OLD = 1_500_000_000
|
||||||
with open(os.path.join(source, "f.txt"), "wb") as fh:
|
with open(os.path.join(source, "f.txt"), "wb") as fh:
|
||||||
fh.write(b"basis-content\n")
|
fh.write(b"basis-content\n")
|
||||||
|
os.utime(os.path.join(source, "f.txt"), (OLD, OLD))
|
||||||
# rsync resolves --compare-dest relative to the destination dir; its
|
# rsync resolves --compare-dest relative to the destination dir; its
|
||||||
# basis file sits at the transfer-relative path.
|
# basis file sits at the transfer-relative path.
|
||||||
os.makedirs(os.path.join(rdst, "basis"), exist_ok=True)
|
os.makedirs(os.path.join(rdst, "basis"), exist_ok=True)
|
||||||
with open(os.path.join(rdst, "basis", "f.txt"), "wb") as fh:
|
with open(os.path.join(rdst, "basis", "f.txt"), "wb") as fh:
|
||||||
fh.write(b"basis-content\n")
|
fh.write(b"basis-content\n")
|
||||||
|
os.utime(os.path.join(rdst, "basis", "f.txt"), (OLD, OLD))
|
||||||
rs = _rsync(["-a", "--compare-dest=basis", source + "/", rdst + "/"])
|
rs = _rsync(["-a", "--compare-dest=basis", source + "/", rdst + "/"])
|
||||||
assert rs.returncode == 0, rs.stderr
|
assert rs.returncode == 0, rs.stderr
|
||||||
assert not os.path.exists(os.path.join(rdst, "f.txt")), \
|
assert not os.path.exists(os.path.join(rdst, "f.txt")), \
|
||||||
@@ -738,6 +743,7 @@ class TestVerifyAndFlip:
|
|||||||
os.makedirs(basis, exist_ok=True)
|
os.makedirs(basis, exist_ok=True)
|
||||||
with open(os.path.join(basis, "f.txt"), "wb") as fh:
|
with open(os.path.join(basis, "f.txt"), "wb") as fh:
|
||||||
fh.write(b"basis-content\n")
|
fh.write(b"basis-content\n")
|
||||||
|
os.utime(os.path.join(basis, "f.txt"), (OLD, OLD))
|
||||||
received = get_dest_received_dir(dest, source)
|
received = get_dest_received_dir(dest, source)
|
||||||
result, _ = run_client(source, dest,
|
result, _ = run_client(source, dest,
|
||||||
flags=["--compare-dest=basis", "--incremental"],
|
flags=["--compare-dest=basis", "--incremental"],
|
||||||
@@ -751,14 +757,17 @@ class TestVerifyAndFlip:
|
|||||||
def test_link_dest_hardlinks_matches_rsync(self, shared_server):
|
def test_link_dest_hardlinks_matches_rsync(self, shared_server):
|
||||||
source = self._src("linkd")
|
source = self._src("linkd")
|
||||||
dest = self._dst("linkd")
|
dest = self._dst("linkd")
|
||||||
|
OLD = 1_500_000_000
|
||||||
with open(os.path.join(source, "f.txt"), "wb") as fh:
|
with open(os.path.join(source, "f.txt"), "wb") as fh:
|
||||||
fh.write(b"link-basis-content\n")
|
fh.write(b"link-basis-content\n")
|
||||||
|
os.utime(os.path.join(source, "f.txt"), (OLD, OLD))
|
||||||
rel = os.path.abspath(source).lstrip(os.sep)
|
rel = os.path.abspath(source).lstrip(os.sep)
|
||||||
basis = os.path.join(dest, "basis", rel)
|
basis = os.path.join(dest, "basis", rel)
|
||||||
os.makedirs(basis, exist_ok=True)
|
os.makedirs(basis, exist_ok=True)
|
||||||
basis_file = os.path.join(basis, "f.txt")
|
basis_file = os.path.join(basis, "f.txt")
|
||||||
with open(basis_file, "wb") as fh:
|
with open(basis_file, "wb") as fh:
|
||||||
fh.write(b"link-basis-content\n")
|
fh.write(b"link-basis-content\n")
|
||||||
|
os.utime(basis_file, (OLD, OLD))
|
||||||
received = get_dest_received_dir(dest, source)
|
received = get_dest_received_dir(dest, source)
|
||||||
result, _ = run_client(source, dest,
|
result, _ = run_client(source, dest,
|
||||||
flags=["--link-dest=basis", "--incremental"],
|
flags=["--link-dest=basis", "--incremental"],
|
||||||
@@ -771,12 +780,11 @@ class TestVerifyAndFlip:
|
|||||||
|
|
||||||
@requires_rsync
|
@requires_rsync
|
||||||
def test_basis_dir_size_only_content_residual(self, shared_server):
|
def test_basis_dir_size_only_content_residual(self, shared_server):
|
||||||
"""Documented residual (RSYNC_COMPAT.md basis-dir rows): FastSync
|
"""rsync parity (default): a basis hit is decided by the metadata
|
||||||
xxHash-verifies a basis hit, while rsync's `--size-only` quick check
|
quick-check alone. With `--size-only`, a same-size, different-content
|
||||||
trusts the size alone. With a same-size, different-content basis,
|
basis is trusted, so rsync links the basis content and FastSync must now
|
||||||
rsync links/copies the wrong basis content while FastSync transfers the
|
do the same instead of xxHash-verifying it. `--verify-basis` restores
|
||||||
source. This test pins both observed behaviors (FastSync is stricter,
|
the stricter content equality (covered by the differential test)."""
|
||||||
so the rows are reclassified Divergent)."""
|
|
||||||
source = self._src("basissz")
|
source = self._src("basissz")
|
||||||
rdest = self._dst("basissz_r")
|
rdest = self._dst("basissz_r")
|
||||||
fdest = self._dst("basissz_f")
|
fdest = self._dst("basissz_f")
|
||||||
@@ -806,9 +814,123 @@ class TestVerifyAndFlip:
|
|||||||
flags=["-a", "--size-only", "--link-dest=basis", "--incremental"],
|
flags=["-a", "--size-only", "--link-dest=basis", "--incremental"],
|
||||||
port=shared_server.port)
|
port=shared_server.port)
|
||||||
assert result.returncode == 0, result.stderr[:300]
|
assert result.returncode == 0, result.stderr[:300]
|
||||||
|
with open(os.path.join(received, "f.txt"), "rb") as fh:
|
||||||
|
assert fh.read() == b"BBBB\n", \
|
||||||
|
"FastSync must trust the metadata quick-check exactly like rsync"
|
||||||
|
|
||||||
|
@requires_rsync
|
||||||
|
def test_verify_basis_restores_content_check(self, shared_server):
|
||||||
|
"""FastSync-only `--verify-basis`: a same-size, same-mtime basis with
|
||||||
|
different content is rejected by the whole-file digest, so the source is
|
||||||
|
transferred instead of installing the wrong basis bytes. The default
|
||||||
|
(no flag) installs the basis content, matching rsync."""
|
||||||
|
source = self._src("vbasis")
|
||||||
|
fdest = self._dst("vbasis_f")
|
||||||
|
with open(os.path.join(source, "f.txt"), "wb") as fh:
|
||||||
|
fh.write(b"AAAA\n")
|
||||||
|
OLD = 1_400_000_000
|
||||||
|
os.utime(os.path.join(source, "f.txt"), (OLD, OLD))
|
||||||
|
rel = os.path.abspath(source).lstrip(os.sep)
|
||||||
|
basis = os.path.join(fdest, "basis", rel)
|
||||||
|
os.makedirs(basis, exist_ok=True)
|
||||||
|
with open(os.path.join(basis, "f.txt"), "wb") as fh:
|
||||||
|
fh.write(b"BBBB\n")
|
||||||
|
os.utime(os.path.join(basis, "f.txt"), (OLD, OLD))
|
||||||
|
received = get_dest_received_dir(fdest, source)
|
||||||
|
result, _ = run_client(source, fdest,
|
||||||
|
flags=["-a", "--link-dest=basis", "--incremental",
|
||||||
|
"--verify-basis"],
|
||||||
|
port=shared_server.port)
|
||||||
|
assert result.returncode == 0, result.stderr[:300]
|
||||||
with open(os.path.join(received, "f.txt"), "rb") as fh:
|
with open(os.path.join(received, "f.txt"), "rb") as fh:
|
||||||
assert fh.read() == b"AAAA\n", \
|
assert fh.read() == b"AAAA\n", \
|
||||||
"FastSync must verify the basis content and transfer the source"
|
"--verify-basis must reject the same-size/different-content basis"
|
||||||
|
|
||||||
|
|
||||||
|
def _stat_bytes(output, key):
|
||||||
|
"""Parse a --stats byte counter (e.g. ``Matched data: 65,536 bytes``)."""
|
||||||
|
for line in output.splitlines():
|
||||||
|
if line.startswith(key + ":"):
|
||||||
|
raw = line.split(":", 1)[1].strip().split()[0]
|
||||||
|
return int(raw.replace(",", ""))
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
class TestFuzzy:
|
||||||
|
"""Track 5b: `-y`/`--fuzzy` is an internal bandwidth optimization with a
|
||||||
|
byte-exact result. FastSync ports rsync 3.4.1's weighted-Levenshtein name
|
||||||
|
heuristic, so where both delta engines admit the candidate the tools pick
|
||||||
|
the same basis (the ``fuzzy_basis`` differential asserts the tree and the
|
||||||
|
Matched/Literal counters match with the block size pinned). The residual is
|
||||||
|
candidate ELIGIBILITY: FastSync's delta size gate (both files >= 16 KiB and
|
||||||
|
a <= 10x size ratio) is narrower than rsync's, which empirically uses a
|
||||||
|
fuzzy basis well beyond 10x and below 16 KiB. These tests pin the window
|
||||||
|
boundary and prove the byte-exact fallback on both sides of it."""
|
||||||
|
|
||||||
|
_BASE = b"the quick brown fox jumps over the lazy dog\n" * 4000
|
||||||
|
|
||||||
|
def _src(self, tag):
|
||||||
|
source = os.path.join(TEST_DATA_DIR, f"fz_{tag}_src")
|
||||||
|
clean_dir(source)
|
||||||
|
return source
|
||||||
|
|
||||||
|
def _dst(self, tag):
|
||||||
|
d = os.path.join(TEST_DATA_DIR, f"fz_{tag}_dst")
|
||||||
|
clean_dir(d)
|
||||||
|
return d
|
||||||
|
|
||||||
|
def _run_both(self, shared_server, source, dest, rdst, payload, sibling,
|
||||||
|
rs_extra=(), fs_extra=()):
|
||||||
|
with open(os.path.join(source, "report_v2.txt"), "wb") as fh:
|
||||||
|
fh.write(payload)
|
||||||
|
for root in (rdst, get_dest_received_dir(dest, source)):
|
||||||
|
os.makedirs(root, exist_ok=True)
|
||||||
|
with open(os.path.join(root, "report_v1.txt"), "wb") as fh:
|
||||||
|
fh.write(sibling)
|
||||||
|
rs = _rsync(["-a", "--no-whole-file", "--fuzzy", "--stats"] +
|
||||||
|
list(rs_extra) + [source + "/", rdst + "/"])
|
||||||
|
assert rs.returncode == 0, rs.stderr[:300]
|
||||||
|
result, _ = run_client(
|
||||||
|
source, dest,
|
||||||
|
flags=["-a", "--incremental", "--delta", "--fuzzy", "--stats"] +
|
||||||
|
list(fs_extra),
|
||||||
|
port=shared_server.port)
|
||||||
|
assert result.returncode == 0, result.stderr[:300]
|
||||||
|
_assert_same_tree(rdst, get_dest_received_dir(dest, source), "(--fuzzy)")
|
||||||
|
return rs, result
|
||||||
|
|
||||||
|
@requires_rsync
|
||||||
|
def test_fuzzy_above_size_window_declines_but_tree_exact(self, shared_server):
|
||||||
|
"""A sibling >10x the source is used by rsync but declined by FastSync's
|
||||||
|
delta size-ratio gate; both destinations stay byte-identical."""
|
||||||
|
n = 65536
|
||||||
|
payload = (self._BASE * ((n // len(self._BASE)) + 1))[:n]
|
||||||
|
sibling = (self._BASE * 200)[: n * 20]
|
||||||
|
source, dest, rdst = (self._src("big"), self._dst("big"),
|
||||||
|
self._dst("big_r"))
|
||||||
|
rs, result = self._run_both(shared_server, source, dest, rdst,
|
||||||
|
payload, sibling)
|
||||||
|
assert _stat_bytes(rs.stdout, "Matched data") > 0, \
|
||||||
|
"rsync should still use a >10x fuzzy basis"
|
||||||
|
assert _stat_bytes(result.stdout, "Matched data") == 0, \
|
||||||
|
"FastSync's 10x delta size-ratio gate must decline the oversized basis"
|
||||||
|
assert _stat_bytes(result.stdout, "Literal data") == n
|
||||||
|
|
||||||
|
@requires_rsync
|
||||||
|
def test_fuzzy_below_delta_minimum_declines_but_tree_exact(self, shared_server):
|
||||||
|
"""A sibling below the 16 KiB delta minimum is used by rsync but never
|
||||||
|
enters FastSync's delta/fuzzy path; both trees stay byte-identical."""
|
||||||
|
n = 8192
|
||||||
|
payload = (self._BASE * ((n // len(self._BASE)) + 1))[:n]
|
||||||
|
source, dest, rdst = (self._src("small"), self._dst("small"),
|
||||||
|
self._dst("small_r"))
|
||||||
|
rs, result = self._run_both(shared_server, source, dest, rdst,
|
||||||
|
payload, payload)
|
||||||
|
assert _stat_bytes(rs.stdout, "Matched data") > 0, \
|
||||||
|
"rsync applies --fuzzy below 16 KiB"
|
||||||
|
assert _stat_bytes(result.stdout, "Matched data") == 0, \
|
||||||
|
"FastSync's 16 KiB delta minimum must bypass the fuzzy basis"
|
||||||
|
assert _stat_bytes(result.stdout, "Literal data") == n
|
||||||
|
|
||||||
|
|
||||||
class TestIgnoreExistingShortCircuit:
|
class TestIgnoreExistingShortCircuit:
|
||||||
|
|||||||
@@ -118,8 +118,8 @@ class TestProtocol:
|
|||||||
shutil.rmtree(dest, ignore_errors=True)
|
shutil.rmtree(dest, ignore_errors=True)
|
||||||
os.makedirs(dest)
|
os.makedirs(dest)
|
||||||
_seed_protocol_source(source)
|
_seed_protocol_source(source)
|
||||||
for bad in ("2.22.0", "2.21.0", "2.20.0", "2.19.0", "2.18.0", "2.17.0", "2.15.0", "2.16.0",
|
for bad in ("2.27.0", "2.26.0", "2.25.0", "2.24.0", "2.23.0", "2.22.0", "2.21.0", "2.20.0",
|
||||||
"216", "31"):
|
"2.19.0", "2.18.0", "2.17.0", "2.15.0", "2.16.0", "216", "31"):
|
||||||
result, _ = run_client(source, dest, flags=[f"--protocol={bad}"],
|
result, _ = run_client(source, dest, flags=[f"--protocol={bad}"],
|
||||||
port=shared_server.port)
|
port=shared_server.port)
|
||||||
assert result.returncode != 0, f"--protocol={bad} should be rejected"
|
assert result.returncode != 0, f"--protocol={bad} should be rejected"
|
||||||
|
|||||||
@@ -340,6 +340,7 @@ static void test_parse_args_protocol_accept_current() {
|
|||||||
static void test_parse_args_protocol_rejects_other_versions() {
|
static void test_parse_args_protocol_rejects_other_versions() {
|
||||||
static const char* const bad_versions[] = {"2.17", "2.16", "2.15.0", "2.16.0", "2.17.0",
|
static const char* const bad_versions[] = {"2.17", "2.16", "2.15.0", "2.16.0", "2.17.0",
|
||||||
"2.18.0", "2.19.0", "2.20.0", "2.21.0", "2.22.0",
|
"2.18.0", "2.19.0", "2.20.0", "2.21.0", "2.22.0",
|
||||||
|
"2.23.0", "2.24.0", "2.25.0", "2.26.0", "2.27.0",
|
||||||
"216", "31", "abc", ""};
|
"216", "31", "abc", ""};
|
||||||
for (size_t i = 0; i < sizeof(bad_versions) / sizeof(bad_versions[0]); i++) {
|
for (size_t i = 0; i < sizeof(bad_versions) / sizeof(bad_versions[0]); i++) {
|
||||||
Config* cfg = valid_client_config();
|
Config* cfg = valid_client_config();
|
||||||
@@ -1045,6 +1046,23 @@ static void test_parse_args_basis_dirs() {
|
|||||||
config_delete(cfg);
|
config_delete(cfg);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* --verify-basis (FastSync-only, long-only): default off; parses on as a plain
|
||||||
|
boolean and leaves the basis implications intact. */
|
||||||
|
static void test_parse_args_verify_basis() {
|
||||||
|
Config* cfg = config_create();
|
||||||
|
EXPECT_FALSE(cfg->verify_basis);
|
||||||
|
config_delete(cfg);
|
||||||
|
|
||||||
|
cfg = config_create();
|
||||||
|
int positional_args[2];
|
||||||
|
int positional_count = 0;
|
||||||
|
char* argv[] = {"fastsync", "--link-dest=prior", "--verify-basis", "/src", "/dst"};
|
||||||
|
EXPECT_EQ_INT(parse_args(cfg, 5, argv, positional_args, &positional_count), 0);
|
||||||
|
EXPECT_TRUE(cfg->verify_basis);
|
||||||
|
EXPECT_TRUE(config_has_basis(cfg));
|
||||||
|
config_delete(cfg);
|
||||||
|
}
|
||||||
|
|
||||||
/* Escaping or degenerate basis-dir values must be rejected up front (they would
|
/* Escaping or degenerate basis-dir values must be rejected up front (they would
|
||||||
resolve outside the destination root on the receiver); an absolute path is
|
resolve outside the destination root on the receiver); an absolute path is
|
||||||
accepted (rsync parity) and canonicalized with its leading '/' preserved. */
|
accepted (rsync parity) and canonicalized with its leading '/' preserved. */
|
||||||
@@ -1132,6 +1150,63 @@ static void test_parse_args_delete_timing_flags() {
|
|||||||
config_delete(cfg);
|
config_delete(cfg);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Plain --delete with no explicit timing defaults to delete-during, matching
|
||||||
|
* rsync's --del default (progressive deletion). --delete-commit is the
|
||||||
|
* FastSync-only long spelling that selects rsync's --delete-after timing (the
|
||||||
|
* late whole-tree commit), and an explicit timing always wins over the default.
|
||||||
|
*/
|
||||||
|
static void test_parse_args_delete_default_timing_and_commit() {
|
||||||
|
Config* cfg = config_create();
|
||||||
|
char* argv[] = {"fastsync", "--delete", "/src", "/dst"};
|
||||||
|
int positional_args[2];
|
||||||
|
int positional_count = 0;
|
||||||
|
EXPECT_EQ_INT(parse_args(cfg, 4, argv, positional_args, &positional_count), 0);
|
||||||
|
EXPECT_TRUE(cfg->use_delete);
|
||||||
|
EXPECT_TRUE(cfg->delete_during);
|
||||||
|
EXPECT_FALSE(cfg->delete_before);
|
||||||
|
EXPECT_FALSE(cfg->delete_delay);
|
||||||
|
EXPECT_FALSE(cfg->delete_after);
|
||||||
|
cfg->send_directory = str_dup("/src");
|
||||||
|
cfg->receive_root_directory = str_dup("/dst");
|
||||||
|
EXPECT_TRUE(validate_config(cfg));
|
||||||
|
config_delete(cfg);
|
||||||
|
|
||||||
|
/* --delete-commit selects the late whole-tree commit (delete_after) and
|
||||||
|
implies --delete. */
|
||||||
|
cfg = config_create();
|
||||||
|
char* argv_commit[] = {"fastsync", "--delete-commit", "/src", "/dst"};
|
||||||
|
positional_count = 0;
|
||||||
|
EXPECT_EQ_INT(parse_args(cfg, 4, argv_commit, positional_args, &positional_count), 0);
|
||||||
|
EXPECT_TRUE(cfg->use_delete);
|
||||||
|
EXPECT_TRUE(cfg->delete_after);
|
||||||
|
EXPECT_FALSE(cfg->delete_before);
|
||||||
|
EXPECT_FALSE(cfg->delete_during);
|
||||||
|
EXPECT_FALSE(cfg->delete_delay);
|
||||||
|
cfg->send_directory = str_dup("/src");
|
||||||
|
cfg->receive_root_directory = str_dup("/dst");
|
||||||
|
EXPECT_TRUE(validate_config(cfg));
|
||||||
|
config_delete(cfg);
|
||||||
|
|
||||||
|
/* An explicit --delete-after alongside plain --delete keeps the late timing:
|
||||||
|
the default never overwrites an explicit timing. */
|
||||||
|
cfg = config_create();
|
||||||
|
char* argv_after[] = {"fastsync", "--delete", "--delete-after", "/src", "/dst"};
|
||||||
|
positional_count = 0;
|
||||||
|
EXPECT_EQ_INT(parse_args(cfg, 5, argv_after, positional_args, &positional_count), 0);
|
||||||
|
EXPECT_TRUE(cfg->use_delete);
|
||||||
|
EXPECT_TRUE(cfg->delete_after);
|
||||||
|
EXPECT_FALSE(cfg->delete_during);
|
||||||
|
config_delete(cfg);
|
||||||
|
|
||||||
|
/* --delete-commit conflicts with a different timing. */
|
||||||
|
cfg = config_create();
|
||||||
|
char* argv_conflict[] = {"fastsync", "--delete-commit", "--delete-during", "/src", "/dst"};
|
||||||
|
positional_count = 0;
|
||||||
|
EXPECT_EQ_INT(parse_args(cfg, 5, argv_conflict, positional_args, &positional_count), 0);
|
||||||
|
EXPECT_FALSE(validate_config(cfg));
|
||||||
|
config_delete(cfg);
|
||||||
|
}
|
||||||
|
|
||||||
/* Two different delete-timing flags on one command line are a conflict, not a
|
/* Two different delete-timing flags on one command line are a conflict, not a
|
||||||
* silent last-one-wins choice. */
|
* silent last-one-wins choice. */
|
||||||
static void test_parse_args_delete_timing_conflict_rejected() {
|
static void test_parse_args_delete_timing_conflict_rejected() {
|
||||||
@@ -4740,6 +4815,7 @@ void test_client_cli() {
|
|||||||
test_parse_args_relative_no_implied_mkpath();
|
test_parse_args_relative_no_implied_mkpath();
|
||||||
test_parse_args_delete_during_alias();
|
test_parse_args_delete_during_alias();
|
||||||
test_parse_args_delete_timing_flags();
|
test_parse_args_delete_timing_flags();
|
||||||
|
test_parse_args_delete_default_timing_and_commit();
|
||||||
test_parse_args_delete_timing_conflict_rejected();
|
test_parse_args_delete_timing_conflict_rejected();
|
||||||
test_parse_args_delete_timing_without_delete_rejected();
|
test_parse_args_delete_timing_without_delete_rejected();
|
||||||
test_parse_args_rejects_unimplemented_options();
|
test_parse_args_rejects_unimplemented_options();
|
||||||
@@ -4831,6 +4907,7 @@ void test_client_cli() {
|
|||||||
test_parse_args_filter_rules();
|
test_parse_args_filter_rules();
|
||||||
test_parse_args_from0_cvs_filter_file_flags();
|
test_parse_args_from0_cvs_filter_file_flags();
|
||||||
test_parse_args_basis_dirs();
|
test_parse_args_basis_dirs();
|
||||||
|
test_parse_args_verify_basis();
|
||||||
test_parse_args_basis_invalid_paths();
|
test_parse_args_basis_invalid_paths();
|
||||||
test_validate_config_basis_rejects_chunk_serialization();
|
test_validate_config_basis_rejects_chunk_serialization();
|
||||||
test_parse_args_delete_policy_flags();
|
test_parse_args_delete_policy_flags();
|
||||||
|
|||||||
+103
-6
@@ -1,6 +1,7 @@
|
|||||||
#include "test_config.h"
|
#include "test_config.h"
|
||||||
#include "config.h"
|
#include "config.h"
|
||||||
#include "delta.h"
|
#include "delta.h"
|
||||||
|
#include "filter.h"
|
||||||
#include "identity.h"
|
#include "identity.h"
|
||||||
#include "multiprocessing.h"
|
#include "multiprocessing.h"
|
||||||
#include "protocol.h"
|
#include "protocol.h"
|
||||||
@@ -2119,6 +2120,48 @@ static void test_config_receive_rejects_oversized_string_budget() {
|
|||||||
config_delete(over_bytes);
|
config_delete(over_bytes);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Pre-auth bounds for the receiver-side filter rule block (protocol 2.28.0).
|
||||||
|
A peer may send `protect`/`risk` rules; the receiver must reject an over-cap
|
||||||
|
count or an over-long pattern before evaluating anything, so a crafted config
|
||||||
|
cannot drive unbounded glob work or install a rule that silently never
|
||||||
|
matches. */
|
||||||
|
static void test_config_receive_rejects_bad_protect_rules() {
|
||||||
|
if (is_running_under_valgrind())
|
||||||
|
return;
|
||||||
|
|
||||||
|
/* Over-cap rule count: one more than MAX_FILTER_RULES rules. */
|
||||||
|
Config* c = config_create();
|
||||||
|
EXPECT_NOT_NULL(c);
|
||||||
|
c->send_directory = str_dup("/src");
|
||||||
|
c->receive_root_directory = str_dup("/dst");
|
||||||
|
c->filters = array_list_create(free);
|
||||||
|
EXPECT_NOT_NULL(c->filters);
|
||||||
|
for (int i = 0; i <= MAX_FILTER_RULES; i++)
|
||||||
|
EXPECT_TRUE(array_list_add(c->filters, str_dup("- *.tmp")));
|
||||||
|
EXPECT_TRUE(roundtrip_config_rejected(c));
|
||||||
|
config_delete(c);
|
||||||
|
|
||||||
|
/* A pattern longer than the receiver's evaluation bound is rejected by the
|
||||||
|
sender (mirroring the receiver's guard) instead of being sent as an inert
|
||||||
|
rule. */
|
||||||
|
c = config_create();
|
||||||
|
EXPECT_NOT_NULL(c);
|
||||||
|
c->send_directory = str_dup("/src");
|
||||||
|
c->receive_root_directory = str_dup("/dst");
|
||||||
|
c->filters = array_list_create(free);
|
||||||
|
EXPECT_NOT_NULL(c->filters);
|
||||||
|
size_t big = MAX_PROTECT_PATTERN_LEN + 1;
|
||||||
|
char* long_rule = malloc(big + 3);
|
||||||
|
EXPECT_NOT_NULL(long_rule);
|
||||||
|
long_rule[0] = '-';
|
||||||
|
long_rule[1] = ' ';
|
||||||
|
memset(long_rule + 2, 'x', big);
|
||||||
|
long_rule[big + 2] = '\0';
|
||||||
|
EXPECT_TRUE(array_list_add(c->filters, long_rule));
|
||||||
|
EXPECT_TRUE(roundtrip_config_rejected(c));
|
||||||
|
config_delete(c);
|
||||||
|
}
|
||||||
|
|
||||||
/* identity_copy_as_refused() is the pure, pre-snapshot refusal predicate: a
|
/* identity_copy_as_refused() is the pure, pre-snapshot refusal predicate: a
|
||||||
--copy-as is refused when the receiver is not root OR the effective super
|
--copy-as is refused when the receiver is not root OR the effective super
|
||||||
mode is OFF (an operator veto), and never when --copy-as is unset. */
|
mode is OFF (an operator veto), and never when --copy-as is unset. */
|
||||||
@@ -2590,6 +2633,46 @@ static bool basis_equal(const Config* a, const Config* b) {
|
|||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* The receiver reconstructs its delete-protection list from the sender's
|
||||||
|
* compiled base rules, so compare the received list against a fresh
|
||||||
|
* filter_base_build() of the sender's raw --filter texts. */
|
||||||
|
static bool filter_rules_equal(const Config* a, const FilterRuleList* got) {
|
||||||
|
int count = a->filters ? a->filters->size : 0;
|
||||||
|
const char** texts = NULL;
|
||||||
|
if (count > 0) {
|
||||||
|
texts = calloc((size_t)count, sizeof(char*));
|
||||||
|
if (!texts)
|
||||||
|
return false;
|
||||||
|
for (int i = 0; i < count; i++)
|
||||||
|
texts[i] = (const char*)a->filters->items[i];
|
||||||
|
}
|
||||||
|
char err[160];
|
||||||
|
FilterRuleList* expected =
|
||||||
|
filter_base_build(texts, count, a->cvs_exclude, a->delete_excluded, err, sizeof(err));
|
||||||
|
free(texts);
|
||||||
|
if (!expected)
|
||||||
|
return false;
|
||||||
|
bool equal = true;
|
||||||
|
int want = expected->count;
|
||||||
|
int have = got ? got->count : 0;
|
||||||
|
if (want != have) {
|
||||||
|
equal = false;
|
||||||
|
} else {
|
||||||
|
for (int i = 0; i < want; i++) {
|
||||||
|
const FilterRule* x = expected->items[i];
|
||||||
|
const FilterRule* y = got->items[i];
|
||||||
|
if (x->action != y->action || x->sides != y->sides || x->anchored != y->anchored ||
|
||||||
|
x->dir_only != y->dir_only || x->negate != y->negate ||
|
||||||
|
!str_opt_equal(x->owner, y->owner) || !str_opt_equal(x->pattern, y->pattern)) {
|
||||||
|
equal = false;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
filter_rule_list_free(expected);
|
||||||
|
return equal;
|
||||||
|
}
|
||||||
|
|
||||||
#define CONFIG_CMP_BOOL(a, b, name) ((a)->name == (b)->name)
|
#define CONFIG_CMP_BOOL(a, b, name) ((a)->name == (b)->name)
|
||||||
#define CONFIG_CMP_INT(a, b, name) ((a)->name == (b)->name)
|
#define CONFIG_CMP_INT(a, b, name) ((a)->name == (b)->name)
|
||||||
#define CONFIG_CMP_RAW(a, b, name) ((a)->name == (b)->name)
|
#define CONFIG_CMP_RAW(a, b, name) ((a)->name == (b)->name)
|
||||||
@@ -2615,6 +2698,7 @@ static bool basis_equal(const Config* a, const Config* b) {
|
|||||||
#define CONFIG_CMP_COPY_AS_ID(a, b, name) (!(a)->copy_as_set || (a)->name == (b)->name)
|
#define CONFIG_CMP_COPY_AS_ID(a, b, name) (!(a)->copy_as_set || (a)->name == (b)->name)
|
||||||
#define CONFIG_CMP_BLOCK_SKIP_SUFFIXES(a, b, name) skip_suffixes_equal((a), (b))
|
#define CONFIG_CMP_BLOCK_SKIP_SUFFIXES(a, b, name) skip_suffixes_equal((a), (b))
|
||||||
#define CONFIG_CMP_BLOCK_BASIS(a, b, name) basis_equal((a), (b))
|
#define CONFIG_CMP_BLOCK_BASIS(a, b, name) basis_equal((a), (b))
|
||||||
|
#define CONFIG_CMP_BLOCK_PROTECT_RULES(a, b, name) filter_rules_equal((a), (b)->name)
|
||||||
#define CONFIG_CMP_BLOCK_IDMAP(a, b, name) \
|
#define CONFIG_CMP_BLOCK_IDMAP(a, b, name) \
|
||||||
idmap_equal((a)->name, (a)->name##_count, (b)->name, (b)->name##_count)
|
idmap_equal((a)->name, (a)->name##_count, (b)->name, (b)->name##_count)
|
||||||
|
|
||||||
@@ -2784,6 +2868,7 @@ static void golden_config_populate(Config* c) {
|
|||||||
c->skip_compress_suffixes[1] = str_dup(".xz");
|
c->skip_compress_suffixes[1] = str_dup(".xz");
|
||||||
EXPECT_EQ_INT(config_basis_append(c, BASIS_DEST_COMPARE, "compare"), 0);
|
EXPECT_EQ_INT(config_basis_append(c, BASIS_DEST_COMPARE, "compare"), 0);
|
||||||
EXPECT_EQ_INT(config_basis_append(c, BASIS_DEST_LINK, "link"), 0);
|
EXPECT_EQ_INT(config_basis_append(c, BASIS_DEST_LINK, "link"), 0);
|
||||||
|
c->verify_basis = true;
|
||||||
c->fuzzy = true;
|
c->fuzzy = true;
|
||||||
c->checksum_algo = CHECKSUM_ALGO_MD5;
|
c->checksum_algo = CHECKSUM_ALGO_MD5;
|
||||||
c->checksum_seed = 0x1122334455667788ULL;
|
c->checksum_seed = 0x1122334455667788ULL;
|
||||||
@@ -2829,6 +2914,14 @@ static void golden_config_populate(Config* c) {
|
|||||||
c->copy_as_set = true;
|
c->copy_as_set = true;
|
||||||
c->copy_as_uid = 111;
|
c->copy_as_uid = 111;
|
||||||
c->copy_as_gid = 222;
|
c->copy_as_gid = 222;
|
||||||
|
/* Compile-through delete-protection rules (protocol 2.28.0). The golden
|
||||||
|
* sender serializes its compiled base rules, so populate a diverse set that
|
||||||
|
* exercises both sides, negate, anchoring and dir-only. */
|
||||||
|
c->filters = array_list_create(free);
|
||||||
|
array_list_add(c->filters, str_dup("P *.log"));
|
||||||
|
array_list_add(c->filters, str_dup("+r **/*.txt"));
|
||||||
|
array_list_add(c->filters, str_dup("H,!secret"));
|
||||||
|
array_list_add(c->filters, str_dup("- /sub/dir/"));
|
||||||
}
|
}
|
||||||
|
|
||||||
/* The pinned golden frame (protocol 2.28.0). The values below are the only
|
/* The pinned golden frame (protocol 2.28.0). The values below are the only
|
||||||
@@ -2836,12 +2929,14 @@ static void golden_config_populate(Config* c) {
|
|||||||
* them ONLY with a PROTOCOL_VERSION bump and a documented reason. The 2.24.0
|
* them ONLY with a PROTOCOL_VERSION bump and a documented reason. The 2.24.0
|
||||||
* delete-plan wave changed only the version string; 2.25.0 appended the
|
* delete-plan wave changed only the version string; 2.25.0 appended the
|
||||||
* report_stats bool, 2.26.0 appended the compression_algo int, 2.27.0 appended
|
* report_stats bool, 2.26.0 appended the compression_algo int, 2.27.0 appended
|
||||||
* the report_deletes bool, and 2.28.0 changed only the version string (the
|
* the report_deletes bool, and 2.28.0 changed only the version string and
|
||||||
* STATUS_STATS body grew, but the config frame layout is unchanged, so the
|
* appended the receiver-side delete-protection rule block (the STATUS_STATS
|
||||||
* frame length is identical). The byte-exact values are recomputed for the
|
* body also grew, but that is not part of this frame). Track 5a appends the
|
||||||
* merged layout. */
|
* FastSync-only verify_basis bool to the basis block WITHOUT a version bump
|
||||||
#define GOLDEN_WIRE_LEN 709
|
* (project decision), so the frame grew by one int to 886 bytes. The
|
||||||
#define GOLDEN_WIRE_HASH 417335736347473203ULL
|
* byte-exact values are recomputed for the merged layout. */
|
||||||
|
#define GOLDEN_WIRE_LEN 886
|
||||||
|
#define GOLDEN_WIRE_HASH 5809509022716816757ULL
|
||||||
|
|
||||||
static unsigned long long fnv1a_64(const unsigned char* buf, size_t len) {
|
static unsigned long long fnv1a_64(const unsigned char* buf, size_t len) {
|
||||||
unsigned long long h = 1469598103934665603ULL;
|
unsigned long long h = 1469598103934665603ULL;
|
||||||
@@ -2990,6 +3085,7 @@ static void test_config_wire_golden_receive() {
|
|||||||
recv->groupmap[0].to_name != NULL && strcmp(recv->groupmap[0].to_name, "root") == 0;
|
recv->groupmap[0].to_name != NULL && strcmp(recv->groupmap[0].to_name, "root") == 0;
|
||||||
ok = ok && recv->basis_count == 2 && recv->basis_dirs[0].type == BASIS_DEST_COMPARE &&
|
ok = ok && recv->basis_count == 2 && recv->basis_dirs[0].type == BASIS_DEST_COMPARE &&
|
||||||
recv->basis_dirs[1].type == BASIS_DEST_LINK;
|
recv->basis_dirs[1].type == BASIS_DEST_LINK;
|
||||||
|
ok = ok && recv->verify_basis;
|
||||||
ok = ok && recv->module != NULL && strcmp(recv->module, "goldenmod") == 0;
|
ok = ok && recv->module != NULL && strcmp(recv->module, "goldenmod") == 0;
|
||||||
ok = ok && recv->copy_as_set && recv->copy_as_uid == 111 && recv->copy_as_gid == 222;
|
ok = ok && recv->copy_as_set && recv->copy_as_uid == 111 && recv->copy_as_gid == 222;
|
||||||
}
|
}
|
||||||
@@ -3250,6 +3346,7 @@ void test_config() {
|
|||||||
test_config_wire_golden_receive();
|
test_config_wire_golden_receive();
|
||||||
test_config_wire_receive_bounds();
|
test_config_wire_receive_bounds();
|
||||||
test_config_receive_rejects_overcap_counts();
|
test_config_receive_rejects_overcap_counts();
|
||||||
|
test_config_receive_rejects_bad_protect_rules();
|
||||||
test_config_wire_roundtrip_all_fields();
|
test_config_wire_roundtrip_all_fields();
|
||||||
test_config_preserve_attribute_wire_roundtrip();
|
test_config_preserve_attribute_wire_roundtrip();
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -17,6 +17,7 @@
|
|||||||
* caller/receiver entry point) describing `dir` with no kept children. */
|
* caller/receiver entry point) describing `dir` with no kept children. */
|
||||||
static void send_plan_frame(int fd, const char* dir) {
|
static void send_plan_frame(int fd, const char* dir) {
|
||||||
EXPECT_TRUE(send_int(fd, 0)); /* has_config */
|
EXPECT_TRUE(send_int(fd, 0)); /* has_config */
|
||||||
|
EXPECT_TRUE(send_int(fd, 1)); /* apply: a real plan */
|
||||||
EXPECT_TRUE(send_wire_str(fd, dir));
|
EXPECT_TRUE(send_wire_str(fd, dir));
|
||||||
EXPECT_TRUE(send_int(fd, 0)); /* kept child dirs */
|
EXPECT_TRUE(send_int(fd, 0)); /* kept child dirs */
|
||||||
EXPECT_TRUE(send_int(fd, 0)); /* kept child files */
|
EXPECT_TRUE(send_int(fd, 0)); /* kept child files */
|
||||||
@@ -216,9 +217,58 @@ static void test_delete_delay_actual_removal_charges_budget(void) {
|
|||||||
config_delete(config);
|
config_delete(config);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Send a config-only carrier frame (apply=false): the per-run config block with
|
||||||
|
* one --delete-missing-args exact path, and no directory walk. */
|
||||||
|
static void send_config_only_frame(int fd, const char* missing_path) {
|
||||||
|
EXPECT_TRUE(send_int(fd, 1)); /* has_config */
|
||||||
|
EXPECT_TRUE(send_int(fd, 0)); /* protected prefixes */
|
||||||
|
EXPECT_TRUE(send_int(fd, 0)); /* size-skipped */
|
||||||
|
EXPECT_TRUE(send_int(fd, 1)); /* missing args */
|
||||||
|
EXPECT_TRUE(send_wire_str(fd, missing_path));
|
||||||
|
EXPECT_TRUE(send_int(fd, 0)); /* apply = false */
|
||||||
|
EXPECT_TRUE(send_wire_str(fd, "."));
|
||||||
|
EXPECT_TRUE(send_int(fd, 0));
|
||||||
|
EXPECT_TRUE(send_int(fd, 0));
|
||||||
|
}
|
||||||
|
|
||||||
|
/* The config-only carrier frame (apply=false) still applies the
|
||||||
|
* --delete-missing-args exact deletions even though it walks no directory. This
|
||||||
|
* is the fix for a --files-from list that synchronizes no directory. */
|
||||||
|
static void test_config_only_frame_applies_missing_args(void) {
|
||||||
|
char root[] = "/tmp/fastsync_dp_cfgonly_XXXXXX";
|
||||||
|
EXPECT_TRUE(mkdtemp(root) != NULL);
|
||||||
|
char gone[1024];
|
||||||
|
snprintf(gone, sizeof(gone), "%s/gone.txt", root);
|
||||||
|
int fd = open(gone, O_WRONLY | O_CREAT | O_TRUNC, 0600);
|
||||||
|
EXPECT_TRUE(fd >= 0);
|
||||||
|
close(fd);
|
||||||
|
|
||||||
|
Config* config = config_create();
|
||||||
|
EXPECT_NOT_NULL(config);
|
||||||
|
config->receive_root_directory = str_dup(root);
|
||||||
|
config->delete_missing_args = true;
|
||||||
|
|
||||||
|
int p[2];
|
||||||
|
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
|
||||||
|
|
||||||
|
DeletePlanSession* session = delete_plan_session_create(config);
|
||||||
|
EXPECT_NOT_NULL(session);
|
||||||
|
send_config_only_frame(p[1], "gone.txt");
|
||||||
|
EXPECT_EQ_INT(delete_plan_session_receive(session, config, p[0]), 0);
|
||||||
|
EXPECT_EQ_INT((int)delete_plan_session_deleted(session), 1);
|
||||||
|
EXPECT_TRUE(lstat(gone, &(struct stat){0}) != 0);
|
||||||
|
|
||||||
|
delete_plan_session_destroy(session);
|
||||||
|
close(p[0]);
|
||||||
|
close(p[1]);
|
||||||
|
rmdir(root);
|
||||||
|
config_delete(config);
|
||||||
|
}
|
||||||
|
|
||||||
void test_delete_plan(void) {
|
void test_delete_plan(void) {
|
||||||
test_delete_delay_refilled_dir_removed_recursively();
|
test_delete_delay_refilled_dir_removed_recursively();
|
||||||
test_delete_delay_removed_file_counted();
|
test_delete_delay_removed_file_counted();
|
||||||
test_delete_delay_max_delete_bounds_actual();
|
test_delete_delay_max_delete_bounds_actual();
|
||||||
test_delete_delay_actual_removal_charges_budget();
|
test_delete_delay_actual_removal_charges_budget();
|
||||||
|
test_config_only_frame_applies_missing_args();
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -6,6 +6,7 @@
|
|||||||
#include "file_receive.h"
|
#include "file_receive.h"
|
||||||
#include "data.h"
|
#include "data.h"
|
||||||
#include "config.h"
|
#include "config.h"
|
||||||
|
#include "charset.h"
|
||||||
#include "utils.h"
|
#include "utils.h"
|
||||||
#include "protocol.h"
|
#include "protocol.h"
|
||||||
#include "test_utils.h"
|
#include "test_utils.h"
|
||||||
@@ -1807,6 +1808,159 @@ static void test_receive_incremental_check_empty_path() {
|
|||||||
config_delete(cfg);
|
config_delete(cfg);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Pure policy helpers behind the basis quick-check / --verify-basis decision. */
|
||||||
|
static void test_file_basis_quick_match_decision() {
|
||||||
|
Config* cfg = config_create();
|
||||||
|
EXPECT_NOT_NULL(cfg);
|
||||||
|
EXPECT_FALSE(file_basis_content_required(cfg));
|
||||||
|
cfg->verify_basis = true;
|
||||||
|
EXPECT_TRUE(file_basis_content_required(cfg));
|
||||||
|
cfg->verify_basis = false;
|
||||||
|
|
||||||
|
struct stat st;
|
||||||
|
memset(&st, 0, sizeof(st));
|
||||||
|
st.st_mtime = 1500000000;
|
||||||
|
#ifdef __linux__
|
||||||
|
st.st_mtim.tv_nsec = 500;
|
||||||
|
#endif
|
||||||
|
/* Equal size is required by the caller; this leg is the mtime / --size-only
|
||||||
|
rule. Equal mtime matches, a different mtime misses by default. */
|
||||||
|
EXPECT_TRUE(file_basis_quick_match(cfg, &st, 1500000000, 500));
|
||||||
|
EXPECT_FALSE(file_basis_quick_match(cfg, &st, 1500000001, 500));
|
||||||
|
cfg->size_only = true;
|
||||||
|
EXPECT_TRUE(file_basis_quick_match(cfg, &st, 1500000001, 500));
|
||||||
|
cfg->size_only = false;
|
||||||
|
cfg->modify_window = 2;
|
||||||
|
EXPECT_TRUE(file_basis_quick_match(cfg, &st, 1500000002, 500));
|
||||||
|
config_delete(cfg);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* End-to-end handshake decision for a same-size, same-mtime, DIFFERENT-content
|
||||||
|
basis. Default (rsync parity): the metadata quick-check is trusted, the
|
||||||
|
receiver answers STATUS_OK and materializes the basis bytes. --verify-basis:
|
||||||
|
the whole-file digest is required, the basis is rejected and the receiver
|
||||||
|
asks for the source (STATUS_NEXT + full transfer). */
|
||||||
|
static void test_receive_incremental_check_basis_quick_check_and_verify() {
|
||||||
|
const char* root = "test_basis_quick_root";
|
||||||
|
const char* basis_dir = "test_basis_quick_root/basis";
|
||||||
|
const char* basis_file = "test_basis_quick_root/basis/f.txt";
|
||||||
|
unlink(basis_file);
|
||||||
|
unlink("test_basis_quick_root/f.txt");
|
||||||
|
rmdir(basis_dir);
|
||||||
|
rmdir(root);
|
||||||
|
EXPECT_EQ_INT(mkdir(root, 0755), 0);
|
||||||
|
EXPECT_EQ_INT(mkdir(basis_dir, 0755), 0);
|
||||||
|
|
||||||
|
const char* src_bytes = "AAAA";
|
||||||
|
const unsigned long long size = 4;
|
||||||
|
const time_t mtime = 1500000000;
|
||||||
|
{
|
||||||
|
FILE* fh = fopen(basis_file, "wb");
|
||||||
|
EXPECT_NOT_NULL(fh);
|
||||||
|
// cppcheck-suppress knownConditionTrueFalse
|
||||||
|
if (fh) {
|
||||||
|
EXPECT_EQ_INT((int)fwrite("BBBB", 1, (size_t)size, fh), (int)size);
|
||||||
|
fclose(fh);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
struct timespec ts[2] = {{mtime, 0}, {mtime, 0}};
|
||||||
|
EXPECT_EQ_INT(utimensat(AT_FDCWD, basis_file, ts, 0), 0);
|
||||||
|
|
||||||
|
char root_abs[PATH_MAX];
|
||||||
|
EXPECT_NOT_NULL(realpath(root, root_abs));
|
||||||
|
int root_fd = open(root_abs, O_RDONLY | O_DIRECTORY | O_CLOEXEC);
|
||||||
|
EXPECT_TRUE(root_fd >= 0);
|
||||||
|
// cppcheck-suppress knownConditionTrueFalse
|
||||||
|
if (root_fd < 0) {
|
||||||
|
unlink(basis_file);
|
||||||
|
rmdir(basis_dir);
|
||||||
|
rmdir(root);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
EXPECT_TRUE(utils_set_authorized_root(root_fd, root_abs));
|
||||||
|
|
||||||
|
uint8_t digest[CHECKSUM_MAX_DIGEST_LEN];
|
||||||
|
size_t digest_len = 0;
|
||||||
|
EXPECT_TRUE(checksum_digest(CHECKSUM_ALGO_XXH64, 0, src_bytes, size, digest, sizeof(digest),
|
||||||
|
&digest_len));
|
||||||
|
|
||||||
|
/* Route protocol I/O through the explicit descriptors (a previous test group
|
||||||
|
may have left io_set_fds() bound to its own pipe). */
|
||||||
|
io_set_fds(-1, -1);
|
||||||
|
io_set_bwlimit(0);
|
||||||
|
|
||||||
|
for (int verify = 0; verify <= 1; verify++) {
|
||||||
|
Config* cfg = config_create();
|
||||||
|
EXPECT_NOT_NULL(cfg);
|
||||||
|
cfg->receive_root_directory = str_dup(root_abs);
|
||||||
|
cfg->checksum = false;
|
||||||
|
cfg->checksum_algo = CHECKSUM_ALGO_XXH64;
|
||||||
|
cfg->checksum_seed = 0;
|
||||||
|
cfg->use_incremental = true;
|
||||||
|
cfg->use_delta = false;
|
||||||
|
cfg->use_metadata = false;
|
||||||
|
cfg->verify_basis = (verify != 0);
|
||||||
|
EXPECT_EQ_INT(config_basis_append(cfg, BASIS_DEST_LINK, "basis"), 0);
|
||||||
|
|
||||||
|
int p[2];
|
||||||
|
EXPECT_EQ_INT(socketpair(AF_UNIX, SOCK_STREAM, 0, p), 0);
|
||||||
|
EXPECT_TRUE(send_wire_str(p[1], "f.txt"));
|
||||||
|
unsigned long long check_size = size;
|
||||||
|
long long check_mtime = (long long)mtime;
|
||||||
|
long long check_mtime_nsec = 0;
|
||||||
|
EXPECT_TRUE(send_n_data(p[1], &check_size, sizeof(check_size)));
|
||||||
|
EXPECT_TRUE(send_n_data(p[1], &check_mtime, sizeof(check_mtime)));
|
||||||
|
EXPECT_TRUE(send_n_data(p[1], &check_mtime_nsec, sizeof(check_mtime_nsec)));
|
||||||
|
/* Only --verify-basis needs the digest (cfg->checksum is false) and the
|
||||||
|
pre-staged fallback full transfer the receiver will request. */
|
||||||
|
if (verify) {
|
||||||
|
uint8_t wire_len = (uint8_t)digest_len;
|
||||||
|
EXPECT_TRUE(send_n_data(p[1], &wire_len, sizeof(wire_len)));
|
||||||
|
EXPECT_TRUE(send_n_data(p[1], digest, digest_len));
|
||||||
|
char* payload_bytes = str_dup(src_bytes);
|
||||||
|
EXPECT_NOT_NULL(payload_bytes);
|
||||||
|
Data* payload = data_create(payload_bytes, size);
|
||||||
|
EXPECT_NOT_NULL(payload);
|
||||||
|
// cppcheck-suppress knownConditionTrueFalse
|
||||||
|
if (payload)
|
||||||
|
EXPECT_TRUE(send_data(p[1], payload));
|
||||||
|
data_destroy(payload);
|
||||||
|
}
|
||||||
|
|
||||||
|
bool skipped = false;
|
||||||
|
File* file = receive_incremental_check(p[0], cfg, &skipped);
|
||||||
|
EXPECT_NOT_NULL(file);
|
||||||
|
// cppcheck-suppress knownConditionTrueFalse
|
||||||
|
if (file) {
|
||||||
|
EXPECT_FALSE(skipped);
|
||||||
|
Status reply = STATUS_ERROR;
|
||||||
|
EXPECT_TRUE(receive_status(p[1], &reply));
|
||||||
|
if (verify) {
|
||||||
|
EXPECT_EQ_INT((int)reply, (int)STATUS_NEXT);
|
||||||
|
EXPECT_NOT_NULL(file->data->data);
|
||||||
|
EXPECT_TRUE(file->data->data != NULL && memcmp(file->data->data, src_bytes, size) == 0);
|
||||||
|
} else {
|
||||||
|
EXPECT_EQ_INT((int)reply, (int)STATUS_OK);
|
||||||
|
EXPECT_TRUE(file->skip);
|
||||||
|
/* The default quick-check hit materializes from the basis PATH at
|
||||||
|
install time (streaming), so no content is buffered on the File. */
|
||||||
|
EXPECT_NOT_NULL(file->basis_link);
|
||||||
|
EXPECT_NULL(file->data->data);
|
||||||
|
}
|
||||||
|
file_destroy(file);
|
||||||
|
}
|
||||||
|
close(p[0]);
|
||||||
|
close(p[1]);
|
||||||
|
config_delete(cfg);
|
||||||
|
}
|
||||||
|
|
||||||
|
utils_set_authorized_root(-1, NULL);
|
||||||
|
close(root_fd);
|
||||||
|
unlink(basis_file);
|
||||||
|
rmdir(basis_dir);
|
||||||
|
rmdir(root);
|
||||||
|
}
|
||||||
|
|
||||||
/* -K/--keep-dirlinks secure open: with an authorized root, a destination path
|
/* -K/--keep-dirlinks secure open: with an authorized root, a destination path
|
||||||
* component that is a symlink to an IN-ROOT directory is used as that directory
|
* component that is a symlink to an IN-ROOT directory is used as that directory
|
||||||
* (its referent is opened through a relative O_NOFOLLOW walk from the root fd,
|
* (its referent is opened through a relative O_NOFOLLOW walk from the root fd,
|
||||||
@@ -2140,6 +2294,8 @@ void test_file() {
|
|||||||
test_dir_time_list();
|
test_dir_time_list();
|
||||||
test_dir_time_list_cap();
|
test_dir_time_list_cap();
|
||||||
test_receive_incremental_check_empty_path();
|
test_receive_incremental_check_empty_path();
|
||||||
|
test_file_basis_quick_match_decision();
|
||||||
|
test_receive_incremental_check_basis_quick_check_and_verify();
|
||||||
test_keep_dirlinks_secure_open();
|
test_keep_dirlinks_secure_open();
|
||||||
test_inplace_overwrite_clears_special_mode_bits();
|
test_inplace_overwrite_clears_special_mode_bits();
|
||||||
test_inplace_overwrite_metadata_strips_special_bits();
|
test_inplace_overwrite_metadata_strips_special_bits();
|
||||||
|
|||||||
@@ -18,10 +18,11 @@
|
|||||||
|
|
||||||
/* P8 config-frame tail: super_mode (4) + copy-as presence (4) + uid (4) + gid (4). */
|
/* P8 config-frame tail: super_mode (4) + copy-as presence (4) + uid (4) + gid (4). */
|
||||||
#define P8_TAIL_BYTES 16
|
#define P8_TAIL_BYTES 16
|
||||||
/* Bytes after the P8 tail: report_dest_info (4), report_stats (4, wire-stats
|
/* Bytes after the P8 tail: report_dest_info (4), report_stats (4),
|
||||||
* wave) and compression_algo (4, codec wave). The P8 fields sit this many
|
* report_deletes (4, --info=del wave), compression_algo (4, codec wave) and the
|
||||||
* bytes before the end of the frame. */
|
* receiver delete-protection count (4, protocol 2.28.0). The P8 fields sit
|
||||||
#define POST_P8_TAIL_BYTES 12
|
* this many bytes before the end of the frame. */
|
||||||
|
#define POST_P8_TAIL_BYTES 20
|
||||||
|
|
||||||
/* Smoke test for chunk_deserialize fuzz target */
|
/* Smoke test for chunk_deserialize fuzz target */
|
||||||
static void test_fuzz_chunk_deserialize() {
|
static void test_fuzz_chunk_deserialize() {
|
||||||
|
|||||||
+27
-13
@@ -1063,14 +1063,18 @@ static void test_incremental_check_fifo_destination_does_not_hang() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/* A server-contacting --dry-run with an alternate basis dir must never read or
|
/* A server-contacting --dry-run with an alternate basis dir must never read or
|
||||||
hash the basis file. An exact (size+mtime+content) basis match would
|
hash the basis file. Under the default metadata quick-check a hit needs no
|
||||||
otherwise let a client probe the basis bytes against its own supplied digest
|
basis bytes, so a compare-dest match is reported as a skip (STATUS_OK) just
|
||||||
(a 1-bit content oracle). The dry-run decision is metadata-only, so even a
|
like a real run -- and still no content is read. Under --verify-basis a hit
|
||||||
byte-identical basis is reported as would-transfer, not a compare-dest skip. */
|
would require hashing the basis against the client-supplied digest (a 1-bit
|
||||||
static void test_incremental_check_dry_run_basis_does_not_read_content() {
|
content oracle), which a dry-run must never do, so even a byte-identical
|
||||||
|
basis is reported as would-transfer. The destination is never materialized
|
||||||
|
in either arm. */
|
||||||
|
static void run_dry_run_basis_check(bool verify, Status expected) {
|
||||||
Config* cfg = config_create();
|
Config* cfg = config_create();
|
||||||
EXPECT_NOT_NULL(cfg);
|
EXPECT_NOT_NULL(cfg);
|
||||||
cfg->dry_run = true;
|
cfg->dry_run = true;
|
||||||
|
cfg->verify_basis = verify;
|
||||||
char* root = make_check_root("dryb");
|
char* root = make_check_root("dryb");
|
||||||
EXPECT_NOT_NULL(root);
|
EXPECT_NOT_NULL(root);
|
||||||
cfg->receive_root_directory = str_dup(root);
|
cfg->receive_root_directory = str_dup(root);
|
||||||
@@ -1086,8 +1090,8 @@ static void test_incremental_check_dry_run_basis_does_not_read_content() {
|
|||||||
EXPECT_EQ_INT(stat(basis_path, &bst), 0);
|
EXPECT_EQ_INT(stat(basis_path, &bst), 0);
|
||||||
EXPECT_EQ_INT(config_basis_append(cfg, BASIS_DEST_COMPARE, "basis"), 0);
|
EXPECT_EQ_INT(config_basis_append(cfg, BASIS_DEST_COMPARE, "basis"), 0);
|
||||||
|
|
||||||
/* The (correct) source digest for the basis bytes: an unfixed dry-run would
|
/* The (correct) source digest for the basis bytes: a buggy dry-run that read
|
||||||
read+hash the basis and treat this as an exact compare-dest hit. */
|
and hashed the basis would treat this as an exact compare-dest hit. */
|
||||||
uint8_t digest[CHECKSUM_MAX_DIGEST_LEN];
|
uint8_t digest[CHECKSUM_MAX_DIGEST_LEN];
|
||||||
size_t digest_len = 0;
|
size_t digest_len = 0;
|
||||||
EXPECT_TRUE(checksum_digest((ChecksumAlgo)cfg->checksum_algo, cfg->checksum_seed, content,
|
EXPECT_TRUE(checksum_digest((ChecksumAlgo)cfg->checksum_algo, cfg->checksum_seed, content,
|
||||||
@@ -1106,7 +1110,8 @@ static void test_incremental_check_dry_run_basis_does_not_read_content() {
|
|||||||
bool skipped = false;
|
bool skipped = false;
|
||||||
bool would_transfer = false;
|
bool would_transfer = false;
|
||||||
File* file = receive_incremental_check_ex(p[0], cfg, &skipped, &would_transfer);
|
File* file = receive_incremental_check_ex(p[0], cfg, &skipped, &would_transfer);
|
||||||
bool ok = file == NULL && !skipped && would_transfer;
|
bool ok = file == NULL && skipped == (expected == STATUS_OK) &&
|
||||||
|
would_transfer == (expected != STATUS_OK);
|
||||||
file_destroy(file);
|
file_destroy(file);
|
||||||
config_delete(cfg);
|
config_delete(cfg);
|
||||||
close(p[0]);
|
close(p[0]);
|
||||||
@@ -1124,13 +1129,16 @@ static void test_incremental_check_dry_run_basis_does_not_read_content() {
|
|||||||
EXPECT_TRUE(send_n_data(p[1], &size, sizeof(size)));
|
EXPECT_TRUE(send_n_data(p[1], &size, sizeof(size)));
|
||||||
EXPECT_TRUE(send_n_data(p[1], &mtime, sizeof(mtime)));
|
EXPECT_TRUE(send_n_data(p[1], &mtime, sizeof(mtime)));
|
||||||
EXPECT_TRUE(send_n_data(p[1], &mtime_nsec, sizeof(mtime_nsec)));
|
EXPECT_TRUE(send_n_data(p[1], &mtime_nsec, sizeof(mtime_nsec)));
|
||||||
uint8_t wire_len = (uint8_t)digest_len;
|
/* The digest is only on the wire when --checksum or --verify-basis needs it
|
||||||
EXPECT_TRUE(send_n_data(p[1], &wire_len, sizeof(wire_len)));
|
(cfg->checksum is false here); the default quick-check arm sends none. */
|
||||||
EXPECT_TRUE(send_n_data(p[1], digest, digest_len));
|
if (verify) {
|
||||||
|
uint8_t wire_len = (uint8_t)digest_len;
|
||||||
|
EXPECT_TRUE(send_n_data(p[1], &wire_len, sizeof(wire_len)));
|
||||||
|
EXPECT_TRUE(send_n_data(p[1], digest, digest_len));
|
||||||
|
}
|
||||||
Status s;
|
Status s;
|
||||||
EXPECT_TRUE(receive_status(p[1], &s));
|
EXPECT_TRUE(receive_status(p[1], &s));
|
||||||
/* A skip here would mean the receiver read+hashed the basis file. */
|
EXPECT_EQ_INT(s, expected);
|
||||||
EXPECT_EQ_INT(s, STATUS_DRY_RUN_TRANSFER);
|
|
||||||
|
|
||||||
int status;
|
int status;
|
||||||
waitpid(pid, &status, 0);
|
waitpid(pid, &status, 0);
|
||||||
@@ -1148,6 +1156,11 @@ static void test_incremental_check_dry_run_basis_does_not_read_content() {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
static void test_incremental_check_dry_run_basis_does_not_read_content() {
|
||||||
|
run_dry_run_basis_check(false, STATUS_OK);
|
||||||
|
run_dry_run_basis_check(true, STATUS_DRY_RUN_TRANSFER);
|
||||||
|
}
|
||||||
|
|
||||||
/* B1: a FIFO planted in a --link-dest basis directory must not block
|
/* B1: a FIFO planted in a --link-dest basis directory must not block
|
||||||
* basis_open_regular() either; the basis match is simply declined. */
|
* basis_open_regular() either; the basis match is simply declined. */
|
||||||
static void test_incremental_check_basis_fifo_does_not_hang() {
|
static void test_incremental_check_basis_fifo_does_not_hang() {
|
||||||
@@ -1278,6 +1291,7 @@ static void test_dry_run_delete_plan_commit_does_not_delete() {
|
|||||||
EXPECT_TRUE(send_int(p[1], 0)); /* size-skipped prefixes */
|
EXPECT_TRUE(send_int(p[1], 0)); /* size-skipped prefixes */
|
||||||
EXPECT_TRUE(send_int(p[1], 1)); /* missing-args exact deletions */
|
EXPECT_TRUE(send_int(p[1], 1)); /* missing-args exact deletions */
|
||||||
EXPECT_TRUE(send_str(p[1], "victim.txt"));
|
EXPECT_TRUE(send_str(p[1], "victim.txt"));
|
||||||
|
EXPECT_TRUE(send_int(p[1], 1)); /* apply: a real plan */
|
||||||
EXPECT_TRUE(send_str(p[1], ".")); /* receive root plan */
|
EXPECT_TRUE(send_str(p[1], ".")); /* receive root plan */
|
||||||
EXPECT_TRUE(send_int(p[1], 0)); /* kept child directories */
|
EXPECT_TRUE(send_int(p[1], 0)); /* kept child directories */
|
||||||
EXPECT_TRUE(send_int(p[1], 0)); /* kept child files */
|
EXPECT_TRUE(send_int(p[1], 0)); /* kept child files */
|
||||||
|
|||||||
@@ -138,7 +138,7 @@ static void test_walker_removes_extras_keeps_manifest_and_protected() {
|
|||||||
DeleteSkipEntry skip = {"prot", false};
|
DeleteSkipEntry skip = {"prot", false};
|
||||||
size_t deleted = 0;
|
size_t deleted = 0;
|
||||||
DeleteWalkResult result =
|
DeleteWalkResult result =
|
||||||
delete_extras_limited(root, manifest, NULL, 100000, &skip, 1, &deleted, NULL);
|
delete_extras_limited(root, manifest, NULL, 100000, &skip, 1, NULL, &deleted, NULL);
|
||||||
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_OK);
|
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_OK);
|
||||||
EXPECT_FALSE(file_exists(root, "a.txt"));
|
EXPECT_FALSE(file_exists(root, "a.txt"));
|
||||||
EXPECT_TRUE(file_exists(root, "keep.txt"));
|
EXPECT_TRUE(file_exists(root, "keep.txt"));
|
||||||
@@ -173,7 +173,7 @@ static void test_walker_keeps_nested_manifest_dirs() {
|
|||||||
EXPECT_NOT_NULL(manifest);
|
EXPECT_NOT_NULL(manifest);
|
||||||
size_t deleted = 0;
|
size_t deleted = 0;
|
||||||
DeleteWalkResult result =
|
DeleteWalkResult result =
|
||||||
delete_extras_limited(root, manifest, NULL, 100000, NULL, 0, &deleted, NULL);
|
delete_extras_limited(root, manifest, NULL, 100000, NULL, 0, NULL, &deleted, NULL);
|
||||||
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_OK);
|
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_OK);
|
||||||
EXPECT_FALSE(file_exists(root, "extra.txt"));
|
EXPECT_FALSE(file_exists(root, "extra.txt"));
|
||||||
EXPECT_TRUE(file_exists(root, "keepdir/deep/keep.txt"));
|
EXPECT_TRUE(file_exists(root, "keepdir/deep/keep.txt"));
|
||||||
@@ -204,7 +204,7 @@ static void test_walker_max_delete_partial_deletes_up_to_cap() {
|
|||||||
size_t deleted = 999;
|
size_t deleted = 999;
|
||||||
size_t skipped = 0;
|
size_t skipped = 0;
|
||||||
DeleteWalkResult result =
|
DeleteWalkResult result =
|
||||||
delete_extras_limited(root, manifest, NULL, 2, NULL, 0, &deleted, &skipped);
|
delete_extras_limited(root, manifest, NULL, 2, NULL, 0, NULL, &deleted, &skipped);
|
||||||
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_LIMIT_REACHED);
|
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_LIMIT_REACHED);
|
||||||
EXPECT_EQ_INT((int)deleted, 2);
|
EXPECT_EQ_INT((int)deleted, 2);
|
||||||
EXPECT_EQ_INT((int)skipped, 1);
|
EXPECT_EQ_INT((int)skipped, 1);
|
||||||
@@ -225,7 +225,8 @@ static void test_walker_max_delete_exact_bound_deletes() {
|
|||||||
ArrayList* manifest = make_manifest_strings(keeps, 0);
|
ArrayList* manifest = make_manifest_strings(keeps, 0);
|
||||||
EXPECT_NOT_NULL(manifest);
|
EXPECT_NOT_NULL(manifest);
|
||||||
size_t deleted = 0;
|
size_t deleted = 0;
|
||||||
DeleteWalkResult result = delete_extras_limited(root, manifest, NULL, 2, NULL, 0, &deleted, NULL);
|
DeleteWalkResult result =
|
||||||
|
delete_extras_limited(root, manifest, NULL, 2, NULL, 0, NULL, &deleted, NULL);
|
||||||
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_OK);
|
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_OK);
|
||||||
EXPECT_EQ_INT((int)deleted, 2);
|
EXPECT_EQ_INT((int)deleted, 2);
|
||||||
EXPECT_FALSE(file_exists(root, "a.txt"));
|
EXPECT_FALSE(file_exists(root, "a.txt"));
|
||||||
@@ -258,7 +259,7 @@ static void test_walker_removes_extraneous_symlinks() {
|
|||||||
EXPECT_NOT_NULL(manifest);
|
EXPECT_NOT_NULL(manifest);
|
||||||
size_t deleted = 0;
|
size_t deleted = 0;
|
||||||
DeleteWalkResult result =
|
DeleteWalkResult result =
|
||||||
delete_extras_limited(root, manifest, NULL, 100000, NULL, 0, &deleted, NULL);
|
delete_extras_limited(root, manifest, NULL, 100000, NULL, 0, NULL, &deleted, NULL);
|
||||||
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_OK);
|
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_OK);
|
||||||
EXPECT_FALSE(file_exists(root, "link_file"));
|
EXPECT_FALSE(file_exists(root, "link_file"));
|
||||||
EXPECT_FALSE(file_exists(root, "link_dir"));
|
EXPECT_FALSE(file_exists(root, "link_dir"));
|
||||||
@@ -294,7 +295,7 @@ static void test_walker_confines_deletion_to_synced_dirs() {
|
|||||||
EXPECT_TRUE(array_list_add(dirs, str_dup("inscope")));
|
EXPECT_TRUE(array_list_add(dirs, str_dup("inscope")));
|
||||||
size_t deleted = 0;
|
size_t deleted = 0;
|
||||||
DeleteWalkResult result =
|
DeleteWalkResult result =
|
||||||
delete_extras_limited(root, manifest, dirs, 100000, NULL, 0, &deleted, NULL);
|
delete_extras_limited(root, manifest, dirs, 100000, NULL, 0, NULL, &deleted, NULL);
|
||||||
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_OK);
|
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_OK);
|
||||||
EXPECT_TRUE(file_exists(root, "rootextra.txt"));
|
EXPECT_TRUE(file_exists(root, "rootextra.txt"));
|
||||||
EXPECT_FALSE(file_exists(root, "inscope/extra.txt"));
|
EXPECT_FALSE(file_exists(root, "inscope/extra.txt"));
|
||||||
@@ -324,6 +325,45 @@ static void test_walker_unlimited_deletes_all() {
|
|||||||
free(root);
|
free(root);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Receiver-side filter protection (protocol 2.28.0): a compiled protect rule
|
||||||
|
shields a DESTINATION-ONLY extra that never appeared on the sender, a risk
|
||||||
|
rule cancels an earlier/later protect (first match wins), and a dir-only
|
||||||
|
protect rule shields the whole subtree. */
|
||||||
|
static void test_walker_protect_rules_shield_dest_only() {
|
||||||
|
char* root = make_walk_root("protectrules");
|
||||||
|
EXPECT_NOT_NULL(root);
|
||||||
|
EXPECT_TRUE(write_file_at(root, "keep.txt", "kept"));
|
||||||
|
EXPECT_TRUE(write_file_at(root, "extra.log", "risk cancels protect"));
|
||||||
|
EXPECT_TRUE(write_file_at(root, "safe.log", "protected"));
|
||||||
|
EXPECT_TRUE(write_file_at(root, "other.txt", "deleted"));
|
||||||
|
EXPECT_EQ_INT(make_subdir(root, "prot"), 0);
|
||||||
|
EXPECT_TRUE(write_file_at(root, "prot/inside.txt", "shielded subtree"));
|
||||||
|
EXPECT_TRUE(write_file_at(root, "prot/deep.log", "shielded subtree"));
|
||||||
|
|
||||||
|
const char* keeps[] = {"keep.txt"};
|
||||||
|
ArrayList* manifest = make_manifest_strings(keeps, 1);
|
||||||
|
EXPECT_NOT_NULL(manifest);
|
||||||
|
const char* rule_text[] = {"R extra.log", "P *.log", "P prot/"};
|
||||||
|
char err[160];
|
||||||
|
FilterRuleList* rules = filter_base_build(rule_text, 3, false, false, err, sizeof(err));
|
||||||
|
EXPECT_NOT_NULL(rules);
|
||||||
|
size_t deleted = 0;
|
||||||
|
DeleteWalkResult result =
|
||||||
|
delete_extras_limited(root, manifest, NULL, 100000, NULL, 0, rules, &deleted, NULL);
|
||||||
|
EXPECT_EQ_INT((int)result, (int)DELETE_WALK_OK);
|
||||||
|
EXPECT_TRUE(file_exists(root, "keep.txt"));
|
||||||
|
EXPECT_FALSE(file_exists(root, "extra.log")); /* risk wins the first match */
|
||||||
|
EXPECT_TRUE(file_exists(root, "safe.log")); /* protect shields the extra */
|
||||||
|
EXPECT_FALSE(file_exists(root, "other.txt"));
|
||||||
|
EXPECT_TRUE(dir_exists(root, "prot"));
|
||||||
|
EXPECT_TRUE(file_exists(root, "prot/inside.txt"));
|
||||||
|
EXPECT_TRUE(file_exists(root, "prot/deep.log"));
|
||||||
|
filter_rule_list_free(rules);
|
||||||
|
array_list_delete(manifest);
|
||||||
|
remove_walk_tree(root);
|
||||||
|
free(root);
|
||||||
|
}
|
||||||
|
|
||||||
typedef struct {
|
typedef struct {
|
||||||
bool eight_bit_output;
|
bool eight_bit_output;
|
||||||
const char* expected;
|
const char* expected;
|
||||||
@@ -626,6 +666,7 @@ void test_shared_utils() {
|
|||||||
test_walker_removes_extraneous_symlinks();
|
test_walker_removes_extraneous_symlinks();
|
||||||
test_walker_confines_deletion_to_synced_dirs();
|
test_walker_confines_deletion_to_synced_dirs();
|
||||||
test_walker_unlimited_deletes_all();
|
test_walker_unlimited_deletes_all();
|
||||||
|
test_walker_protect_rules_shield_dest_only();
|
||||||
test_loopback_helpers();
|
test_loopback_helpers();
|
||||||
test_fd_peer_ip();
|
test_fd_peer_ip();
|
||||||
|
|
||||||
|
|||||||
+2
-1
@@ -210,7 +210,8 @@ static void test_xattr_receive_drops_acl_without_preserve_acls() {
|
|||||||
|
|
||||||
/* MINOR-2: a --link-dest / -H copy fallback (linkat refused) must still apply
|
/* MINOR-2: a --link-dest / -H copy fallback (linkat refused) must still apply
|
||||||
* the per-file xattrs and --fake-super stat. A DIRECTORY basis forces linkat
|
* the per-file xattrs and --fake-super stat. A DIRECTORY basis forces linkat
|
||||||
* to fail with EPERM, exercising the byte-copy fallback deterministically.
|
* to fail with EPERM, exercising the byte-copy fallback deterministically (the
|
||||||
|
* basis is not a regular file, so the fallback uses the caller's bytes).
|
||||||
* Guarded on filesystem xattr support. */
|
* Guarded on filesystem xattr support. */
|
||||||
static void test_link_copy_fallback_preserves_xattrs() {
|
static void test_link_copy_fallback_preserves_xattrs() {
|
||||||
const char* dest = "test_link_xattr_dest.txt";
|
const char* dest = "test_link_xattr_dest.txt";
|
||||||
|
|||||||
Reference in New Issue
Block a user